The hottest CI/CD Substack posts right now

Ensure all necessary steps are taken before landing a pull request to the main branch, such as passing all tests and code reviews.
Deploy new software versions gradually to production, starting with a small number of machines first.
Consider implementing CI/CD for continuous deployment to improve observability, but balance it with on-demand deployments to ensure all changes are attended to.

Data Engineers may have a love-hate relationship with AWS Lambdas due to their versatility but occasional limitations.
AWS Lambdas are under-utilized in Data Engineering but offer benefits like cheap solutions, ease of use, and driving better practices.
AWS Lambdas are handy for processing small datasets, running data quality checks, and executing quick logic while reducing architecture complexity and cost.

DAST tools were valuable in the past for their ability to discover OWASP top 10 defects
Now, the way software is built has changed, making DAST less effective in CI/CD pipelines
There are still some ways to integrate DAST tools effectively, like repurposing them for low-hanging fruit and using them alongside Pentesters

PostgreSQL 16 released with significant enhancements and new features
Hydra 1.0 now generally available, bringing columnar tables to Postgres for analytical reports
Linux Foundation introduces OpenTofu as an alternative to Terraform

Drone CI is a self-service Continuous Integration platform that simplifies CI pipeline configuration by using a .drone.yml file in the project's root directory.
Make sure the Drone service is public-facing with port 443 accessible, and use a read-through cache with Docker to ensure resilience to Docker Hub outages.
Add yourself as an administrator to new Drone installations to have full access to API features, set up cron interval as needed, and utilize the CLI tool for more advanced capabilities.

Get a weekly roundup of the best Substack posts, by hacker news affinity:

Consider alternatives to Jenkins for new software projects due to limitations in plugin complexity.
Evaluate your R&D department's expertise and resources to manage Jenkins installation and perform updates.
Assess the security risks and maintenance requirements of Jenkins installations to prevent potential breaches.

The project successfully migrated a critical GPS tracking service from Elixir to AnyCable, enabling real-time features and smoother maintenance.
The team optimized the infrastructure using AWS ECS, Fargate, and CloudFormation, delivering improvements in performance, scalability, and resource management.
AnyCable deployment was streamlined within the project's infrastructure, bringing in monitoring features and helping speed up the CI/CD pipeline.

Building docker images can be time-consuming, especially in CI/CD pipelines.
To speed up the process, consider using cache, layering your Dockerfile, and utilizing multi-stage builds.
Ensure your Dockerfile is correctly set up, specifying versions, exposing necessary ports, and addressing platform compatibility.

Bytewax started with an open-source library to simplify streaming data processing.
The Bytewax community has grown with positive feedback from developers.
The next step for Bytewax is to build a platform that eliminates deployment overhead.

Each pipeline step in DroneCI can use different container images, allowing for versatile tasks like testing across multiple platforms.
Base64 encoding secrets in DroneCI is a useful technique for securely handling sensitive information like SSH keys.
Monitoring DroneCI pipelines can be enhanced by utilizing Prometheus to track status, duration, and using a Push Gateway to export build metrics.