The hottest Data Pipelines Substack posts right now

And their main takeaways
Category
Top Technology Topics

The "Brittleness" Problem in Data Pipelines.

Data Engineering Central • 157 implied HN points • 24 Apr 23
🕹 Technology Data Engineering Data Pipelines Resources
  1. Brittleness in data pipelines can lead to various issues like data quality problems, difficult debugging, and slow performance.
  2. To overcome brittle pipelines, focus on addressing data quality issues through monitoring, sanity checks, and using tools like Great Expectations.
  3. Development issues such as lack of tests, poor documentation, and bad code practices contribute to brittle pipelines; implementing best practices like unit testing and Docker can help improve pipeline reliability.

Data Pipelines - Streams to Parquet

Bytewax • 19 implied HN points • 19 Dec 23
🕹 Technology Data Pipelines Stream Processing Data Storage Data processing
  1. One common use case for stream processing is transforming data into a format for different systems or needs.
  2. Bytewax is a Python stream processing framework that allows real-time data processing and customization.
  3. Bytewax enables creating custom connectors for data sources and sinks, making it versatile for various data processing tasks.

What Suboptimal Public Transit and Your Data Pipelines Have in Common

Bytes, Data, Action! • 19 implied HN points • 05 Sep 23
🕹 Technology Data Pipelines Public Transit Monitoring Data Orchestration Data Management
  1. Public transit and data pipelines both aim to move things from point A to point B smoothly and quickly.
  2. Issues like delays, lack of visibility, and missed connections can disrupt the experiences of both public transit and data pipelines.
  3. Efficient, transparent, and reliable practices are key to ensuring a smooth journey for both public transit users and data pipelines.

Orchestration isn’t going anywhere

Data People Etc. • 36 HN points • 24 Apr 23
🕹 Technology Data Management Data Pipelines Orchestration Software Engineering Organizational dynamics
  1. Orchestration is essential and will continue to be important in the future of managing data pipelines.
  2. Orchestration involves coordinating and managing multiple systems and tasks to execute workflows.
  3. Tools like Dagster provide a control plane for managing data assets and metadata, ensuring a structured and cohesive data platform.
Get a weekly roundup of the best Substack posts, by hacker news affinity: