The hottest Data Pipelines Substack posts right now

And their main takeaways
Category
Top Technology Topics

Backfills - The Necessary Evil of Data Engineering

SeattleDataGuy’s Newsletter • 906 implied HN points • 23 Feb 26
🕹 Technology Data Pipelines
  1. Backfills are an unavoidable part of data work — you need them when source data is corrected, pipelines have bugs, or schemas and logic change.
  2. They’re hated because they can be expensive, slow, and risky at scale, can disrupt downstream users, and erode stakeholder trust when numbers shift unexpectedly.
  3. Design for safe backfills by building parameterized, rerunnable pipelines, adding strong data quality checks, communicating changes clearly, and using table-swaps or other strategies when partitions or immutable storage formats make in-place fixes risky.

Why Alerting is Key for enabling Generative AI and Machine Learning

The Orchestra Data Leadership Newsletter • 79 implied HN points • 23 Apr 24
🕹 Technology Data Pipelines
  1. Alerting and governance are crucial for the success of Data and AI initiatives, as highlighted by the high failure rates of AI projects and Data Science projects not making it to production.
  2. Building trust between Data Teams and Business Stakeholders is essential, and alerting plays a key role in this by ensuring effective communication and collaboration during data pipeline failures.
  3. Effective alerting systems should be proactive, asset-based, and granular, allowing for quick detection and communication of issues to build trust and reliability in Data and AI products.

The "Brittleness" Problem in Data Pipelines.

Data Engineering Central • 157 implied HN points • 24 Apr 23
🕹 Technology Data Pipelines
  1. Brittleness in data pipelines can lead to various issues like data quality problems, difficult debugging, and slow performance.
  2. To overcome brittle pipelines, focus on addressing data quality issues through monitoring, sanity checks, and using tools like Great Expectations.
  3. Development issues such as lack of tests, poor documentation, and bad code practices contribute to brittle pipelines; implementing best practices like unit testing and Docker can help improve pipeline reliability.

A dbt cloud alternative with no-code ELT: end-to-end data pipelines in Coalesce on Snowflake

The Orchestra Data Leadership Newsletter • 39 implied HN points • 09 Jan 24
🕹 Technology Data Pipelines
  1. The article discusses building a data release pipeline to analyze Hubspot data using Coalesce, a no-code ELT tool on Snowflake.
  2. One key issue encountered was the challenges with Hubspot's data model when trying to consolidate form fill data and messages into a meaningful view.
  3. Setting up Coalesce involves defining storage mappings, granting access to Coalesce users, and carefully handling environments to prevent data overwriting when working between development and production.
Get a weekly roundup of the best Substack posts, by hacker news affinity:

How I use Gen AI as a Data Engineer

The Orchestra Data Leadership Newsletter • 19 implied HN points • 05 Apr 24
🕹 Technology Data Pipelines
  1. Generative AI can help Data Engineers summarize vast quantities of structured and unstructured data, expanding the breadth and depth of data available.
  2. Feature engineering using Generative AI involves ingesting unstructured data like call notes, making API calls, and transforming the data for analysis in existing pipelines.
  3. Utilizing Generative AI for webscraping can help teams extract information efficiently from the internet, enabling monitoring of new data sources and optimizing business processes.

How to build a dual Incremental + snapshot data ingestion pipeline

Practical Data Engineering Substack • 59 implied HN points • 01 Oct 23
🕹 Technology Data Pipelines
  1. You can improve data accuracy by using two pipelines: one for getting recent updates quickly and another for regularly loading the entire dataset. This helps in keeping the data reliable over time.
  2. It's essential to manage pipeline scheduling based on your business's needs, like how often you need updates. You can choose faster updates or less frequent full reloads depending on how critical the data is.
  3. Using tools like Apache Airflow can help organize these pipelines efficiently. You can simplify tasks by dynamically generating them from a list, making it easier to handle many data tables.

Data Pipelines - Streams to Parquet

Bytewax • 19 implied HN points • 19 Dec 23
🕹 Technology Data Pipelines
  1. One common use case for stream processing is transforming data into a format for different systems or needs.
  2. Bytewax is a Python stream processing framework that allows real-time data processing and customization.
  3. Bytewax enables creating custom connectors for data sources and sinks, making it versatile for various data processing tasks.

What Suboptimal Public Transit and Your Data Pipelines Have in Common

Bytes, Data, Action! • 19 implied HN points • 05 Sep 23
🕹 Technology Data Pipelines
  1. Public transit and data pipelines both aim to move things from point A to point B smoothly and quickly.
  2. Issues like delays, lack of visibility, and missed connections can disrupt the experiences of both public transit and data pipelines.
  3. Efficient, transparent, and reliable practices are key to ensuring a smooth journey for both public transit users and data pipelines.

How we think about Data Pipelines is changing

The Orchestra Data Leadership Newsletter • 0 implied HN points • 08 Nov 23
🕹 Technology Data Pipelines
  1. Data pipelines are transitioning towards a focus on reliability and efficiency, similar to software engineering practices.
  2. Continuous Data Integration and Delivery in data engineering involves releasing data into production in response to code changes in a simple manner.
  3. Observability and metadata gathering play a crucial role in ensuring data quality and preventing issues before they occur in data pipelines.

Managing Dependencies Between Data Pipelines

Practical Data Engineering Substack • 0 implied HN points • 26 Aug 23
🕹 Technology Data Pipelines
  1. Managing dependencies between data pipelines is crucial for ensuring that upstream tasks are completed before downstream tasks start. This avoids issues with incomplete or faulty data.
  2. There are different techniques to manage these dependencies, ranging from simple time-based scheduling to more complex orchestrations that adjust based on the successful completion of previous tasks.
  3. Choosing the right method for managing pipeline dependencies depends on the complexity of the data workflows and the need for independence between different teams and tasks.