The hottest Stream Processing Substack posts right now

And their main takeaways
Category
Top Technology Topics

SAI Notes #01: Watermarks in Stream Processing, SQL Query order of Execution.

SwirlAI Newsletter • 255 implied HN points • 07 May 23
🕹 Technology Data processing Stream Processing Data Engineering Data Systems
  1. Watermarks in Stream Processing help handle event lateness and decide when to treat data as 'late data'.
  2. In SQL Query execution, the order is FROM and JOIN, WHERE, GROUP BY, HAVING, SELECT, ORDER BY, LIMIT.
  3. To optimize SQL Queries, reduce dataset sizes for joins and use subqueries for pre-filtering.

The stream processing model behind Google Cloud Dataflow

VuTrinh. • 39 implied HN points • 27 Apr 24
🕹 Technology Data processing Cloud Computing Big Data Software Engineering Stream Processing
  1. Google Cloud Dataflow is a service that helps process both streaming and batch data. It aims to ensure correct results quickly and cost-effectively, useful for businesses needing real-time insights.
  2. The Dataflow model separates the logical data processing from the engine that runs it. This allows users to choose how they want to process their data while still using the same fundamental tools.
  3. Windowing and triggers are important features in Dataflow. They help organize and manage how data is processed over time, allowing for better handling of events that come in at different times.

GroupBy #25: From Samza to Flink: A Decade of Stream Processing, DoorDash’s In-House Search Engine,Meta's DotSlash, Designing Metrics Trees

VuTrinh. • 19 implied HN points • 05 Mar 24
🕹 Technology Data Engineering Software Development Stream Processing Data Visualization
  1. Stream processing has evolved significantly over the years, with frameworks like Samza and Flink leading the way in handling real-time data streams.
  2. DoorDash developed its own search engine using Apache Lucene, achieving impressive performance improvements, like reduced latency and lower hardware costs.
  3. Understanding metrics trees is essential for businesses as they visually represent how different inputs contribute to outputs, helping in decision-making.

Limits of the Event-Driven Orchestrator

Data People Etc. • 106 implied HN points • 03 Apr 23
🕹 Technology Data processing Workflow Orchestration Stream Processing
  1. Event-driven orchestrators are not suitable for stream processing because they cannot handle tasks with definite starts and ends.
  2. Event-driven applications operate asynchronously by triggering tasks based on events like files appearing in a directory.
  3. Unlike stream processors, orchestrators like Airflow and Dagster do not have the ability to hold state, distribute tasks for parallel execution, or shuffle data between tasks.
Get a weekly roundup of the best Substack posts, by hacker news affinity:

Data Pipelines - Streams to Parquet

Bytewax • 19 implied HN points • 19 Dec 23
🕹 Technology Data Pipelines Stream Processing Data Storage Data processing
  1. One common use case for stream processing is transforming data into a format for different systems or needs.
  2. Bytewax is a Python stream processing framework that allows real-time data processing and customization.
  3. Bytewax enables creating custom connectors for data sources and sinks, making it versatile for various data processing tasks.

Memphis: Building Stream Processing that Doesn't Suck (aka a Delightful Dev First Experience)...Welcome to the Boldstart Family!

Software Snack Bites • 50 implied HN points • 28 Jun 23
🕹 Technology Stream Processing Developer Experience Data Engineering Real-Time Data
  1. Memphis provides a better developer experience for stream processing.
  2. Memphis is designed for quick setup, cost efficiency, and user-friendly monitoring.
  3. Memphis is a platform of choice for companies looking to replace or enhance their streaming platforms.

GroupBy #21: How to design resilient and large scale data systems, What Data Modeling is NOT

VuTrinh. • 0 implied HN points • 06 Feb 24
🕹 Technology Data Systems Data Modeling Stream Processing Data Engineering Big Data
  1. Designing data systems requires resilience and scalability, which means they should handle growth and failures efficiently.
  2. Data modeling is more than just making diagrams; it's about understanding the entire system and how data flows within it.
  3. Using tools like DuckDB in the browser can open up new possibilities for data processing, making it more accessible and flexible.

Operational Use case Patterns for Apache Kafka and Flink — Part 1

Tributary Data • 0 implied HN points • 03 Jan 23
🕹 Technology Data Platforms Stream Processing Microservices Data Management
  1. Operational use cases with Kafka and Flink are crucial for business operations due to their message ordering, low latency, and exactly-once delivery guarantees.
  2. Using polyglot persistency with different data stores for read and write purposes can help solve the mismatch between write and read paths in microservices data management.
  3. Implementing a backend rate limiter using Flink as a Kafka consumer can help prevent exhausting an external system (e.g., a database) due to high message arrival rates from Kafka.