The hottest Data Engineering Substack posts right now

And their main takeaways
Category
Top Technology Topics

How does Uber build real-time infrastructure to handle petabytes of data every day?

VuTrinh. • 659 implied HN points • 23 Mar 24
🕹 Technology Data Engineering Infrastructure Real-Time Processing Open Source Big Data
  1. Uber handles huge amounts of data by processing real-time information from drivers, riders, and restaurants. This helps them make quick decisions, like adjusting prices based on demand.
  2. They use a mix of open-source tools like Apache Kafka for data streaming and Apache Flink for processing, which allow them to scale their operations smoothly as the business grows.
  3. Uber values data consistency, high availability, and quick response times in their infrastructure. This means they need reliable systems that work well even when they're overloaded with data.

Data Science Weekly - Issue 529

Data Science Weekly Newsletter • 999 implied HN points • 12 Jan 24
🕹 Technology Data science Artificial Intelligence Machine Learning Data Engineering Software Development
  1. Using ChatGPT can help you budget better. It can track and categorize your spending easily.
  2. When coding, it's important to find a balance between moving quickly and keeping your code well-structured. This is a real challenge for many developers.
  3. Language models, like GPT-4, are becoming very advanced, but there are big philosophical questions about what that really means for intelligence and understanding.

GroupBy #43: Uber | Kafka - The Tiered Storage

VuTrinh. • 139 implied HN points • 09 Jul 24
🕹 Technology Data Engineering Software Development Cloud Computing Information Systems Big Data
  1. Uber recently introduced Kafka Tiered Storage, which allows storage and compute resources to work separately. This means you can add storage without needing to upgrade processing power.
  2. The tiered storage system has two parts: local storage for fast access and remote storage for long-term data. This setup helps manage data efficiently and keeps the local storage less cluttered.
  3. When you need older data, it can be accessed directly from the remote storage, allowing faster performance for applications that need quick access to recent messages.

GroupBy #44: Meta | The Data Stack

VuTrinh. • 119 implied HN points • 16 Jul 24
🕹 Technology Data Engineering Infrastructure Data Analytics Software Development Real-Time Processing
  1. Meta uses a complex data warehouse to manage millions of tables and keeps data only as long as it's needed. Data is organized into namespaces for efficient querying.
  2. They built tools like iData for data discovery and Scuba for real-time analytics. These tools help engineers find and analyze data quickly.
  3. Data engineers at Meta develop pipelines mainly with SQL and Python, using internal tools for orchestration and monitoring to ensure everything runs smoothly.

Data Science Weekly - Issue 527

Data Science Weekly Newsletter • 959 implied HN points • 29 Dec 23
🕹 Technology Data science Machine Learning Artificial Intelligence Data Engineering Analytics
  1. This week, there's a focus on using data science techniques for practical decision-making, highlighted by an interview with Steven Levitt, who discusses making tough choices using data.
  2. There's a roundup of AI developments from 2023, showing how the field has evolved over the past year, which can help professionals stay updated.
  3. Understanding data quality is essential, as it directly impacts how useful data is for decision-making and analysis in any organization.
Get a weekly roundup of the best Substack posts, by hacker news affinity:

GroupBy #40: Data Infrastructure at Airbnb

VuTrinh. • 179 implied HN points • 18 Jun 24
🕹 Technology Data Engineering Software Development Infrastructure Open Source Scalability
  1. Airbnb focuses on using open-source tools and contributing back to the community. This helps them build a strong and collaborative data infrastructure.
  2. Their data infrastructure prioritizes scalability and uses specific clusters for different types of jobs. This approach ensures that critical tasks run efficiently without overwhelming the system.
  3. Airbnb has improved their data processing performance significantly, reducing costs while increasing speed. This was achieved through careful planning and migration of their Hadoop clusters.

How does Uber handle petabytes of Spark shuffle data every day?

VuTrinh. • 159 implied HN points • 22 Jun 24
🕹 Technology Data Engineering Big Data Cloud Computing Software Development Distributed Systems
  1. Uber uses a Remote Shuffle Service (RSS) to handle large amounts of Spark shuffle data more efficiently. This means data is sent to a remote server instead of being saved on local disks during processing.
  2. By changing how data is transferred, the new system helps reduce failures and improve the lifespan of hardware. Now, servers can handle more jobs without crashing and SSDs last longer.
  3. RSS also streamlines the process for the reduce tasks, as they now only need to pull data from one server instead of multiple ones. This saves time and resources, making everything run smoother.

Data Science Weekly - Issue 554

Data Science Weekly Newsletter • 119 implied HN points • 04 Jul 24
🕹 Technology Data science AI Machine Learning Data Engineering Visualization
  1. Staying updated in data science, AI, and machine learning is essential for improving skills and knowledge. Weekly newsletters provide curated articles and resources that help you keep up with the latest trends.
  2. Effective structuring of data science teams can greatly enhance productivity. Learning from past experiences on team reorganizations can help in clarifying roles and increasing effectiveness.
  3. Building interactive dashboards in Python can make data more accessible. Using tools like PostgreSQL and specific libraries can simplify the process and enhance data visualization.

Issue #10 - The Data Lifecycle

The Data Ecosystem • 159 implied HN points • 16 Jun 24
🕹 Technology Data science Data Management Data security Data Analysis Data Engineering Data Visualization
  1. The data lifecycle includes all the steps from when data is created until it is no longer needed. This helps organizations understand how to manage and use their data effectively.
  2. Different people and companies might describe the data lifecycle in slightly different ways, which can be confusing. It's important to have a clear understanding of what each term means in context.
  3. Properly managing data involves stages like storage, analysis, and even disposal or archiving. This ensures data remains useful and complies with regulations.

Data Science Weekly - Issue 550

Data Science Weekly Newsletter • 179 implied HN points • 07 Jun 24
🕹 Technology Data science AI Machine Learning Computing Data Engineering
  1. Curiosity in data science is important. It's essential to critically assess the quality and reliability of the data and models we use, especially when making claims about complex issues like COVID-19.
  2. New fields, like neural systems understanding, are blending different disciplines to explore complex questions. This approach can help unravel how understanding works in both humans and machines.
  3. Understanding AI advancements requires keeping track of evolving resources. It’s helpful to have a well-organized guide to the latest in AI learning resources as the field grows rapidly.

Data Science Weekly - Issue 555

Data Science Weekly Newsletter • 99 implied HN points • 11 Jul 24
🕹 Technology Data science AI Machine Learning Data Engineering Data Visualization
  1. Large language models can sometimes create false or confusing information, a problem known as hallucination. Understanding the cause of these mistakes can help improve their accuracy.
  2. Good data visualizations are important to effectively communicate patterns and insights. Poorly designed visuals can lead to misunderstandings, especially among those not familiar with graphics.
  3. There's an ongoing debate about copyright in the context of generative AI. Many believe it would be better to focus on finding compromises rather than pursuing strict legal battles.

Data Science Weekly - Issue 552

Data Science Weekly Newsletter • 139 implied HN points • 20 Jun 24
🕹 Technology Data science Artificial Intelligence Machine Learning Data Engineering Data Visualization
  1. Notebooks can be easy to use, but they might make you lazy in coding. It's important to follow good practices even when using them.
  2. When handling large datasets, it's crucial to learn how to scale effectively. Knowing how to use resources wisely can help you reach your goals faster.
  3. Retrieval Augmented Generation (RAG) can improve how models generate information. It's complex, but understanding it can boost the performance of your projects.

Data Science Weekly - Issue 556

Data Science Weekly Newsletter • 79 implied HN points • 18 Jul 24
🕹 Technology Data science Artificial Intelligence Machine Learning Programming Data Engineering
  1. AI research in China is progressing rapidly, but it hasn't received much attention compared to developments in the US. There are many complexities in understanding the implications of this advancement.
  2. There are new methods to improve large language models (LLMs) using production data, which can enhance their performance over time. A structured approach to analyzing data quality can lead to better outcomes.
  3. Evaluating modern machine learning models can be challenging, leading to some questionable research practices. It's important to understand these issues to ensure more accurate and reproducible results.

The Architecture of Apache Druid

VuTrinh. • 139 implied HN points • 15 Jun 24
🕹 Technology Data Engineering Data architecture Big Data
  1. Apache Druid is built to handle real-time analytics on large datasets, making it faster and more efficient than Hadoop for certain tasks.
  2. Druid uses a variety of node types—like real-time, historical, broker, and coordinator nodes—to manage data, process queries, and ensure everything runs smoothly.
  3. The architecture allows for quick data retrieval while maintaining high availability and performance, making it a strong choice for applications that need fast, interactive data exploration.

sqlmesh init -t dlt --dlt-pipeline bluesky duckdb

davidj.substack • 71 implied HN points • 05 Dec 24
🕹 Technology Software Data Engineering APIs Databases Automation
  1. Using dlt to work with Bluesky API allows for easy data extraction. It saves time by handling metadata and schema changes automatically.
  2. dlt simplifies dealing with nested data by creating separate tables. This makes it easier to manage complex data structures.
  3. sqlmesh can quickly generate SQL models based on dlt pipelines. This feature streamlines the workflow and reduces manual setup time.

dlt windsurfing

davidj.substack • 71 implied HN points • 04 Dec 24
🕹 Technology Software Data Engineering AI Programming APIs
  1. dlt is a Python tool that helps organize messy data into clear, structured datasets. It's easy to use and can quickly load data from many sources.
  2. Using AI tools like Windsurf can make coding feel more collaborative. They help you find solutions faster and reduce the burden of coding from scratch.
  3. Storing data in formats like parquet can make processing much quicker. Simplifying your data handling can save you a lot of time and resources.

sqlmesh migrate

davidj.substack • 47 implied HN points • 20 Dec 24
🕹 Technology Software Data Engineering Programming Cloud Computing Analytics
  1. If you're using dbt to run analytics, switching to sqlmesh is a good idea. It offers more features and is easy to learn while still being compatible with dbt.
  2. sqlmesh helps manage data environments and is more comprehensive in handling analytics tasks compared to dbt. It's simpler to transition from dbt to sqlmesh than from older methods like stored procedures.
  3. When using sqlmesh, think about where to run it and how to store its state. You have choices like using a different database or a cloud service, which can save you money and hassle.

Data Science Weekly - Issue 549

Data Science Weekly Newsletter • 159 implied HN points • 31 May 24
🕹 Technology Data science Artificial Intelligence Machine Learning Data Engineering Cloud Computing Software Development
  1. Mediocre machine learning can be very risky for businesses, as it may lead to significant financial losses. Companies need to ensure their ML products are reliable and efficient.
  2. Understanding logistic regression can be made easier by using predicted probabilities. This approach helps in clearly presenting data analysis results, especially to those who may not be familiar with technical terms.
  3. Data quality management is becoming essential in today's data-driven world. It's important to keep track of how data is tested and monitored to maintain trust and accuracy in business decisions.

Data Science Weekly - Issue 553

Data Science Weekly Newsletter • 99 implied HN points • 27 Jun 24
🕹 Technology Data science Artificial Intelligence Machine Learning Data Engineering Data Visualization
  1. Data visualization can show important patterns, like changes in night and daylight globally. Understanding these trends helps us appreciate our environment better.
  2. In AI engineering, simplifying data preparation is crucial. Many new AI applications can be built without structured data, which might lead to rushed expectations about their effectiveness.
  3. Aquaculture technology is evolving with better methods to track and analyze fish behavior. New approaches like deep learning are making monitoring more accurate and efficient.

Growing From Analyst To Data Engineer In 100 Days

SeattleDataGuy’s Newsletter • 1165 implied HN points • 02 Jan 24
🕹 Technology Data Engineering Skills Development Community Building
  1. Breaking into data engineering may be easier through lateral moves, like from data analyst to data engineer.
  2. The 100-day plan discussed is not meant to master data engineering but to help commit to learning and identify areas for improvement.
  3. The plan includes reviewing basics, diving deeper, building a mini project, surveying tools, best practices, and committing to a final project.

Data Science Weekly - Issue 541

Data Science Weekly Newsletter • 279 implied HN points • 05 Apr 24
🕹 Technology Data science AI Machine Learning Software Development Data Engineering
  1. AI agents have unique challenges that traditional laws may not effectively solve. New rules and systems are needed to ensure they are managed properly.
  2. JS-Torch is a new JavaScript library that makes deep learning easier for developers familiar with PyTorch. It allows building and training neural networks directly in the browser.
  3. Data acquisition is crucial for AI start-ups to succeed. There are strategies outlined to help these businesses gather the right data efficiently.

GroupBy #41: Uber’s Batch Data Infrastructure with Google Cloud Platform

VuTrinh. • 99 implied HN points • 25 Jun 24
🕹 Technology Data Engineering Cloud Computing Machine Learning Infrastructure Analytics
  1. Uber is moving its huge amount of data to Google Cloud to keep up with its growth. They want a smooth transition that won't disrupt current users.
  2. They are using existing technologies to make sure the change is easy. This includes tools that will help keep data safe and accessible during the move.
  3. Managing costs is a big concern for Uber. They plan to track and control spending carefully as they switch to cloud services.

Data Science Weekly - Issue 543

Data Science Weekly Newsletter • 219 implied HN points • 19 Apr 24
🕹 Technology Data science Machine Learning AI Analytics Data Engineering
  1. Statistical ideas have a big impact on the world. Learning about important papers can help us understand how statistics shape modern research and decision-making.
  2. Machine Learning teams have different roles that face unique challenges. Understanding these personas can help leaders support their teams better.
  3. Using vector embeddings can greatly improve search experiences in apps. They simplify processes that previously seemed too complex and highlight their usefulness in technology.

I spent 5 hours understanding more about the Delta Lake table format

VuTrinh. • 179 implied HN points • 04 May 24
🕹 Technology Data Engineering Database Performance optimization Software Development
  1. Delta Lake is designed to solve problems with traditional cloud object storage. It provides ACID transactions, making data operations like updates and deletions safe and reliable.
  2. Using Delta Lake, data is stored in Apache Parquet format, allowing for efficient reading and writing. The system tracks changes through a transaction log, which keeps everything organized and easy to manage.
  3. Delta Lake supports advanced features like time travel, allowing users to see and revert to past versions of data. This makes it easier to recover from mistakes and manage data over time.

Data Science Weekly - Issue 548

Data Science Weekly Newsletter • 139 implied HN points • 24 May 24
🕹 Technology Data science Machine Learning Artificial Intelligence Data Visualization Data Engineering
  1. Good communication is key for statisticians to explain their complex work to non-experts. Finding ways to relate data to everyday situations can make it easier for others to understand.
  2. Using histograms can speed up the training process for gradient boosted machines in data science. This simple technique can improve efficiency significantly.
  3. There are efforts to use machine learning algorithms to detect type 1 diabetes in children earlier. This can help avoid serious health issues by improving recognition of symptoms.

GroupBy #38: Modernizing Uber’s Batch Data Infrastructure with Google Cloud Platform, Apache Iceberg - What Is It

VuTrinh. • 119 implied HN points • 04 Jun 24
🕹 Technology Data Engineering Cloud Computing Data Infrastructure Machine Learning Open Source
  1. Uber is upgrading its data system by moving from its huge Hadoop setup to Google Cloud Platform for better efficiency and performance.
  2. Apache Iceberg is an important tool for managing data efficiently, and it can help create a more organized data environment.
  3. Building data products requires a strong foundation in data engineering, which includes understanding the tools and processes involved.

Procella - The query engine at YouTube

VuTrinh. • 79 implied HN points • 29 Jun 24
🕹 Technology Data Engineering Cloud Computing Database Systems Analytics
  1. YouTube built Procella to combine different data processing needs into one powerful SQL query engine. This means they can handle many tasks, like analytics and reporting, without needing separate systems for each task.
  2. Procella is designed for high performance and scalability by keeping computing and storage separate. This makes it faster and more efficient, allowing for quick data access and analysis.
  3. The engine uses clever techniques to reduce delays and improve response times, even when many users are querying at once. It constantly optimizes and adapts, making sure users get their data as quickly as possible.

GroupBy #36: Agoda- How We Solve Load Balancing Challenges in Apache Kafka, How to reduce your Snowflake cost

VuTrinh. • 139 implied HN points • 21 May 24
🕹 Technology Data Engineering Software Development Cloud Computing Infrastructure Cost Optimization
  1. Working on pet projects is fun, but it's important to have clear learning goals to actually gain knowledge from them.
  2. When using tools like Spark or Airflow, always ask what problem they solve to understand their value better.
  3. To make your projects more effective, think like a user and check if they get what they need from your data systems.

Data Science Weekly - Issue 539

Data Science Weekly Newsletter • 259 implied HN points • 22 Mar 24
🕹 Technology Data science AI Machine Learning Data Engineering Data Visualization
  1. Data storytelling is important for sharing insights, and AI can help people create better stories. The research looks at how different tools assist in each storytelling stage.
  2. Switching from R to Python in data science isn't just about learning new syntax; it's a mindset change. New Python tools can help make this transition smoother for users coming from R's tidyverse.
  3. Emerging technologies often face skepticism, as seen throughout history. New inventions have raised concerns about their impact, but they eventually become part of everyday life.

Data Science Weekly - Issue 532

Data Science Weekly Newsletter • 379 implied HN points • 02 Feb 24
🕹 Technology Data science Machine Learning Artificial Intelligence Data Engineering Data Visualization
  1. Forecasting in data science is challenging because time series data can be non-stationary. Using the right evaluation methods can help bridge the gap between traditional and modern forecasting techniques.
  2. It's important to consider the smartness of your data structures. Creating overly complicated dashboards that ultimately just produce simple outputs may not be the best use of time.
  3. There are clear distinctions between well-built data pipelines and amateur setups. Understanding what makes a pipeline production-grade can improve the quality and reliability of data processing.

Monthly Python Data Engineering, September 2024

Monthly Python Data Engineering • 2 HN points • 26 Sep 24
🕹 Technology Data Engineering Open Source Python Software Development Libraries
  1. A new free book called 'How Data Platforms Work' is being created for Python developers. It will explain the inner workings of data platforms in simple terms, with one chapter released each month.
  2. The Ibis library has removed the Pandas backend and now uses DuckDB, which is faster and has fewer dependencies. This change is expected to improve performance and usability.
  3. Several popular libraries in Python, such as GreatTables and Shiny, have released updates with new features and improvements, focusing on better usability and integration with modern technologies.

Data Science Weekly - Issue 533

Data Science Weekly Newsletter • 339 implied HN points • 09 Feb 24
🕹 Technology Data science Machine Learning Artificial Intelligence AI Research Data Engineering
  1. Satellite data is important for machine learning and should be treated as a unique area of research. Recognizing this can help improve how we use this data.
  2. Many data science and machine learning projects fail from the start due to common mistakes. Learning from past experiences can help increase the chances of success.
  3. Open source software plays a crucial role in advancing AI technology. It's important to support and protect open source AI from regulations that could harm its progress.

How to grow from a mid-level to senior Data Engineer

SeattleDataGuy’s Newsletter • 694 implied HN points • 14 Feb 24
🕹 Technology Data Engineering Software Engineering Learning Teamwork Leadership
  1. To grow from mid to senior level, it's important to continuously learn and improve, share new knowledge, work on code improvements, and become an expert in a certain domain.
  2. Making the team better is crucial - focus on mentoring, sharing knowledge, and creating a positive team environment. Think beyond individual tasks to impact the overall team outcomes.
  3. Seniority includes building not just technical solutions, but solutions that customers love. Challenge requirements, understand the business and product, and take initiative in problem-solving.

Data Science Weekly - Issue 544

Data Science Weekly Newsletter • 159 implied HN points • 26 Apr 24
🕹 Technology Data science Machine Learning Artificial Intelligence Data Engineering Data Visualization
  1. Evaluating AI models can be expensive, but tools like lm-buddy and Prometheus help do it on cheaper hardware without high costs.
  2. Installing and deploying LLaMA 3 is made simple with clear guides that cover everything from setup to scaling effectively.
  3. Understanding best practices in machine learning is essential, and resources like the 'Rules of Machine Learning' provide valuable guidelines for beginners.

Finding Data Bugs in dbt Pull Requests

clkao@substack • 39 implied HN points • 17 Aug 24
🕹 Technology Data Engineering Software Development Open Source Machine Learning Quality Assurance
  1. Data bugs can be costly for companies, with bad data potentially costing up to 25% of their revenue. These issues often arise from problems in data-centric systems like dbt.
  2. Using dbt allows data engineers to implement software practices like version control and testing, helping to ensure the correctness of their data transformations. However, relying solely on post-processing tests has its limits.
  3. Manual spot checks are still crucial in ensuring data accuracy during code reviews. Tools like Recce aim to streamline this process, making it easier for developers to validate and document their changes.