The hottest Data Quality Substack posts right now

And their main takeaways
Category
Top Technology Topics

My take on data quality

Minimal Modeling • 304 implied HN points • 15 Mar 26
🕹 Technology Data Quality
  1. Treat queries as functions and start by defining anchors: maintain a compact one‑column list of unique IDs for each entity and document retention/archive rules so input data quality is clear.
  2. Represent attributes and links as clean two‑column datasets (anchor ID + value or anchor ID + anchor ID), filter out NULLs and sentinel values, canonicalize values, use only atomic types, and ensure uniqueness.
  3. Materialize those compact datasets and keep them updated with a pipeline so your data is correct by construction; from these trusted pieces you can build flat tables while avoiding common issues like duplicates, unclear identity, and messy JSON.

Backfills - The Necessary Evil of Data Engineering

SeattleDataGuy’s Newsletter • 906 implied HN points • 23 Feb 26
🕹 Technology Data Quality
  1. Backfills are an unavoidable part of data work — you need them when source data is corrected, pipelines have bugs, or schemas and logic change.
  2. They’re hated because they can be expensive, slow, and risky at scale, can disrupt downstream users, and erode stakeholder trust when numbers shift unexpectedly.
  3. Design for safe backfills by building parameterized, rerunnable pipelines, adding strong data quality checks, communicating changes clearly, and using table-swaps or other strategies when partitions or immutable storage formats make in-place fixes risky.

Why Data Pipelines Exist

SeattleDataGuy’s Newsletter • 788 implied HN points • 09 Feb 26
🕹 Technology Data Quality
  1. Data pipelines exist to create trust in your data by making it timely, accurate, consistent, recoverable, and scalable.
  2. They centralize and integrate siloed data so analysts, automations, and models can access well‑modeled, usable datasets.
  3. Build pipelines with clear business outcomes and ownership or they become costly technical liabilities; examples include reducing discounts, improving onboarding, and cutting support costs.

Is Undervalued Vacant Land just a Baltimore problem?

Progress and Poverty • 692 implied HN points • 12 Feb 26
🇺🇸 U.S. Politics Data Quality
  1. Chronic undervaluation of vacant land is a Baltimore-specific problem — other Maryland counties do not show the same widespread under-assessments.
  2. The state appraisal office has acknowledged the issue in Baltimore and begun fixes, which means the problem is correctable rather than systemic across SDAT.
  3. Fixes focus on better data quality and sales validation, proper use of the allocation method (use a single local land rate derived from prevailing improved-property values), and mapping land values to spot side-by-side inconsistencies.

Common Data Pipeline Patterns You’ll See in the Real World

SeattleDataGuy’s Newsletter • 859 implied HN points • 05 Jan 26
🕹 Technology Data Quality
  1. Data pipelines come in many shapes — from source standardization and amalgamation to enrichment, operational syncs, and even manual Excel-based processes — each built for different business needs.
  2. Common challenges are mapping and standardizing varied formats, keeping reliable IDs and timing for joins, and handling data quality and system-specific ingestion limits.
  3. Despite the variety, pipelines all aim to move and transform source data into usable outputs for analytics, operations, or ML, and they often follow the same extract-transform-load steps that can be automated and productionized.
Get a weekly roundup of the best Substack posts, by hacker news affinity:

Issue #16 - The Data Quality Conundrum (Part 2 – Solving)

The Data Ecosystem • 439 implied HN points • 28 Jul 24
🕹 Technology Data Quality
  1. Data quality isn't just a simple fix; it's a complex issue that requires a deep understanding of the entire data landscape. You can't just throw money at it and expect it to get better.
  2. It's crucial to identify and prioritize your most important data assets instead of trying to fix everything at once. Focusing on what truly matters will help you allocate resources effectively.
  3. Implementing tools for data quality is important but should come after you've set clear standards and strategies. Just using technology won’t solve problems if you don’t understand your data and its needs.

Issue #15 – The Data Quality Conundrum (Part 1 – Root Causes)

The Data Ecosystem • 399 implied HN points • 21 Jul 24
💼 Business Data Quality
  1. Poor data quality is a big problem for organizations, but it's often misunderstood. It's not just about fixing bad data; you need to figure out what's causing the issues.
  2. Data quality has many aspects, like accuracy and completeness. Good data helps businesses make better decisions, while bad data can cost a lot of money.
  3. To solve data quality issues, you need a complete approach that looks at different root causes. Simply fixing one part won't fix everything, and different sources might create new problems.

VC's Framework for Evaluating an AI Startup's Tech Stack. | VC Jobs

Venture Curator • 419 implied HN points • 06 Jun 24
🕹 Technology Data Quality
  1. The value proposition of AI companies now lies not just within models but predominantly in underpinning datasets, emphasizing the importance of data quality.
  2. When evaluating AI startups, VCs use frameworks to assess data quality, considering relevance, accuracy, coverage, and bias in the datasets used to train the AI models.
  3. To avoid investing in ineffectual AI startups, VCs focus on evaluating the processes behind data generation by asking questions about data automation, storage, access, processing, governance, and management.

Claude found me an error in the ANES

Never Met a Science • 72 implied HN points • 03 Feb 26
🇺🇸 U.S. Politics Data Quality
  1. An AI code assistant detected a subtle data error in a major survey where one variable was overwritten, preventing a misleading analysis result.
  2. AI tools are highly useful for routine data processing and quality control, catching problems automatically that researchers might otherwise miss.
  3. AI works best when given specific, domain-relevant examples or code, because vague checks can produce false positives or flag legitimate, documented values as errors.

Issue #12 – The Three Biggest Data Problems Companies Face

The Data Ecosystem • 239 implied HN points • 30 Jun 24
💼 Business Data Quality
  1. Companies often struggle with a data operating model that doesn't connect well with their other teams. This leads to isolation among data specialists, making it hard to work effectively.
  2. Data models, which are important for understanding and using data correctly, are often overlooked. When organizations don’t reference these models, they can drift further away from their goals.
  3. Many data quality issues come from deeper problems within the organization, like poor data governance and inconsistent processes. Fixing just the visible data quality issues won't solve the bigger problems.

Does data quality matter?

benn.substack • 1099 implied HN points • 22 Nov 24
🕹 Technology Data Quality
  1. Data quality is important for making both strategic and operational decisions, as inaccurate data can lead to poor outcomes. Good data helps companies know what customers want and improve their services.
  2. AI models can tolerate some bad data better than traditional methods because they average out inaccuracies. This means these models might not break as easily if some of the input data isn’t perfect.
  3. Businesses now care more about AI than they used to about regular data reporting. This shift in focus might make data quality feel more important, even if it doesn’t technically impact AI model performance as much.

How To Measure Data Quality - Issue 185

Data Analysis Journal • 235 implied HN points • 07 Feb 24
🕹 Technology Data Quality
  1. Data quality metrics are essential for measuring data governance and analytics success.
  2. There is no industry standard for defining poor-quality data; it varies based on context.
  3. Having specific KPIs for data quality is crucial to scale data governance initiatives and improve the state of data quality.

Why Alerting is Key for enabling Generative AI and Machine Learning

The Orchestra Data Leadership Newsletter • 79 implied HN points • 23 Apr 24
🕹 Technology Data Quality
  1. Alerting and governance are crucial for the success of Data and AI initiatives, as highlighted by the high failure rates of AI projects and Data Science projects not making it to production.
  2. Building trust between Data Teams and Business Stakeholders is essential, and alerting plays a key role in this by ensuring effective communication and collaboration during data pipeline failures.
  3. Effective alerting systems should be proactive, asset-based, and granular, allowing for quick detection and communication of issues to build trust and reliability in Data and AI products.

Lessons from customer evaluations of an AI product

The AI Frontier • 59 implied HN points • 18 Apr 24
🕹 Technology Data Quality
  1. Customers who have experience with AI products often have a better understanding of what to look for. They know what works and what doesn't, so they can more easily evaluate new AI tools.
  2. The quality of data is super important for AI performance. If the data is good, the answers will be better, so paying attention to data quality is key.
  3. Expectations around AI products can be tricky. Some people think AI is not useful, while others expect it to know everything. It's important to set clear expectations about what AI can do.

3 Years of ChatGPT

Olshansky's Newsletter • 22 implied HN points • 03 Dec 25
🕹 Technology Data Quality
  1. AI is already here as an amplifier of human intelligence and is being used daily across personal and professional tasks; agent-driven tools have massively increased productivity, especially for coding.
  2. High-quality, unique data and expert-labeled "golden" datasets are the most valuable assets for building useful AI systems; simple benchmarks and naive fine-tuning are limited, while reinforcement fine-tuning and dedicated context engineering will drive real gains.
  3. Practical changes are coming in the next few years: local inference stations, agentic e-commerce, consolidation of tooling, and new roles like context engineers and AI bootcamps; foundational roles like architects will remain and superintelligence isn’t expected soon.

Revisiting A/B tests in SaaS 💯

The SaaS Baton • 117 implied HN points • 26 Apr 23
🕹 Technology Data Quality
  1. Running A/B tests on SaaS products has unique challenges beyond just having enough users for statistically significant results.
  2. Incorporating minimal clear constraints in projects can drive creativity and productivity, as seen in Buffer's Build Week.
  3. Establishing indirect growth channels, like Gusto did with accounting firms, can create network effects and be a win-win for both parties.

Choosing a Data Quality Tool

Sarah's Newsletter • 359 implied HN points • 22 Feb 22
🕹 Technology Data Quality
  1. Data quality tools are essential for maintaining trust in data and preventing stakeholders from resorting to workaround solutions.
  2. Choosing the right data quality tool involves understanding the specific needs of your organization and considering factors like budget, technical resources, and overall data quality goals.
  3. There are different types of data quality tools available, including auto-profiling data tools, pipeline testing tools, infrastructure monitoring tools, and integrated solutions, each with unique characteristics and considerations for selection.

Will we ever have clean data?

benn.substack • 511 implied HN points • 28 Jul 23
🕹 Technology Data Quality
  1. Data quality is a tradeoff in balancing stability and agility.
  2. Data resiliency tools like SDF focus on tracing data lineage to improve debugging and fixing issues.
  3. Managing messy data often requires making choices between stability and adaptability in data infrastructure.

Joe's Nerdy Rants #4

Joe Reis • 78 implied HN points • 10 Jun 23
🕹 Technology Data Quality
  1. Encourage kids and others to interact more in real life, consider alternatives to college, find careers that can't be easily automated, and learn to coexist with AI.
  2. Embrace lifelong learning and be open to change in order to adapt to evolving technologies and industries.
  3. Read up on interesting articles about tech, AI, data, and business topics for insights and inspiration.

The Making of Whisper: An In-Depth Exploration of its Training Data and Process

Amgad’s Substack • 39 implied HN points • 22 Dec 23
🕹 Technology Data Quality
  1. OpenAI's Whisper ASR model stands out for its accuracy, made possible by releasing both its architecture and checkpoints under an open-source license, setting a new standard of innovation in the field.
  2. The training of AI models can be divided into supervised and unsupervised approaches, each with its unique strengths and limitations, with significant implications for achieving high-quality results.
  3. Data curation is a critical aspect of model training, with OpenAI showcasing the importance of maintaining data integrity through a meticulous process of automated filtering, manual inspection, and guarding against data leakage.

What is SaaS Observability?

Sarah's Newsletter • 139 implied HN points • 30 Aug 22
🕹 Technology Data Quality
  1. SaaS Observability sheds light on the health of all data and automations in SaaS tools.
  2. Business teams should not need to rely on technical-heavy tools to ensure their systems are working correctly.
  3. Having bad data quality and anomalies in automations can impact business operations significantly and require constant monitoring.

The Froyo Data Shop

Sarah's Newsletter • 159 implied HN points • 22 Mar 22
🕹 Technology Data Quality
  1. Self-service is about making choices with clear explanations and options.
  2. Raw data without context can lead to misinterpretation and flawed analysis.
  3. Data democratization needs testing, context building, and ongoing data literacy.

Shift-Left Analytics

SUP! Hubert’s Substack • 50 implied HN points • 22 Nov 24
🕹 Technology Data Quality
  1. Shift-left analytics means doing analysis early in the data process. This helps in getting insights faster and making quick decisions.
  2. It focuses on checking data quality right away, so only reliable data is used. This leads to more accurate insights and avoids problems caused by bad data.
  3. Collaboration between teams is encouraged in this approach. By working together from the start, everyone can ensure their analyses are useful and aligned with business goals.

The State of Data Testing in 2024: Why We Still Have Broken Windows

Inside Data by Mikkel Dengsøe • 49 implied HN points • 18 Nov 24
🕹 Technology Data Quality
  1. Data teams are overwhelmed by too many alerts from test failures. This leads to important issues being overlooked.
  2. It's crucial to focus on the right tests that have significant business impact rather than just mechanical tests. This means deeper insights into the data are needed.
  3. Sharing the responsibility for data quality across teams can improve the situation. When everyone understands their role, issues are resolved faster.

Should Data Teams care about Data Contracts?

The Orchestra Data Leadership Newsletter • 19 implied HN points • 05 Nov 23
🕹 Technology Data Quality
  1. Consider data contracts if your internal data changes often to ensure collaboration between software engineering and data engineering teams.
  2. If you have important metrics that depend on software engineering actions, like defining 'Active Users,' data contracts can help maintain data quality.
  3. In cases where software engineering and data engineering roles overlap, implementing data contracts can streamline data ingestion processes and improve data quality.

Why Snowflake’s Clone command changes the game for CI/CD in Data

The Orchestra Data Leadership Newsletter • 19 implied HN points • 26 Oct 23
🕹 Technology Data Quality
  1. The Snowflake Clone command allows for cheap and quick testing of data during Continuous Integration flows, showing significant cost and time improvement compared to traditional create table commands.
  2. Continuous Deployment can be facilitated through Snowflake clones, as they are relatively inexpensive, removing cost barriers for Data Teams in implementing effective CD processes.
  3. The Clone command, available since at least 2016, is invaluable for Data Teams as it enables CI/CD pipelines, essential for deploying data into production reliably and efficiently via a data release pipeline.

A Digital Cohort on Personalized Nutrition

Digital Epidemiology • 19 implied HN points • 02 Jun 23
🔬 Science Data Quality
  1. The study focused on personalized nutrition with a digital cohort of 1,000+ participants tracking various data for glucose level management.
  2. Developing a digital cohort requires intricate digital infrastructure and investment in user-friendly applications for high retention rates.
  3. Data quality assessment is crucial for multi-modal data collection, and the study achieved high completion rates with a focus on improving nutrition tracking.

Gradient Flow #46: Smarter Language Models; Data Engineering Trends

Gradient Flow • 99 implied HN points • 04 Nov 21
🕹 Technology Data Quality
  1. Data scientists should transition into social scientists in addition to being computer scientists.
  2. The report presents insights from a global online survey of 372 respondents on data engineering trends and challenges.
  3. Information on improvements in large language models, modernizing data integration, and the importance of data quality is shared in the podcast.

Data Quality

Data Thoughts • 39 implied HN points • 21 Jan 23
🕹 Technology Data Quality
  1. Data quality is all about how useful the data is for the specific task at hand. What is considered high quality in one situation might not be in another.
  2. There are several key aspects of data quality, including accuracy, completeness, consistency, and uniqueness. Each of these factors helps to determine how reliable the data is.
  3. Improving data quality involves preventing errors, detecting them when they occur, and repairing them. It's about making sure the data is accurate and useful over time.

The Future of Vector Search

Gradient Flow • 2 HN points • 13 Jun 24
🕹 Technology Data Quality
  1. When choosing a vector search system, focus on features like deployment scalability and performance efficiency to meet specific needs.
  2. To ensure reliability and security, opt for systems that offer built-in embedding pipelines and integrate with data governance tools.
  3. Prioritize data quality and transparency in AI applications, emphasizing reproducibility through sharing code, data, and detailed documentation.

Understanding Isn't Enough

Sarah's Newsletter • 59 implied HN points • 08 Feb 22
🕹 Technology Data Quality
  1. Value in data products comes from taking action, not just providing information.
  2. Vendors and data tools add significant value by influencing processes and saving time for users.
  3. Analytics products should aim to change behaviors by answering critical questions, prioritizing effectively, and continuously refining to ensure effectiveness.

March, Etc.

Data People Etc. • 53 implied HN points • 15 Mar 23
🕹 Technology Data Quality
  1. Intermediate data modeling can be valuable following Kimball design principles.
  2. Attending events like Data Council can provide insights and networking opportunities.
  3. Engaging in ongoing discussions and being part of a community can enhance the writing and learning experience.