The hottest Data Quality Substack posts right now

And their main takeaways
Category
Top Technology Topics

Improving data quality in online studies

UX Psychology • 19 implied HN points • 23 Nov 21
🔬 Science Data Quality
  1. In online studies, factors like distractions, poor equipment, and cheating can impact data quality.
  2. Engagement levels, accuracy, outliers, and speed of responses are key indicators to assess data quality in online studies.
  3. Strategies like consistency measures, attention checks, bot detection, and serious response checks can help improve data quality in online studies.

Sort Feature Overview: Better Data Through Collaboration

Database Engineering by Sort • 15 implied HN points • 01 Mar 24
🕹 Technology Data Quality
  1. Data quality is crucial for businesses as it influences customer experience, decision-making, and AI outcomes.
  2. Collaboration is key for improving data quality, as automated tools can only address a portion of data issues.
  3. Sort provides a platform for transparent collaboration on databases, allowing for public and private database sharing, issue tracking, proposing and reviewing database changes.

Gradient Flow #23: AI Liabilities, Data Quality, Robust Language Models

Gradient Flow • 19 implied HN points • 03 Dec 20
🕹 Technology Data Quality
  1. Adversarial attacks in NLP models and computer vision models have been a growing concern, leading to research on generating defences and examples.
  2. Tools like the SDV library from MIT can generate synthetic data for testing various applications beyond just machine learning models.
  3. Companies and startups are increasingly addressing the importance of high-quality data through projects like Apache Griffin and Deequ.
Get a weekly roundup of the best Substack posts, by hacker news affinity:

Why Data Quality Is More Important Than Ever in an AI-Driven World

Data Products • 5 implied HN points • 08 Jan 24
🕹 Technology Data Quality
  1. Data quality is crucial for machine learning projects and can have negative impacts on both society and individuals.
  2. Advances in Generative AI highlight the importance of high-quality data and the potential shortage of such data.
  3. Data quality affects the machine learning product development cycle, including ongoing maintenance costs of ML pipelines.

The extensive guide for Server-Side Tracking

timo's substack • 1 HN point • 16 May 23
🕹 Technology Data Quality
  1. Take control of event data by implementing server-side tracking for better data quality and faster implementation.
  2. Incorporate the development team in tracking projects from the start to achieve more effective server-side tracking implementations.
  3. Consider different strategies for implementing server-side tracking, such as close to the API layer, stream, database, third-party applications, or application code.

The True Cost of Data Debt

Data Products • 5 implied HN points • 11 Oct 23
🕹 Technology Data Quality
  1. Data should be seen as an asset, not just a resource.
  2. Data debt can lead to serious consequences like trust issues and organizational chaos.
  3. Data developers need to focus on data quality tools like data contracts to prevent and manage data debt.

The Data Quality Resolution Process

Data Products • 3 implied HN points • 11 Dec 23
🕹 Technology Data Quality
  1. Stakeholders surface data quality issues, managers must balance responsiveness without burnout.
  2. Prioritize issues by determining urgency, impact, and potential solutions.
  3. Communicate clearly with technical stakeholders, implement fixes cautiously, and maintain trust through thorough communication.

The Consumer-Defined Data Contract

Data Products • 3 implied HN points • 04 Dec 23
🕹 Technology Data Quality
  1. Producers need to move towards consumer-defined data contracts to improve data quality and alignment with user needs.
  2. A phased approach of awareness, collaboration, and contract ownership helps in successful data contract adoption.
  3. Starting with consumer-defined contracts drives communication, awareness, and problem visibility, leading to long-term benefits.

Data Contracts Book - Early Release

Data Products • 2 implied HN points • 27 Feb 24
🕹 Technology Data Quality
  1. Chad Sanderson announced an upcoming book on Data Contracts with O'Reilly, covering topics like what data contracts are, how they work, implementation, examples, and the future implications. The book will delve into Data Quality and Governance.
  2. The first two chapters of the book are available for free on the O'Reilly website. They cover the importance of data contracts and the real goals of data quality initiatives, totaling about 45 pages of content.
  3. Chad Sanderson is currently selecting technical reviewers for the book. Interested individuals can reach out to him to share their thoughts on an advance copy.

A step towards self-improving LLMs

Artificial Fintelligence • 4 implied HN points • 07 Mar 23
🕹 Technology Data Quality
  1. Models need to generate data by themselves for self-improvement, seen in examples like AlphaZero.
  2. Models should adapt to new domains without requiring vast existing data, like the CLIP model.
  3. Improving efficiency of models, like auto regressive sampling, is crucial for advancement in AI development.

Peak Data

East Wind • 2 HN points • 25 Oct 23
🕹 Technology Data Quality
  1. The quality and percentage of human-generated data on the internet may have reached a peak, affecting the efficacy of future AI models.
  2. Models may face challenges with outdated training data and lack of relevant information for solving newer problems.
  3. Potential solutions include leveraging RAG models, proactive data contribution by platform vendors, and maintaining incentives for human contributions on user-generated content platforms.

Gradient Flow #32: Data Cascades, Demand for Data Engineers, Exploiting ML models

Gradient Flow • 0 implied HN points • 08 Apr 21
🕹 Technology Data Quality
  1. Data quality is essential for great AI products and services, emphasizes the need for tools like Great Expectations for validation and testing.
  2. There is a rising demand for data engineers, illustrated by the funding announcements of Streamlit, Flatfile, and Snorkel.
  3. Exploiting machine learning pickle files is a concern, with an open source tool discussed to reverse engineer and test these files.

GroupBy #4: Polars and Pandas, 1.8 trillion events, data quality

VuTrinh. • 0 implied HN points • 10 Oct 23
🕹 Technology Data Quality
  1. Polars and Pandas are tools for data processing, but they have different performance levels. Understanding when to use each can help manage large datasets better.
  2. Data quality is crucial for successful data engineering. Companies like Google and Uber have strategies in place to ensure their data is accurate and reliable.
  3. Learning SQL execution order can really help in data tasks. It outlines the steps SQL takes to process a query, which is key for optimizing database interactions.

The buck stops here

Brave New Teams • 0 implied HN points • 25 Jan 26
🕹 Technology Data Quality
  1. AI has made basic competence—drafting, summarising and producing text—cheap and abundant, so markets now reward people who deliver real results, not just plausible outputs. That shifts value toward asking the right questions and owning the consequences of decisions.
  2. Three human scarcities remain valuable: setting ends and moral choices (and taking the blame), verifying models with fresh real-world signals, and winning acceptance through trust and relationships. These tasks require being inside institutions and doing hard fieldwork, not just producing words.
  3. Work will shift from content production to governance: people will be paid to edit, test, decide and take responsibility while AI handles generation. The mediocre who only produce plausible text without owning outcomes will be displaced, while skilled operators who bind AI to reality, responsibility and trust will win.