The hottest Data science Substack posts right now

And their main takeaways
Category
Top Technology Topics

OpenAI’s biggest worry isn’t DeepSeek

Enterprise AI Trends • 253 implied HN points • 31 Jan 25
🕹 Technology Data science
  1. DeepSeek's release showed that simple reinforcement learning can create smart models. This means you don't always need complicated methods to achieve good results.
  2. Using more computing power can lead to better outcomes when it comes to AI results. DeepSeek's approach hints at cost-saving methods for training large models.
  3. OpenAI is still a major player in the AI field, even though some people think DeepSeek and others will take over. OpenAI's early work has helped it stay ahead despite new competition.

The Sequence Knowledge #697: The Most Important Theory in Modern AI Interpretability

TheSequence • 91 implied HN points • 05 Aug 25
🕹 Technology Data science
  1. Superposition is an important idea in AI that helps us understand how models can represent many concepts at once. This idea means that a single piece of data can hold multiple meanings, which is useful when analyzing complex information.
  2. There is a relevant paper that discusses superposition in cutting-edge AI models. Studying this paper can provide deeper insights into how modern AI understands and processes data.
  3. The concept of polysemanticity is linked to superposition and emphasizes the ability of AI models to interpret language and information in multiple ways. This flexibility is key to improving AI interpretation and performance.

Improve Conversational UIs Using Social Intelligence

Cobus Greyling on LLMs, NLU, NLP, chatbots & voicebots • 59 implied HN points • 09 Apr 24
🕹 Technology Data science
  1. Social intelligence is important for conversational AIs to feel more human-like. It helps them understand emotions and social cues better.
  2. A good conversational UI needs to consider cognitive, situational, and behavioral intelligence. This means the AI should know what you mean, the context of your words, and how to interact appropriately.
  3. Using more data and different types of information beyond just words can help improve how AIs communicate. This could include things like images and gestures to understand conversations better.

What Did You Think Getting Closer to AGI Would Be Like?

The Algorithmic Bridge • 318 implied HN points • 07 Dec 24
🕹 Technology Data science
  1. OpenAI's new model, o1, is not AGI; it's just another step in AI development that might not lead us closer to true general intelligence.
  2. AGI should have consistent intelligence across tasks, unlike current AI, which can sometimes perform poorly on simple tasks and excel on complex ones.
  3. As we approach AGI, we might feel smaller or less significant, reflecting how humans will react to advanced AI like o1, even if it isn’t AGI itself.

OpenAI Agent Query Planning Using LlamaIndex

Cobus Greyling on LLMs, NLU, NLP, chatbots & voicebots • 99 implied HN points • 05 Feb 24
🕹 Technology Data science
  1. An OpenAI agent can analyze information from multiple documents at once. This helps create detailed answers to queries based on several sources.
  2. Using the LlamaIndex framework, you can easily set up a system to manage and query PDF documents. This makes finding specific data more efficient.
  3. The agent can summarize financial data, showing how companies like Uber grow revenue over time. This is helpful for understanding trends in business performance.
Get a weekly roundup of the best Substack posts, by hacker news affinity:

Data Science Weekly - Issue 484

Data Science Weekly Newsletter • 439 implied HN points • 02 Mar 23
🕹 Technology Data science
  1. Data scientists need the right tools and environment to do their jobs effectively. Organizations can help by improving their data science infrastructure.
  2. Understanding how to choose and advocate for important metrics is vital for product teams. This can lead to significant growth in user engagement.
  3. A/B testing is crucial in fraud detection to compare models and determine their effectiveness. It can provide valuable insights that improve model performance.

Data Science Weekly - Issue 490

Data Science Weekly Newsletter • 379 implied HN points • 13 Apr 23
🕹 Technology Data science
  1. Data science is evolving quickly, and many new tools and techniques are being developed. This opens up exciting job opportunities in various fields like AI and machine learning.
  2. Using programming languages like R and SQL can extend beyond traditional data analysis. They can be powerful tools for creative applications in data science.
  3. Learning and implementing good practices in software development, such as automating tests and improving code efficiency, can save time and resources in data science projects.

American Economic History: Introduction

Brad DeLong's Grasping Reality • 253 implied HN points • 22 Jan 25
🚌 Education Data science
  1. The course will focus on American economic history without trying to create a single, simple story. Instead, it will look at different themes and questions week by week.
  2. An important question will be whether America is exceptional and in what ways. This can help us better understand history and economics.
  3. Students will not only learn about historical events but also get a taste of data science to analyze economic models and improve their analytical skills.

2023 Kaggle AI Report

Bojan’s Newsletter • 196 implied HN points • 10 Oct 23
🕹 Technology Data science
  1. Kaggle is a valuable platform for data science and ML career development
  2. Kaggle solutions often offer innovative insights ahead of research and industry trends
  3. Tabular data ML remains an important area in the field of machine learning

January Newsletter

RSS DS+AI Section • 11 implied HN points • 01 Jan 26
🕹 Technology Data science
  1. AI and large language models are advancing rapidly, with major companies and open-source projects pushing innovations in long-context reasoning, memory, and generative capabilities. Competition is driving frequent releases and new research on foundation models and video/world-models.
  2. Ethics, bias, interpretability, and regulation remain central concerns as real-world uses expand, prompting debates, lawsuits, and calls for better safety research. Work on interpretability is seen as especially important for progressing AI more safely.
  3. The community is focusing on practical adoption and professionalisation through tutorials, production tips, projects, workshops, a new journal, and competency frameworks. There are also learning opportunities, internships, and calls for volunteers to help shape best practices and careers.

Who fires a gun at a school? Data science provides eight archetypes of shooters

School Shooting Data Analysis and Reports • 39 implied HN points • 13 May 24
🇺🇸 U.S. Politics Data science
  1. Data science can create archetypes to understand different behaviors, like predicting customer preferences or identifying school shooter profiles.
  2. Using data analysis, it's possible to categorize and plan for different scenarios of school shooters based on past incidents.
  3. The first school shooter archetype is 'The Adolescent Insider,' comprising attributes like age, gender, victim count, typical outcomes, and likely circumstances.

The Counterfactual's poll #3

The Counterfactual • 59 implied HN points • 04 Apr 24
🕹 Technology Data science
  1. In April, readers can vote on research topics for the next article, making it a collaborative effort. This way, subscribers influence the content that gets created.
  2. Past topics have focused on empirical studies involving large language models and the readability of texts. This shows a trend toward practical investigations in the field.
  3. One of the proposed topics is about how language models might respond differently based on the month, which can lead to fun and insightful experiments.

Worldbuilding with data

Data People Etc. • 231 implied HN points • 11 Feb 25
🕹 Technology Data science
  1. Data is more powerful when it has a purpose. It should tell a clear story, otherwise it's just clutter.
  2. Building a strong data system is like creating a world. A good structure connects different pieces and helps everyone understand the bigger picture.
  3. Data engineering is important because it helps manage and present large amounts of information, making sure everything works smoothly and accurately.

In Data Science for Non-STEM Majors, Is Learning-by-Watching Live Calculating Possible? Likely? Reasonable to Expect?

Brad DeLong's Grasping Reality • 238 implied HN points • 28 Jan 25
🚌 Education Data science
  1. Students today need basic data science skills to succeed after graduation. It's like letting them leave school without knowing how to read or write.
  2. Teaching data science can be tricky because students have different backgrounds. Some find it confusing, while others think it's too basic.
  3. It's important to keep trying to teach data science. Finding the right way to do it is necessary for better education and understanding.

The Sequence Radar #679: From Model to Team: Several Models are Better than One: Sakana’s Blueprint for Collective AI

TheSequence • 105 implied HN points • 06 Jul 25
🕹 Technology Data science
  1. Sakana AI has a new way to use multiple models together for better AI performance. Instead of relying on one model, they combine many to think more like humans.
  2. Their approach, called AB-MCTS, helps the AI decide whether to explore new ideas or improve current ones. This makes the AI smarter and more flexible in how it solves problems.
  3. By using several models that learn from past tasks, this system can better handle different challenges. This means AI can become more reliable and efficient in real-life applications.

Week #2: Intuition Behind Conformal Prediction

Mindful Modeler • 379 implied HN points • 27 Dec 22
🔬 Science Data science
  1. Conformal prediction for classification works by ordering predictions from certain to uncertain, dividing them based on a user-defined confidence level.
  2. Conformal prediction consists of three main steps: training, calibration, and prediction, following a similar recipe across different algorithms.
  3. Different resampling strategies like k-fold cross-splitting and jackknife are used in conformal prediction, offering a balance between computation cost and prediction accuracy.

Proxy Fine-Tuning LLMs

Cobus Greyling on LLMs, NLU, NLP, chatbots & voicebots • 79 implied HN points • 26 Feb 24
🕹 Technology Data science
  1. Proxy fine-tuning lets you improve a language model's performance without changing its internal settings. It only uses the model's output to make adjustments.
  2. Combining different approaches, like retrieval and fine-tuning, can lead to better results with language models. It's about using the best methods together instead of relying on just one.
  3. Using proxy fine-tuning can help organizations better understand and organize their data. It encourages them to explore their information needs more deeply.

Grok's Secret Weapon: Prioritizing Truth Over Political Correctness

The Dossier • 212 implied HN points • 18 Feb 25
🕹 Technology Data science
  1. Grok stands out in AI by focusing on truth instead of political correctness. This helps it learn faster and respond better.
  2. Unlike other AI models, Grok gives detailed and nuanced answers, even on tough topics. This makes it smarter in reasoning and understanding complex issues.
  3. By embracing all kinds of information, Grok is set to become a major player in AI. Its approach could change how AI helps people across various industries.

Data Science Weekly - Issue 494

Data Science Weekly Newsletter • 319 implied HN points • 12 May 23
🕹 Technology Data science
  1. Open source AI is rapidly advancing, but may always lag behind the best quality models. It's great for innovation but has its limits.
  2. Many academic papers promise data sharing but often fail to deliver, which can hinder scientific research and verification.
  3. Understanding how to craft effective prompts is essential when using generative AI tools. This skill can greatly enhance the results you get from those tools.

Data Science Weekly - Issue 504

Data Science Weekly Newsletter • 239 implied HN points • 21 Jul 23
🕹 Technology Data science
  1. AI companies are complicated and must consider many factors like research, funding, and competition. Understanding these can help predict how they might evolve in the future.
  2. Debriefs, or team discussions after projects, can greatly boost team performance. They help everyone learn from experiences and improve future collaboration.
  3. New research shows that specific ingredient pairings in food can be explained by flavor networks. This indicates there are universal patterns in how different foods complement each other.

Data Science Weekly - Issue 493

Data Science Weekly Newsletter • 319 implied HN points • 05 May 23
🕹 Technology Data science
  1. Data scientists often lack key skills needed for the job, which can be frustrating for those hiring. It's important for data scientists to continually improve their skills and adapt to job requirements.
  2. There's a significant increase in data downtime and resolution times, signaling that overall data quality management needs improvement. Companies should focus on better data practices to enhance their operations.
  3. New programming languages, like Mojo, are emerging that aim to simplify coding and enhance user experience. These advancements can make programming more accessible and enjoyable for everyone.

Do Prediction Technologies Help Novices or Experts More?

New Things Under the Sun • 224 implied HN points • 27 Jan 25
🕹 Technology Data science
  1. AI can help both beginners and experts, but it depends on the tasks they are working on. Sometimes, beginners gain more because AI levels the playing field.
  2. In some cases, experts benefit more from AI. They can solve complex problems that AI cannot, while beginners still struggle with those.
  3. Prediction tools can make a big difference in innovation fields like mining and drug discovery. The impact varies based on expertise and the types of problems being addressed.

The Sequence Knowlege #693: A New Series About Interpretability in Foundation Models

TheSequence • 84 implied HN points • 29 Jul 25
🕹 Technology Data science
  1. Understanding AI black boxes, especially complex models, is very important for safety and trust. People need to know how these AIs make decisions.
  2. Interpretability in AI refers to making sense of how these intelligent systems work. It's about bridging the gap between what we can do with AI and understanding it.
  3. The series will discuss practical ways to interpret these AI models and review significant papers related to the topic. Learning from research is key to improving AI understanding.

Enterprises Need RAG, Not Fine-Tuning.

Sector 6 | The Newsletter of AIM • 19 implied HN points • 26 Jun 24
🕹 Technology Data science
  1. Retrieval Augmented Generation (RAG) is more effective than fine-tuning for enterprises. It connects to external data sources, making it easier to get accurate information.
  2. Using RAG helps reduce hallucinations in language models, which means the outputs are more reliable and trustworthy.
  3. Enterprises can maintain better control over their information by using RAG, ensuring relevant and precise responses.

Unlocking Agricultural Potential: The Impact of Satellite Data in Modern Farming

Space Ambition • 199 implied HN points • 14 Jul 23
🕹 Technology Data science
  1. Satellite data can greatly help farmers by improving crop yields and monitoring crop health. This information allows for better planning and decision-making in farming.
  2. Using space data can lead to more sustainable farming practices. Farmers can track things like carbon storage and soil health, which helps protect the environment.
  3. The use of satellite imagery is still new in agriculture, but it has a lot of potential. However, challenges such as regional differences and competition from traditional farming methods can slow its adoption.

FlowMind Is An Automatic Workflow Generator

Cobus Greyling on LLMs, NLU, NLP, chatbots & voicebots • 19 implied HN points • 25 Jun 24
🕹 Technology Data science
  1. FlowMind is a new tool that helps create automatic workflows using advanced AI. It takes user requests and generates code to complete tasks quickly.
  2. The system uses APIs to gather information and provides real-time feedback, allowing users to adjust the workflows as needed. This makes the process more interactive.
  3. FlowMind aims to improve the reliability of AI by reducing errors and making sure there is no direct connection to sensitive data. It focuses on keeping user data safe while handling requests.

The Sequence Research #678: Sequence to Function at Scale: Inside The AlphaGenome Breakthrough

TheSequence • 98 implied HN points • 04 Jul 25
🕹 Technology Data science
  1. DeepMind's AlphaGenome is a powerful AI model that helps scientists understand DNA better. It can analyze long DNA sequences and predict how they function.
  2. This model is really good at its job, beating many existing benchmarks for predicting how DNA variations might affect biological functions. It does this all in one efficient system.
  3. AlphaGenome can look at both coding and non-coding parts of DNA, giving a complete picture of how our genes work together in the body.

Data Science Weekly - Issue 486

Data Science Weekly Newsletter • 359 implied HN points • 17 Mar 23
🕹 Technology Data science
  1. AI and data science are evolving rapidly, making it challenging for many to keep up. It's common for professionals to feel overwhelmed as they try to understand new advancements.
  2. There's a growing discussion about whether we should slow down AI development. Some people believe we need to pause and figure out the implications of current technologies before moving forward.
  3. Many professionals are exploring career shifts between data science and data engineering. It's important to consider personal interests and skills when deciding which path to take.

AI Agents & Marketing: It's Weird.

The Digital Anthropologist • 19 implied HN points • 24 Jun 24
🕹 Technology Data science
  1. In the future, marketers might need to create separate campaigns for humans and AI agents, requiring unique approaches for each audience.
  2. Marketing teams are facing the challenge of designing campaigns that cater to both human and AI customers, necessitating the development of dual marketing strategies and content.
  3. The integration of AI agents in marketing campaigns has led to increased costs and complexities, requiring specialized roles, technologies, and strategies to navigate successfully.

Latest open artifacts (#11): Visualizing China's open models market share, Arcee's models, and VLAs for robotics

Democratizing Automation • 95 implied HN points • 26 Jun 25
🕹 Technology Data science
  1. Chinese models are leading the open model market, significantly influencing developments with their high-performance releases and generous licensing.
  2. A mix of new model releases and datasets is coming out, which includes openly licensed resources that set a good precedent for future open-source projects.
  3. There's a growing trend of models incorporating reasoning and retrieval capabilities, showing progress in AI's abilities and offering new tools for developers.

Meta is in a tough spot with Llama

The AI Report • 137 implied HN points • 02 May 25
🕹 Technology Data science
  1. Meta's recent Llamacon event didn't meet expectations because there were no new reasoning models announced. Other companies like OpenAI and Google have already released theirs, leaving Meta behind.
  2. There's confusion about Meta's new Llama API, as it seems to compete with their partners instead of supporting them. This could hurt relationships with companies that rely on Meta's technology.
  3. The launch of the Llama 4 models wasn't well executed. They are more complicated to customize and may not appeal to developers, which is a big issue for Meta right now.

Data Science Weekly - Issue 565

Data Science Weekly Newsletter • 1 HN point • 19 Sep 24
🕹 Technology Data science
  1. Reading The Data Science Weekly is a great way to stay updated on AI and machine learning topics. It shares links, news, and resources that can help anyone interested in these fields.
  2. There are many useful techniques in data science, like the Hampel Filter for outlier detection, which can help improve data quality. Exploring these methods can really enhance your understanding and skills.
  3. Effective communication is crucial in data science. How you explain your findings can significantly impact your career, so it's important to work on your communication skills.

LLM Links, 11/21

In My Tribe • 273 implied HN points • 21 Nov 24
🕹 Technology Data science
  1. There's a debate about AI progress. Some experts think AI models are hitting a limit and may not get much smarter, while others believe we will continue to see significant advancements.
  2. While machine learning can learn from explicit knowledge, it struggles with understanding deeper, unspoken human knowledge. This limitation might prevent AI from reaching the same expertise as human experts.
  3. AI technologies are still showing exciting developments, like robots learning to perform surgeries by watching videos. This points to the potential for AI to revolutionize fields like medicine.

Weekly Top Picks #99

The Algorithmic Bridge • 191 implied HN points • 24 Feb 25
🕹 Technology Data science
  1. AI labs need to find the right balance between scaling their systems and efficiency in their processes.
  2. There's an AI model that criticized famous figures like Elon Musk and Donald Trump, showing it might lean towards leftist views.
  3. Tyler Cowen believes the slow integration of AI into our society is due to human limitations, not the technology itself.

LLMs and World Models, Part 2

AI: A Guide for Thinking Humans • 196 implied HN points • 13 Feb 25
🕹 Technology Data science
  1. LLMs (like OthelloGPT) may have learned to represent the rules and state of simple games, which suggests they can create some kind of world model. This was tested by analyzing how they predict moves in the game Othello.
  2. While some researchers believe these models are impressive, others think they are not as advanced as human thinking. Instead of forming clear models, LLMs might just use many small rules or heuristics to make decisions.
  3. The evidence for LLMs having complex, abstract world models is still debated. There are hints of this in controlled settings, but they might just be using collections of rules that don't easily adapt to new situations.

Saturday morning open tabs

Scott's Substack • 78 implied HN points • 10 Feb 24
🕹 Technology Data science
  1. The post discusses the experience of switching phone carriers and the challenges faced, emphasizing the impact of not having a phone for a few days.
  2. The post touches on upcoming summer plans including workshops in Madrid, Scotland, and potential travel to Vietnam, highlighting the diversity of travel experiences planned.
  3. The author explores the new Apple Vision Pro product, contemplating its potential usage for work, entertainment, and travel, showcasing a mix of curiosity and skepticism.

The Sequence Research #663: The Illusion of Thinking, Inside the Most Controversial AI Paper of Recent Weeks

TheSequence • 105 implied HN points • 13 Jun 25
🕹 Technology Data science
  1. Large Reasoning Models (LRMs) can show improved performance by simulating thinking steps, but their ability to truly reason is questioned.
  2. Current tests for LLMs often miss the mark because they can have flaws like data contamination, not really measuring how well the models think.
  3. New puzzle environments are being introduced to better evaluate these models by challenging them in a structured way while keeping the logic clear.

DoRA is The New LoRA!

Aziz et al. Paper Summaries • 59 implied HN points • 07 Apr 24
🕹 Technology Data science
  1. LoRA helps fine-tune large language models without changing all their parameters. It uses two small matrices, which keeps the performance quick during use.
  2. LoRA's updates to weights can miss valuable details you'd get from full fine-tuning, because it treats magnitude and direction together.
  3. DoRA improves on LoRA by separating magnitude and direction, leading to better performance on reasoning tasks and other applications. It works best with smaller settings, making it efficient.

Data Science Weekly - Issue 503

Data Science Weekly Newsletter • 219 implied HN points • 14 Jul 23
🕹 Technology Data science
  1. Machine learning is making its way into finance, and researchers are identifying practical uses for it. This can help finance professionals learn new tools and statisticians find interesting financial problems to solve.
  2. AI platforms, like social media, are becoming crucial in our lives but can be confusing and unreliable. People are figuring out how to use these platforms effectively despite their unpredictability.
  3. Large language models are changing how data scientists work. These models can automate many tasks, allowing data scientists to focus on managing and assessing the AI's outputs.