The hottest Data Analysis Substack posts right now

And their main takeaways
Category
Top Technology Topics

Taking time series modeling and stream processing mainstream

Gradient Flow • 139 implied HN points • 10 Nov 22
🕹 Technology Data Analysis
  1. The global market for time series analysis software is growing significantly, presenting opportunities for companies and startups
  2. There is a need to focus on stream processing to gain competitive advantages in making quick decisions and leveraging incoming data
  3. Open source tools and collaborations play a key role in advancing fields like time series modeling and stream processing

Data Deep Dive: Predicting China's Exports Using AIS Vessel Tracking Data (+ Other Uses)

The Data Score • 59 implied HN points • 28 Jun 23
🕹 Technology Data Analysis
  1. AIS vessel tracking data can predict China's exports, monitor global trade, and understand real-time economic activity.
  2. Data cleansing is crucial for turning raw AIS data into actionable insights. Cleaning the data involves filtering out anomalies and ensuring accuracy.
  3. It's important to consider limitations like the exclusive focus on large commercial ships, uncertainties in cargo data, and vessel behavior anomalies when analyzing AIS data.

Address Not Found (Part 2)

Cybernetic Forests • 59 implied HN points • 18 Jun 23
🕹 Technology Data Analysis
  1. Communication technologies historically categorized into one-to-one, one-to-many, and many-to-many transmission systems.
  2. Artificial Intelligence operates in a unique structure called many-to-one-to-one, where data from multiple sources shapes responses for individual users.
  3. AI systems, despite the appearance of one-to-one engagement, actually function asynchronously and as a blend of many-to-one transmission, controlled by the operators and designers.

The Bull Case for Alternative Data

The Data Score • 59 implied HN points • 11 Apr 23
💰 Finance Data Analysis
  1. The Alternative Data industry is currently facing challenges but has the potential for long-term success by emphasizing clear client outcomes and building network effects.
  2. Understanding the value clients gain from data insights is crucial, as insights drive decisions and financial outcomes.
  3. Creating network effects and aligning data teams with critical client outcomes are key factors for the Alternative Data industry to move towards sustained growth and productivity.
Get a weekly roundup of the best Substack posts, by hacker news affinity:

Is ChatGPT Getting worse? A Case Study on Confirmation Bias

The Data Score • 59 implied HN points • 20 Jul 23
🕹 Technology Data Analysis
  1. Testing and improving AI models, like ChatGPT, is crucial as our reliance on AI grows. Ensuring model performance and explainability is key for professionals in the field.
  2. Machine learning and AI models face challenges with explainability, especially in the context of large language models like ChatGPT. Specific wording and temperature settings can greatly impact model outputs.
  3. Confirmation bias is a common human tendency to search for and interpret information that aligns with existing beliefs. It's important to recognize and manage biases when assessing AI model performance.

Microsoft Sentinel SOC 101: How to Detect and Mitigate Drive-by Download Attacks with Microsoft Sentinel

Rod’s Blog • 59 implied HN points • 04 Oct 23
🕹 Technology Data Analysis
  1. Drive-by download attacks exploit vulnerabilities to download malicious code without user knowledge. They can lead to data breaches and install malware.
  2. Mitigation strategies include user education, enforcing security policies, monitoring network traffic, and using SIEM services like Microsoft Sentinel.
  3. Microsoft Sentinel can help detect drive-by download attacks by collecting relevant data, enriching it, analyzing with rules and ML, visualizing results, and automating incident response.

18 months since - funding statistics for the 2021 space startup cohort

Intersections (by Filip) • 59 implied HN points • 17 Aug 23
🕹 Technology Data Analysis
  1. Raw data reveals the financial landscape of space startups in 2021, with significant amounts invested and patterns in subsequent funding rounds.
  2. Data collection discrepancies highlight the importance of a diverse dataset and the impact of subsequent funding on company growth and investor interest.
  3. Understanding the timeline of funding, company failures, loyal investors, and the need for continuous rounds sheds light on the complexities and challenges faced by space startups.

The Paradox of Finding Surprising Insights in Alternative Data

The Data Score • 59 implied HN points • 22 Jun 23
💼 Business Data Analysis
  1. Institutional investors need to find surprising insights in data but also be skeptical of them to ensure accuracy and avoid errors.
  2. When using alternative data to make predictions, it's crucial to verify if the insights answer the right questions and differ from the market consensus.
  3. Digging into the data through various methods like independent validation, error margin assessment, and data integrity checks is essential for investors to ensure the reliability of surprising insights.

Home EV Charging

Solar Powered Data • 6 HN points • 29 Jun 24
🕹 Technology Data Analysis
  1. Consider home EV charging to save on fuel costs and enjoy convenient charging at home
  2. Understanding your EV's energy consumption can help you track savings and make informed decisions
  3. EVs like the Hyundai Ioniq 5 can offer significant savings compared to traditional gas-powered vehicles through lower operating costs

Data used in nuclear war piece

Myth Pilot • 58 implied HN points • 26 Apr 23
🕹 Technology Data Analysis
  1. The author wanted to understand the impact of a theoretical nuclear war on voter demographics
  2. Access to geographic data by locality required decoding binary files from a game simulator
  3. To access specific data for research, the author wrote a script to extract the needed information

The Future of Network Observability and Network Automation

Internet Dynamics • 58 implied HN points • 06 Sep 23
🕹 Technology Data Analysis
  1. Network observability is crucial for network automation to handle real-time mitigation and remediation.
  2. Observability solutions need to consider topology, alerts, correlation, suppression, policy, and meta-data for effective network monitoring.
  3. Future approaches to observability and automation should recognize and manifest common components like Topology, CMDBs and Meta-data.

Data Oracles

Datent • 58 implied HN points • 24 May 23
🕹 Technology Data Analysis
  1. The best predictions come from deep analysis of today's data challenges and trends.
  2. Data oracles provide valuable insights for the future by understanding present data trends.
  3. Data writers like Davenport, Moses, Madsen, and Thomas offer grounded observations and advice on data topics.

Fraud, Incompetence, or Both?

Cremieux Recueil • 229 implied HN points • 31 Jan 24
🔬 Science Data Analysis
  1. Fraud can happen in scientific research through deliberate misrepresentation of results.
  2. Being critical in research is important, but it's crucial to back up claims with thorough analysis and evidence.
  3. Failure to fully analyze data and make accurate conclusions can indicate either fraud or incompetence in a study.

Vulnerability Management and Developer Toil

Resilient Cyber • 119 implied HN points • 02 Apr 23
🕹 Technology Data Analysis
  1. Vulnerability management is crucial for security but often overwhelms developers with too much information. It’s important to focus on vulnerabilities that really pose a risk, instead of just following strict checklists.
  2. The number of vulnerabilities has exploded in recent years, but most are never exploited. Organizations need better ways to prioritize which vulnerabilities to address based on actual risk, rather than just severity scores.
  3. Security teams should work more closely with developers to reduce friction and support their efforts. Improving communication and providing context can make security a partner, not a blocker.

People Should Not Prioritize What They Think Are "Vibes" Over Actual Data: The Economic Situation Really Is—Relative to All Other Years since 2000—Excellent

Brad DeLong's Grasping Reality • 215 implied HN points • 10 Feb 24
🇺🇸 U.S. Politics Data Analysis
  1. Don't prioritize 'vibes' over actual data - the economy is actually excellent compared to past years.
  2. Partisanship influences perceptions of the economy - Democrats more optimistic than Republicans.
  3. Journalists sometimes emphasize negative news, even when data shows a positive economic situation.

Synthesizing Introspection: The AI Mediated Self

Cybernetic Forests • 99 implied HN points • 04 Dec 22
🕹 Technology Data Analysis
  1. The challenge of using AI for introspection is knowing what you are really asking and understanding the limitations of the technology.
  2. Conversing with AI to simulate interactions with younger versions of oneself may not provide personalized or beneficial insights.
  3. Relying on AI for deep introspection or personal growth may present risks of misunderstanding, projection, and avoidance of true self-care.

The KQL Mysteries: The Holiday 2023 Episode Part 3

Rod’s Blog • 39 implied HN points • 13 Dec 23
🕹 Technology Data Analysis
  1. The mysterious numbers given by the hacker were not random, but dates with a hidden significance, leading to a revelation about impending events.
  2. Through identifying patterns in network traffic using KQL, Jon and Sarah uncovered a hacker exploiting a security vulnerability and resolved to apply a critical patch.
  3. The duo set a trap to stop the hacker's planned attack, showcasing the importance of proactive security measures in monitoring and defending against cyber threats.

The KQL Mysteries: The Holiday 2023 Episode Part 2

Rod’s Blog • 39 implied HN points • 12 Dec 23
🕹 Technology Data Analysis
  1. The hacker in the story had a personal connection to one of the characters, making the situation more intense and personal.
  2. Using Kusto Query Language (KQL), the characters tried to analyze the hacker's network traffic and database activity to uncover clues about the hacker's identity and location.
  3. Despite challenges in decoding the hacker's data, the characters discovered a message from the hacker in the database logs, prompting them to solve a mysterious puzzle involving numbers.

Why you should analyze the distribution of your Data [Math Mondays]

Technology Made Simple • 59 implied HN points • 14 Mar 23
🕹 Technology Data Analysis
  1. Analyzing the distribution of your data is crucial for accurate analysis results, helps in choosing the right statistical tests, identifying outliers, and confirming data collection systems.
  2. Common techniques to analyze data distribution include histograms, boxplots, quantile-quantile plots, descriptive statistics, and statistical tests like Shapiro-Wilk or Kolmogorov-Smirnov.
  3. Common mistakes in analyzing data distribution include ignoring or dropping outliers, using the wrong statistical test, and not visualizing data to identify patterns and trends.

Surveillance Education (w/ Nolan Higdon and Allison Butler) | The Chris Hedges Report

The Chris Hedges Report • 89 implied HN points • 20 Nov 24
🚌 Education Data Analysis
  1. Technology in schools can invade student privacy. Many tools are designed for safety but can monitor students in ways they might not agree with.
  2. Surveillance tools can discriminate against students of color and those from poor neighborhoods. They often increase the risk of negative consequences for these groups.
  3. The culture of constant monitoring can stifle curiosity and free expression in classrooms, turning them into places where students just comply rather than learn actively.

Unveiling the Mystery of Compressed Regions

Vesuvius Challenge • 20 implied HN points • 15 Aug 25
🔬 Science Data Analysis
  1. They are using very small scans to understand why some layers of ancient papyrus look blurry. This helps them figure out how to get clearer images.
  2. The blurriness in the scans seems to come from the structure of the papyrus fibers, which scatter the X-rays. Identifying this can help improve future scanning methods.
  3. The team is developing tools to manage and analyze the huge amounts of data from their scans. This makes it easier to work with and improves their chances of reading the ancient texts.

How do AI security products being sold to schools really work?

School Shooting Data Analysis and Reports • 19 implied HN points • 12 Mar 24
🕹 Technology Data Analysis
  1. School administrators are facing pressure to evaluate AI security products but may lack expert knowledge to do so.
  2. Understanding how AI models are trained, the probability threshold, and error rates are crucial when assessing AI security solutions.
  3. The high stakes of AI security decisions for schools underscore the importance of asking detailed questions about the technology being implemented.

How to use Machine Learning for your Small Business [Storytime Saturdays]

Technology Made Simple • 79 implied HN points • 17 Dec 22
🕹 Technology Data Analysis
  1. Machine Learning can be effective for small businesses too, not just large corporations, opening up opportunities for growth and innovation.
  2. Understanding the process of implementing AI can benefit professionals across various roles, not just those directly working in AI fields.
  3. Having the right skills and knowledge about AI implementation can significantly increase your chances of success and career advancement.

Finding fake followers (Bluesky edition)

Conspirador Norteño • 68 implied HN points • 18 Jan 25
🕹 Technology Data Analysis
  1. You can spot fake followers on Bluesky by looking for accounts with similar join dates and generic profiles. These accounts often have no posts and repetitive bios.
  2. Using a method where you track the followers of suspected fake accounts can help identify whole networks of fake followers. By downloading and filtering their followers, you can map out these networks.
  3. The Bluesky platform has a real-time feature called the firehose that makes it easier to catch fake follower activity as it happens. However, this can give some false positives, so users need to be careful.

Daily Chartbook #393

Daily Chartbook • 183 implied HN points • 02 Mar 24
💰 Finance Data Analysis
  1. Daily Chartbook provides 30 charts to summarize the day's events for paid subscribers.
  2. The content on Daily Chartbook can be accessed through a subscription link on their website.
  3. To view specific posts on Daily Chartbook, individuals need to be paid subscribers.

Freddie Mac House Price Index Declined in May; Up 2.2% Year-over-year

CalculatedRisk Newsletter • 28 implied HN points • 30 Jun 25
💰 Finance Data Analysis
  1. The Freddie Mac House Price Index dropped by 0.23% in May but is still up 2.2% compared to last year. This shows that while prices are currently declining, there has been some growth over the past year.
  2. Florida and Texas are experiencing significant price declines, with 17 out of the 30 cities with the biggest drops located in these states. This trend indicates that real estate markets in these areas are facing challenges.
  3. Overall, 31 states and Washington D.C. have seen house prices fall since their peak. With inventory increasing and low sales, housing price growth may slow down even more in the future.

🤘ACDC (not that one)

Gonzo ML • 63 implied HN points • 29 Jan 25
🕹 Technology Data Analysis
  1. The paper introduces a method called ACDC that automates the process of finding important circuits in neural networks. This can help us better understand how these networks work.
  2. Researchers follow a three-step workflow to study model behavior, and ACDC fully automates the last step which helps identify connections that matter for a specific task.
  3. While ACDC shows promise, it isn't perfect. It may miss some important connections and needs adjustments for different tasks to improve its accuracy.

DRAFT: Notes: What Is the Techno-Optimist Slant on “AI”?

Brad DeLong's Grasping Reality • 169 implied HN points • 14 Mar 24
🕹 Technology Data Analysis
  1. Very large-scale, high-dimension regression and classification analysis will be game-changing, transforming bureaucracy to algorithms with significant impacts across sectors from finance to healthcare.
  2. Natural-language interfaces to databases may be challenging to control but offer more intuitive access to vast information repositories, potentially enhancing user efficiency.
  3. Autocomplete technology provides substantial time savings for white-collar workers, illustrating the significant productivity boost modern technologies can offer.

A Sneak Peak into New Spatial Analysis Tools

LatchBio • 23 implied HN points • 23 Jul 25
🕹 Technology Data Analysis
  1. There's an upcoming webinar on July 29, 2025, focused on a new tool for analyzing spatial datasets. It's hosted by Takara Bio and LatchBio.
  2. The webinar will showcase various methods like image alignment and gene expression analysis, so attendees can learn about these important topics.
  3. Participants will get to see live demonstrations of how to use these new analysis methods, which can be very helpful for anyone working with the Seeker™ and Trekker™ datasets.

RLHF progress: Scaling DPO to 70B, DPO vs PPO update, Tülu 2, Zephyr-β, meaningful evaluation, data contamination

Democratizing Automation • 213 implied HN points • 22 Nov 23
🕹 Technology Data Analysis
  1. Reinforcement learning from human feedback (RLHF) is a technology that is still unknown and undocumented.
  2. Scaling DPO to 70B parameters showed strong performance by directly integrating the data and using lower learning rates.
  3. DPO and PPO have differences in their approaches, with DPO showing potential for enhancing chat evaluations and happy users of Tulu and Zephyr models.

Lateral thinking

Logging the World • 79 implied HN points • 12 Nov 22
🔬 Science Data Analysis
  1. Lateral flow tests had a much lower false positive rate than many initially assumed, around 0.03%, showing their effectiveness.
  2. Data on PCR retests of positive lateral flow tests revealed a positive predictive value of 82% even at low prevalence, supporting the reliability of lateral flow tests.
  3. A rise in prevalence due to variants like delta and omicron, as well as ease in lockdown restrictions, contributed to the wider acceptance of lateral flow tests for controlling the pandemic.

OSD 330: The rise of sensorship

Open Source Defense • 28 implied HN points • 17 Jun 25
🕹 Technology Data Analysis
  1. Sensors help us understand and measure things better. The more accurate our sensors are, the more we can improve our products and practices.
  2. In different fields, the use of sensors is at various stages. Some areas, like competition shooting, are advanced, while others, like non-lethal weapons, have much room for growth.
  3. Using objective measurements can change our understanding of different situations. By having clear data, we can make better decisions and improve our overall knowledge.

Modifying readability with large language models (pt. 1)

The Counterfactual • 19 implied HN points • 29 Feb 24
🕹 Technology Data Analysis
  1. Large language models can change text to make it easier or harder to read. It's important to check if these changes actually help with understanding.
  2. By comparing modified texts to their original versions, it's clear that 'Easy' texts are generally simpler than 'Hard' texts. However, it can be harder to make texts significantly simpler than they originally are.
  3. Despite the usefulness of these models, they might sometimes lose important information when simplifying texts. Future studies should involve human judgments to see if the changes maintain the original meaning.

Does Generative AI Pose a Threat to Industry Analyst Firms?

The Security Industry • 20 implied HN points • 04 Aug 25
🕹 Technology Data Analysis
  1. AI can help with many tasks that industry analysts do, like researching and analyzing market conditions. This means analysts might use AI more and improve their work.
  2. While AI is good at some things, it can struggle with completeness, like listing all companies in a market. Analysts still have an edge in this area if they have complete data.
  3. The future of industry analysis might shift as AI changes how information is processed and shared. Analysts will need to adapt to this new landscape to stay relevant.