The hottest Neural Networks Substack posts right now

And their main takeaways
Category
Top Technology Topics

A Visual Guide to Mixture of Experts (MoE)

Exploring Language Models • 3289 implied HN points • 07 Oct 24
đź•ą Technology Neural Networks
  1. Mixture of Experts (MoE) uses multiple smaller models, called experts, to help improve the performance of large language models. This way, only the most relevant experts are chosen to handle specific tasks.
  2. A router or gate network decides which experts are best for each input. This selection process makes the model more efficient by activating only the necessary parts of the system.
  3. Load balancing is critical in MoE because it ensures all experts are trained equally, preventing any one expert from becoming too dominant. This helps the model to learn better and work faster.

A knockout blow for LLMs?

Marcus on AI • 47783 implied HN points • 07 Jun 25
đź•ą Technology Neural Networks
  1. LLMs have a hard time solving complex problems reliably, like the Tower of Hanoi, which is concerning because it shows their reasoning abilities are limited.
  2. Even with new reasoning models, LLMs struggle to think logically and produce correct answers consistently, highlighting fundamental issues with their design.
  3. For now, LLMs can be useful for certain tasks like coding or brainstorming, but they can't be relied on for tasks needing strong logic and reliability.

Now I Really Won That AI Bet

Astral Codex Ten • 30146 implied HN points • 08 Jul 25
đź•ą Technology Neural Networks
  1. In 2022, a bet was made on whether AI could create complex images by 2025. The challenge was to generate images that matched detailed prompts.
  2. Over the years, various AI models were tested, and the results showed both progress and limitations. Improvements were made, but some details were still missed.
  3. By June 2025, an updated AI model finally met all the conditions of the bet, showing that AI can achieve a high level of image generation based on specific instructions.

Deep Learning Legends: Ilya's List, or 90% of Everything That Matters in AI

Gonzo ML • 252 implied HN points • 08 Feb 26
đź•ą Technology Neural Networks
  1. A compact, curated reading list of landmark papers can teach roughly 90% of the core ideas and techniques in deep learning, offering a fast path to real understanding.
  2. The essential topics span sequence models (RNNs/LSTMs/NTM), attention and transformers, convolutional vision models, theory of complexity and description length, training methods and scaling, and multimodal/speech work.
  3. The publicly available partial list misses several important areas — notably reinforcement learning and meta-learning — so it should be supplemented with RL classics and recent advances like scaling laws, compute‑optimal training, mixture‑of‑experts, distillation, and key optimization tricks.

The Sequence Opinion #810: The Black Box Behind the Mask: The Imperative of Post-Training Interpretability

TheSequence • 161 implied HN points • 19 Feb 26
đź•ą Technology Neural Networks
  1. AI development has two stages: pre-training builds a raw base model, and post-training (like SFT and RLHF) puts a behavioral "mask" on it so it acts helpful, safe, and fluent.
  2. Post-training interpretability is a distinct focus that studies how knowledge is modulated, suppressed, or amplified during fine-tuning, asking not just what the model knows but why it chose to say one thing instead of another.
  3. As models get more capable and the alignment cost falls, understanding post-training interventions becomes increasingly important and is becoming a key research frontier with new techniques emerging.
Get a weekly roundup of the best Substack posts, by hacker news affinity:

Quick Essay: Large Language Models, How to Train Them, and xAI’s Grok

chamathreads • 3321 implied HN points • 31 Jan 24
đź•ą Technology Neural Networks
  1. Large language models (LLMs) are neural networks that can predict the next sequence of words, specialized for tasks like generating responses to questions.
  2. LLMs work by representing words as vectors, capturing meanings and context efficiently using techniques like 'self-attention'.
  3. To build an LLM, it goes through two stages: training (teaching the model to predict words) and fine-tuning (specializing the model for specific tasks like answering questions).

Universal Reasoning Model

Gonzo ML • 252 implied HN points • 05 Jan 26
đź•ą Technology Neural Networks
  1. A Universal Transformer–style model (URM) repeatedly applies a shared transformer layer with ACT, combining ConvSwiGLU and truncated backprop through loops to get very deep effective computation while keeping parameter count low.
  2. ConvSwiGLU injects a small depthwise convolution into the SwiGLU gating to mix local token context, and TBPTL reduces memory and training cost by only backpropagating through the final iterations.
  3. The model outperforms prior HRM/TRM baselines on tasks like Sudoku and ARC-AGI and Muon speeds convergence, but differences in evaluation protocols and some unclear experimental details mean independent verification is still needed.

LLMs – Part 3: Context Matters — Self Attention

Vasu’s Newsletter • 78 implied HN points • 25 Jan 26
đź•ą Technology Neural Networks
  1. Each token creates query, key, and value vectors so it can ask what it needs, match that against other tokens, and gather useful information.
  2. Tokens compare their query to every key to get raw scores, convert those scores to attention weights with softmax, and use the weights to take a weighted sum of value vectors to produce a new contextual vector.
  3. Self-attention makes token meanings contextual (helping with pronouns, disambiguation, and long-range links), and models use multiple attention heads plus feed-forward layers to capture different relation patterns and refine each token's representation.

Linearity: Why Batching Works

Software Bits Newsletter • 103 implied HN points • 03 Jan 26
đź•ą Technology Neural Networks
  1. Linearity lets you process many inputs as one big matrix multiply, so batching is nearly free and GPUs can run large batches with high efficiency.
  2. Differentiation is linear, so per-sample gradients can be summed and scaled — enabling gradient accumulation, distributed training, and efficient backprop.
  3. Non-linearities are required for expressivity, so networks interleave cheap, element-wise nonlinear functions with batch-friendly linear layers and prefer operations (like LayerNorm) that preserve batching advantages.

Import AI 374: China's military AI dataset; platonic AI; brainlike convnets

Import AI • 339 implied HN points • 27 May 24
đź•ą Technology Neural Networks
  1. UC Berkeley researchers discovered a suspicious Chinese military dataset named 'Zhousidun' with specific images of American destroyers, presenting potential implications for military use of AI.
  2. Research suggests that as AI systems scale up, their representations of reality become more similar, with bigger models better approximating the world we exist in.
  3. Convolutional neural networks are shown to align more with primate visual cortexes than transformers, indicating architectural biases that can lead to better understanding the brain.

Commutativity: Why Transformers Need Positional Encodings

Software Bits Newsletter • 103 implied HN points • 01 Jan 26
đź•ą Technology Neural Networks
  1. Self-attention treats all positions symmetrically, so permuting tokens just permutes outputs; because attention is permutation‑equivariant, Transformers need positional encodings to learn token order.
  2. Commutativity is a deliberate design trade‑off: it enables parallelization and is perfect for unordered data like point clouds, sets, and graphs, but it destroys order information so you must use non‑commutative models or inject positions when order matters (language, time series).
  3. Commutativity shows up across ML: global pooling gives useful invariance but loses location, gradient aggregation and distributed training rely on commutative sums, and floating‑point associativity issues can still cause small nondeterminism.

Why Graphs are great for Fraud Detection [Math Mondays]

Technology Made Simple • 639 implied HN points • 01 Jan 24
đź•ą Technology Neural Networks
  1. Graphs are efficient at encoding and representing relationships between entities, making them useful for fraud detection tasks.
  2. Graph Neural Networks excel at fraud detection due to their ability to visualize strong correlations among fraudulent activities that share common properties, adapt to new fraud patterns, and offer transparency in AI systems.
  3. Graph Neural Networks require less labeled data and feature engineering compared to other techniques, have better explainability, and work well with semi-supervised learning, making them a powerful tool for fraud detection.

Can AI learn language like we do?

AI Supremacy • 491 implied HN points • 09 Feb 24
🔬 Science Neural Networks
  1. An AI model was trained using video footage from a baby to learn language and concepts.
  2. The AI model demonstrated the ability to link words to their visual counterparts based on limited real-world experiences.
  3. This study could help reshape our understanding of how AI and humans learn language and concepts.

The Transformer Zoo Revisited

Gonzo ML • 126 implied HN points • 29 Nov 25
đź•ą Technology Neural Networks
  1. Transformer models can be either encoder-decoder types or decoder-only types. Right now, decoder-only models like GPT are very popular, but there are still reasons to explore the full encoder-decoder architecture.
  2. In initial tests, decoder-only models often perform better during the pretraining stage. They have an advantage in tasks like zero-shot and few-shot learning because of their training setup.
  3. After fine-tuning, encoder-decoder models show improved performance and efficiency. They handle long contexts better and can generate outputs more effectively, suggesting they might be a strong choice for future models.

Analog Chip Design is an Art. Can AI Help?

The Asianometry Newsletter • 2707 implied HN points • 12 Feb 24
đź•ą Technology Neural Networks
  1. Analog chip design is a complex art form that often takes up a significant portion of the total design cost of an integrated circuit.
  2. Analog design involves working with continuous signals from the real world and manipulating them to create desired outputs.
  3. Automating analog chip design with AI is a challenging task that involves using machine learning models to assist in tasks like circuit sizing and layout.

How does ChatGPT work? A brief history of computational language understanding

prakasha • 648 implied HN points • 23 Feb 23
đź•ą Technology Neural Networks
  1. A brief history of computational language understanding dates back to collaboration between linguists and computer scientists.
  2. Language models like ChatGPT use word embeddings to predict and generate text, allowing for effective context analysis.
  3. Neural networks, like Transformers, have revolutionized NLP tasks, enabling advancements in machine translation and language understanding.

The Sequence AI of the Week #785: Gradient Highway Maintenance: Inside DeepSeek’s Latest Breakthrough

TheSequence • 35 implied HN points • 07 Jan 26
đź•ą Technology Neural Networks
  1. DeepSeek's mHC challenges established assumptions about AI scaling and suggests new architectural ideas that could change how larger models are built and trained.
  2. Residual connections are the unsung scaffolding of modern deep networks, providing a 'gradient highway' that keeps training stable across many layers.
  3. The simple rule y = f(x) + x—adding the input back to a layer's output—was revolutionary because it preserves signals and gradients, making very deep networks trainable.

self learning models (1)

Eternal Sunshine of the Stochastic Mind • 119 implied HN points • 02 May 24
đź•ą Technology Neural Networks
  1. Machine Learning is a leap of faith in Computer Science where data shapes the outcome rather than instructions.
  2. In machine learning, viewing yourself as a neural network model can offer insights into self-improvement.
  3. Understanding machine learning concepts can help in identifying learning failures, training the mind, and reflecting on personal objectives.

The Sequence AI of the Week #793: DeepSeek's New Paper: Storing 100B Parameters on CPU RAM

TheSequence • 21 implied HN points • 21 Jan 26
đź•ą Technology Neural Networks
  1. The current LLM trend is to scale models huge and use sparsity tricks like Mixture-of-Experts so only a small part of the model activates per token, reducing FLOPs.
  2. Reusing an old technique — storing large, static lookup-like memories on CPU RAM and conditionally accessing them — can let models hold around 100B parameters off-GPU and avoid expensive dense computation.
  3. The key insight is that many LLM costs come from simulating static lookup tables with neural computation, so replacing that simulation with real conditional lookups makes models much more efficient.

Horse rides Astronaut, redux

Marcus on AI • 1462 implied HN points • 13 Feb 24
đź•ą Technology Neural Networks
  1. DALL-E 2 and Gemini Ultra struggled with complex prompts and concepts, showing limitations in language understanding.
  2. Proper prompts and iterations are crucial to achieve desired results with AI models like Gemini Ultra.
  3. Despite progress in some areas, challenges persist in neural networks' factuality and compositionality.

AI and neuroscience

Technically • 21 implied HN points • 13 Jan 26
đź•ą Technology Neural Networks
  1. Neural networks are deliberately inspired by the brain: they use many simple "neurons" wired together to detect patterns and process information.
  2. This brain-inspired approach has a long history and has been applied to real problems since early work by neuroscientists and engineers, showing the idea actually works in practice.
  3. The brain is still poorly understood, so AI only roughly approximates biological brains, and many researchers think learning more about the brain could be key to building far more powerful intelligence.

Infinite Context Length 🤯

Sector 6 | The Newsletter of AIM • 99 implied HN points • 18 Apr 24
đź•ą Technology Neural Networks
  1. Meta has introduced MEGALODON, a new neural architecture that allows for infinite context length in AI, making it more efficient than previous models.
  2. With developments from Microsoft, Google, and Meta, the focus will shift away from which model has the highest context length, as all will likely have infinite capabilities soon.
  3. The upcoming Llama-3 model is expected to continue this trend by also supporting infinite context length, enhancing its utility in various applications.

So... what is the lottery ticket hypothesis [Math Mondays]

Technology Made Simple • 159 implied HN points • 05 Feb 24
đź•ą Technology Neural Networks
  1. The Lottery Ticket Hypothesis proposes that within deep neural networks, there are subnetworks capable of achieving high performance with fewer parameters, leading to smaller and faster models.
  2. Successful application of the Lottery Ticket Hypothesis relies on iterative magnitude pruning strategies, with potential benefits like faster learning and higher accuracy.
  3. The hypothesis works due to factors like favorable gradients, implicit regularization, and data alignment, but challenges like scalability and interpretability remain towards practical implementation.

The Case for Uninterpretable Machine Learning

Mindful Modeler • 319 implied HN points • 03 Oct 23
đź•ą Technology Neural Networks
  1. Machine learning excels because it's not interpretable, not in spite of it.
  2. Embracing complexity in models like neural networks can effectively capture the intricacies of real-world tasks that lack simple rules or semantics.
  3. Interpretable models can outperform complex ones with smaller datasets and ease of debugging, but being open to complex models can lead to better performance.

Be Not Afraid: Gods, Monsters, and Generative AI

jonstokes.com • 154 implied HN points • 13 Jul 25
đź•ą Technology Neural Networks
  1. AI is just a tool, nothing more. It's not a god or the end of the world; it's like another stage in our technology growth, similar to the industrial revolution.
  2. Using AI should be like a search process where you drive the interaction. You're the one guiding the conversation or output, not the AI speaking to you like a human.
  3. We need to take responsibility for AI's impact. It can either help us improve how we communicate and create, or it can lead us to shallow experiences if we let it.

LLMs - Part 2: Order Matters - Positional Encoding

Vasu’s Newsletter • 13 implied HN points • 11 Jan 26
đź•ą Technology Neural Networks
  1. Large language models process tokens in parallel and need positional encoding to know word order; without it, reordered sentences look the same to the model.
  2. Positional encodings (like sinusoidal functions or methods such as RoPE and ALiBi) give each position a unique vector that’s combined with token embeddings, so the same word at different positions produces different vectors and relative distances can be inferred.
  3. Positional encoding only makes order visible — it doesn’t compute relationships or context; deciding which words matter to each other is handled next by self-attention.

How In-Context Learning Emerges

Last Week in AI • 437 implied HN points • 21 Jul 23
đź•ą Technology Neural Networks
  1. In-context learning (ICL) allows Large Language Models to learn new tasks without additional training.
  2. ICL is exciting because it enables versatility, generalization, efficiency, and accessibility in AI systems.
  3. Three key factors that enable and enhance ICL abilities in large language models are model architecture, model scale, and data distribution.

Perceptrons, XOR, and the first "AI winter"

The Counterfactual • 139 implied HN points • 17 Jan 24
đź•ą Technology Neural Networks
  1. AI systems are getting better, but there are still limits to what they can do. For example, some tasks might just be impossible for current AI technology.
  2. The history of AI shows that there have been times of excitement followed by periods of reduced interest, called 'AI winters'. This happens especially when expectations exceed reality.
  3. Early AI models, like perceptrons, were limited in their abilities, which led to skepticism about their potential. Understanding these past limitations helps us think more critically about today's AI capabilities.

One True Love

Don't Worry About the Vase • 940 implied HN points • 09 Feb 24
đź•ą Technology Neural Networks
  1. The story discusses a man's use of AI to find his One True Love by having the AI communicate with women on his behalf.
  2. The man's approach included filtering potential matches based on various criteria, leading to improved results over time.
  3. Ultimately, the AI suggested he propose to his chosen partner, which he did, and she said yes.

Should we stop interpreting ML models because XAI methods are imperfect?

Mindful Modeler • 199 implied HN points • 31 Oct 23
đź•ą Technology Neural Networks
  1. Don't let a pursuit of perfection in interpreting ML models hinder progress. It's important to be pragmatic and make decisions even in the face of imperfect methods.
  2. Consider the balance of benefits and risks when interpreting ML models. Imperfect methods can still provide valuable insights despite their limitations.
  3. While aiming for improvements in interpretability methods, it's practical to use the existing imperfect methods that offer a net benefit in practice.

Understanding Large Language Models

Startup Pirate by Alex Alexakis • 216 implied HN points • 12 May 23
đź•ą Technology Neural Networks
  1. Large Language Models (LLMs) revolutionized AI by enabling computers to learn language characteristics and generate text.
  2. Neural networks, especially transformers, played a significant role in the development and success of LLMs.
  3. The rapid growth of LLMs has led to innovative applications like autonomous agents, but also raises concerns about the race towards Artificial General Intelligence (AGI).

What are embeddings?

Normcore Tech • 1353 implied HN points • 07 Jun 23
đź•ą Technology Neural Networks
  1. The author delved deep into the concept of embeddings in deep learning.
  2. The author's journey in understanding embeddings involved a significant amount of research and work.
  3. The author hopes that others can benefit from their learning about embeddings as well.

LLMs and World Models, Part 1

AI: A Guide for Thinking Humans • 247 implied HN points • 13 Feb 25
đź•ą Technology Neural Networks
  1. In the past, AI systems often used shortcuts to solve problems rather than truly understanding concepts. This led to unreliable performance in different situations.
  2. Today’s large language models are debated to either have learned complex world models or just rely on memorizing and retrieving data from their training. There’s no clear agreement on how they think.
  3. A 'world model' helps systems understand and predict real-world behaviors. Different types of models exist, with some capable of capturing causal relationships, but it's unclear how well AI systems can do this.

Learning the Language of Rain

Daoist Methodologies • 176 implied HN points • 17 Oct 23
🔬 Science Neural Networks
  1. Huawei's Pangu AI model shows promise in weather prediction, outperforming some standard models in accuracy and speed.
  2. Google's Metnet models, using neural networks, excel in predicting weather based on images of rain clouds, showcasing novel ways to approach weather simulation.
  3. Neural networks are efficient in processing complex data, like rain cloud images, to extract detailed information and act as entropy sinks, providing insights into real-world phenomena simulation.

The Sequence Knowlege #693: A New Series About Interpretability in Foundation Models

TheSequence • 84 implied HN points • 29 Jul 25
đź•ą Technology Neural Networks
  1. Understanding AI black boxes, especially complex models, is very important for safety and trust. People need to know how these AIs make decisions.
  2. Interpretability in AI refers to making sense of how these intelligent systems work. It's about bridging the gap between what we can do with AI and understanding it.
  3. The series will discuss practical ways to interpret these AI models and review significant papers related to the topic. Learning from research is key to improving AI understanding.

Can you explain GPT with ... GPT?

Mindful Modeler • 199 implied HN points • 16 May 23
đź•ą Technology Neural Networks
  1. OpenAI experimented with using GPT-4 to interpret the functionality of neurons in GPT-2, showcasing a unique approach to understanding neural networks.
  2. The process involved analyzing activations for various input texts, selecting specific texts to explain neuron activations, and evaluating the accuracy of these explanations.
  3. Interpreting complex models like LLMs with other complex models, such as using GPT-4 to understand GPT-2, presents challenges but offers a method to evaluate and improve interpretability.

LLMs and World Models, Part 2

AI: A Guide for Thinking Humans • 196 implied HN points • 13 Feb 25
đź•ą Technology Neural Networks
  1. LLMs (like OthelloGPT) may have learned to represent the rules and state of simple games, which suggests they can create some kind of world model. This was tested by analyzing how they predict moves in the game Othello.
  2. While some researchers believe these models are impressive, others think they are not as advanced as human thinking. Instead of forming clear models, LLMs might just use many small rules or heuristics to make decisions.
  3. The evidence for LLMs having complex, abstract world models is still debated. There are hints of this in controlled settings, but they might just be using collections of rules that don't easily adapt to new situations.