The hottest Optimization Substack posts right now

And their main takeaways
Category
Top Technology Topics

All paths point downhill

arg min • 218 implied HN points • 31 Oct 24
🕹 Technology Optimization
  1. In optimization, there are three main approaches: local search, global optimization, and a method that combines both. They all aim to find the best solution to minimize a function.
  2. Gradient descent is a popular method in optimization that works like local search, by following the path of steepest descent to improve the solution. It can also be viewed as a way to solve equations or approximate values.
  3. Newton's method, another optimization technique, is efficient because it converges quickly but requires more computation. Like gradient descent, it can be interpreted in various ways, emphasizing the interconnectedness of optimization strategies.

Basic Linear Algebra Subprogramming

arg min • 178 implied HN points • 29 Oct 24
🕹 Technology Optimization
  1. Understanding how optimization solvers work can save time and improve efficiency. Knowing a bit about the tools helps you avoid mistakes and make smarter choices.
  2. Nonlinear equations are harder to solve than linear ones, and methods like Newton's help us get approximate solutions. Iteratively solving these systems is key to finding optimal results in optimization problems.
  3. The speed and efficiency of solving linear systems can greatly affect computational performance. Organizing your model in a smart way can lead to significant time savings during optimization.

Size Does Matter: Why -Os Beat -O2 on My ESP32-S3

atomic14 • 346 implied HN points • 07 Mar 26
🕹 Technology Optimization
  1. On the ESP32-S3, compiling with -Os (optimize for size) gave better results than using -O2 (optimize for speed).
  2. Binary size can matter more than you might expect on constrained microcontrollers, so smaller builds can be preferable.
  3. This challenges the common assumption that higher optimization levels focused on speed are always the best choice for embedded targets.

The Shape of Stats to Come

arg min • 634 implied HN points • 10 Oct 24
🚌 Education Optimization
  1. Statistics often involves optimizing methods to get the best results. Many statistical techniques can actually be viewed as optimization problems.
  2. Choosing a statistical method isn't just about the math—it's also based on beliefs about reality. This philosophical side is important but often overlooked.
  3. There's a danger in relying too much on tools and models we can solve. Sometimes, we force the data to fit our preferred methods instead of being open to the actual complexities.

Designed Interactions

arg min • 257 implied HN points • 15 Oct 24
🚌 Education Optimization
  1. Experiment design is about choosing the right measurements to get useful data while reducing errors. It's important in various fields, including medical imaging and randomized trials.
  2. Statistics play a big role in how we analyze and improve measurement processes. They help us understand the noise in our data and guide us in making our experiments more reliable.
  3. Optimization is all about finding the best way to minimize errors in our designs. It's a practical approach rather than just seeking perfection, and we need to accept that some questions might remain unanswered.
Get a weekly roundup of the best Substack posts, by hacker news affinity:

Convex Optimization at the Midpoint

arg min • 198 implied HN points • 17 Oct 24
🚌 Education Optimization
  1. Modeling is really important in optimization classes. It's better to teach students how to set up real problems instead of just focusing on abstract theories.
  2. Introducing programming assignments earlier can help students understand optimization better. Using tools like cvxpy can make solving problems easier without needing to know all the underlying algorithms.
  3. Convex optimization is heavily used in statistics, but there's not much focus on control systems. Adding a section on control applications could help connect optimization with current interests in machine learning.

Interpolation Is All You Need

arg min • 317 implied HN points • 08 Oct 24
🕹 Technology Optimization
  1. Interpolation is a process where we find a function that fits a specific set of input and output points. It's a useful tool for solving problems in optimization.
  2. We can build more complex function fitting problems by combining simple interpolation constraints. This allows for greater flexibility in how we define functions.
  3. Duality in convex optimization helps solve interpolation problems, enabling efficient computation and application in areas like machine learning and control theory.

Fine-tuning LLMs with 32-bit, 8-bit, and Paged AdamW Optimizers

The Kaitchup – AI on a Budget • 259 implied HN points • 07 Oct 24
🕹 Technology Optimization
  1. Using 8-bit and paged AdamW optimizers can save a lot of memory when training large models. This means you can run more complex models on cheaper, lower-memory GPUs.
  2. The 8-bit optimizer is almost as effective as the 32-bit version, showing similar results in training. You can get great performance with less memory required.
  3. Paged optimizers help manage memory efficiently by moving data only when needed. This way, you can keep training even if you don't have enough GPU memory for everything.

An Inversion Cookbook

arg min • 297 implied HN points • 04 Oct 24
🚌 Education Optimization
  1. Using modularity, we can tackle many inverse problems by turning them into convex optimization problems. This helps us use simple building blocks to solve complex issues.
  2. Linear models can be a good approximation for many situations, and if we rely on them, we can find clear solutions to our inverse problems. However, we should be aware that they don't always represent reality perfectly.
  3. Different regression techniques, like ordinary least squares and LASSO, allow us to handle noise and sparse data effectively. Tuning the right parameters can help us balance accuracy and manageability in our models.

Method Dispatch in Swift: The Complete Guide

Jacob’s Tech Tavern • 2624 implied HN points • 01 Dec 25
🕹 Technology Optimization
  1. Swift has four types of method dispatch that determine how function calls are executed, and understanding these can help improve your code's performance.
  2. The Swift compiler and runtime perform many optimizations behind the scenes, making some traditional coding tips less important.
  3. Learning about method dispatch can help you write faster, more efficient code and build a better intuition about how Swift works.

Integer programming easily encloses horse

DYNOMIGHT INTERNET NEWSLETTER • 968 implied HN points • 15 Jan 26
🕹 Technology Optimization
  1. The horse-enclosure puzzle can be encoded as an integer program using binary variables for walls and for whether a tile can escape, with linear constraints that enforce adjacency and boundaries, so solvers can quickly find and certify optimal enclosures.
  2. Integer programming is a hugely practical and powerful tool for discrete optimization: even though it’s NP-hard in theory, modern solvers solve many real-world instances very fast and reliably.
  3. Whether a combinatorial problem is fun depends on legibility and the right level of difficulty, and many NP-complete problems can be made engaging with a good interface; it’s not obvious whether this specific puzzle is provably NP-complete.

Inverse frontiers

arg min • 158 implied HN points • 07 Oct 24
🕹 Technology Optimization
  1. Convex optimization has benefits, like collecting various modeling tools and always finding a reliable solution. However, not every problem fits neatly into a convex framework.
  2. Some complex problems, like dictionary learning and nonlinear models, often require nonconvex optimization, which can be tricky to handle but might be necessary for accurate results.
  3. Using machine learning methods can help solve inverse problems because they can learn the mapping from measurements to states, making it easier to compute solutions later, though training the model initially can take a lot of time.

Song Pong

Victor Tao • 273 HN points • 28 Aug 24
🕹 Technology Optimization
  1. You can make a pong game more exciting by syncing the ball's movements to music. This allows paddles to dance to the beat as they hit the ball.
  2. Using math and optimization techniques can help you decide where the paddles should hit the ball. It ensures that the game looks good while still following all the rules.
  3. Changing the physics of the game doesn't have to be hard. You just update the rules in your math model, making it easy to test new ideas and keep improving the game.

Deep Learning Legends: Ilya's List, or 90% of Everything That Matters in AI

Gonzo ML • 252 implied HN points • 08 Feb 26
🕹 Technology Optimization
  1. A compact, curated reading list of landmark papers can teach roughly 90% of the core ideas and techniques in deep learning, offering a fast path to real understanding.
  2. The essential topics span sequence models (RNNs/LSTMs/NTM), attention and transformers, convolutional vision models, theory of complexity and description length, training methods and scaling, and multimodal/speech work.
  3. The publicly available partial list misses several important areas — notably reinforcement learning and meta-learning — so it should be supplemented with RL classics and recent advances like scaling laws, compute‑optimal training, mixture‑of‑experts, distillation, and key optimization tricks.

The LLM Context Tax: Best Tips for Tax Avoidance

Nicolas Bustamante • 104 implied HN points • 11 Feb 26
🕹 Technology Optimization
  1. Context tokens are expensive and degrade performance as they accumulate, so treat context as a scarce resource and keep prompts stable and append-only; move dynamic pieces (like timestamps) to the end so you preserve KV cache hits.
  2. Architect agents to minimize tokens by storing tool outputs as files, using precise two-step tools that return metadata before full content, delegating work to cheaper subagents, reusing templates, batching or parallelizing tool calls, and caching common responses at the application level.
  3. Clean and compact data before sending it to the model, place critical information at the beginning or end to avoid the lost-in-the-middle problem, use summarization/compaction before hitting pricing cliffs, and set strict output token limits to control costly outputs.

Universal Reasoning Model

Gonzo ML • 252 implied HN points • 05 Jan 26
🕹 Technology Optimization
  1. A Universal Transformer–style model (URM) repeatedly applies a shared transformer layer with ACT, combining ConvSwiGLU and truncated backprop through loops to get very deep effective computation while keeping parameter count low.
  2. ConvSwiGLU injects a small depthwise convolution into the SwiGLU gating to mix local token context, and TBPTL reduces memory and training cost by only backpropagating through the final iterations.
  3. The model outperforms prior HRM/TRM baselines on tasks like Sudoku and ARC-AGI and Muon speeds convergence, but differences in evaluation protocols and some unclear experimental details mean independent verification is still needed.

Domain Transformations: The Art of Finding Easier Spaces

Software Bits Newsletter • 103 implied HN points • 05 Jan 26
🕹 Technology Optimization
  1. Transform hard problems into easier ones by moving to a different domain, doing the simpler computation there, and (if needed) transforming the result back; this is worth it when the transform cost plus the easier computation is less than solving the original problem.
  2. Use well-known transforms to fix numerical and computational issues: log-space turns tiny-product underflow into stable sums (use the log-sum-exp trick to add probabilities safely), Fourier turns convolution into cheap pointwise multiplication, and embeddings or kernels lift data so linear methods work.
  3. Always check that a transform preserves what you need and that the round-trip cost is justified; the best algorithms exploit problem structure by finding the space where the computation becomes simple.

Linearity: Why Batching Works

Software Bits Newsletter • 103 implied HN points • 03 Jan 26
🕹 Technology Optimization
  1. Linearity lets you process many inputs as one big matrix multiply, so batching is nearly free and GPUs can run large batches with high efficiency.
  2. Differentiation is linear, so per-sample gradients can be summed and scaled — enabling gradient accumulation, distributed training, and efficient backprop.
  3. Non-linearities are required for expressivity, so networks interleave cheap, element-wise nonlinear functions with batch-friendly linear layers and prefer operations (like LayerNorm) that preserve batching advantages.

Ilya on deep learning in 2015

Democratizing Automation • 760 implied HN points • 28 Jun 25
🕹 Technology Optimization
  1. Deep learning is not as complicated as it seems; the basic ideas are pretty straightforward and can be learned quickly with the right guidance. You don't need years of study to understand how it works.
  2. Getting the right random initialization for neural networks is crucial. If the initialization is too small, the signal can decay and become unnoticeable, making it hard for the model to learn effectively.
  3. Machine learning focuses on achieving good enough results rather than perfect solutions. It’s more about finding practical and useful models with the resources available.

One Law to Rule All Code Optimizations

Confessions of a Code Addict • 817 implied HN points • 08 Jun 25
🕹 Technology Optimization
  1. Code optimization can be unpredictable, and not every change will guarantee improved performance. It's important to understand why an optimization might succeed or fail.
  2. The Iron Law of Performance provides a framework for evaluating software optimizations. It focuses on three key factors: the number of instructions, cycles per instruction, and cycle time.
  3. Optimizations like loop unrolling and function inlining reduce the number of instructions executed and can increase instruction throughput. However, they might also lead to some challenges like register spills and increased cache pressure.

Bridging the Gap: From Statistical Distributions to Machine Learning Loss Functions

Mindful Modeler • 818 implied HN points • 14 Nov 23
🕹 Technology Optimization
  1. Understanding the distribution of the target variable is key in choosing statistical analysis or machine learning loss functions.
  2. Certain loss functions in machine learning correspond to maximum likelihood estimation for specific distributions, creating a bridge between statistical modeling and machine learning.
  3. While connecting distributions to loss functions is insightful, the real power in machine learning lies in the flexibility to design custom loss functions rather than being constrained by specific distributions.

Attracted To The Desert (Or The Forest)

Software Design: Tidy First? • 1634 implied HN points • 12 Nov 24
🕹 Technology Optimization
  1. Software development has different styles that often lead to similar outcomes, guided by underlying trends called attractors. These attractors influence how teams change over time, pulling them towards certain approaches.
  2. It’s not just about adding more value in software projects. Instead, the focus should be on removing waste and improving efficiency in how teams work together.
  3. The environment where a team operates, whether it's a productive forest or a limiting desert, greatly affects their potential for growth. The forest offers more opportunities for improvement than the desert.

7 perspectives on machine learning

Mindful Modeler • 279 implied HN points • 09 Apr 24
🕹 Technology Optimization
  1. Machine learning is about building prediction models. It covers a wide range of applications, but may not be perfect for unsupervised learning.
  2. Machine learning is about learning patterns from data. This view is useful for understanding ML projects beyond just prediction.
  3. Machine learning is automated decision-making at scale. It emphasizes the purpose of prediction, which is to facilitate decision-making.

You need to neglect mostly everything to win big

Play Permissionless • 319 implied HN points • 18 Mar 24
💼 Business Optimization
  1. To win big, you only need to get a small number of things right and can afford to mess up everything else. This applies to both companies and individuals.
  2. Winning big often requires unlearning traditional schooling strategies and focusing on doing a great job at a few key aspects while neglecting the rest.
  3. Removing non-essential tasks and focusing solely on what helps deliver better and faster results can lead to significant improvements and ultimately winning big.

The God's-eye View of the Economy

In My Tribe • 410 implied HN points • 08 Jul 25
💰 Finance Optimization
  1. Economists often view individuals and firms as 'optimizers' who try to get the best out of their choices. This means they make decisions to maximize their satisfaction and profits.
  2. Pareto Optimality is a key concept where resources are distributed in a way that no one can be made better off without making someone else worse off. However, just because a situation is Pareto Optimal doesn't mean it’s fair or ideal.
  3. Governments are seen as having the role to correct market failures and redistribute wealth for a fairer society. But not everyone agrees on whether governments actually have the capability or motivation to do this well.

The Mathematics of Intelligence: A Deep Dive into LLM Training

Recommender systems • 26 implied HN points • 31 Jan 26
🕹 Technology Optimization
  1. Pre-training builds a base "world model" by predicting next tokens across huge text corpora, minimizing cross-entropy (negative log-likelihood) so the model learns facts, grammar, and reasoning.
  2. Supervised fine-tuning (SFT) teaches the model to follow instructions, and LoRA makes this efficient by adding small low-rank adapter matrices so you can adapt behavior without updating the entire model.
  3. Reinforcement approaches (like PPO) use a reward model, advantage estimates, clipping, and a KL penalty to safely push adapters toward human preferences, while Direct Preference Optimization (DPO) skips the reward model and trains a new adapter using a log-ratio objective between preferred and unpreferred responses.

@adlrocha - Is N=NP?

@adlrocha Weekly Newsletter • 64 implied HN points • 14 Dec 25
🕹 Technology Optimization
  1. Complexity theory measures how much time and memory algorithms need so we can tell which problems scale feasibly and which become intractable. It separates problems that are merely computable from those that are practically solvable before resources run out.
  2. P contains problems solvable in polynomial time, while NP contains problems whose solutions can be verified quickly even if they seem hard to find. NP-Complete problems are the hardest in NP because every NP problem can be reduced to them, and NP-Hard problems are at least that hard but not necessarily verifiable quickly.
  3. If P = NP, many cryptographic systems would break because one-way functions would no longer exist. At the same time, P = NP would let us solve huge optimization and AI problems exactly and efficiently, radically changing many fields.

How fast is your shell?

Register Spill • 294 implied HN points • 14 Jan 24
🕹 Technology Optimization
  1. Check how fast your shell starts up by running specific commands.
  2. Optimize your shell startup time by running as few commands as possible, keeping the prompt simple, and doing less.
  3. Profile your shell and tweak your configuration files to improve performance.

Why This Old Python Performance Trick Doesn’t Matter Anymore

Confessions of a Code Addict • 264 implied HN points • 28 Jun 25
🕹 Technology Optimization
  1. Performance optimization in Python has changed a lot due to improvements in the Python virtual machine. Tricks that helped in the past may not be needed anymore.
  2. Creating local aliases for functions can speed up access, but recent Python updates have made this less important. In many cases, the performance difference is small now.
  3. Not all lookups are the same—using direct local references or importing functions can still be faster than accessing them through module paths. Always consider readability vs. speed based on your code's needs.

A quick introduction to Memory Pools [Math Mondays]

Technology Made Simple • 179 implied HN points • 27 Feb 24
🕹 Technology Optimization
  1. Memory pools are a way to pre-allocate and reuse memory blocks in software, which can significantly enhance performance.
  2. Benefits of memory pools include reduced fragmentation, quick memory management, and improved performance in programs with frequent memory allocations.
  3. Drawbacks of memory pools include fixed-size blocks, overhead in management, and potential for memory exhaustion if not carefully managed.

A Guide to Optimising your Spark Application Performance (Part 2)

SwirlAI Newsletter • 314 implied HN points • 06 Aug 23
🕹 Technology Optimization
  1. Choose the right file format for your data storage in Spark like Parquet or ORC for OLAP use cases.
  2. Understand and utilize encoding techniques like Run Length Encoding and Dictionary Encoding in Parquet for efficient data storage.
  3. Optimize Spark Executor Memory allocation and maximize the number of executors for improved application performance.

Bitcoin Tech Talk #380

jimmysong • 137 implied HN points • 22 Jan 24
🔮 Crypto Optimization
  1. Neuroscience data can be meaningless due to flawed methods and captured academia.
  2. Getting stuck in life traps is common, but overcoming them is crucial for growth.
  3. Balancing exploration and exploitation is key in life's decision-making process.

The best Python feature you cannot use

Bite code! • 1223 implied HN points • 17 Jun 23
🕹 Technology Optimization
  1. Python has a powerful feature with the assert keyword for contract-based programming.
  2. Using assert in Python can help catch bugs and remove checks in production with PYTHONOPTIMIZE.
  3. The community is unaware of this feature, leading to potential misuse of assert statements.

Age of Invention: Cash Cows

Age of Invention, by Anton Howes • 1008 implied HN points • 10 Aug 23
💾 History Optimization
  1. Robert Bakewell had an 'improving mentality' when it came to breeding animals, focusing on optimizing profit and efficiency.
  2. Bakewell selectively bred cows and sheep to maximize valuable meat and minimize feeding costs.
  3. The improving mentality led Bakewell to continuously optimize all aspects of his farm, from animal breeding to farm layout and operations.

A fun thought on optimization

Sunday Letters • 79 implied HN points • 22 Jan 24
🕹 Technology Optimization
  1. Avoid optimizing too early in the design process. This can lead to wasted efforts and complicated designs.
  2. In the world of AI, focusing too much on costs can lead to weak solutions. It's better to have a solid, simple design from the start.
  3. Instead of worrying about future needs, consider how hard it will be to make changes later. It's important to find a balance between planning and flexibility.