The hottest Experimental Design Substack posts right now

Where and how you ask matters: public, informal polls (like Twitter) invite people to joke or troll on simple/funny questions, while private or more formal surveys tend to get more accurate answers.
Some questions are especially vulnerable to ego or incentives—people give more flattering or different answers when they expect feedback or visibility (e.g., claiming to be above average or reporting horniness), but other sensitive items (like certain sexual fantasies) may not change much.
There’s no one-size-fits-all rule for survey reliability; good survey design requires thinking about your audience’s incentives and visibility, testing specific questions, and adjusting phrasing or format to reduce trolling and bias.

The structure of scientific papers can create a misleading impression of how research actually happens. Often, real research involves lots of trial and error, not just a straight path from question to answer.
The amyloid cascade hypothesis, which suggests that amyloid plaques in the brain cause Alzheimer's, has been heavily focused on, but recent studies suggest it might not be the whole story. This has led to wasted research and funding on treatments that may not work.
When reading scientific papers, it's important to think critically and not just accept the conclusions presented. Questions about what is missing or what alternative explanations exist can reveal more about the validity of the research.

A beautiful experiment is efficient and clever, showing that you can get more useful information from it than the effort put in. This idea is not just about being smart; it's also about designing experiments that yield significant results.
The qualities that make an experiment beautiful include clarity, simplicity, and decisiveness. A good experiment should be easy to understand and should clearly show the results or answers it seeks.
Historically, the appreciation of experiments has shifted. In the past, the focus was on revealing nature's beauty, but now it's more about the design and ingenuity behind the experiment itself.

Statistical controls aren’t a magic solution; using them incorrectly can lead to wrong conclusions. It's important to understand the underlying relationships between variables before just plugging numbers into an equation.
Matching groups in studies to control for variables often isn't enough. You might still end up with biases if the controls aren’t comprehensive or well-measured.
Over-controlling or trying to account for too many factors can confuse the results. Sometimes, less control can provide a clearer picture, just like how comparing fast food and fine dining should keep their unique qualities intact.

Social tipping points may not be as impactful as commonly believed
The idea of a committed minority of 25% causing rapid societal change could be overrated
Experimental evidence suggests that the 25% threshold for social tipping points is not highly applicable to real-world social issues

Get a weekly roundup of the best Substack posts, by hacker news affinity:

The use of machine learning and adaptive experimental design is revolutionizing battery technology for more efficient, reliable, and sustainable energy storage solutions.
Machine learning enhances consumer electronics by optimizing battery life and performance, showing practical benefits in devices like smartphones and electric vehicles.
The combination of machine learning and adaptive experimental design leads to quicker research and innovation in battery technology, making advancements more tailored, responsive, and impactful across industries.

Relying only on randomized experiments can be limiting. It's important to consider all types of evidence based on their quality.
Not every decision needs a complex A/B test; sometimes simpler data or even gut feelings are enough.
We should weigh the cost of getting reliable data against the value it provides. For some choices, high-quality data is a must, but for others, less rigorous information can do the job.

A/B tests are important for businesses because they help test ideas and make informed decisions. Many companies have seen significant revenue increases by using A/B tests.
It's crucial to define the right performance metrics for A/B tests to ensure long-term success. Focus on metrics that show real customer engagement, not just short-term results.
Pay close attention to statistical principles when running A/B tests. Misunderstanding p-values and making hasty conclusions can lead to incorrect results and poor decisions.

A/B testing compares two treatments to measure impact.
Frequentist A/B testing involves hypothesis formulation, experiment design, and statistical testing.
Bayesian A/B testing incorporates prior beliefs to estimate probabilities directly.

Supervised machine learning helps us understand how inputs relate to outputs, but just because two things move together doesn't mean one causes the other.
To prove something causes another, experiments are the best way, but we can also make educated guesses using causal diagrams, like trees that show how different factors connect.
Machine learning models are great at predictions but aren't designed to show cause and effect; we can use them to help create clearer models for understanding these relationships.