Mindful Modeler

Mindful Modeler focuses on enhancing machine learning practices through statistical thinking, critical data analysis, and model interpretability. It delves into methods like conformal prediction, quantile regression, and handling imbalanced data, emphasizing the importance of uncertainty estimation, thoughtful data treatment, and leveraging inductive biases for resilient, informative modeling.

Machine Learning Statistical Modeling Data Analysis Model Interpretability Uncertainty Quantification Research and Development Career Development Writing and Documentation

The hottest Substack posts of Mindful Modeler

And their main takeaways

From Theory to Practice: Inductive Biases in Machine Learning

639 implied HN points • 23 Apr 24
🕹 Technology Machine Learning Algorithms Data Bias Modeling
  1. Different machine learning models exhibit varying behaviors when extrapolating features, influenced by their inductive biases.
  2. Inductive biases in machine learning influence the learning algorithm's direction, excluding certain functions or preferring specific forms.
  3. Understanding inductive biases can lead to more creative and data-friendly modeling practices in machine learning.

Statistical modeling seen through inductive biases

419 implied HN points • 28 May 24
🔬 Science Statistics Modeling Machine Learning
  1. Statistical modeling involves modeling distributions and assuming relationships between features and the target with a few interpretable parameters.
  2. Distributions shape the hypothesis space by restricting the range of models compatible with specific distributions like a zero-inflated Poisson distribution.
  3. Parameterization in statistical modeling simplifies estimation, interpretation, and inference of model parameters by making them more interpretable and allowing for confidence intervals.

My perfectly imperfect note-taking system for ML papers

838 implied HN points • 12 Mar 24
🕹 Technology Note Taking
  1. Developing a note-taking system that works for you is essential, especially in fast-paced fields like ML research.
  2. Using software tools like Firefox, Zotero, and Obsidian can streamline the process of note-taking and organization.
  3. Having flexible note-taking 'rules' like using only bullet points, describing reading status, and avoiding copy-pasting can help streamline the note-taking process and encourage understanding.

Inductive biases of the Random Forest and their consequences

379 implied HN points • 21 May 24
🕹 Technology Machine Learning Algorithms
  1. Machine learning models like Random Forest have inductive biases that impact interpretability, robustness, and extrapolation.
  2. Random Forest's inductive biases come from decision tree learning algorithms, random factors like bootstrapping and column sampling, and ensembling of trees.
  3. Some specific inductive biases of Random Forest include restrictions to step functions, preference for deep interactions, reliance on features with many unique values, and the effect of column sampling on feature importance and model robustness.

Ignore inductive biases at your own peril

399 implied HN points • 07 May 24
🕹 Technology Machine Learning
  1. Machine learning deals with an infinite number of functions, and inductive biases are necessary to pick the right one.
  2. Inductive biases guide machine learning algorithms on where to search in the hypothesis space, impacting model choices like feature engineering and architecture.
  3. Ignoring inductive biases can lead to misunderstanding nuances in models and failing to grasp important model assumptions.
Get a weekly roundup of the best Substack posts, by hacker news affinity:

Shedding light on "Impossibility Theorems for Feature Attribution"

199 implied HN points • 18 Jun 24
🕹 Technology Data science Interpretation
  1. The limitations of feature attribution methods like SHAP and Integrated Gradients have been studied, particularly focusing on their reliability for explaining predictions as a sum of attributions.
  2. Tasks such as algorithmic recourse, characterizing model behavior, and identifying spurious feature identification all revolve around how predictions change with slight feature alterations, making SHAP unsuitable for these specific tasks.
  3. It's important to avoid using SHAP for questions related to minor changes in feature values or counterfactual analysis, as it may yield unreliable results in such scenarios.

How to make use of inductive biases

219 implied HN points • 04 Jun 24
🔬 Science Machine Learning Interpretability Forecasting Modeling
  1. Inductive biases play a crucial role in model robustness, interpretability, and leveraging domain knowledge.
  2. Choosing inherently interpretable models can enhance model understandability by restricting the hypothesis space of the learning algorithm.
  3. By selecting inductive biases that reflect the data-generating process, models can better align with reality and improve performance.

How I made peace with quantile regression

778 implied HN points • 16 Jan 24
🔬 Science Statistics Machine Learning Estimation Modeling
  1. Quantile regression can be understood through the lens of loss optimization, specifically with the pinball loss function.
  2. In machine learning, quantile regression is essentially regression with the unique pinball loss function that emphasizes absolute differences between actual and predicted values.
  3. The asymmetry of the pinball loss function, controlled by the parameter tau, dictates how models should handle under- and over-predictions, making quantile regression a tool to optimize different quantiles of a distribution.

Take the inductive leap

279 implied HN points • 30 Apr 24
🔬 Science Machine Learning Induction Prediction Philosophy
  1. In a 2-day universe, predicting the future is uncertain and relies on assumptions, highlighting the challenge of inductive reasoning.
  2. The problem of induction questions the idea that the future will always mirror the past, emphasizing the need to critically assess assumptions.
  3. Taking an inductive leap involves making predictions based on past observations and acknowledging the inherent uncertainty and need to challenge assumptions in our understanding of the world.

Book Launch: ML for Science 🐦‍⬛

499 implied HN points • 06 Feb 24
🔬 Science Machine Learning Philosophy of science Interpretability Causality Domain Knowledge
  1. The book discusses the justification and strengths of using machine learning in science, emphasizing prediction and adaptation to data
  2. Machine learning lacks inherent transparency and causal understanding, but tools like interpretability and causality modeling can enhance its utility in research
  3. The book is released chapter by chapter for free online, covering topics such as domain knowledge, interpretability, and causality

Bridging the Gap: From Statistical Distributions to Machine Learning Loss Functions

818 implied HN points • 14 Nov 23
🕹 Technology Machine Learning Statistical Analysis Optimization
  1. Understanding the distribution of the target variable is key in choosing statistical analysis or machine learning loss functions.
  2. Certain loss functions in machine learning correspond to maximum likelihood estimation for specific distributions, creating a bridge between statistical modeling and machine learning.
  3. While connecting distributions to loss functions is insightful, the real power in machine learning lies in the flexibility to design custom loss functions rather than being constrained by specific distributions.

7 perspectives on machine learning

279 implied HN points • 09 Apr 24
🕹 Technology Machine Learning Data interpretation Automation Optimization Computing
  1. Machine learning is about building prediction models. It covers a wide range of applications, but may not be perfect for unsupervised learning.
  2. Machine learning is about learning patterns from data. This view is useful for understanding ML projects beyond just prediction.
  3. Machine learning is automated decision-making at scale. It emphasizes the purpose of prediction, which is to facilitate decision-making.

No Free Dessert in Machine Learning

399 implied HN points • 20 Feb 24
🔬 Science Machine Learning Generalization Modeling Data
  1. Generalization in machine learning is essential for a model to perform well on unseen data.
  2. There are different types of generalization in machine learning: from training data to unseen data, from training data to application, and from sample data to a larger population.
  3. The No Free Lunch theorem in machine learning highlights that assumptions and effort are always needed for generalization, and there's no free lunch when it comes to achieving further generalization.

Don't "fix" your imbalanced data

818 implied HN points • 05 Sep 23
🕹 Technology Data science Machine Learning
  1. Avoid trying to fix imbalanced data through sampling methods like oversampling or undersampling. It can distort your model's calibration and reduce information for the majority class.
  2. SMOTE, a common method for imbalanced data, works well only with weak classifiers, not strong ones. It may not be suitable if calibration is crucial for your model.
  3. Consider doing nothing when faced with imbalanced data as a default strategy. Sometimes in machine learning, less is more.

to kaggle, or not to kaggle

379 implied HN points • 13 Feb 24
🕹 Technology Machine Learning Modeling Competitions Data science AI
  1. There are conflicting views on Kaggle - some see it as a playground while others believe it produces top machine learning results.
  2. Participating in Kaggle competitions can be beneficial to learn core supervised machine learning concepts.
  3. The decision to focus on Kaggle competitions should depend on how much daily tasks align with Kaggle-style work.

How to deal with non-i.i.d data in machine learning

479 implied HN points • 09 Jan 24
🕹 Technology Machine Learning Data Modeling Data interpretation Model Evaluation
  1. Dealing with non-i.i.d data in machine learning can prevent data leakage, overfitting, and overly optimistic performance evaluation.
  2. For modeling data with dependencies, classical statistical approaches like mixed effect models can be used to correctly estimate coefficients.
  3. In non-i.i.d. data situations, the data splitting setup must align with the real-world use case of the model to avoid issues like row-wise leakage and over-optimistic model performance.

How to get from evaluation to final model

279 implied HN points • 19 Mar 24
🕹 Technology Machine Learning Data science Model Deployment Model Evaluation
  1. When moving from model evaluation to the final model, there are various approaches with trade-offs.
  2. Options include using all data for training the final model with best hyperparameters, deploying an ensemble of models, or a lazy approach of choosing one from cross-validation.
  3. Each approach like inside-out, parameter donation, or ensemble has its pros and cons, highlighting the complexity of transitioning from evaluation to the final model.

How to sell bread with quantile regression

339 implied HN points • 23 Jan 24
🕹 Technology Machine Learning Data science Modeling Predictions
  1. Quantile regression can be used for robust modeling to handle outliers and predict tail behavior, helping in scenarios where underestimation or overestimation leads to loss.
  2. It is important to choose quantile regression when predicting specific quantiles, such as upper quantiles, for scenarios like bread sales where under or overestimating can have financial impacts.
  3. Quantile regression can also be utilized for uncertainty quantification, and combining it with conformal prediction can improve coverage, making it useful for understanding and managing uncertainty in predictions.

Machine learning never cheats but it may play flawed games

259 implied HN points • 27 Feb 24
🕹 Technology Machine Learning Data Analysis Statistics Data generation
  1. Machine learning models may use shortcuts or exploit quirks in data, but it's important to consider them as playing the game according to the rules set by the data.
  2. Detecting flaws in prediction games is crucial, as models can unintentionally learn and act on misleading information from the data.
  3. Designing prediction games effectively requires a deep understanding of the data-generating process, tools like sampling theory, design of experiments, and a statistical mindset can be valuable in shaping prediction tasks.

SHAP Is Not All You Need

898 implied HN points • 07 Feb 23
🕹 Technology Machine Learning Interpretability Research Books Critique
  1. It's important to avoid assuming one method is always the best for all interpretation contexts when working with machine learning interpretability tools like SHAP.
  2. Different interpretability methods like SHAP and permutation feature importance (PFI) have unique goals and can provide different insights, so it's crucial to choose the method that aligns with the specific question you want to answer.
  3. Research on interpretability should be more driven by questions rather than methods, to ensure that the tools used provide meaningful insights based on the context.

Week #1: Getting Started With Conformal Prediction For Classification

1018 implied HN points • 20 Dec 22
🕹 Technology Machine Learning Data Analysis Predictive Modeling Python AI
  1. Model predictions should consider uncertainty to make informed decisions. Decisions relying only on point predictions can be risky.
  2. Conformal prediction is a method that can provide rigorous uncertainty scores, giving probabilistic guarantees of covering the true outcome.
  3. Conformal prediction is simple to apply, often with just 3 lines of code. It is model-agnostic, distribution-free, and comes with coverage guarantees.

Imbalanced data? Why "Do Nothing" should be the default

419 implied HN points • 19 Sep 23
🔬 Science Data science Machine Learning Classification Model Evaluation
  1. For imbalanced classification tasks, 'Do Nothing' should be the default approach, especially when dealing with calibration, strong classifiers, and class-based metrics.
  2. Addressing imbalanced data should be considered in scenarios where misclassification costs vary, metrics are impacted by imbalance, or weaker classifiers are used.
  3. Instead of using oversampling methods like SMOTE, adjusting data weighting, using cost-sensitive machine learning, and threshold tuning are more effective ways to handle class imbalance.

A Grandmaster's Guide to Machine Learning Challenges

339 implied HN points • 07 Nov 23
🕹 Technology Machine Learning Data Analysis Competition Podcast Challenges
  1. Focus on creating an end-to-end pipeline first, experiment with simple models, and then scale up gradually for better results in machine learning challenges.
  2. Success in a challenge correlates with time invested, so choose challenges that motivate you and spend time understanding the data before committing.
  3. Adopt a strategy to pick challenges that interest you, prioritize an experimentation loop, and aim to optimize later for overall success.

How much I've made with Modeling Mindsets

379 implied HN points • 22 Aug 23
💼 Business Entrepreneurship Book Publishing Passive Income Financial Stability Writing
  1. The author shared the earnings from their book 'Modeling Mindsets,' revealing they earned $14,155 in total.
  2. The book received positive feedback with 73 reviews, 40 on Amazon and 33 on Leanpub.
  3. Despite not getting rich, the author found financial stability through writing and digital assets, hinting at the potential for future income from the book.

Proofreading an entire book with GPT-4 for $6.88

479 implied HN points • 02 May 23
🕹 Technology AI Automation Tools Costs Workflow
  1. Proofreading an entire book with GPT-4 can help automate tasks like improving grammar, language, and cutting clutter in a draft.
  2. Using prompts to guide LLMs like GPT-4 is important for specific and successful outcomes in automated editing.
  3. The economic benefit of using GPT-4 for proofreading can be significant compared to hiring a professional proofreader, offering a balance between capabilities and cost.

Use interpretability to improve and debug your ML model

279 implied HN points • 05 Dec 23
🕹 Technology Machine Learning Interpretability Debugging
  1. Identify target leakage using feature importance to prevent accidental data pre-processing errors that leak target information into features.
  2. Debug your model by utilizing ML interpretability to spot errors in feature coding, such as incorrect signs on feature effects.
  3. Gain insights for feature engineering by understanding important features, and know which ones to focus on for creating new informative features.

But have you considered writing your own evaluation metric?

299 implied HN points • 21 Nov 23
🕹 Technology Machine Learning Evaluation Metrics Customization Domain Knowledge Model optimization
  1. Consider writing your own evaluation metric in machine learning to better align with your specific goals and domain knowledge.
  2. Off-the-shelf metrics like mean squared error come with assumptions that may not always fit your model's needs, so customizing metrics can be beneficial.
  3. Communication with domain experts and incorporating domain knowledge into evaluation metrics can lead to more effective model performance assessments.

A new chapter on generalization

99 implied HN points • 16 Apr 24
🔬 Science Machine Learning Generalization Statistics Interpretability Data Analysis
  1. Many COVID-19 classification models based on X-ray images during the pandemic were found to be ineffective due to various issues like overfitting and bias.
  2. Generalization in machine learning goes beyond just low test errors and involves understanding real-world complexities and data-generating processes.
  3. Generalization of insights from machine learning models to real-world phenomena and populations is a challenging process that requires careful consideration and assumptions.

Machine learning interpretability from first principles

359 implied HN points • 26 Sep 23
🕹 Technology Machine Learning Interpretability Models Methods
  1. Machine learning models can be understood as mathematical functions that can be broken down into simpler parts
  2. Interpretation methods address the behavior of these simplified components to enhance model interpretability
  3. Techniques like Permutation Feature Importance (PFI), SHAP values, and Accumulated Local Effect Plots use decomposition to explain the importance of features in prediction models

Use ML interpretability to gain data insights

239 implied HN points • 12 Dec 23
🕹 Technology Data insights Model performance
  1. ML interpretability can help gain insights about data, along with model improvement and justification.
  2. There are two scenarios for data insights: explorative scenario for general insights and inference scenario for specific, reliable answers.
  3. To achieve inference via ML interpretability, a theory is needed that links model interpretation to the real-world data-generating process.

A Pragmatic View of Uncertainty in Machine Learning

359 implied HN points • 06 Jun 23
🕹 Technology Machine Learning Uncertainty Modeling Prediction Calibration
  1. Machine learning models have uncertainty in predictions, categorized into aleatoric and epistemic uncertainty.
  2. Defining and distinguishing between aleatoric and epistemic uncertainty is a complex task influenced by deterministic and random factors.
  3. Conformal prediction methods capture both aleatoric and epistemic uncertainty, providing prediction intervals reflecting model uncertainty.

A short history of SHAP

359 implied HN points • 30 May 23
🕹 Technology Machine Learning Explainable AI AI Ethics Research Interpretability
  1. Shapley values originated in game theory in 1953 and contributed to fair resource distribution methods.
  2. In 2010, Shapley values were introduced to explain machine learning predictions, but didn't gain traction until the SHAP method in 2017.
  3. SHAP gained popularity for its new estimator for Shapley values, unification of existing methods, and efficient computation, leading to widespread adoption in machine learning interpretation.

The Case for Uninterpretable Machine Learning

319 implied HN points • 03 Oct 23
🕹 Technology Machine Learning Complexity Interpretability Flexibility Neural Networks
  1. Machine learning excels because it's not interpretable, not in spite of it.
  2. Embracing complexity in models like neural networks can effectively capture the intricacies of real-world tasks that lack simple rules or semantics.
  3. Interpretable models can outperform complex ones with smaller datasets and ease of debugging, but being open to complex models can lead to better performance.