Overfitting, Bias, and Variance: Why More Flexible Models Can Get Worse

Overfitting happens when a model captures patterns specific to the training data rather than relationships that generalize to new examples. The important part is to understand what is measured, what is inferred, and what remains unknown.

The core idea

A flexible model can represent many functions, which is useful when the real relationship is complex. But flexibility also gives the model enough capacity to fit noise or accidental quirks. Bias-variance language provides one way to reason about the tension between overly rigid models and overly sensitive models.

Treat this as a design problem before treating it as a coding problem. Write the assumptions down. A short experiment can often settle a question that a long argument cannot.

How it works

Start by comparing training and validation performance. A large gap can be evidence that the model fits the training set much more closely than new data.

Then vary model complexity. A shallow tree may underfit; a very deep tree may memorize. Regularization, early stopping, feature selection, and more representative data can alter the balance.

Finally remember that bias and variance are explanatory concepts, not two buttons. The observed generalization behavior also depends on noise, distribution shift, sampling, evaluation quality, and the loss function.

A concrete example

A decision tree can keep splitting until it isolates individual training examples. Training error can approach zero while validation error rises. Limiting depth or requiring more samples per leaf can reduce this memorization and produce a model that generalizes better.

Change one input or one assumption and predict the result before testing it. This is a compact way to turn passive reading into an active learning loop.

Common mistakes

• Choosing the model with the highest training accuracy.
• Treating one train/validation split as definitive when the dataset is small.
• Assuming regularization fixes a dataset with severe sampling bias or label noise.

A student project that makes it stick

Create a small reproducible experiment around the mechanism. Store the dataset or fixture, the code, the measurement method, and the result. If the experiment cannot be rerun, the lesson is harder to verify later.

Where it connects

This topic connects to the surrounding engineering stack: data, networking, security, software design, and operations. The most useful concepts are the ones that explain behavior across several layers rather than only one framework.

What to remember

1. Define the objective before selecting the technique.
2. Make hidden assumptions explicit.
3. Preserve a baseline so improvements are measurable.
4. Inspect failure cases, not only averages.
5. Keep the experiment small enough to understand end to end.

Limitations

No simplified guide can capture every implementation detail. Results vary with data, versions, hardware, workload, and configuration. The sources below provide the normative or technical reference; use them when a production decision depends on details omitted here.

Related Observatory reads

• train validation test splits for machine learning
• feature engineering without data leakage
• transformers and attention the core idea

Primary sources

• scikit-learn — Decision tree learning
• scikit-learn — Learning curves