Bias-Variance Tradeoff

Why model error splits into bias and variance, and why reducing one often increases the other.

What it is

A model's prediction error decomposes into bias (error from overly simplistic assumptions — underfitting) and variance (error from being too sensitive to the training data's noise — overfitting), plus irreducible noise.

Key points

High bias: the model is too simple to capture the underlying pattern — high training and test error.
High variance: the model fits training data (including its noise) too closely — low training error, high test error.
Model complexity is the main knob: increasing it (more parameters, deeper trees, higher-degree polynomials) trades bias for variance.
Regularization, more training data, and ensembling are the standard levers for managing this tradeoff without just tuning complexity.