Regularization (L1/L2, Dropout)

What it is

Regularization adds a penalty for model complexity to the training objective, discouraging the model from fitting noise in the training data.

Key points

• L2 (Ridge): penalizes the sum of squared weights, shrinking all weights smoothly toward zero — good default, keeps all features.
• L1 (Lasso): penalizes the sum of absolute weights, which drives some weights exactly to zero — performs implicit feature selection.
• Elastic Net combines both.
• Dropout (neural nets): randomly zeroes out a fraction of activations during training, forcing the network to not rely on any single unit — a different mechanism achieving a similar effect to weight penalties.
• Regularization strength is itself a hyperparameter — too much causes underfitting, too little doesn't fix overfitting.