Overview
Ridge regression minimizes squared error plus a penalty proportional to the squared coefficient norm, reducing variance when features are correlated or data is limited.
CoreSource checked
Ridge Regression
Fits linear regression with an L2 penalty that shrinks coefficients.
Quick Facts
Compact metadata for implementation and discovery.
- Category
- Machine Learning
- Difficulty
- Intermediate
- Family
- Regularized Linear Model
- Tier
- Core
- Status
- Verified
- Review
- Source Checked
- Sources
- 1
- Formulas
- 1
- Relationships
- 5
- Code example
- Ready
- Visualization
- Ready
Core Concepts
linear modelL2 penaltyregularization strengthsquared error
Quick facts
Complexity
Runtime and memory behavior at a glance.
Best
O(n d^2)
Average
O(n d^2)
Worst
O(d^3 + n d^2)
Space
O(d^2)
When to Use
- Use when linear regression overfits or features are strongly correlated.
- Use when coefficient shrinkage is acceptable and all features can be scaled consistently.
When Not to Use
- Avoid when sparse feature selection is the goal; Lasso-style penalties are more appropriate.
- Avoid when the relationship is strongly nonlinear without feature engineering.
What It Does
- Learns linear weights
- Adds an L2 weight penalty
- Reduces coefficient variance
Use Cases
- Regression with correlated features
- Stable baseline models
- Small or noisy datasets
Inputs
- Feature matrix
- Target values
- Regularization strength
Outputs
- Linear weights
- Predictions
Key Formulas (1)
Ridge objective
loss = mean((y − Xw)2) + λ · ∑(wi2)
loss = mean((y - Xw)^2) + lambda * sum(w_i^2)Adds an L2 penalty to ordinary squared-error regression.
Pseudocode
- 1.Split training and validation data
- 2.Scale features using training statistics
- 3.Choose a regularization value lambda
- 4.Minimize squared error plus lambda times squared weight norm
- 5.Evaluate validation error and retune lambda if needed
- 6.Return the fitted coefficients
Canonical Example
- Input
- two correlated features predicting house price
- Output
- smaller, more stable coefficients than ordinary least squares
- Explanation
- The L2 penalty discourages large weights that fit noise in correlated inputs.
Implementation Notes
- Scale features using training data statistics only.
- Tune lambda or alpha on validation data.
- Track whether the solver uses closed form, gradient descent, or factorization.
Common Pitfalls
- Regularization strength must be chosen with validation, not training error.
- Unscaled features receive uneven effective penalties.
- The intercept is usually not penalized in the same way as feature weights.
Interactive Mechanism Walkthrough
Iterative Convergence & State TrajectoryShow a fitted line with coefficients shrinking as regularization increases.
Step 1 of 6: Split training and validation data
model-trainingNotes
Ridge regression improves stability through shrinkage; it does not make a linear model nonlinear.
Sources (1)
What each reference supports for this entry.
The Elements of Statistical Learning
BookStatistical learning reference for the model family and tradeoffs of Ridge Regression.
Open sourceExplore Next
Good next jumps from the relationship graph, ranked for useful discovery.
Relationships
View full relationship pageAlternative toLasso Regression
Ridge and Lasso use different regularization penalties.
Variant ofLinear Regression
Ridge regression adds L2 regularization to linear regression.
Related Algorithms
Tags
Machine LearningRegressionSupervised LearningRegularization