Skip to content
Algorithm Atlas
Machine LearningAll categories
FoundationSource checked

Linear Regression

Fits a linear model that predicts a numeric target from input features.

Explore relationshipsOpen constellationBrowse atlas

Quick Facts

Compact metadata for implementation and discovery.

Category
Machine Learning
Difficulty
Beginner
Family
Supervised Learning
Tier
Foundation
Status
Verified
Review
Source Checked
Sources
1
Formulas
2
Relationships
11
Code example
Not ready
Visualization
Ready

Overview

Linear Regression chooses coefficients so a weighted sum of features approximates a numeric target, commonly by minimizing squared error.

Core Concepts

linear modelleast squaresresidualregularization

Quick facts

Complexity

Runtime and memory behavior at a glance.

Best

O(nd)

Average

O(ndt)

Worst

O(ndt)

Space

O(d)

When to Use

  • Use it as a transparent baseline for numeric prediction.
  • Use it when relationships are roughly linear or linear features are meaningful.
  • Use regularization when features are many or correlated.

When Not to Use

  • Avoid it when the target relationship is strongly nonlinear and features do not capture that structure.
  • Avoid interpreting coefficients causally without a causal design.
  • Avoid it when outliers dominate and no robust loss or preprocessing is used.

What It Does

  • Learns a linear relationship
  • Predicts continuous values

Use Cases

  • Forecasting numeric outcomes
  • Trend modeling
  • Baseline prediction models

Inputs

  • Feature matrix
  • Numeric target values

Outputs

  • Learned coefficients
  • Numeric predictions

Key Formulas (2)

Linear prediction

ŷ = dot(weights, features) + bias
y_hat = dot(weights, features) + bias

Computes a continuous prediction from weighted input features.

Squared error

loss = mean((y − ŷ)2)
loss = mean((y - y_hat)^2)

Measures fit by averaging squared prediction errors.

Pseudocode

  1. 1.Initialize model weights
  2. 2.Predict target values from features
  3. 3.Measure prediction error
  4. 4.Adjust weights to reduce error
  5. 5.Repeat until the fit stabilizes

Canonical Example

Input
examples = [[size, price], ...], target = label or value
Output
model predictions for new examples
Explanation
Linear Regression: The algorithm learns a pattern from examples and applies it to unseen data.

Implementation Notes

  • Separate training, validation, and test data before fitting.
  • Standardize features when regularization or gradient optimization is used.
  • Inspect residuals to check whether the linear model is appropriate.

Common Pitfalls

  • Ignoring feature scaling and multicollinearity.
  • Evaluating only training error.
  • Treating correlation in fitted coefficients as causal evidence.

Interactive Mechanism Walkthrough

Iterative Convergence & State Trajectory

Show a line moving closer to scattered data points.

Linear Regression mechanism walkthrough step 1 of 5x₀x₁x₂x*
Step 1 of 5: Initialize model weights
fit-line

Notes

Linear regression is often valuable because its failures are easy to inspect.

Sources (1)

What each reference supports for this entry.

The Elements of Statistical Learning

Book

Statistical learning reference for regression, classification, clustering, trees, and ensemble methods.

Open source

Explore Next

Good next jumps from the relationship graph, ranked for useful discovery.

Open discoveryOpen map
AlternativesGaussian Process Regression

Machine Learning · Core

Linear regression gives a parametric baseline, while GPR uses a nonparametric kernel prior.

Compare with this
AlternativesPerceptron

Machine Learning · Core

Perceptron and linear regression both learn linear models, but target classification and regression respectively.

Compare with this
Alternative toRANSAC

Optimization · Core

RANSAC is a robust fitting alternative when outliers would distort linear regression.

Compare with this
GeneralizesLasso Regression

Machine Learning · Core

Lasso regression adds L1 regularization to linear regression.

Compare with this
GeneralizesRidge Regression

Machine Learning · Core

Ridge regression adds L2 regularization to linear regression.

Compare with this
Conceptual neighborsGaussian Naive Bayes

Machine Learning · Core

Gaussian Naive Bayes and Linear Regression address nearby machine learning problems but make different implementation tradeoffs.

Compare with this

Relationships

View full relationship page
GeneralizesLasso Regression

Lasso regression adds L1 regularization to linear regression.

Alternative toRANSAC

RANSAC is a robust fitting alternative when outliers would distort linear regression.

GeneralizesRidge Regression

Ridge regression adds L2 regularization to linear regression.

Related Algorithms

Logistic RegressionGradient Descent

Tags

Machine LearningSupervised LearningRegressionOptimization