Skip to content
Algorithm Atlas
OptimizationAll categories
CoreSource checked

Stochastic Gradient Descent

Updates parameters using gradients from one example or a small batch at a time.

Explore relationshipsOpen constellationBrowse atlas

Quick Facts

Compact metadata for implementation and discovery.

Category
Optimization
Difficulty
Intermediate
Family
First-Order Optimization
Tier
Core
Status
Verified
Review
Source Checked
Sources
1
Formulas
1
Relationships
7
Code example
Not ready
Visualization
Ready
Aliases
SGD

Overview

Stochastic Gradient Descent trades exact full-dataset gradients for cheaper noisy updates that scale to large datasets.

Core Concepts

mini-batch gradientlearning ratenoisy updateiterative optimizationobjective functiongradient updatelearned parametersrandom sampling

Quick facts

Complexity

Runtime and memory behavior at a glance.

Best

O(bg)

Average

O(tbg)

Worst

O(tbg)

Space

O(p)

When to Use

  • Use Stochastic Gradient Descent when you need to sample training examples or mini-batches.
  • Use it for training machine learning models with inputs like objective or loss function.
  • Use it when you can define a score, objective, constraint, or feasibility test clearly.
  • Use it when the First-Order Optimization tradeoff is acceptable for the problem size.

When Not to Use

  • Avoid it when repeatability is required but seeds and random sources are not controlled.
  • Avoid it when the objective is noisy or changing and the algorithm assumes stable feedback.
  • Avoid it when the required output is not optimized parameters.

What It Does

  • Samples training examples or mini-batches
  • Updates parameters with noisy gradients

Use Cases

  • Training machine learning models
  • Large-scale optimization
  • Online learning

Inputs

  • Objective or loss function
  • Training data
  • Learning rate

Outputs

  • Optimized parameters

Key Formulas (1)

Mini-batch update

θnext = θ − α · ∇(Jbatch(θ))
theta_next = theta - alpha * gradient(J_batch(theta))

Updates parameters using one example or a mini-batch instead of the full dataset.

Pseudocode

  1. 1.Initialize parameters
  2. 2.Choose a random example or mini-batch
  3. 3.Compute its gradient
  4. 4.Move parameters opposite the gradient
  5. 5.Repeat over many updates

Canonical Example

Input
objective = minimize (x - 3)^2, start = x = 0
Output
x moves toward 3
Explanation
Stochastic Gradient Descent: The algorithm repeatedly improves a candidate solution according to the objective.

Implementation Notes

  • Keep the input contract explicit: Objective or loss function; Training data; Learning rate.
  • Use this step as the implementation spine: Initialize parameters.
  • Then preserve the next state change: Choose a random example or mini-batch.
  • Thread a seedable random source through the implementation for repeatable tests.
  • Log objective values across iterations so regressions and stalled progress are visible.

Common Pitfalls

  • Learning rate schedules often matter as much as the base algorithm.
  • Stopping after one good-looking result without checking convergence or constraints.
  • Define edge-case behavior for empty, invalid, or boundary inputs before relying on the result.

Interactive Mechanism Walkthrough

Iterative Convergence & State Trajectory

Show noisy parameter steps trending downhill.

Stochastic Gradient Descent mechanism walkthrough step 1 of 5x₀x₁x₂x*
Step 1 of 5: Initialize parameters
noisy-descent

Notes

Stochastic Gradient Descent operates in O(tbg) time and O(p) space within the First-Order Optimization family. Key governing relation: theta_next = theta - alpha * gradient(J_batch(theta)). Often compared with Gradient Descent and Adam Optimizer.

Sources (1)

What each reference supports for this entry.

Bottou, Curtis, and Nocedal 2018

Paper

Optimization survey covering stochastic gradient methods for large-scale learning.

Open source

Explore Next

Good next jumps from the relationship graph, ranked for useful discovery.

Open discoveryOpen map
Variant ofGradient Descent

Optimization · Foundation

SGD applies gradient descent updates using samples or mini-batches.

Compare with this
Alternative toAdam Optimizer

Optimization · Core

SGD is a simpler optimizer often compared with Adam.

Compare with this
Used byWord2Vec Skip-Gram

Machine Learning · Core

Skip-gram models are typically optimized with stochastic gradient updates.

Compare with this
Often paired withBackpropagation

Machine Learning · Foundation

Neural networks commonly combine backpropagated gradients with SGD updates.

Compare with this
Often paired withMatrix Factorization

Machine Learning · Core

Large matrix-factorization models are often trained with stochastic gradient updates.

Compare with this

Relationships

View full relationship page
Alternative toAdam Optimizer

SGD is a simpler optimizer often compared with Adam.

Often paired withBackpropagation

Neural networks commonly combine backpropagated gradients with SGD updates.

Variant ofGradient Descent

SGD applies gradient descent updates using samples or mini-batches.

Related Algorithms

Gradient DescentAdam OptimizerBackpropagationGradient Clipping

Tags

OptimizationGradient BasedMachine LearningStochastic