Overview
Gradient Descent evaluates the gradient of a loss function and updates parameters in the direction that locally decreases the loss. The step size controls how far each update moves.
FoundationSource checked
Gradient Descent
Minimizes an objective by repeatedly moving parameters opposite the gradient.
Quick Facts
Compact metadata for implementation and discovery.
- Category
- Optimization
- Difficulty
- Intermediate
- Family
- First-Order Optimization
- Tier
- Foundation
- Status
- Verified
- Review
- Source Checked
- Sources
- 1
- Formulas
- 1
- Relationships
- 21
- Code example
- Not ready
- Visualization
- Ready
Core Concepts
objective functiongradientlearning rateiterative update
Quick facts
Complexity
Runtime and memory behavior at a glance.
Best
O(tg)
Average
O(tg)
Worst
O(tg)
Space
O(p)
When to Use
- Use it when the objective is differentiable or has useful gradient estimates.
- Use it for model fitting, optimization, and numerical parameter tuning.
- Use stochastic or mini-batch variants when data is large.
When Not to Use
- Avoid it when gradients are unavailable or too noisy to guide progress.
- Avoid fixed large learning rates that make the objective diverge.
- Avoid expecting global optima for nonconvex objectives without additional analysis.
What It Does
- Computes a direction of steepest increase
- Moves parameters in the opposite direction
Use Cases
- Training machine learning models
- Fitting numerical parameters
- Minimizing smooth loss functions
Inputs
- Objective function
- Initial parameters
- Learning rate
Outputs
- Improved parameters
- Objective value history
Key Formulas (1)
Gradient descent update
θnext = θ − α · ∇(J(θ))
theta_next = theta - alpha * gradient(J(theta))Moves parameters in the direction that most quickly reduces the objective function.
Pseudocode
- 1.Choose initial parameters
- 2.Compute the gradient of the objective
- 3.Move parameters opposite the gradient
- 4.Repeat until progress is small or a limit is reached
- 5.Return the final parameters
Canonical Example
- Input
- objective = minimize (x - 3)^2, start = x = 0
- Output
- x moves toward 3
- Explanation
- Gradient Descent: The algorithm repeatedly improves a candidate solution according to the objective.
Implementation Notes
- Track the objective value and gradient norm during iterations.
- Use learning-rate schedules or adaptive methods when fixed steps are unstable.
- Keep parameter updates separate from gradient computation for easier testing.
Common Pitfalls
- Choosing a learning rate that is too large or too small.
- Failing to scale features in problems where gradient magnitudes differ widely.
- Stopping without checking convergence or validation behavior.
Interactive Mechanism Walkthrough
Iterative Convergence & State TrajectoryShow a point stepping downhill on a curve or surface.
Step 1 of 5: Choose initial parameters
loss-surfaceNotes
Gradient descent is a family pattern; batch, stochastic, and mini-batch variants trade stability for per-step cost.
Sources (1)
What each reference supports for this entry.
Numerical Optimization
BookNocedal and Wright textbook reference for gradient descent, quasi-Newton methods, and nonlinear optimization.
Open sourceExplore Next
Good next jumps from the relationship graph, ranked for useful discovery.
Has variantsMomentum Gradient Descent
Optimization · Core
Momentum Gradient Descent is a gradient-family method that changes the update rule used by Gradient Descent.
Compare with thisHas variantsNesterov Accelerated Gradient
Optimization · Core
Nesterov Accelerated Gradient is a gradient-family method that changes the update rule used by Gradient Descent.
Compare with thisHas variantsRMSProp
Optimization · Core
RMSProp is a gradient-family method that changes the update rule used by Gradient Descent.
Compare with thisHas variantsSubgradient Method
Optimization · Core
Subgradient Method is a gradient-family method that changes the update rule used by Gradient Descent.
Compare with thisRelationships
View full relationship pagePrerequisite ofAdam Optimizer
Adam builds on gradient updates with adaptive moment estimates.
Prerequisite ofBackpropagation
Backpropagation computes gradients that are often consumed by gradient descent.
Alternative toNewton's Method
Both iteratively minimize objectives, but Newton's Method uses second-order curvature.
Prerequisite ofStochastic Gradient Descent
Stochastic gradient descent adapts gradient updates to sampled data.
Related Algorithms
Tags
OptimizationIterativeMachine Learning