Overview
Gradient Clipping rescales or caps gradients before an optimizer applies them, commonly preventing exploding gradients in neural networks.
CoreSource checked
Gradient Clipping
Limits gradient size so optimization updates do not become unstable.
Quick Facts
Compact metadata for implementation and discovery.
- Category
- Optimization
- Difficulty
- Beginner
- Family
- Optimization Stabilization
- Tier
- Core
- Status
- Verified
- Review
- Source Checked
- Sources
- 1
- Formulas
- 1
- Relationships
- 3
- Code example
- Ready
- Visualization
- Ready
Core Concepts
gradient normthresholdrescalingtraining stability
Quick facts
Complexity
Runtime and memory behavior at a glance.
Best
O(n)
Average
O(n)
Worst
O(n)
Space
O(1) extra
When to Use
- Use when occasional large gradients destabilize an otherwise useful training loop.
- Use when training recurrent networks is a central requirement.
When Not to Use
- Avoid when gradients are consistently small or the real problem is poor feature scaling.
- Avoid when the objective is noisy, discontinuous, or violates the method's assumptions.
What It Does
- Measures gradient magnitude
- Caps or rescales large gradients
- Passes the clipped gradient to an optimizer
Use Cases
- Training recurrent networks
- Stabilizing deep learning
- Preventing rare large updates
Inputs
- Gradient vector
- Clipping threshold
Outputs
- Clipped gradient vector
Key Formulas (1)
Norm clipping
gclipped = g · min(1, threshold / norm(g))
g_clipped = g * min(1, threshold / norm(g))Rescales the gradient only when it exceeds the threshold.
Pseudocode
- 1.Compute the gradient
- 2.Measure its norm
- 3.If the norm exceeds the threshold, rescale the gradient
- 4.Apply the optimizer update with the clipped gradient
Canonical Example
- Input
- gradient norm = 12, threshold = 3
- Output
- clipped norm = 3
- Explanation
- The gradient direction is preserved while its magnitude is reduced.
Implementation Notes
- Log both unclipped and clipped gradient norms during tuning.
- Track objective value, step size, and convergence criteria at each iteration.
- Keep a tiny example with expected output beside the implementation.
Common Pitfalls
- Setting the threshold too low can stop learning by shrinking most updates.
- Stopping too early or using unscaled variables can make a good method look broken.
- Not defining boundary, empty, or degenerate inputs before implementation.
Interactive Mechanism Walkthrough
Iterative Convergence & State TrajectoryShow a large gradient vector shortened to fit inside a norm circle.
Step 1 of 4: Compute the gradient
optimization-landscapeNotes
Clipping controls update magnitude but does not fix a badly chosen objective or learning rate by itself.
Sources (1)
What each reference supports for this entry.
Pascanu, Mikolov, and Bengio 2013
PaperReference for exploding gradients and gradient clipping in recurrent network training.
Open sourceExplore Next
Good next jumps from the relationship graph, ranked for useful discovery.
Relationships
View full relationship pageOften paired withAdam Optimizer
Gradient clipping can be applied before adaptive optimizer updates.
Often paired withBackpropagation
Gradient clipping is commonly applied to backpropagated gradients in deep networks.
Often paired withRecurrent Neural Network
Recurrent networks often use clipping to control exploding gradients.
Related Algorithms
Tags
OptimizationGradient BasedMachine LearningRegularization