Overview
Backpropagation records intermediate activations during the forward pass, then propagates loss gradients backward so each parameter receives its derivative.
FoundationSource checked
Backpropagation
Computes neural-network gradients by applying the chain rule backward through layers.
Quick Facts
Compact metadata for implementation and discovery.
- Category
- Machine Learning
- Difficulty
- Advanced
- Family
- Neural Network Training
- Tier
- Foundation
- Status
- Verified
- Review
- Source Checked
- Sources
- 1
- Formulas
- 1
- Relationships
- 12
- Code example
- Not ready
- Visualization
- Ready
- Aliases
- Backprop
Core Concepts
chain ruleforward passbackward passparameter gradient
Quick facts
Complexity
Runtime and memory behavior at a glance.
Best
O(w)
Average
O(w)
Worst
O(w)
Space
O(w)
When to Use
- Use it to train differentiable neural networks.
- Use automatic differentiation systems when implementing complex models.
- Use it when parameters affect the loss through a chain of differentiable operations.
When Not to Use
- Avoid it for nondifferentiable pipelines unless surrogate gradients are justified.
- Avoid manual implementations for large systems when autodiff is available.
- Avoid ignoring exploding or vanishing gradients in deep networks.
What It Does
- Propagates error signals backward
- Computes gradients for network weights
Use Cases
- Training neural networks
- Deep learning
- Learning feature representations
Inputs
- Neural network
- Training example
- Loss function
Outputs
- Gradients for model parameters
Key Formulas (1)
Chain rule gradient
dLoss/dw = dLoss/dactivation · dactivation/dw
dLoss/dw = dLoss/dactivation * dactivation/dwPropagates gradients through each layer using the chain rule.
Pseudocode
- 1.Run a forward pass through the network
- 2.Compute the loss at the output
- 3.Compute the output layer gradient
- 4.Move backward through each layer
- 5.Store gradients for each weight
Canonical Example
- Input
- examples = [[size, price], ...], target = label or value
- Output
- model predictions for new examples
- Explanation
- Backpropagation: The algorithm learns a pattern from examples and applies it to unseen data.
Implementation Notes
- Keep forward-pass caches tied to the operations that need them during the backward pass.
- Check gradients with finite differences on small examples.
- Separate gradient accumulation, regularization, and optimizer updates.
Common Pitfalls
- Dropping cached forward values needed for gradients.
- Getting tensor shapes or broadcasting rules wrong.
- Updating parameters before all gradients that depend on old values are computed.
Interactive Mechanism Walkthrough
Iterative Convergence & State TrajectoryShow error signals flowing backward through network layers.
Step 1 of 5: Run a forward pass through the network
neural-network-gradientsNotes
Backpropagation is the gradient-computation procedure; an optimizer such as SGD or Adam decides how parameters change.
Sources (1)
What each reference supports for this entry.
Rumelhart, Hinton, and Williams 1986
PaperFoundational backpropagation paper for learning internal representations in neural networks.
Open sourceExplore Next
Good next jumps from the relationship graph, ranked for useful discovery.
UsesGradient Descent
Optimization · Foundation
Backpropagation supplies gradients for gradient-based optimizers.
Compare with thisUsed byMultilayer Perceptron
Machine Learning · Core
MLPs are commonly trained with backpropagation.
Compare with thisOften paired withCross-Entropy Loss
Machine Learning · Foundation
Classification networks often backpropagate cross-entropy gradients.
Compare with thisOften paired withAdam Optimizer
Optimization · Core
Adam commonly updates neural network weights using backpropagated gradients.
Compare with thisRelationships
View full relationship pagePrerequisite ofConvolutional Neural Network
CNN training commonly relies on backpropagation.
Often paired withCross-Entropy Loss
Classification networks often backpropagate cross-entropy gradients.
UsesGradient Descent
Backpropagation supplies gradients for gradient-based optimizers.
Prerequisite ofRecurrent Neural Network
RNN training uses backpropagation through time.
Related Algorithms
Tags
Machine LearningNeural NetworksOptimization