CoreSource checked

Adam Optimizer

Adapts gradient updates using moving averages of gradients and squared gradients.

Explore relationships Open constellationBrowse atlas

Quick Facts

Compact metadata for implementation and discovery.

Category: Optimization
Difficulty: Intermediate
Family: Adaptive Gradient Optimization
Tier: Core
Status: Verified
Review: Source Checked
Sources: 1
Formulas: 2
Relationships: 6
Code example: Not ready
Visualization: Ready
Aliases: Adaptive Moment Estimation

Overview

Adam combines momentum-like smoothing with per-parameter step scaling, making it a common optimizer for neural networks.

Core Concepts

moment estimatesadaptive step sizebias correctiongradient updateobjective functionlearned parametersadaptive gradient optimizationiterative improvement

Quick facts

Complexity

Runtime and memory behavior at a glance.

Best

O(p)

Average

O(tp)

Worst

O(tp)

Space

O(p)

When to Use

Use Adam Optimizer when you need to track first and second gradient moments.
Use it for deep learning with inputs like parameters.
Use it when you can define a score, objective, constraint, or feasibility test clearly.
Use it when the Adaptive Gradient Optimization tradeoff is acceptable for the problem size.

When Not to Use

Avoid it when the objective is noisy or changing and the algorithm assumes stable feedback.
Avoid it when the required output is not updated parameters.

What It Does

Tracks first and second gradient moments
Scales parameter updates adaptively

Use Cases

Deep learning
Noisy objective optimization
Fast model prototyping

Inputs

Parameters
Gradients
Learning rate and decay settings

Outputs

Updated parameters

Key Formulas (2)

Moment update

m_t = beta1 · m_prev + (1 − beta1) · gradient

m_t = beta1 * m_prev + (1 - beta1) * gradient

Tracks an exponential moving average of recent gradients.

Adaptive step

θ_next = θ − α · m̂ / (√v̂ + ε)

theta_next = theta - alpha * m_hat / (sqrt(v_hat) + epsilon)

Scales the update by an adaptive second-moment estimate.

Pseudocode

1.Compute the current gradient
2.Update the moving average of gradients
3.Update the moving average of squared gradients
4.Bias-correct both averages
5.Apply the scaled parameter update

Canonical Example

Input: objective = minimize (x - 3)^2, start = x = 0
Output: x moves toward 3
Explanation: Adam Optimizer: The algorithm repeatedly improves a candidate solution according to the objective.

Implementation Notes

Keep the input contract explicit: Parameters; Gradients; Learning rate and decay settings.
Use this step as the implementation spine: Compute the current gradient.
Then preserve the next state change: Update the moving average of gradients.
Log objective values across iterations so regressions and stalled progress are visible.

Common Pitfalls

Adam is convenient, but tuned SGD can still generalize well in some settings.
Stopping after one good-looking result without checking convergence or constraints.
Define edge-case behavior for empty, invalid, or boundary inputs before relying on the result.

Interactive Mechanism Walkthrough

Iterative Convergence & State Trajectory

Show per-parameter step sizes changing during descent.

Step 1 of 5: Compute the current gradient

adaptive-steps

Notes

Adam Optimizer operates in O(tp) time and O(p) space within the Adaptive Gradient Optimization family. Key governing relation: m_t = beta1 * m_prev + (1 - beta1) * gradient. Often compared with Stochastic Gradient Descent and Backpropagation.