Overview
Proximal Gradient takes a gradient step on the smooth term, then applies a proximal operator for the nonsmooth part.
SpecializedSource checked
Proximal Gradient Method
Optimizes objectives with a smooth part and a nonsmooth regularization term.
Quick Facts
Compact metadata for implementation and discovery.
- Category
- Optimization
- Difficulty
- Advanced
- Family
- Convex Optimization
- Tier
- Specialized
- Status
- Verified
- Review
- Source Checked
- Sources
- 1
- Formulas
- 1
- Relationships
- 4
- Code example
- Ready
- Visualization
- Ready
Core Concepts
smooth termproximal operatorregularizationsoft thresholding
Quick facts
Complexity
Runtime and memory behavior at a glance.
Best
O(n)
Average
O(k n)
Worst
O(k n)
Space
O(n)
When to Use
- Use when a useful objective contains a nonsmooth term such as an L1 penalty.
- Use when sparse regression is a central requirement.
When Not to Use
- Avoid when the proximal operator is unavailable or expensive compared with direct methods.
- Avoid when the objective is noisy, discontinuous, or violates the method's assumptions.
What It Does
- Computes convex optimization within Convex Optimization workloads (O(k n) time, O(n) space).
- Works by Optimizes objectives with a smooth part and a nonsmooth regularization term.
- Maintains key invariants across smooth term, proximal operator, regularization.
Use Cases
- Sparse regression
- L1-regularized models
- Constrained convex optimization
Inputs
- Smooth objective
- Nonsmooth penalty
- Step size
Outputs
- Optimized parameters
Key Formulas (1)
Proximal gradient step
xnext = prox_{α·g}(x − α · ∇(f(x)))
x_next = prox_{alpha*g}(x - alpha * gradient(f(x)))Combines a smooth gradient step with the proximal operator for the nonsmooth term.
Pseudocode
- 1.Compute gradient of the smooth term
- 2.Take a normal gradient step
- 3.Apply the proximal operator for the penalty
- 4.Repeat until convergence
Canonical Example
- Input
- weights = [2.0, 0.2], L1 penalty
- Output
- small weight shrinks toward zero
- Explanation
- The proximal step encourages sparse parameters after the gradient step.
Implementation Notes
- For L1 penalties, start with soft-thresholding as the proximal operator.
- Track objective value, step size, and convergence criteria at each iteration.
- Keep a tiny example with expected output beside the implementation.
Common Pitfalls
- Using plain gradient descent on a nonsmooth penalty can converge poorly.
- Stopping too early or using unscaled variables can make a good method look broken.
- Not defining boundary, empty, or degenerate inputs before implementation.
Interactive Mechanism Walkthrough
Iterative Convergence & State TrajectoryShow a gradient step followed by a shrinkage or projection step.
Step 1 of 4: Compute gradient of the smooth term
optimization-landscapeNotes
The proximal operator must match the penalty or constraint being used.
Sources (1)
What each reference supports for this entry.
Parikh and Boyd 2014
PaperReference for proximal algorithms and proximal-gradient updates.
Open sourceExplore Next
Good next jumps from the relationship graph, ranked for useful discovery.
Variant ofGradient Descent
Optimization · Foundation
Proximal gradient adds a proximal step to a gradient update.
Compare with thisHas variantsFISTA
Optimization · Specialized
FISTA is a gradient-family method that changes the update rule used by Proximal Gradient Method.
Compare with thisRelationships
View full relationship pageVariant ofGradient Descent
Proximal gradient adds a proximal step to a gradient update.
Often paired withLasso Regression
Lasso-style penalties are a common use case for proximal updates.
Related Algorithms
Tags
OptimizationGradient BasedRegularizationIterative