Overview
BFGS is a quasi-Newton method that builds an inverse Hessian approximation from gradient differences.
SpecializedSource checked
BFGS
Optimizes smooth functions by updating an approximation of second-order curvature.
Quick Facts
Compact metadata for implementation and discovery.
- Category
- Optimization
- Difficulty
- Advanced
- Family
- Quasi-Newton Optimization
- Tier
- Specialized
- Status
- Verified
- Review
- Source Checked
- Sources
- 1
- Formulas
- 1
- Relationships
- 6
- Code example
- Ready
- Visualization
- Ready
Core Concepts
quasi-newton updateinverse Hessianline searchgradient difference
Quick facts
Complexity
Runtime and memory behavior at a glance.
Best
O(n^2)
Average
O(k n^2)
Worst
O(k n^2)
Space
O(n^2)
When to Use
- Use when gradients are available and curvature information would speed convergence.
- Use when smooth unconstrained optimization is a central requirement.
When Not to Use
- Avoid when the parameter dimension is so large that storing a dense matrix is impractical.
- Avoid when the objective is noisy, discontinuous, or violates the method's assumptions.
What It Does
- Tracks gradient changes
- Updates an inverse-Hessian estimate
- Chooses curvature-aware search directions
Use Cases
- Smooth unconstrained optimization
- Parameter fitting
- Medium-scale numerical optimization
Inputs
- Objective function
- Gradient function
- Initial parameters
Outputs
- Estimated minimizer
- Objective value
Key Formulas (1)
Search direction
pk = −Hk · ∇(f(xk))
p_k = -H_k * gradient(f(x_k))Uses the inverse-Hessian estimate to choose a descent direction.
Pseudocode
- 1.Start with an identity inverse-Hessian estimate
- 2.Compute a search direction from the gradient
- 3.Line search along that direction
- 4.Update parameters and gradient
- 5.Update the inverse-Hessian approximation
Canonical Example
- Input
- start = [3, 3] on a smooth bowl
- Output
- parameters move toward the minimum
- Explanation
- BFGS uses gradient changes to improve each search direction.
Implementation Notes
- Check the curvature condition before applying the matrix update.
- Track objective value, step size, and convergence criteria at each iteration.
- Keep a tiny example with expected output beside the implementation.
Common Pitfalls
- A poor line search can make the curvature update unstable.
- Stopping too early or using unscaled variables can make a good method look broken.
- Not defining boundary, empty, or degenerate inputs before implementation.
Interactive Mechanism Walkthrough
Iterative Convergence & State TrajectoryShow search directions bending according to the accumulated curvature estimate.
Step 1 of 5: Start with an identity inverse-Hessian estimate
optimization-landscapeNotes
BFGS is powerful for smooth problems but stores a dense matrix.
Sources (1)
What each reference supports for this entry.
Numerical Optimization
BookNocedal and Wright textbook reference for gradient descent, quasi-Newton methods, and nonlinear optimization.
Open sourceExplore Next
Good next jumps from the relationship graph, ranked for useful discovery.
Alternative toGradient Descent
Optimization · Foundation
BFGS can converge faster when curvature estimates are reliable.
Compare with thisAlternativesLevenberg-Marquardt
Optimization · Specialized
Both use curvature-like information but target different nonlinear optimization settings.
Compare with thisRelationships
View full relationship pageAlternative toGradient Descent
BFGS can converge faster when curvature estimates are reliable.
GeneralizesL-BFGS
L-BFGS is a memory-limited specialization of BFGS.
ApproximatesNewton's Method
BFGS approximates Newton's curvature update with an inverse-Hessian estimate.
Related Algorithms
Tags
OptimizationQuasi-NewtonGradient BasedIterative