Overview
Gradient boosting trains each new weak learner on the loss gradient or residual left by the current ensemble.
CoreSource checked
Gradient Boosting
Builds an ensemble by adding weak models that reduce current prediction error.
Quick Facts
Compact metadata for implementation and discovery.
- Category
- Machine Learning
- Difficulty
- Intermediate
- Family
- Ensemble Learning
- Tier
- Core
- Status
- Verified
- Review
- Source Checked
- Sources
- 2
- Formulas
- 1
- Relationships
- 7
- Code example
- Ready
- Visualization
- Ready
Core Concepts
additive modelweak learnerresidual fittingloss gradient
Quick facts
Complexity
Runtime and memory behavior at a glance.
Best
O(t n log n)
Average
O(t n log n)
Worst
O(t n log n)
Space
O(trees)
When to Use
- Use when training predictive models needs additive model.
- Use when the input matches the assumptions behind ensemble Learning.
When Not to Use
- Avoid when the dataset is too small or poorly labeled for the model capacity being used.
- Avoid when a simpler algorithm gives the required result with less setup.
What It Does
- Builds an ensemble by adding weak models that reduce current prediction error
- Uses additive model and weak learner to guide the work.
- Produces trained model.
Use Cases
- Training predictive models
- Finding patterns in data
- Building learned representations
Inputs
- Training data
- Model parameters
- Loss or scoring rule
Outputs
- Trained model
- Predictions
- Training diagnostics
Key Formulas (1)
Additive model update
Fm(x) = F_{m−1}(x) + learningrate · hm(x)
F_m(x) = F_{m-1}(x) + learning_rate * h_m(x)Adds a new weak model to correct errors left by the current ensemble.
Pseudocode
- 1.Prepare the inputs and state needed for Gradient Boosting.
- 2.Apply the additive model step while the stopping condition has not been met.
- 3.Update intermediate state and discard invalid candidates.
- 4.Return trained model.
Canonical Example
- Input
- training examples with residual errors after a first weak tree
- Output
- an updated ensemble with another tree fitted to the residual pattern
- Explanation
- Each new learner targets the current model's errors, then is added with a learning-rate weight.
Implementation Notes
- Track validation loss across boosting rounds.
- Fit each weak learner to the negative gradient of the loss.
- Tune learning rate, depth, and number of estimators together.
Common Pitfalls
- Adding too many learners without validation-based stopping.
- Using a learning rate and tree depth that overfit small datasets.
- Ignoring loss choice when residual behavior does not match squared error.
Interactive Mechanism Walkthrough
Iterative Convergence & State TrajectoryShow data, model state, and the training signal that drives updates.
Step 1 of 4: Prepare the inputs and state needed for Gradient Boosting.
machine-learning-mechanismNotes
Gradient Boosting should be used with the assumptions and tradeoffs of ensemble Learning in mind.
Sources (2)
What each reference supports for this entry.
Gradient Boosting Machine
PaperOriginal gradient boosting paper supporting additive models fit to functional gradients.
Open sourceThe Elements of Statistical Learning
BookStatistical learning reference covering Gradient Boosting additive model, ensemble behavior, or supervised-learning objective.
Open sourceExplore Next
Good next jumps from the relationship graph, ranked for useful discovery.
Alternative toAdaBoost
Machine Learning · Core
Gradient Boosting and AdaBoost are sequential ensemble methods with different weighting and loss views.
Compare with thisAlternativesRandom Forest
Machine Learning · Core
Random forests average independently trained trees, while boosting trains trees sequentially.
Compare with thisRelationships
View full relationship pageAlternative toAdaBoost
Gradient Boosting and AdaBoost are sequential ensemble methods with different weighting and loss views.
Often paired withDecision Tree
Gradient boosting often uses decision trees as weak learners.
GeneralizesXGBoost-Style Gradient Boosted Trees
XGBoost-style boosted trees are a specialized implementation of gradient boosting.
Related Algorithms
Tags
Machine LearningEnsembleBoostingSupervised Learning