Overview
Cross-Entropy Loss penalizes assigning low probability to the correct class. It is commonly paired with sigmoid or softmax outputs for classification.
FoundationSource checked
Cross-Entropy Loss
Measures how well predicted probabilities match the true class distribution.
Quick Facts
Compact metadata for implementation and discovery.
- Category
- Machine Learning
- Difficulty
- Beginner
- Family
- Loss Function
- Tier
- Foundation
- Status
- Verified
- Review
- Source Checked
- Sources
- 1
- Formulas
- 1
- Relationships
- 4
- Code example
- Ready
- Visualization
- Ready
Core Concepts
log likelihoodprobability distributionclassification losssoftmax
Quick facts
Complexity
Runtime and memory behavior at a glance.
Best
O(k)
Average
O(n k)
Worst
O(n k)
Space
O(1)
When to Use
- Use it for probabilistic classification models.
- Use binary cross-entropy for independent binary labels.
- Use categorical cross-entropy with softmax-style multiclass predictions.
When Not to Use
- Avoid it when outputs are not probabilities or logits convertible to probabilities.
- Avoid unstable probability calculations that take the log of zero.
- Avoid it when the task's cost is not well represented by likelihood.
What It Does
- Measures how poorly predicted probabilities match the true labels
- Uses probability target and negative log likelihood to guide the work.
- Produces trained model.
Use Cases
- Training predictive models
- Finding patterns in data
- Building learned representations
Inputs
- Training data
- Model parameters
- Loss or scoring rule
Outputs
- Trained model
- Predictions
- Training diagnostics
Key Formulas (1)
Cross-entropy loss
L = −sumi yi · log(pi)
L = -sum_i y_i * log(p_i)Penalizes predicted probabilities that assign low probability to the true label.
Pseudocode
- 1.Prepare the inputs and state needed for Cross-Entropy Loss.
- 2.Apply the probability target step while the stopping condition has not been met.
- 3.Update intermediate state and discard invalid candidates.
- 4.Return trained model.
Canonical Example
- Input
- true class = cat, predicted probabilities = {cat: 0.8, dog: 0.2}
- Output
- loss = -log(0.8)
- Explanation
- The loss penalizes the model according to the probability assigned to the correct class.
Implementation Notes
- Use stable combined loss functions such as log-softmax plus negative log likelihood.
- Clip probabilities only as a last-resort display or safeguard, not as the primary numerical strategy.
- Check whether the library expects logits or probabilities.
Common Pitfalls
- Computing softmax and log separately in a numerically unstable way.
- Using the wrong target encoding for binary versus multiclass classification.
- Ignoring class imbalance when rare classes matter.
Interactive Mechanism Walkthrough
Iterative Convergence & State TrajectoryShow data, model state, and the training signal that drives updates.
Step 1 of 4: Prepare the inputs and state needed for Cross-Entropy Loss.
machine-learning-mechanismNotes
Cross-entropy is a loss function, not a model; it is optimized by another training algorithm.
Sources (1)
What each reference supports for this entry.
Deep Learning Book
BookDeep learning reference for neural model, loss, or training concepts in Cross-Entropy Loss.
Open sourceExplore Next
Good next jumps from the relationship graph, ranked for useful discovery.
Relationships
View full relationship pageOften paired withBackpropagation
Cross-entropy gradients are often propagated through neural networks.
Often paired withSoftmax Classifier
Softmax classifiers commonly optimize cross-entropy loss.
Related Algorithms
Tags
Machine LearningLoss FunctionClassificationSupervised Learning