Overview
Principal Component Analysis centers data and finds orthogonal axes ordered by explained variance. Keeping the leading components gives a lower-dimensional representation that preserves the strongest linear variation.
CoreSource checked
Principal Component Analysis
Projects data onto orthogonal directions that capture the largest variance.
Quick Facts
Compact metadata for implementation and discovery.
- Category
- Machine Learning
- Difficulty
- Intermediate
- Family
- Dimensionality Reduction
- Tier
- Core
- Status
- Verified
- Review
- Source Checked
- Sources
- 1
- Formulas
- 1
- Relationships
- 7
- Code example
- Not ready
- Visualization
- Ready
- Aliases
- PCA
Core Concepts
learned parametersprojection basismatrix operationdimensionality reductionmodel fittraining examplesprediction rule
Quick facts
Complexity
Runtime and memory behavior at a glance.
Best
O(nd^2)
Average
O(nd^2 + d^3)
Worst
O(nd^2 + d^3)
Space
O(d^2)
When to Use
- Use when linear directions of variance are meaningful for the data.
- Use when reducing correlated numeric features helps modeling or visualization.
When Not to Use
- Avoid when important structure is nonlinear or not tied to variance.
- Avoid fitting PCA before train/test splitting if leakage matters.
What It Does
- Centers and optionally scales feature data
- Finds eigenvectors or singular vectors of the covariance structure
- Projects examples onto the leading components
Use Cases
- Dimensionality reduction
- Feature visualization
- Noise reduction under a linear variance model
Inputs
- Numeric data matrix
- Number of components
Outputs
- Principal components
- Projected data
Key Formulas (1)
Covariance matrix
C = (XcenteredT · Xcentered) / (n − 1)
C = (X_centered^T * X_centered) / (n - 1)Summarizes how centered features vary together before extracting principal directions.
Pseudocode
- 1.Split data before fitting preprocessing steps
- 2.Center and optionally scale the training matrix
- 3.Compute covariance eigenvectors or the SVD
- 4.Sort components by explained variance
- 5.Project data onto the selected leading components
Canonical Example
- Input
- 2D points clustered along a diagonal line
- Output
- one principal axis along the diagonal
- Explanation
- The first component captures the direction where the centered points vary most.
Implementation Notes
- Center training data and apply the same centering to held-out data.
- Use SVD for numerical stability when practical.
- Track explained variance ratio and reconstruction error.
Common Pitfalls
- Unscaled features can dominate components only because of units.
- Explained variance is not the same as predictive usefulness.
- Fitting PCA on all data before validation leaks information.
Interactive Mechanism Walkthrough
Iterative Convergence & State TrajectoryShow points projected onto the strongest variance direction.
Step 1 of 5: Split data before fitting preprocessing steps
component-projectionNotes
PCA is unsupervised; it can discard low-variance features that still matter for prediction.
Sources (1)
What each reference supports for this entry.
The Elements of Statistical Learning
BookStatistical learning reference for the model family and tradeoffs of Principal Component Analysis.
Open sourceExplore Next
Good next jumps from the relationship graph, ranked for useful discovery.
UsesSingular Value Decomposition
Numerical Methods · Core
PCA is commonly computed through SVD.
Compare with thisOften paired withK-Means
Machine Learning · Foundation
PCA is often used for visualization or preprocessing before clustering.
Compare with thisConceptual neighborsAutoencoder
Machine Learning · Core
Both learn compressed representations of data.
Compare with thisConceptual neighborMatrix Factorization
Machine Learning · Core
Both represent data through lower-dimensional linear factors.
Compare with thisRelationships
View full relationship pageConceptual neighborMatrix Factorization
Both represent data through lower-dimensional linear factors.
Often paired withK-Means
PCA is often used for visualization or preprocessing before clustering.
Related Algorithms
Tags
Machine LearningDimensionality ReductionLinear Algebra