Overview
The skip-gram Word2Vec model trains embeddings so a center word is useful for predicting words around it. Negative sampling or hierarchical softmax makes the training objective practical on large vocabularies.
CoreSource checked
Word2Vec Skip-Gram
Learns word embeddings by predicting nearby context words from a center word.
Quick Facts
Compact metadata for implementation and discovery.
- Category
- Machine Learning
- Difficulty
- Intermediate
- Family
- Representation Learning
- Tier
- Core
- Status
- Verified
- Review
- Source Checked
- Sources
- 2
- Formulas
- 0
- Relationships
- 2
- Code example
- Ready
- Visualization
- Ready
Core Concepts
word embedding trainingrepresentation learningmachine learning
Quick facts
Complexity
Runtime and memory behavior at a glance.
Best
O(n)
Average
O(n log n)
Worst
O(n^2)
Space
O(n)
When to Use
- Use when local word co-occurrence is a useful signal for downstream tasks.
- Use when a compact static embedding is enough.
When Not to Use
- Avoid when context-dependent meanings are essential.
- Avoid when the corpus is too small or biased for stable embeddings.
What It Does
- Creates training pairs from center and context words
- Updates embeddings to score observed contexts higher
- Learns vector representations from co-occurrence patterns
Use Cases
- Word embeddings for text features
- Semantic similarity and analogy experiments
- Pretraining lightweight language representations
Inputs
- Tokenized text corpus and context window radius c
- Embedding dimension d and negative-sampling distribution
Outputs
- Trained target and context word embedding matrices W, W'
- Dense vector representations for vocabulary tokens
Pseudocode
- 1.Build vocabulary and initialize word vectors
- 2.For each sentence and center word
- 3.Create context-word pairs within the window
- 4.Sample negative words if using negative sampling
- 5.Update embeddings to raise context scores and lower negative scores
- 6.Return the learned embedding table
Canonical Example
- Input
- sentence 'the cat sat' with window size 1
- Output
- training pairs (cat, the) and (cat, sat)
- Explanation
- The center word 'cat' is trained to predict its neighboring context words.
Implementation Notes
- Build a vocabulary with frequency thresholds.
- Generate center-context pairs from a fixed window policy.
- Track validation on a downstream task or held-out objective.
Common Pitfalls
- Subsampling and negative-sampling choices strongly affect learned vectors.
- Rare words may receive poor embeddings without enough context.
- Evaluating only nearest neighbors can hide downstream task failures.
Interactive Mechanism Walkthrough
Iterative Convergence & State TrajectoryShow predicting context words from a target word embedding on a small word embedding training example.
Step 1 of 6: Build vocabulary and initialize word vectors
machine-learning-mechanismNotes
Skip-gram embeddings are static; they do not represent different meanings of a word in different contexts.
Sources (2)
What each reference supports for this entry.
Efficient Estimation of Word Representations
PaperOriginal Word2Vec reference for skip-gram training and distributed word embeddings.
Open sourceDeep Learning
BookDeep learning reference supporting Word2Vec Skip-Gram model architecture, training objective, gradient behavior, or representation-learning role.
Open sourceExplore Next
Good next jumps from the relationship graph, ranked for useful discovery.
Relationships
View full relationship pageConceptual neighborMatrix Factorization
Word embedding training is related to factorizing word-context co-occurrence information.
Skip-gram models are typically optimized with stochastic gradient updates.
Related Algorithms
Tags
Machine Learning