Overview
A recurrent neural network updates a hidden state as it reads each item, letting earlier context influence later predictions.
CoreSource checked
Recurrent Neural Network
Processes sequences by carrying hidden state from one step to the next.
Quick Facts
Compact metadata for implementation and discovery.
- Category
- Machine Learning
- Difficulty
- Advanced
- Family
- Neural Network
- Tier
- Core
- Status
- Verified
- Review
- Source Checked
- Sources
- 2
- Formulas
- 1
- Relationships
- 8
- Code example
- Ready
- Visualization
- Ready
- Aliases
- RNN
Core Concepts
hidden statesequence dependencyshared weightsbackpropagation through time
Quick facts
Complexity
Runtime and memory behavior at a glance.
Best
O(T h^2)
Average
O(T h^2)
Worst
O(T h^2)
Space
O(T h)
When to Use
- Use when training predictive models needs hidden state.
- Use when the input matches the assumptions behind neural Network.
When Not to Use
- Avoid when the dataset is too small or poorly labeled for the model capacity being used.
- Avoid when a simpler algorithm gives the required result with less setup.
What It Does
- Processes sequences by carrying hidden state from one step to the next
- Uses hidden state and sequence dependency to guide the work.
- Produces trained model.
Use Cases
- Training predictive models
- Finding patterns in data
- Building learned representations
Inputs
- Training data
- Model parameters
- Loss or scoring rule
Outputs
- Trained model
- Predictions
- Training diagnostics
Key Formulas (1)
Recurrent update
ht = f(Wx xt + Wh h_{t−1} + b)
h_t = f(W_x x_t + W_h h_{t-1} + b)Updates hidden state from the current input and previous state.
Pseudocode
- 1.Prepare the inputs and state needed for Recurrent Neural Network.
- 2.Apply the hidden state step while the stopping condition has not been met.
- 3.Update intermediate state and discard invalid candidates.
- 4.Return trained model.
Canonical Example
- Input
- Recurrent Neural Network with a minimal inspectable input
- Output
- Trained model
- Explanation
- Recurrent Neural Network applies hidden state to produce trained model.
Implementation Notes
- Separate training, validation, and test data before tuning model choices.
- Keep the hidden state step separate enough to test independently.
Common Pitfalls
- Evaluating only on training data hides overfitting and data leakage.
- Forgetting to check edge cases can make Recurrent Neural Network look correct on only ideal inputs.
Interactive Mechanism Walkthrough
Iterative Convergence & State TrajectoryShow data, model state, and the training signal that drives updates.
Step 1 of 4: Prepare the inputs and state needed for Recurrent Neural Network.
machine-learning-mechanismNotes
Recurrent Neural Network should be used with the assumptions and tradeoffs of neural Network in mind.
Sources (2)
What each reference supports for this entry.
Deep Learning
BookDeep learning reference covering Recurrent Neural Network model architecture, objective, gradient flow, or training behavior.
Open sourceBengio Long-Term Dependencies
PaperReference for recurrent neural networks, sequence learning, and the long-term dependency problem.
Open sourceExplore Next
Good next jumps from the relationship graph, ranked for useful discovery.
AlternativesTransformer Attention
Machine Learning · Specialized
Transformer attention models sequence dependencies without recurrent state updates.
Compare with thisUsesBackpropagation
Machine Learning · Foundation
RNNs are commonly trained with backpropagation through time.
Compare with thisOften paired withGradient Clipping
Optimization · Core
Recurrent networks often use clipping to control exploding gradients.
Compare with thisRelationships
View full relationship pageUsesBackpropagation
RNNs are commonly trained with backpropagation through time.
GeneralizesGRU
GRU is a gated recurrent architecture that specializes RNNs.
GeneralizesLong Short-Term Memory
LSTM is a gated recurrent architecture that specializes RNNs.
Related Algorithms
Tags
Machine LearningNeural NetworksSequence ModelingSupervised Learning