Overview
Q-Learning updates an estimate of long-term action value using the reward from a transition and the best estimated value of the next state.
SpecializedSource checked
Q-Learning
Learns action values from rewards without needing a model of the environment.
Quick Facts
Compact metadata for implementation and discovery.
- Category
- Machine Learning
- Difficulty
- Intermediate
- Family
- Reinforcement Learning
- Tier
- Specialized
- Status
- Verified
- Review
- Source Checked
- Sources
- 1
- Formulas
- 1
- Relationships
- 4
- Code example
- Ready
- Visualization
- Ready
Core Concepts
state-action valuereward signaltemporal differenceexploration
Quick facts
Complexity
Runtime and memory behavior at a glance.
Best
O(1) per transition
Average
O(transitions)
Worst
O(transitions)
Space
O(states * actions)
When to Use
- Use when an agent can learn by trial and error from reward feedback.
- Use when small reinforcement learning tasks is a central requirement.
When Not to Use
- Avoid when unsafe exploration or sparse rewards make trial-and-error learning impractical.
- Avoid when the data size, labels, or feature assumptions do not match the model.
What It Does
- Stores state-action values
- Updates values from observed rewards
- Chooses actions from learned estimates
Use Cases
- Small reinforcement learning tasks
- Gridworld agents
- Model-free control
Inputs
- States
- Actions
- Rewards
- Learning rate
- Discount factor
Outputs
- Action-value table
- Policy derived from values
Key Formulas (1)
Q update
Q(s,a) = Q(s,a) + α · (reward + γ · maxa Q(snext,a) − Q(s,a))
Q(s,a) = Q(s,a) + alpha * (reward + gamma * max_a Q(s_next,a) - Q(s,a))Updates an action value using a temporal-difference target.
Pseudocode
- 1.Initialize Q values
- 2.Choose and take an action
- 3.Observe reward and next state
- 4.Update the Q value toward the reward plus discounted future value
- 5.Repeat while exploring and exploiting
Canonical Example
- Input
- state = cell A, action = right, reward = 1
- Output
- Q[A,right] increases
- Explanation
- The action value moves toward the observed reward plus estimated future value.
Implementation Notes
- Use epsilon-greedy exploration before moving to more advanced policies.
- Split data for validation and keep preprocessing identical between training and inference.
- Keep a tiny example with expected output beside the implementation.
Common Pitfalls
- A greedy policy too early can prevent the agent from discovering better actions.
- Evaluating only on training data can hide overfitting and poor generalization.
- Not defining boundary, empty, or degenerate inputs before implementation.
Interactive Mechanism Walkthrough
Iterative Convergence & State TrajectoryShow a grid of action values changing as an agent receives rewards.
Step 1 of 5: Initialize Q values
model-trainingNotes
Exploration strategy matters because the agent only learns from experienced transitions.
Sources (1)
What each reference supports for this entry.
Watkins and Dayan 1992
PaperFoundational Q-learning paper for reinforcement learning with action-value updates.
Open sourceExplore Next
Good next jumps from the relationship graph, ranked for useful discovery.
Alternative toMonte Carlo Simulation
Simulation · Foundation
Q-learning learns value estimates from experience rather than pure simulation averages.
Compare with thisConceptual neighborsActor-Critic Method
Machine Learning · Core
Actor-Critic methods and Q-Learning both learn from temporal reward feedback but optimize different representations.
Compare with thisRelationships
View full relationship pageConceptual neighborBellman-Ford
Both repeatedly update value estimates using Bellman-style recurrences.
Alternative toMonte Carlo Simulation
Q-learning learns value estimates from experience rather than pure simulation averages.
Related Algorithms
Tags
Machine LearningReinforcement LearningOptimizationDynamic Programming