A Markov Decision Process is the mathematical frame for sequential decision-making — states, actions, rewards, and transitions — and the formal problem that all of reinforcement learning for robots is trying to solve.
An MDP describes a decision problem as: you're in a situation (state), you pick an action, you get a reward and land in a new situation. Solve it and you have a rule for what to do in every situation to earn the most reward over time.
🎯 Quick challenge
The 'Markov' property means the next state depends on…
Before a robot can learn to act, we need a precise way to state the problem "what should I do, over and over, to do well?" That statement is the Markov Decision Process — the foundation of all reinforcement learning.
The five ingredients
An MDP is defined by:
States (S) — the situations the robot can be in.
Actions (A) — what it can do in each state.
Transitions (P) — the (often uncertain) rule for which state an action leads to.
Rewards (R) — the numeric feedback for each step.
Discount (γ) — how much future reward is worth versus immediate reward.
The agent–environment loop of an MDP
The agent observes a state, takes an action, receives a reward and a new state — then repeats. Solving the MDP means choosing actions that maximize total discounted reward.
The Markov property
The defining assumption is that the future is memoryless: the next state depends only on the current state and action, not the entire history. This is what makes the problem tractable — you only need to reason about where you are now, not everything that led here. (When the robot can't fully observe its state, it becomes a POMDP, the harder partially-observed cousin.)
What "solving" it means
The goal is a policy — a rule mapping each state to an action — that maximizes expected total discounted reward. The tools that find it define whole fields: value iteration and dynamic programming when the model is known; Q-learning and policy gradient methods when it isn't and the robot must learn from experience. All of them build on the Bellman equation, which relates the value of a state to the values of the states it leads to.
Why it matters for robots
Locomotion, manipulation, and navigation are all naturally MDPs: a stream of states, actions, and rewards. Framing a robot task as an MDP is the first step to applying reinforcement learning — and understanding the frame is what makes the algorithms make sense.
