Artificial Intelligence Stack Exchange is a question and answer site for people interested in conceptual questions about life and challenges in a world where "cognitive" functions can be mimicked in purely digital environment. It only takes a minute to sign up.
Sign up to join this community
I'm reading an article on reinforcement learning, and I don't understand why the agent's policy $\pi$ is not part of definition of Markov Decision process(MDP):
Bu, Lucian, Robert Babu, and Bart De Schutter. "A comprehensive survey of multiagent reinforcement learning." IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews) 38.2 (2008): 156-172.
My question is:
Why the policy is not a part of the MDP definition?
The MDP defines the environment (which corresponds to the task that you need to solve), so it defines e.g. the states of the environment, the actions that you can take in those states, the probabilities of transitioning from one state to the other and the probabilities of getting a reward when you take a certain action in a certain state.
The policy corresponds to a strategy that the RL agent can follow to act in that environment. Note that the MDP doesn't define what the agent does in each state. That's why you need the policy! An optimal policy for a specific MDP corresponds to the strategy that, if followed, is guaranteed to give you the highest amount of reward in that environment. However, there are multiple strategies, most of them are not optimal. This should clarify why the policy is not part of the definition of the MDP.
Aside from the points raised in nbro's answer, I'd like to point out that for a single MDP (a single instance of a "problem"), it may be sensible to study it from perspectives that include no policy at all, or multiple different policies.
For instance, if I have an MDP, I may be interested in studying it by looking at various inherent properties of the environment. And if I then have multiple different MDPs, all without any policies or anything like that, I could compare them based on those properties. For example, I might simply want to measure the sizes of the state and action spaces. Or write out something like a game tree, and measure properties like the branching factor and the average / min / max / median depth at which we can find a terminal state.
On the other hand, it can also be interesting sometimes to study multiple different policies all for the same MDP. A very common example would be any off-policy learning algorithm (like $Q$-learning): they all involve at least one "target policy" (for which they're learning the $Q(s, a)$ values -- usually the greedy policy with respect to the values learned so far), and at least one "behaviour policy" (which they're using to generate experience -- often something like an $\epsilon$-greedy policy). A more complex example would be population-based training setups, like the one DeepMind used for their StarCraft 2 training; here they have a large population of different policies that they're all using in a complex training setup (and technically I suppose we should say they also have many different MDPs, where every combination of StarCraft 2 level + training opponent would formally be a different MDP).
As already explained by others, the policy accounts for the agent's decisions, which are not set by the enviornment.
The only requirement of an MDP is to define a space of possible policies from which the agent can sample from (the "D" of an MDP). This is usually skipped in literature as it is assumed that the agent can sample any policy, even if this is true only under strong conditions.
Instead, a Markov Reward Process (MRP) is an MDP with a fixed policy $\pi$, and so is a Markov Process (MP), which is an MRP without rewards.
I think David Silver's slides explain this very plainly, you might find it useful: https://www.davidsilver.uk/wp-content/uploads/2020/03/MDP.pdf
