2
$\begingroup$

I'm reading an article on reinforcement learning, and I don't understand why the agent's policy $\pi$ is not part of definition of Markov Decision process(MDP):

enter image description here

Bu, Lucian, Robert Babu, and Bart De Schutter. "A comprehensive survey of multiagent reinforcement learning." IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews) 38.2 (2008): 156-172.

My question is:

Why the policy is not a part of the MDP definition?

$\endgroup$
7
$\begingroup$

The MDP defines the environment (which corresponds to the task that you need to solve), so it defines e.g. the states of the environment, the actions that you can take in those states, the probabilities of transitioning from one state to the other and the probabilities of getting a reward when you take a certain action in a certain state.

The policy corresponds to a strategy that the RL agent can follow to act in that environment. Note that the MDP doesn't define what the agent does in each state. That's why you need the policy! An optimal policy for a specific MDP corresponds to the strategy that, if followed, is guaranteed to give you the highest amount of reward in that environment. However, there are multiple strategies, most of them are not optimal. This should clarify why the policy is not part of the definition of the MDP.

| improve this answer | |
$\endgroup$
2
$\begingroup$

Aside from the points raised in nbro's answer, I'd like to point out that for a single MDP (a single instance of a "problem"), it may be sensible to study it from perspectives that include no policy at all, or multiple different policies.

For instance, if I have an MDP, I may be interested in studying it by looking at various inherent properties of the environment. And if I then have multiple different MDPs, all without any policies or anything like that, I could compare them based on those properties. For example, I might simply want to measure the sizes of the state and action spaces. Or write out something like a game tree, and measure properties like the branching factor and the average / min / max / median depth at which we can find a terminal state.

On the other hand, it can also be interesting sometimes to study multiple different policies all for the same MDP. A very common example would be any off-policy learning algorithm (like $Q$-learning): they all involve at least one "target policy" (for which they're learning the $Q(s, a)$ values -- usually the greedy policy with respect to the values learned so far), and at least one "behaviour policy" (which they're using to generate experience -- often something like an $\epsilon$-greedy policy). A more complex example would be population-based training setups, like the one DeepMind used for their StarCraft 2 training; here they have a large population of different policies that they're all using in a complex training setup (and technically I suppose we should say they also have many different MDPs, where every combination of StarCraft 2 level + training opponent would formally be a different MDP).

| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.