3
$\begingroup$

If a policy is fixed, it is said that an MDP becomes an MRP.

Why is this so? Aren't the transitions and rewards still parameterized by the action and current state? In other words, aren't the transition and reward matrices still cubes?

From my current train of thought, the only thing that is different is that the policy is not changing (the agent is not learning the policy). Everything else is the same.

How is it that it switches to an MRP which is not affected by actions?

I am reading "Deep Reinforcement Learning Hands-On" by Maxim Lapan which states this. I have also found this statement in online articles but I cannot seem to wrap my head around it. I am brand new to RL so please bear with me :).

$\endgroup$
1
$\begingroup$

If a policy is fixed, it is said that an MDP becomes an MRP.

I would change the phrasing slightly here, to:

If a policy is fixed, an MDP can be accurately modeled as an MRP.

Why is this so? Aren't the transitions and rewards still parameterized by the action and current state? In other words, aren't the transition and reward matrices still cubes?

The transition and reward matrices remain the same in the MDP, but it is possible to flatten them into an equivalent MRP, because in terms of observations of next state and reward, the action that is taken is just part of the transition rules - if the policy is fixed, then so are all the probabilities for next state and reward.

More concretely, if you have an MDP with $|\mathcal{A}|$ transition matrices $P_{ss'}^a$ and a fixed policy $\pi(a|s)$, then you can create a combined transition matrix with a sum:

$$P_{ss'} = \sum_{a \in \mathcal{A}} \pi(a|s) P_{ss'}^a$$

and you can similarly reduce the reward function. Once you have done so you have data that describe an MRP.

How is it that it switches to an MRP which is not affected by actions?

If the MDP represents a real system where actions are still being taken by an agent, then of course those are still present within the system, and still affect it. The difference is that if you know the agent's policy, then the action choice is predictable, and the MRP representation covers the full definition of probabilities of observed state transitions and rewards.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.