Is my understanding correct regarding the difference between policy and plan?

Question

I am confused regarding the difference between policy and plan in reinforcement learning. According to my understanding, when we calculate the value of state using Bellman equation in deterministic environment :

The plan in this case will be the strict state action pair, that is gathered using finding the max. value for each action in every state and it will be something like the below image in a maze game as an example :

However, in a stochastic environment the Bellman equation will be:

And then we will have something like this:

In this case, to develop the policy we will need to know the state-action pair for every state like the above image plus mentioning the probability distribution for all the actions at every state and we need to keep in mind that the actions in the above image will not always happen due to the stochastic nature of the environment.

Is my understanding correct regarding the difference between policy and plan?

Answer 1

TL;DR

I don't think your understanding of the notion of plan and policy is correct. These notions are independent of the nature of the environment but you seem to think that you can only find plans in deterministic environments and you need a policy for stochastic ones.

Long answer

Your post is very confusing because apparently you're very confused and mixing a lot of concepts. I'll provide clarifications for everything that I think is confusing.

1. The first equation is just a special case of the second, where $P(a, s, s') = 1$ for some $s'$ and $P(a, s, s') = 0$ for all others. So, in reality, we can just use the second (more general) equation and forget about the first. However, it's true: we can use the first equation if you know the environment is deterministic.

2. A plan is a sequence of actions. A policy $\pi$ is a function that maps states to actions (or probability distributions over actions). So, given a sequence of states $s_1, \dots, s_N$, a deterministic policy will give you a sequence of actions $\pi(s_1), \dots, \pi(s_N) = a_1, \dots, a_N$, which can be thought of as a plan. If $\pi$ is stochastic, you need to sample actions, so the same policy can give you multiple different plans. For more info about the notion of a plan in RL, see this answer.

3. If you have a plan or not does not depend on whether your environment is stochastic or not. You can find plans in stochastic and deterministic environments. See this answer for more info about the notion of planning and how it's related to search.

4. There are stochastic and deterministic policies. There are stochastic and deterministic environments. These are difference concepts. For example, there are stochastic environments where the optimal policy is actually deterministic and there are stochastic environment where it's not.

5. There's no difference between your first and second diagrams, with respect to the definition of a plan or policy. They both look like plans, or sequences of actions generated by a policy.

6. To plan in an MDP, you usually need $P$ (the transition function), even if it's deterministic. You can use a dynamic programming algorithm like policy iteration to find a plan. However, you could also use a reinforcement learning algorithm like Q-learning to find a policy, which will give you a plan, as defined above.

7. So, how come that Q-learning is usually not considered a planning algorithm? I think this is more a convention. We could consider all algorithms that find sequence of actions planning algorithms. However, if an algorithm needs to explore the environment by taking random actions, we don't consider it a planning algorithm. So, planning involves also some notion of thinking ahead before executing given a model, but Q-learning does not really have a model and does not really think ahead.

8. You're wrong that to find a policy you need to know $P$ (the transition function). Q-learning finds policies but does not know or use $P$.