# What is the difference between the state transition of an MDP and an action-value?

Let's say we have MDP where we have a state transition matrix.

How is this state transition different from action value in reinforcement learning? Is the state transition in MDP stochastic transition, meaning transition to some other state without taking any action?

Transition Probabilities: Consider that you are at state $$s$$ and from that state take an action $$a$$.Then there are some probability you will land up at state $$s_{1}'$$ or $$s_{2}'$$ ($$s'$$ indicate the next states). Those probabilities are called transition probabilities. In this example, the transition matrix is just a 3D array since it depends on your state and action($$p(s, a)$$).

Action value function $$Q_{\pi}(s, a)$$: It is the expected total reward you get from state $$s$$, taking action $$a$$ and thereafter following the policy, $$\pi$$.

Is the state transition in MDP stochastic transition, meaning transition to some other state without taking any action?

The environment can be stochastic or deterministic. If the environment is stochastic then those transitions are stochastic. If the environment is deterministic then those transitions are deterministic.

• Yes I am aware of transition probabilities, but what I am asking is how does that transition occur? Is the transition occur by taking "action" or is a stochastic phenomena? In other words how is the transition matrix populated? In other words the transition probability matrix arises if we record action over large period of time or its just that if you in state 1 there is simply stocastic chance that you will end up in state say 7. How do you end up in state 7 from state 1? by taking an action? Apr 28 '20 at 16:35
• The transition occurs by taking action. For example, if you take action 'left' then there is some chance you land up in the left state and there is also some probability that you will land up in other states than the left state. This thing will happen if the environment is stochastic. In reinforcement learning, if you use dynamic programming, then you have to know the dynamics of the system(means the transition matrix and rewards). Or another thing you can do is go to the environment collect a lot of samples and calculate transition matrix from your collected samples.. Apr 28 '20 at 17:01
• ok so in MDF we build the transition matrix by taking actions and recording their frequency? Apr 28 '20 at 17:19
• yes, we can find the transition matrix by using those samples(this is called model-based reinforcement learning). Apr 28 '20 at 17:22
• ok makes sense. Apr 28 '20 at 17:49