0
$\begingroup$

Let's say we have MDP where we have a state transition matrix.

How is this state transition different from action value in reinforcement learning? Is the state transition in MDP stochastic transition, meaning transition to some other state without taking any action?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
0
$\begingroup$

Transition Probabilities: Consider that you are at state $s$ and from that state take an action $a$.Then there are some probability you will land up at state $s_{1}'$ or $s_{2}'$ ($s'$ indicate the next states). Those probabilities are called transition probabilities. In this example, the transition matrix is just a 3D array since it depends on your state and action($p(s, a)$).

Action value function $Q_{\pi}(s, a)$: It is the expected total reward you get from state $s$, taking action $a$ and thereafter following the policy, $\pi$.

Is the state transition in MDP stochastic transition, meaning transition to some other state without taking any action?

The environment can be stochastic or deterministic. If the environment is stochastic then those transitions are stochastic. If the environment is deterministic then those transitions are deterministic.

Improve this answer
$\endgroup$
5
  • $\begingroup$ Yes I am aware of transition probabilities, but what I am asking is how does that transition occur? Is the transition occur by taking "action" or is a stochastic phenomena? In other words how is the transition matrix populated? In other words the transition probability matrix arises if we record action over large period of time or its just that if you in state 1 there is simply stocastic chance that you will end up in state say 7. How do you end up in state 7 from state 1? by taking an action? $\endgroup$
    – gfdsal
    Commented Apr 28, 2020 at 16:35
  • $\begingroup$ The transition occurs by taking action. For example, if you take action 'left' then there is some chance you land up in the left state and there is also some probability that you will land up in other states than the left state. This thing will happen if the environment is stochastic. In reinforcement learning, if you use dynamic programming, then you have to know the dynamics of the system(means the transition matrix and rewards). Or another thing you can do is go to the environment collect a lot of samples and calculate transition matrix from your collected samples.. $\endgroup$
    – Swakshar Deb
    Commented Apr 28, 2020 at 17:01
  • $\begingroup$ ok so in MDF we build the transition matrix by taking actions and recording their frequency? $\endgroup$
    – gfdsal
    Commented Apr 28, 2020 at 17:19
  • $\begingroup$ yes, we can find the transition matrix by using those samples(this is called model-based reinforcement learning). $\endgroup$
    – Swakshar Deb
    Commented Apr 28, 2020 at 17:22
  • $\begingroup$ ok makes sense. $\endgroup$
    – gfdsal
    Commented Apr 28, 2020 at 17:49

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .