4
$\begingroup$

I've read that for MDPs the state transition function $P_a(s, s')$ is a probability. This seems strange to me for modeling because most environments (like video games) are deterministic.

Now, I'd like to assert that most systems we work with are deterministic given enough information in the state (i.e. in a video game, if you had the random number seed, you could predict 'rolls', and then everything else follows game logic).

So, my guess for why would MDP state transitions are probabilities is because the state given to the MDP is typically a subset (i.e. from feature engineering) of total information available. That, and of course to model non-deterministic systems.

Is my understanding correct?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
2
$\begingroup$

Your understanding is right!

Using a probabilistic transition function allows the model to explore a bigger search space before making a decision. One of the most important use cases of MDP is in POS tagging in NLP using a Hidden Markov Model.

In case of a deterministic model, search space is limited by the number of transitions and hence at each step, a definite decision is made. This does not take into regard the possibility of relationship between the previous states, rather only deals with the current and next state. These model are good for solving a certain range of tasks like decision trees etc.

When it come to tasks such as weather prediction, there is significance to the historical weather data. In such cases, we cannot make use of a deterministic approach. You always predict the chance of rainfall etc.

Weather prediction

This example can also be extended to predict the weather for future days

Improve this answer
$\endgroup$
2
  • $\begingroup$ To clarify your answer, I believe you have confirmed my understanding and then contributed two more things. First you pointed out that we want use the MDP to explore different options and solutions, so the probabilistic model enables this. Secondly you gave an example about weather, in which the fact that we don't store historical weather data is why our state is insufficient to allow our MDP state transition to be deterministic. Is this correct? $\endgroup$
    – personjerry
    Jun 17, 2019 at 17:58
  • $\begingroup$ @personjerry the second part of your question that "we dont store historical weather data is why our state is insufficient to allow our MDP state transition to be deterministic", this is not correct. Even if you have recorded all the historical data, your weather prediction will still be indeterministic hence we make use of probability/chance. Your initial understanding that the world is deterministic is wrong, we do have purely stochastic systems $\endgroup$
    – user0193
    May 20, 2022 at 11:32

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .