1
$\begingroup$

I am looking at a lecture on POMDP, and the context is that, when the quadcopter can't see the landmarks, it has to use reckoning. And then he mentions the transition model is not deterministic, hence the uncertainty grows.

Can transition models in MDP be deterministic?

Improve this question
$\endgroup$

2 Answers 2

Reset to default
0
$\begingroup$

Yes, in which case, it will be more like a search problem, if it is not POMDP and have finite number of states. Or you can use the same framework (used for POMDP) with constrained (deterministic) transition matrix modeling for model-based systems.

If you think about it after you train any model/agent with an MDP modeling, during test time the optimal strategy is generally deterministic, i.e., given a feature/state you will take a particular action even if it has more than one non-zero element in each row of the transition matrix.

Improve this answer
$\endgroup$
6
  • $\begingroup$ Alright, when you say deterministic strategy you mean we select argmax or the maximum result? Can you please elaborate constrained (deterministic) transition matrix $\endgroup$
    – gfdsal
    Jun 26, 2020 at 14:36
  • $\begingroup$ 1. Yes, I was referring to the best strategy obtained with argmax. $\endgroup$
    – saiRegrefree
    Jun 28, 2020 at 0:37
  • $\begingroup$ 2. When you have a deterministic transition, the actions are irrelevant, since choosing an action is equivalent to reaching a state, so you can in simple cases use dynamic programing/Beam search to solve the problem. In complex cases, the reward can be infinite and you can use value iteration to solve the problem. I found some paper on the topic which might help you related work "Max-Plus Matching Pursuit for Deterministic Markov Decision Processes" arxiv.org/pdf/1906.08524.pdf $\endgroup$
    – saiRegrefree
    Jun 28, 2020 at 0:44
  • $\begingroup$ allright makes sense! I accepted your answer as now it is clear. I have a small followup question. How can a transition be deterministic if the environment is stochastic? So say if I move left there is 30% chance that i will transition to front instead of left. So when you say actions are irrelevant, you mean to say we dont care about the 30% chance of transitioning to front if we intend to move left? $\endgroup$
    – gfdsal
    Jun 28, 2020 at 12:19
  • 1
    $\begingroup$ I think there is a misunderstanding. Transition matrix being deterministic means when you take an action you are certain that you will end up in only one state, which means the environment is deterministic. In this case (deterministic environment/transition matrix) actions are irrelevant. "So say if I move left there is 30% chance that i will transition to front instead of left." - this means the environment/transition matrix is stochastic. $\endgroup$
    – saiRegrefree
    Jul 1, 2020 at 0:25
0
$\begingroup$

Yes , a transition model shows that our environment is stochastic in nature, and with that model we know the probability of entering a state when an action is taken.

Improve this answer
$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .