1
$\begingroup$

I am interested in learning how transition probabilities/mdps are constructed in continuous state and action space model-based learning setting. There is some literature available on this matter, but they do not explicitly construct the model to simulate the environment, through policy gradient.

The closest literature that builds a model that I found is on continuous state space and finite action space, which is slightly different than what I aim to do. Furthermore even in this case the main problem that I face is, transition probability is assumed to give you the probability of obtaning next state, which does not necessarily have to be non-zero for instance when the transition probability is non-atomic transition probability.

I will appreciate if someone atleast points me relevant literature in this direction.

Improve this question
$\endgroup$
4
  • $\begingroup$ I don't see any question here. Can you please ask a specific question? $\endgroup$
    – nbro
    Jul 25 at 9:24
  • $\begingroup$ but they do not explicitly construct the model to simulate the environment probably because it's not that trivial... you can approximate it with variational models, but still you will have a very hard time building a model that works in the continuous space (model in the RL setting, thus a env simulator) $\endgroup$
    – Alberto
    Jul 25 at 11:22
  • $\begingroup$ but maybe something like this might give a glance arxiv.org/abs/1509.02971 (however, not theoretical definition of a continuous MDP is given) $\endgroup$
    – Alberto
    Jul 25 at 11:24
  • $\begingroup$ Please clarify your specific problem or provide additional details to highlight exactly what you need. As it's currently written, it's hard to tell exactly what you're asking. $\endgroup$
    – Community Bot
    Jul 25 at 15:22

1 Answer 1

Reset to default
2
$\begingroup$

You can use function approximation like neural networks to learn the whole environment, i.e. both the transition function, $p(s'\mid s, a)$, and the reward model, $r(s,a,s')$: $$p(s',r\mid s,a)$$ In principle, you can learn it just by plain supervised learning in which given a state-action pair $(s,a)$ the prediction targets are the next state, $s'$, and reward, $r$. Once you learn a suitable model you can use it to replace your actual environment, or even to plan (i.e., query the learned simulator instead of the true env to see what would happen next.), and then use the usual model-free RL to learn a policy.

In your case I think that the most effective model-based algorithms are the ones based on either world models or dreamer. These approaches learn a generative model (e.g., a VAE) to mimic the response of the true env, and after, or at same time, also learn a control policy.

The thing you want to keep in mind is that the learned simulator is always flawed, meaning that at each timestep it will make a (small) error that cumulate with time. So there are other approaches in which the simulator is used for $K$ steps at most, or that consider the uncertainty in the predictions such that to account for the mis-modelling of the learned model.

Improve this answer
$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .