# Model-based learning in continuous state and action spaces

I am interested in learning how transition probabilities/mdps are constructed in continuous state and action space model-based learning setting. There is some literature available on this matter, but they do not explicitly construct the model to simulate the environment, through policy gradient.

The closest literature that builds a model that I found is on continuous state space and finite action space, which is slightly different than what I aim to do. Furthermore even in this case the main problem that I face is, transition probability is assumed to give you the probability of obtaning next state, which does not necessarily have to be non-zero for instance when the transition probability is non-atomic transition probability.

I will appreciate if someone atleast points me relevant literature in this direction.

• I don't see any question here. Can you please ask a specific question?
– nbro
Jul 25 at 9:24
• but they do not explicitly construct the model to simulate the environment probably because it's not that trivial... you can approximate it with variational models, but still you will have a very hard time building a model that works in the continuous space (model in the RL setting, thus a env simulator) Jul 25 at 11:22
• but maybe something like this might give a glance arxiv.org/abs/1509.02971 (however, not theoretical definition of a continuous MDP is given) Jul 25 at 11:24
• Please clarify your specific problem or provide additional details to highlight exactly what you need. As it's currently written, it's hard to tell exactly what you're asking.
– Community Bot
Jul 25 at 15:22

You can use function approximation like neural networks to learn the whole environment, i.e. both the transition function, $$p(s'\mid s, a)$$, and the reward model, $$r(s,a,s')$$: $$p(s',r\mid s,a)$$ In principle, you can learn it just by plain supervised learning in which given a state-action pair $$(s,a)$$ the prediction targets are the next state, $$s'$$, and reward, $$r$$. Once you learn a suitable model you can use it to replace your actual environment, or even to plan (i.e., query the learned simulator instead of the true env to see what would happen next.), and then use the usual model-free RL to learn a policy.
The thing you want to keep in mind is that the learned simulator is always flawed, meaning that at each timestep it will make a (small) error that cumulate with time. So there are other approaches in which the simulator is used for $$K$$ steps at most, or that consider the uncertainty in the predictions such that to account for the mis-modelling of the learned model.