# Isn't a simulation a great model for model-based reinforcement learning?

Most reinforcement learning agents are trained in simulated environments. The goal is to maximize performance in (often) the same environment, preferably with a minimum amount of interactions. Having a good model of the environment allows to use planning and thus drastically improves the sample efficiency!

Why is the simulation not used for planning in these cases? It is a sampling model of the environment, right? Can't we try multiple actions at each or some states, follow the current policy to look several steps ahead and finally choose the action with the best outcome? Shouldn't this allow us to find better actions more quickly compared to policy gradient updates?

In this case, our environment and the model are kind of identical and this seems to be the problem. Or is the good old curse of dimensionality to blame again? Please help me figure out, what I'm missing.

• Hi and welcome to AI SE! What makes you think that simulated environments are or can not be used for planning? – nbro Apr 9 '20 at 13:06
• Hi @nbro! Thank you for the warm comment and the helpful question, which I'm happy to answer: afaiu, the most popular deep RL algorithms are model-free. These are sample inefficient as they learn from pure interactions with the environment. An agent with a good model of the environment could use the model to plan ahead. This way the agent would get experience without interacting with the environment and thus increase the sample efficiency. When this is true, why are popular algorithms like PPO still very sample inefficient? I think I'm missing something. Please help : ) – Ray Walker Apr 9 '20 at 16:56

It depends on the nature of the simulation. If the simulation models a car as a solid body moving with three $$(x,y,\theta)$$ degrees of freedom in a plane (hopefully, if it doesn't hit anything and propel vertically), the three ordinary differential equations of solid body motion can be solved quite quickly, compared to a simulation used to model the path of least resistance of a ship on wavy sea, where fluid dynamics equations must be solved, that require a huge amount of resources. OK, the response time needed for a ship is much longer, than for a car, yes, but to compute it predictively, one needs a huge amount of computational power.