During the course of training a DQN agent, all visited states are stored in a replay buffer. Therefore would it be practically possible for a CNN, given a reasonable amount of data, to predict the next RL state (in the form of an image) for a given action?

For example, take the game of Atari as shown below - enter image description here Here the agent can take 2 major actions - go left and go right. Would a CNN be generate the image of the bar going right/left for the respective actions? My practical knowledge of CNNs is quite limited and therefore I'm trying to gauge the abilities of CNNs before I take up a project.


It sounds like what you're suggesting is similar to what is done in methods that use a planner. These methods looks to learn the dynamics of the MDP to use to plan during training; that is they want to be able to learn the transition probabilities $p(s'| s, a)$.

In this paper that I read recently they note that learning to predict environment dynamics when the state/action space is high dimensional, as is the case with images, is difficult; so whilst it may be possible in theory it would be difficult to do and if you were predicting many steps into the future then the error would compound.

A way around this, as is done in the referenced paper, is to use predict environment dynamics in a latent space. This means that they use a latent variable to predict the next state using e.g. Variational Autoencoders.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.