Are there RL algorithms that also try to predict the next state?

Question

So far I've developed simple RL algorithms, like Deep Q-Learning and Double Deep Q-Learning. Also, I read a bit about A3C and policy gradient but superficially.

If I remember correctly, all these algorithms focus on the value of the action and try to get the maximum one. Is there an RL algorithm that also tries to predict what the next state will be, given a possible action that the agent would take?

Then, in parallel to the constant training for getting the best reward, there will also be constant training to predict the next state as accurately as possible? And then have that prediction of the next state always be passed as an input into the NN that decides on the action to take. Seems like a useful piece of information.

Answer 1

Yes, there are algorithms that try to predict the next state. Usually this will be a model based algorithm -- this is where the agent tries to make use of a model of the environment to help it learn. I'm not sure on the best resource to learn about this but my go-to recommendation is always the Sutton and Barto book.

This paper introduces PlanGAN; the idea of this model is to use a GAN to generate a trajectory. This will include not only predicting the next state but all future states in a trajectory.

This paper introduces a novelty function to incentivise the agent to visit unexplored states. The idea is that for unexplored states, a model that predicts the next state from the state-action tuple will have high error (measured by Euclidean distance from true next state) and they add this error to the original reward to make a modified reward.

This paper introduces Dreamer. This is where all learning is done in a latent space and so the transition dynamics of this latent space must be learned, another example of needing to learn the next state.

These are just some examples of papers that try to predict the next state, there are many more out there that I would recommend you look for.

Answer 2

Check out Imagination-Augmented Agents paper - seems like it does what you are talking about. The agent itself is the standard A3C that you are familiar with. The novelty is the "imagination" environment model which is trained to predict the behavior of the environment.