Is there a standardized method to train a reinforcement learning NN by demonstration?

Question

I'm less familiar with reinforcement learning compared to other neural network learning approaches, so I'm unaware of anything exactly like what I want for an approach. I'm wondering if there are any ways to train a Deep-Q neural network on, say, OpenAI Gym, where the model is given a recorded demonstration to learn from. That is, I'd like to do the following (with a Mario NES example):

• Play the first level of Mario and somehow record this (maybe as an input sequence?)
• Run the model on the level a few times until it passes
• Send the model to the next level, let it train for as long as possible unless it continues to fail -- if so, I'll play the level and then let it train again
• Repeat

Are there any approaches similar to this that currently exist? And, would this be infeasible because the model may fail to generalize? I'd like to accomplish this with much more complicated games, but if I could use this approach to save on defining endless amounts of reward/penalty policies and ROM hacking to find the memory address of everything I want the model to use, it would be extremely helpful.

Answer 1

Yes, this is known as imitation learning, which can be divided into

• inverse RL (i.e. learn a reward function from demonstrations, then apply RL), and
• behaviour cloning (supervised learning applied to RL).

I don't know the state-of-the-art (I am not an expert in this topic) or whether IL is a good approach in your case, but you can check e.g. the pre-print survey Imitation Learning: Progress, Taxonomies and Opportunities (2021) by Boyuan Zheng et al., which seems to nicely describe the topic.