What are the state-of-the-art meta-reinforcement learning methods?

Question

This question can seem a little bit too broad, but I am wondering what are the current state-of-the-art works on meta reinforcement learning. Can you provide me with the current state-of-the-art in this field?

Answer 1

One of the most recent papers on meta-RL is meta-Q-learning. This paper introduces Meta-Q-Learning (MQL), a new off-policy algorithm for meta-reinforcement learning (meta-RL). MQL builds upon three simple ideas.

• Q-learning is competitive with state-of-the-art meta-RL algorithms if given access to a context variable that is a representation of the past trajectory.

• Using a multi-task objective to maximize the average reward across the training tasks is an effective method to meta-train RL policies.

• past data from the meta-training replay buffer can be recycled to adapt the policy on a new task using off-policy updates

Experiments on standard continuous-control benchmarks suggest that MQL compares favorably with state-of-the-art meta-RL algorithms.

I think that other references to other work on meta-RL are present in the experiments part of the MQL paper.

Answer 2

Meta-Reinforcement Learning can refer to a broad range of ideas. Also, different algorithms are SOTA under different evaluation metrics (sample efficiency, agent performance, adaptation speed on a new task, etc)

Assuming that you are referring to the problem of quickly learning/adapting to a new task by training an agent on a distribution of related tasks, the following are some popular algorithms

• PEARL [Rakelly et al., 2019]
• VariBAD [Zintgraf et al., 2020]
• Meta-Q-Learning [Fakoor et al., 2020]

References:

1. K Rakelly, A Xhou, D Quillen, C Finn, S Levine - Efficient Off-Policy Meta-Reinforcement Learning via Probabilistic Context Variables, ICML 2019.
2. L Zintgraf et al., - VariBAD: A Very Good Method for Bayes-Adaptive Deep RL via Meta-Learning, ICLR 2020.
3. R Fakoor, P Chaudhari, S Soatto, A J Smola - Meta-Q-Learning, ICLR 2020.