Can I add expert data to the replay buffer used by the DDPG algorithm in order to make it converge faster?

Question

I am working on a restricted reinforcement learning environment, i.e. the environment breaks very often (i.e.: the communication between the simulator and reinforcement learning agent breaks after some time). So, it is getting difficult for me to continue training in this environment.

The continuous state-space is $\mathcal{S} \subseteq \mathbb{R}^{10}$ and the continuous action-space $\mathcal{A} \subseteq \mathbb{R}^{2}$.

What I want to know is whether I can add expert data to the replay buffer, given that DDPG is an off-policy algorithm?

Or I should go with the behavior cloning technique to train the actor-network only, so that it converges rapidly?

I just want to get the work done first and then I can think of exploring the environment.

Answer 1

What I want to know is whether I can add expert data to the replay buffer, given that DDPG is an off-policy algorithm?

You certainly can, that is indeed one of the advantages of off-policy learning algorithms; they're still "correct", regardless of which policy generated the data that you're learning from (and a human expert providing the experience to learn from can also be viewed as such a "policy").

There are potential issues to be aware of though. For example, if you just put some expert-generated data in there and don't allow your agent to explore by itself, the experiences that you can learn from may be quite limited in the parts of the state-action space that they explore. So if your expert does not sufficiently explore the entire space, you cannot expect the agent to learn how to act if for whatever reason it ever ends up in some unexplored space. This is no different from what would happen if you trained with an agent that had too little exploration (like a greedy agent).

Or I should go with the behavior cloning technique to train the actor network only, so that it converges rapidly?

I cannot confidently say which approach would work better, so I cannot really answer this... I imagine the answer may also be different for specific different problem domains. But the basic principle of learning from expert data with an off-policy algorithm is not inherently wrong.