# In imitation learning, do you simply inject optimal tuples of experience $(s, a, r, s')$ into your experience replay buffer?

Due to my RL algorithm having difficulties learning some control actions, I've decided to use imitation learning/apprenticeship learning to guide my RL to perform the optimal actions. I've read a few articles on the subject and just want to confirm how to implement it.

Do I simply sample a state $$s$$, then perform the optimal action $$a^*$$ in that state $$s$$, calculate the reward for the action $$r$$, and then observe the next state $$s'$$, and finally put that into the experience replay?

If this is the case, I am thinking of implementing it as follows:

1. Initialize the optimal replay buffer $$D_O$$
2. Add the optimal tuple of experience $$(s, a^*, r, s')$$ into the replay buffer $$D_O$$
3. Initialize the normal replay buffer $$D_N$$
4. During the simulation, initially sample $$(s, a^*, r, s')$$ only from the optimal replay buffer $$D_O$$, while populating the normal replay buffer $$D_N$$ with the simulation results.
5. As training/learning proceeds, anneal out the use of the optimal replay buffer, and sample only from the normal replay buffer.

Would such an architecture work?

• For completeness, you may want to edit your post to include the name of the RL algorithm you were using at the time (if you still remember).
– nbro
Nov 5 '20 at 23:56