Due to my RL algorithm having difficulties learning some control actions, I've decided to use imitation learning/apprenticeship learning to guide my RL to perform the optimal actions. I've read a few articles on the subject and just want to confirm how to implement it.
Do I simply sample a state $s$, then perform the optimal action $a^*$ in that state $s$, calculate the reward for the action $r$, and then observe the next state $s'$, and finally put that into the experience replay?
If this is the case, I am thinking of implementing it as follows:
- Initialize the optimal replay buffer $D_O$
- Add the optimal tuple of experience $(s, a^*, r, s')$ into the replay buffer $D_O$
- Initialize the normal replay buffer $D_N$
- During the simulation, initially sample $(s, a^*, r, s')$ only from the optimal replay buffer $D_O$, while populating the normal replay buffer $D_N$ with the simulation results.
- As training/learning proceeds, anneal out the use of the optimal replay buffer, and sample only from the normal replay buffer.
Would such an architecture work?