due to my RL having difficulties learning some control actions, I've decided to use Imitation learning / apprenticeship learning to guide my RL to perform the optimal actions. I've read a few articles on the subject and just want to confirm how to implement it.
Do I simply just simply sample states, then perform the optimal action in that state, calculate the reward for the action and then observe the s', and then put that into the experience replay?
Ex: Observe states, I perform the optimal action, calculate reward for my action, observe s'. Feed [s, a*, r, s'] into the replay buffer?
If this is the case, I am thinking of implementing it as follows:
1) Initialize optimal replay buffer
2) Introduce optimal [s, a*, r, s'] into buffer
3) Initialize normal replay buffer
4) During simulation, initially sample s, a*, r, s' only from the optimal replay buffer. While populating the normal replay buffer with the simulation results.
5) As episodes -> infinite, anneal out the use of the optimal replay buffer, and sample only from the normal replay buffer.
Would such an architecture work?