Due to my RL algorithm having difficulties learning some control actions, I've decided to use imitation learning/apprenticeship learning to guide my RL to perform the optimal actions. I've read a few articles on the subject and just want to confirm how to implement it.

Do I simply sample a state $s$, then perform the optimal action $a^*$ in that state $s$, calculate the reward for the action $r$, and then observe the next state $s'$, and finally put that into the experience replay?

If this is the case, I am thinking of implementing it as follows:

  1. Initialize the optimal replay buffer $D_O$
  2. Add the optimal tuple of experience $(s, a^*, r, s')$ into the replay buffer $D_O$
  3. Initialize the normal replay buffer $D_N$
  4. During the simulation, initially sample $(s, a^*, r, s')$ only from the optimal replay buffer $D_O$, while populating the normal replay buffer $D_N$ with the simulation results.
  5. As training/learning proceeds, anneal out the use of the optimal replay buffer, and sample only from the normal replay buffer.

Would such an architecture work?

  • $\begingroup$ For completeness, you may want to edit your post to include the name of the RL algorithm you were using at the time (if you still remember). $\endgroup$
    – nbro
    Nov 5, 2020 at 23:56

1 Answer 1


That seems to be functional.

That is a great approach, as long as you are using an off-policy algorithm (since the samples you are using to learn are not the policy currently being performed), like Q-learning.

By annealing the sample rate from the optimal buffer to the regular one, you introduce noise into the network and emphasize exploration (albeit more limited). This is helpful when you (the researcher) have no access to optimal policies, but merely "good" ones, and you still want the network to try and improve on those.

  • 1
    $\begingroup$ Have you tried implementing this before? $\endgroup$
    – Rui Nian
    Sep 4, 2018 at 16:02
  • $\begingroup$ I have, with both DQN and DDPG. Worked nicely with DQN, not that well with DDPG (although I don't think it was about the transfer learning, the problem was just too complex). $\endgroup$
    – BlueMoon93
    Sep 4, 2018 at 16:43

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .