4
$\begingroup$

due to my RL having difficulties learning some control actions, I've decided to use Imitation learning / apprenticeship learning to guide my RL to perform the optimal actions. I've read a few articles on the subject and just want to confirm how to implement it.

Do I simply just simply sample states, then perform the optimal action in that state, calculate the reward for the action and then observe the s', and then put that into the experience replay?

Ex: Observe states, I perform the optimal action, calculate reward for my action, observe s'. Feed [s, a*, r, s'] into the replay buffer?

If this is the case, I am thinking of implementing it as follows:

1) Initialize optimal replay buffer

2) Introduce optimal [s, a*, r, s'] into buffer

3) Initialize normal replay buffer

4) During simulation, initially sample s, a*, r, s' only from the optimal replay buffer. While populating the normal replay buffer with the simulation results.

5) As episodes -> infinite, anneal out the use of the optimal replay buffer, and sample only from the normal replay buffer.

Would such an architecture work?

$\endgroup$
2
$\begingroup$

That seems to be functional.

That is a great approach, as long as you are using an off-policy algorithm (since the samples you are using to learn are not the policy currently being performed), like Q-learning.

By annealing the sample rate from the optimal buffer to the regular one, you introduce noise into the network and emphasize exploration (albeit more limited). This is helpful when you (the researcher) have no access to optimal policies, but merely "good" ones, and you still want the network to try and improve on those.

$\endgroup$
  • $\begingroup$ Thanks for the reply! Have you tried implementing this before? $\endgroup$ – Rui Nian Sep 4 '18 at 16:02
  • $\begingroup$ I have, with both DQN and DDPG. Worked nicely with DQN, not that well with DDPG (although I don't think it was about the transfer learning, the problem was just too complex). $\endgroup$ – BlueMoon93 Sep 4 '18 at 16:43
  • $\begingroup$ Ok thanks, the main problem is that I have a integral state in my system. Hopefully this will solve it. $\endgroup$ – Rui Nian Sep 4 '18 at 18:52
  • $\begingroup$ Ok, good luck. If this answer has helped you, don't forget to upvote or mark it as the accepted answer ^^ $\endgroup$ – BlueMoon93 Sep 4 '18 at 21:10
  • $\begingroup$ Sure thing, I'll upvote it. Once I implement it and if it works, I'll definitely accept it. $\endgroup$ – Rui Nian Sep 4 '18 at 23:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.