I have skimmed through a bunch of deep learning books, but I have not yet understood whether we must use the experience replay buffer with the A3C algorithm.

The approached I used is the following:

  • I have some threaded agents that play their own copy of an enviroment; they all use a shared NN to predict 'best' move given a current state;
  • At the end of each episode, they push the episode (in the form of a list of steps) in a shared memory
  • A separated thread reads from the shared memory and executes the train step on the shared NN, training it episode after episode.

Is this an appropriate approach? More specifically, do I need to sample data to train the NN from the shared memory? Or should I push in the shared memory each step, just when it's done by a single agent? In this last case, I wonder how I could calculate discounted rewards.

I'm afraid that with my current approach I'm doing nothing more that presenting n episodes to the NN, with the hope that each agent explores the enviroment differently from other agents (so that NN is presented a richer variety of states).

  • $\begingroup$ A3C is an actor critic reinforcement learning algorithm with asynchronous updates, hence it is not really relevant for a DL book to cover it. You'll want to check the original paper for details (arxiv.org/abs/1602.01783). There are many implementations which cover the details of the algo, random eg: medium.com/emergent-future/…. $\endgroup$
    – vega
    May 2 '19 at 11:09
  • $\begingroup$ Being on-policy AC, it's not using a replay buffer like off-policy methods such as DQN. However, It does collect data in order to compute discounted rewards. After you update your network, you should not reuse old experiences (that would be off-policy). $\endgroup$
    – vega
    May 2 '19 at 11:09
  • 1
    $\begingroup$ Thanks vega for your quick answer. I'll try to follow the tutorial you suggested as well as the original paper. $\endgroup$
    – Scorpio76
    May 2 '19 at 11:37
  • $\begingroup$ It's good to try to implement the algo, good learning experience. But be warned, it is notoriously difficult. Best to use a known stable version, like ikostrikov, or try a better algo like ppo. Good luck! $\endgroup$
    – vega
    May 2 '19 at 13:26
  • $\begingroup$ Well, from the few experiments done so far, I noticed that RL is really difficult. Let me ask another question: if I understood it right, the example you linked proposes a scenario where each agent has its own copy of the model. In examples I've read until now, there is only a single NN copy, and each agent is, more or less, an "example producer" (i.e provides a list of steps). What am I missing? $\endgroup$
    – Scorpio76
    May 2 '19 at 15:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.