I have skimmed through a bunch of deep learning books, but I have not yet understood whether we must use the experience replay buffer with the A3C algorithm.
The approached I used is the following:
- I have some threaded agents that play their own copy of an enviroment; they all use a shared NN to predict 'best' move given a current state;
- At the end of each episode, they push the episode (in the form of a list of steps) in a shared memory
- A separated thread reads from the shared memory and executes the train step on the shared NN, training it episode after episode.
Is this an appropriate approach? More specifically, do I need to sample data to train the NN from the shared memory? Or should I push in the shared memory each step, just when it's done by a single agent? In this last case, I wonder how I could calculate discounted rewards.
I'm afraid that with my current approach I'm doing nothing more that presenting n episodes to the NN, with the hope that each agent explores the enviroment differently from other agents (so that NN is presented a richer variety of states).