Understanding the configuration of replay memory and epsilon in deep reinforcement learning

Question

I am tentatively reusing a codebase of pacman to train my own deep reinforcement learning model. While most of the components seems reasonable and understandable to me, there are two things that seem obscure to me:

1. How to decide the size of the replay memory? Currently, since I set the total step of learning as 4000 (note that in the referred codebase this value is set as 4000000), so I just proportionally decrease the replay_memory_size as 400. Would that make sense?

2. What is the return value epsilon when calling function PiecewiseSchedule? I also proportionally decrease its parameters as follows:

        epsilon = PiecewiseSchedule([(0, 1.0),
                                     (40, 1.0), # since we start training at 10000 steps
                                     (80, 0.4),
                                     (200, 0.2),
                                     (400, 0.1),
                                     (2000, 0.05)], outside_value=0.01)
        replay_memory = PrioritizedReplayBuffer(replay_memory_size, replay_alpha)

where the original function call is like this:

        epsilon = PiecewiseSchedule([(0, 1.0),
                                     (10000, 1.0), # since we start training at 10000 steps
                                     (20000, 0.4),
                                     (50000, 0.2),
                                     (100000, 0.1),
                                     (500000, 0.05)], outside_value=0.01)
        replay_memory = PrioritizedReplayBuffer(replay_memory_size, replay_alpha)

And in general, what is the principle (guideline) behind setting a good size of "replay memory" and calling function PiecewiseSchedule? Thank you!

Answer 1

Replay memory is a recording of the games that the agents has played. It is this data that is used to train the neural network (or whatever machine learning method you are using). The training is performed by looking at the games played and the resulting reward signal (in Pacman, this might be your score), and then learning which game strategies worked well and which did not. It is difficult to say exactly how big the replay memory should be, this is something that you will need to experiment with. The larger it is, then the more examples will be available to be learned from during a particular training period, but of course it takes longer to acquire them.

Epsilon is a parameter that governs the trade-off between exploration and exploitation. By this, I mean that for each action that your agent takes, it needs to decide whether to do the best thing that it has already discovered to do (exploitation), or try something new (exploration). Usually, a value of epsilon near to 1 means that the agent will choose to explore more, and a value closer to 0 will mean that the agent will exploit more.

You can imagine therefore that at the beginning of an agent's training you might want epsilon ~1, because it has lots of new strategies to try to discover, but as it gets better you would reduce it to ~0 so that it can explore strategies that are only reachable after a long sequence of game steps. Naturally, when the agent is fully trained and ready to be deployed then epsilon would = 0. In the example that you posted, you can see that Pacman starts off doing nothing but exploring, but after 500000 games it exploits 95% of the time.