I am tentatively reusing a codebase of pacman to train my own deep reinforcement learning model. While most of the components seems reasonable and understandable to me, there are two things that seem obscure to me:
How to decide the size of the replay memory? Currently, since I set the total step of learning as 4000 (note that in the referred codebase this value is set as 4000000), so I just proportionally decrease the
replay_memory_sizeas 400. Would that make sense?
What is the return value
epsilonwhen calling function
PiecewiseSchedule? I also proportionally decrease its parameters as follows:
epsilon = PiecewiseSchedule([(0, 1.0), (40, 1.0), # since we start training at 10000 steps (80, 0.4), (200, 0.2), (400, 0.1), (2000, 0.05)], outside_value=0.01) replay_memory = PrioritizedReplayBuffer(replay_memory_size, replay_alpha)
where the original function call is like this:
epsilon = PiecewiseSchedule([(0, 1.0), (10000, 1.0), # since we start training at 10000 steps (20000, 0.4), (50000, 0.2), (100000, 0.1), (500000, 0.05)], outside_value=0.01) replay_memory = PrioritizedReplayBuffer(replay_memory_size, replay_alpha)
And in general, what is the principle (guideline) behind setting a good size of "replay memory" and calling function
PiecewiseSchedule? Thank you!