I have an environment where an agent faces an equal opponent, and while I've achieved OK performance implementing DQN and treating the opponent as a part of the environment, I think performance would improve if the agent trains against itself iteratively. I've seen posts about it, but never detailed implementation notes. My thoughts were to implement the following (agent and opponent are separate networks for now):
- Bootstrap agent and opponent with initial weights (either random or trained against CPU, not sure)
- Use Annealing Epsilon Greedy strategy for N iterations
- After M iterations (M > N), copy agent network's weights to opponent's network
- Reset annealing epsilon (i.e. start performing randomly again to explore new opponent)?
- Repeat steps 2-4
Would something like this work? Some specific questions are:
- Should I "reset" my annealing epsilon strategy every time the opponent is updated? I feel like this is needed because the agent needs sufficient time to explore new strategies for this "new" opponent.
- Should the experience replay buffer be cleared out when the opponent is updated? Again, I think this is needed.
Any pointers would be appreciated.