I am training an agent to play a simple game using double deep q learning. However, the variance in agent performance is very high, even for agents trained with same model parameters. For example, I can train agent A and agent B using the exact parameters and agent A beats B 800 to 200.
I think I understand why this is happening, when training starts the model is initialized with different weights, and this leads the model to find different local max/min.
The above makes it difficult to find optimal parameters.
What are the strategies to reduce this variance? What parameters should I look at tweaking?
More details about the environment:
This is a two player game (Zombie Dice); however, in my implementation so far the agents are learning to maximize expected score on their turn, so the actions and score of the opponent is ignored.
The variance is higher when I am using purely greedy strategy with no exploitation at all. Though it exists in both cases. I would say roughly 2/3 wins for stronger side with greedy and 3/5 with exploration out of 1000 matches.
The environment is stochastic; I have not done many assessment runs maybe 20 or 30, it is mostly eyeballing, but the differences are fairly large; therefore, I am confident that this is not due to chance.
I tested the models against themselves, and I get scores very close to 50/50. However, two different models trained with same parameters give results very different from 50/50. I tested this with models trained with different types of parameters and it is generally the same problem.