I am implementing an actor-critic reinforcement learning algorithm for winning a two player tic-tac-toe like game. The agent is trained against a min-max player and after a number of episodes is able to learn a set of rules which lead it to winning a good majority of games.
However, as soon as I play against the trained agent by using even a slightly different playing style, it looses miserably. In other words, it is evident the agent overfitted with respect to the deterministic behaviour of the min-max player. It is clear to me what are the roots of the problem, but I would like to get an overview of the different methodologies which can be applied to overcome (or mitigate) this issue.
The two solutions I would like to try are the following:
1. Training the agent with different opponents for fixed amounts of episodes (or time) each. So that for example I train the agent by using a depth 2 min-max player for the first 10000 episodes, then I use a random playing agent for the next 10000 episodes, then I use a depth 4 min-max player for other 10000 episodes and repeat the process.
2. Starting episodes from different initial configurations. In this way the agents will play a much wider set of sampled games and will be more difficult for the agent to overfit.
Are these two reasonable approaches? Are there other tricks/good practices to try out?