I am trying to implement a DQN agent for playing standard 3x3 Tic-Tac-Toe (it is a double DQN with experience replay, and using a target network). I got the hyperparameters to the point where the agent tends to reach a 92-93% win percentage against a random opponent, which is not quite optimal (a tabular Q-learner wins 99.5% and ties the remaining 0.5%, as does minimax). Soon after reaching the peak, it tends to crash down to abysmal performance, and never quite recovers. I am just wondering how typical these results are. Should I expect that it is possible to obtain an agent that plays optimally using DQN, or is the algorithm inherently so noisy that it can only be expected to get good but never quite perfect?
Thank you :)