I'm trying to use DQN to solve the cart-pole environment. I have 2 networks (target and behavior). Both of them have 3 hidden layers with 24 neurons, using the ReLU activation. The loss is MSE and the optimizer is Adam. I copy the weights of the behavior network to the target network every 15 backpropagation steps.
My agent learns. Below you can see the total reward and running average plots.
However, it has a lot of "drops". Usually, after a couple of perfect sequences, it just "kills" the running average with a couple of very short episodes. What may be the reason for this behavior?