If the accumulated reward increases, the loss increases and vice versa. This is a strange behaviour. See the figure below for an example. What is the possibility of having this behaviour in DQN?

enter image description here

I have $3000$ episodes. An episode lasts $20$ time steps. So, in total, I have $60000$ time steps. I am using the ADAM optimizer with a learning rate of $5\times10^{-6}$. The target update frequency is once every $2000$ time steps (once every $100$ episodes). The training happens every episode (once every $20$ time steps). I am using double DQN with the duelling architecture and the prioritized experience replay buffer. The size of the buffer is $300$ and the batch size is $128$.

I tried to modify some hyperparameters but for now I still have this strange behaviour.

  • $\begingroup$ What is your loss function? Also, your replay buffer is way too small. Can you train it with a rp buffer size of ~10,000 and check if the error persists? $\endgroup$ – Leon Shams Sep 14 at 8:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.