I'm using Q-learning with some extensions such as noisy linear layers, n-steps and double DQN.

The training, however, isn't that successful, my rewards are descreasing over time after a steep increase at the beginning:

enter image description here

But what's interesting is that my td loss is also descreasing:

enter image description here

The sigma magnitudes of the noisy linear layers which control the exploration are strangely increasing, and also, seems to converge. I expected it to reduce uncertainty over time, but the opposite is the case.

enter image description here

Another intresting thing, and that's probably why my loss is decreasing: The model tends to generate always the same transition, which is why the episodes are ending early and the rewards are getting lower. My experience replay is full of this single transition (around 99 percent of the buffer).

What could be the reason? Which things I should check? Is there anything I could try? I'm also willing to add information, just comment what could be of interest.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.