I'm using Q-learning with some extensions such as noisy linear layers, n-steps and double DQN.
The training, however, isn't that successful, my rewards are descreasing over time after a steep increase at the beginning:
But what's interesting is that my td loss is also descreasing:
The sigma magnitudes of the noisy linear layers which control the exploration are strangely increasing, and also, seems to converge. I expected it to reduce uncertainty over time, but the opposite is the case.
Another intresting thing, and that's probably why my loss is decreasing: The model tends to generate always the same transition, which is why the episodes are ending early and the rewards are getting lower. My experience replay is full of this single transition (around 99 percent of the buffer).
What could be the reason? Which things I should check? Is there anything I could try? I'm also willing to add information, just comment what could be of interest.