If deep Q-learning starts to choose only one action, is this a sign that the algorithm diverged?

Question

I'm working on a deep q-learning model in an infinite horizon problem, with a continous state space and 3 possible actions. I'm using a neural network to approximate the action-value function. Sometimes it happens that, after a few steps, the algorithm starts choosing only one between the possible actions (apart from a few steps where I suppose it explores, given the epsilon-greedy policy it follows), leading to bad results in terms of cumulative rewards. Is this a sign that the algorithm diverged?

Answer 1

Is this a sign that the algorithm diverged?

It is a common sign of a problem with learning process. That includes divergence due to poor hyper-parameters, even just bad luck. But it can also point to a design/architecture problem.

Other common causes of algorithm failing with a fixed action choice include:

• Neural network inputs not scaled before use.

• Large reward values causing large initial squared errors (either re-scale rewards or reduce learning rate to fix)

• State representation too far from Markov property assumption

• A bug in code (almost anywhere, unfortunately)

• Catastrophic forgetting due to focusing too much on specific results and generalising from them incorrectly. Your agent might be suffering from this if it starts to learn correctly, reaching some level of competence at the task before failing.