1
$\begingroup$

I'm working on a deep q-learning model in an infinite horizon problem, with a continous state space and 3 possible actions. I'm using a neural network to approximate the action-value function. Sometimes it happens that, after a few steps, the algorithm starts choosing only one between the possible actions (apart from a few steps where I suppose it explores, given the epsilon-greedy policy it follows), leading to bad results in terms of cumulative rewards. Is this a sign that the algorithm diverged?

$\endgroup$
3
$\begingroup$

Is this a sign that the algorithm diverged?

It is a common sign of a problem with learning process. That includes divergence due to poor hyper-parameters, even just bad luck. But it can also point to a design/architecture problem.

Other common causes of algorithm failing with a fixed action choice include:

  • Neural network inputs not scaled before use.

  • Large reward values causing large initial squared errors (either re-scale rewards or reduce learning rate to fix)

  • State representation too far from Markov property assumption

  • A bug in code (almost anywhere, unfortunately)

  • Catastrophic forgetting due to focusing too much on specific results and generalising from them incorrectly. Your agent might be suffering from this if it starts to learn correctly, reaching some level of competence at the task before failing.

$\endgroup$
1
  • $\begingroup$ I'm in the "deadly triad" case, so it is not such a surprise that it can easily diverge. I still have to try more sophisticated reward functions so I hope that I can soft that tendency in this way. $\endgroup$ – aandre_90 May 31 '20 at 12:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.