I am using Open AI's code to do a RL task on an environment that I built myself.

I tried some network architectures, and they all converge, faster or slower on CartPole.

On my environment, the reward seems not to converge, and keeps flickering forever.

I suspect the neural network is too small, but I want to confirm my belief before going the route of researching the architecture.

How can I confirm that the architecture is the problem and not anything else in a neural network reinforcement learning task?


Check the function loss.

It might be that your environment is impossible to learn. However, most likely the network simply can't handle it. By measuring the loss during the learning stage, if you find it is always very high and does not decrease, it's a strong indication this might be the issue.

Because the network is too simple, when you optimize for some states, you ruin others. There is not formal way to find out if this is the case, but since the same algorithm works elsewhere, it's either a problem of your environment, or of the network.

  • $\begingroup$ Thanks! Can a network be "too large?" $\endgroup$
    – Gulzar
    Mar 3 '19 at 9:23
  • $\begingroup$ @Gulzar it can. Not by virtue of being unable to learn, but by taking too long to do all the calculations do to its size. Ideally, we want the smallest and fastest network possible that can learn the function we are modelling $\endgroup$
    – BlueMoon93
    Mar 3 '19 at 11:09

Loss not decreasing during the training does not signify that network don't train. It may mean network is in the exploration mode. I have observed that situation - accumulated reward growing steadily, which mean network is training well, but loss is not decreasing.

If you already know solution to some (simpler or other) version of your environment, you can train network in supervised manner to reproduce that solution. If network unable reproduce existing solution that is strong indication that network is too small or otherwise not good.

The other reason for accumulated reward oscillation could be network overfitting on latest training samples. In that case bigger replay buffer or more slow update of target network(if target network is used) may help.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.