I am trying to use Deep-Q learning environment to learn Super Mario Bros. The implementation is on Github.

I have a neural network that Q values update within an episode for a very small learning rate (0.00005). However, even if I increase the learning rate to 0.00025, the Q values do not change within an episode as they are predicting the same Q values regardless of what state it is in. For example, if Mario moves right, the Q value is the same. When I start a new episode, the Q values change though.

I think that the Q values should be changing within an episode as the game should be seeing different parts and taking different actions. Why don't I observe this?

  • 1
    $\begingroup$ Learning rate should only have an effect on the magnitude of updates to your Q-values, it should not influence when updates are triggered. What makes you think that it does? If you are certain that you have correctly observed this phenomenon, where did you get the code from that does this? $\endgroup$ – Dennis Soemers Sep 12 '18 at 7:41
  • $\begingroup$ @DennisSoemers if I had a larger learning rate, would the magnitude of updates to my Q-Values be smaller or bigger? As in my own code, the higher learning rate results in the Q-values not updating or staying the same. I've done something wrong, was just trying to find the root of it $\endgroup$ – rtz Sep 12 '18 at 7:50
  • $\begingroup$ @NeilSlater the environment is Super Mario Bros. I am using Deep-Q learning $\endgroup$ – rtz Sep 12 '18 at 7:51
  • $\begingroup$ @rtz Typically, we compute some sort of error (something like target_output - current_output), multiply that with the learning rate, and then update accordingly (that's the basic idea). So, a larger learning rate should result in larger updates. A learning rate of $0$ would mean no updating whatsoever. All of this is of course you mean the same thing as I do when you say "learning rate"... though I don't think it's really an ambiguous term generally $\endgroup$ – Dennis Soemers Sep 12 '18 at 7:54
  • 1
    $\begingroup$ @rtz You can put the code somewhere and I'll probably have time somewhere today to have a quick look through it (though not right now). Inspecting a DQN implementation line-by-line would probably be a lot of work, but can at least see if I happen to notice something in a quick glance. $\endgroup$ – Dennis Soemers Sep 12 '18 at 8:05

I was attaining negative values out of my convolutional layers and then using relu on them resulted in the gradient of the activation being 0. Hence, my Q values were not being updated. I've since updated my activations to be ELU. Thanks for the help.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.