# Should Q values be changing within an epoch/episode or should they change after one episode/epoch?

I am trying to use Deep-Q learning environment to learn Super Mario Bros. The implementation is on Github.

I have a neural network that Q values update within an episode for a very small learning rate (0.00005). However, even if I increase the learning rate to 0.00025, the Q values do not change within an episode as they are predicting the same Q values regardless of what state it is in. For example, if Mario moves right, the Q value is the same. When I start a new episode, the Q values change though.

I think that the Q values should be changing within an episode as the game should be seeing different parts and taking different actions. Why don't I observe this?

• Learning rate should only have an effect on the magnitude of updates to your Q-values, it should not influence when updates are triggered. What makes you think that it does? If you are certain that you have correctly observed this phenomenon, where did you get the code from that does this? – Dennis Soemers Sep 12 '18 at 7:41
• @DennisSoemers if I had a larger learning rate, would the magnitude of updates to my Q-Values be smaller or bigger? As in my own code, the higher learning rate results in the Q-values not updating or staying the same. I've done something wrong, was just trying to find the root of it – rtz Sep 12 '18 at 7:50
• @NeilSlater the environment is Super Mario Bros. I am using Deep-Q learning – rtz Sep 12 '18 at 7:51
• @rtz Typically, we compute some sort of error (something like target_output - current_output), multiply that with the learning rate, and then update accordingly (that's the basic idea). So, a larger learning rate should result in larger updates. A learning rate of $0$ would mean no updating whatsoever. All of this is of course you mean the same thing as I do when you say "learning rate"... though I don't think it's really an ambiguous term generally – Dennis Soemers Sep 12 '18 at 7:54
• @rtz You can put the code somewhere and I'll probably have time somewhere today to have a quick look through it (though not right now). Inspecting a DQN implementation line-by-line would probably be a lot of work, but can at least see if I happen to notice something in a quick glance. – Dennis Soemers Sep 12 '18 at 8:05