I am trying to use Deep-Q learning environment to learn Super Mario Bros. The implementation is on Github.
I have a neural network that Q values update within an episode for a very small learning rate (0.00005). However, even if I increase the learning rate to 0.00025, the Q values do not change within an episode as they are predicting the same Q values regardless of what state it is in. For example, if Mario moves right, the Q value is the same. When I start a new episode, the Q values change though.
I think that the Q values should be changing within an episode as the game should be seeing different parts and taking different actions. Why don't I observe this?