Just a passing thought about Q-learning. In the tabular Q-learning, what if I play around and modify any Q-values as I am using them to take actions? Would it be a violation of any (1) theoretical rule? (2) a reduction of efficiency?

Update: By modification I mean the corresponding value after the action is taken. I am not referring to the Bellman update step. I am assuming we have filled our Q-table and now implementing in real scenario. And in this real scenario upon deciding on an action can I update the corresponding Q-value by say adding some constant factor? Not updating through Bellman eqn. Only after certain time steps I will re-train (if that's the correct word) and refresh my Q-table using the Bellman. I am thinking of like training and testing kind of phases here, if that make sense. I say so because my rewards are dependent on a random dynamic parameter and so I cannot just re-update my Q-table right after each time-step (to prevent unnecessary swaying of Q-values) and want to do so after certain n time-steps. In between, I just use the table to look up and take the actions without updating the corresponding Bellman update equation. But since I have & want to do something to reflect some criteria upon taking the action, I am thinking what if I just modify the corresponding Q-values only by adding some constant factor until I retrain. This would help take better actions even until the n time-steps are over and ready for re-train....

  • 1
    $\begingroup$ Could you clarify what you mean by modify: Changing the values stored in the table, or changing the values used to decide current action? What kinds of change are you considering - e.g. is it based on some mathematical property of the state, or some other collected data or random mutation? How often/by how much will you be changing the values? Without those details it is very hard to answer your questions (1) and (2). $\endgroup$ Jun 13 at 6:21
  • $\begingroup$ @NeilSlater I have added the required details. Please see it and give any suggestions. $\endgroup$ Jun 13 at 12:56
  • $\begingroup$ Thanks for the update. I don't understand why you would not want to use the Bellman update equation in your scenario? It is usually OK to use it in a live environment, and if you want to change the Q values, it would be the usual way to change them, whilst adding some constant does not seem to serve any purpose that I can think of. What are your concerns, and what do you think you will gain by adding this constant? You mention a Q-table a few times, but you have also tagged the question DQN, where there is a neural network instead of a Q-table - this may be an important detail. $\endgroup$ Jun 13 at 15:05
  • 1
    $\begingroup$ @NeilSlater oh yes, I deleted the tag. I was arriving at a similar conclusion to what you said but I think I will first process what you said w.r.t my application. Will edit my code & then get back here to comment better on what you asked. $\endgroup$ Jun 13 at 19:04


You must log in to answer this question.