In the vanilla version of deep Q-learning, there are three places where the Q-network is queried:
a. When calculating the optimal value of the state reached by an action (so as to compute a target discounted reward).
b. When calculating the optimal Q-value for a given state, during training (so as to nudge the network weights and better reproduce the observed reward).
Now, during which steps should batch normalization and dropout be activated?
I couldn't find anything through a Google search.
Here are my guesses, for each step:
When exploring: activate batch normalization and dropout: this lets the normalizations to be learned, and gives a chance to uncertain Q-values to be selected even if they are relatively low (because the dropout can result in a Q-value prediction higher than its average).
a. Do not activate batch normalization and dropout for calculating the optimal state value of the state reached by an action, because we want the Bellman equation to converge faster and therefore prefer stable (optimal state value) targets.
b. Activate batch normalization and dropout when calculating the Q-value of a chosen action, as this is the whole idea of dropout (we use it during training).
What is the common wisdom on this?