When to activate batch normalization and dropout in deep Q-learning?

Question

In the vanilla version of deep Q-learning, there are three places where the Q-network is queried:

1. When exploring.

2. When training:

   a. When calculating the optimal value of the state reached by an action (so as to compute a target discounted reward).

   b. When calculating the optimal Q-value for a given state, during training (so as to nudge the network weights and better reproduce the observed reward).

Now, during which steps should batch normalization and dropout be activated?

I couldn't find anything through a Google search.

Here are my guesses, for each step:

1. When exploring: activate batch normalization and dropout: this lets the normalizations to be learned, and gives a chance to uncertain Q-values to be selected even if they are relatively low (because the dropout can result in a Q-value prediction higher than its average).

2. When training:

   a. Do not activate batch normalization and dropout for calculating the optimal state value of the state reached by an action, because we want the Bellman equation to converge faster and therefore prefer stable (optimal state value) targets.

   b. Activate batch normalization and dropout when calculating the Q-value of a chosen action, as this is the whole idea of dropout (we use it during training).

What is the common wisdom on this?

Answer 1

Batch Normalization should be applied between all layers and their activation functions excluding the output layer. This squishes the ranges of numbers in a better range for neural network to build appropriate sized gradients.

I've not seen much use of Dropout in Deep RL because the networks are usually small and overfitting isn't as much of a problem as in supervised learning.