1
$\begingroup$

In the vanilla version of deep Q-learning, there are three places where the Q-network is queried:

  1. When exploring.

  2. When training:

    a. When calculating the optimal value of the state reached by an action (so as to compute a target discounted reward).

    b. When calculating the optimal Q-value for a given state, during training (so as to nudge the network weights and better reproduce the observed reward).

Now, during which steps should batch normalization and dropout be activated?

I couldn't find anything through a Google search.

Here are my guesses, for each step:

  1. When exploring: activate batch normalization and dropout: this lets the normalizations to be learned, and gives a chance to uncertain Q-values to be selected even if they are relatively low (because the dropout can result in a Q-value prediction higher than its average).

  2. When training:

    a. Do not activate batch normalization and dropout for calculating the optimal state value of the state reached by an action, because we want the Bellman equation to converge faster and therefore prefer stable (optimal state value) targets.

    b. Activate batch normalization and dropout when calculating the Q-value of a chosen action, as this is the whole idea of dropout (we use it during training).

What is the common wisdom on this?

$\endgroup$
1
$\begingroup$

Batch Normalization should be applied between all layers and their activation functions excluding the output layer. This squishes the ranges of numbers in a better range for neural network to build appropriate sized gradients.

I've not seen much use of Dropout in Deep RL because the networks are usually small and overfitting isn't as much of a problem as in supervised learning.

$\endgroup$
13
  • $\begingroup$ This is quite true, but this doesn't seem to answer the question of when to let the batch normalization layers learn their normalization, during reinforcement learning. $\endgroup$ Aug 28 at 19:55
  • $\begingroup$ I would always try it. As far as I know there is no one size fits all solution. So in general just treat using BatchNorm as a hyperparameter and test your agent with and without. $\endgroup$
    – tnfru
    Aug 29 at 9:10
  • $\begingroup$ Hmm, I should have written "where" should batch norm be used (not "when"): the algorithm itself uses the network in 3 different places (the the original question): the question is in which of these places should batch normalization be activated. $\endgroup$ Aug 30 at 13:58
  • $\begingroup$ If you use BatchNorm, then use it during all phases. Only Dropout is something you turn off during evaluation. $\endgroup$
    – tnfru
    Aug 30 at 14:06
  • $\begingroup$ Batch norm is indeed always used, but the question is really about the activation of its updates (i.e. training vs evaluation mode): which of the three different parts of the basic Reinforcement Learning algorithm should update the running measurement of the input's statistics? I'm not sure it matters much, but maybe this is incorrect. $\endgroup$ Aug 30 at 19:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.