Timeline for When to activate batch normalization and dropout in deep Q-learning?
Current License: CC BY-SA 4.0
15 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Sep 17, 2021 at 7:41 | comment | added | Eric O. Lebigot | … or more or less, depending on how the replay buffer is sampled. All in all, it looks like when batch norm accumulates statistics might not matter too much. One question could be whether we want the batch norm statistics to be based on the exact samples on which we learn, or be skewed towards all the samples encountered during exploration episodes. If anybody has any opinion on this, that would be interesting. | |
| Sep 13, 2021 at 13:13 | comment | added | tnfru | Because you would do it twice then. | |
| Sep 13, 2021 at 10:08 | comment | added | Eric O. Lebigot | Since the agent constantly interact with the environment, what I mean by "exploration phase" is simply the early part of training, where the agent accumulates experiences (in the "memory" you refer to--the replay buffer). When the learning takes place, batch norm will indeed learn statistics. But then why not accumulate batch norm statistics when accumulating experiences too? Since no sampling takes place, this even looks like a good idea, no? | |
| Sep 4, 2021 at 15:59 | comment | added | tnfru | There is no reason to accumulate statistics during interaction with environment (this is what you call exploration phase?). This is because your experienced timesteps will be stored in some kind of memory to train on. When our agents then trains on them, batchnorm will train on the data. | |
| Sep 3, 2021 at 10:29 | comment | added | Eric O. Lebigot | Indeed, for exploring we do not calculate a gradient, but there is the choice of whether we want to accumulate statistics for the batch normalization, hence the question (the same question also holds for training, when calculating the expected discounted reward of the state reached by an action). | |
| Sep 2, 2021 at 9:41 | comment | added | tnfru | What exactly do you mean by 'during exploration'? In general your get_action method called in rollout should not calculate a gradient. For calculating state value you should use a target net for stability, so it's use of batchnorm depends on if your state value function uses it. | |
| Sep 1, 2021 at 18:20 | comment | added | Eric O. Lebigot | … and also possibly when calling it during exploration and when computing the state value of the state reached by an action (for Bellman's equation). Hence the question. | |
| Aug 31, 2021 at 17:03 | comment | added | tnfru | I'm not really sure I understand your question. It will be always updated by backprop whenever your network is updated. | |
| Aug 30, 2021 at 19:59 | comment | added | Eric O. Lebigot | Batch norm is indeed always used, but the question is really about the activation of its updates (i.e. training vs evaluation mode): which of the three different parts of the basic Reinforcement Learning algorithm should update the running measurement of the input's statistics? I'm not sure it matters much, but maybe this is incorrect. | |
| Aug 30, 2021 at 14:06 | comment | added | tnfru | If you use BatchNorm, then use it during all phases. Only Dropout is something you turn off during evaluation. | |
| Aug 30, 2021 at 13:58 | comment | added | Eric O. Lebigot | Hmm, I should have written "where" should batch norm be used (not "when"): the algorithm itself uses the network in 3 different places (the the original question): the question is in which of these places should batch normalization be activated. | |
| Aug 29, 2021 at 9:10 | comment | added | tnfru | I would always try it. As far as I know there is no one size fits all solution. So in general just treat using BatchNorm as a hyperparameter and test your agent with and without. | |
| Aug 28, 2021 at 19:55 | comment | added | Eric O. Lebigot | This is quite true, but this doesn't seem to answer the question of when to let the batch normalization layers learn their normalization, during reinforcement learning. | |
| S Aug 26, 2021 at 18:34 | review | First answers | |||
| Aug 26, 2021 at 20:37 | |||||
| S Aug 26, 2021 at 18:34 | history | answered | tnfru | CC BY-SA 4.0 |