1
$\begingroup$

I would like to train a neural network (NN) so that it learns the policy and value function for my agent.

Since I am using reinforcement learning and do not want to prefer certain actions in certain states at the beginning of the learning, ideally, my NN should be initialized in a way that it predicts a uniform policy for all of the actions in every state and then during training, it will adjust its weights based on the observations.

The idea for this weight initialization is to speed up the training process by not "delearning" random initial policy that can be off by quite some margin and to also guarantee equal exploration from every state straight from the beginning.

I would like to ask two questions about this topic:

  • Is this a good idea?
  • Are there any available tools for achieving this?
Improve this question
$\endgroup$
6
  • $\begingroup$ what you describe sounds like a simple uniform weight initialization, which all deep learning libraries allow out of the shelf. Be aware though that initializing a network to output uniform stuff doesn't necessary mean faster training. It's an assumption that sounds logical to us but it's not correlated at all with the optimization problem math the network is suppose to learn. $\endgroup$
    – Edoardo Guerriero
    Commented Jan 24, 2022 at 15:25
  • $\begingroup$ Can you please put your specific question in the title? "Weight initialization for Deep Reinforcement Learning" is not a question and it's quite general. $\endgroup$
    – nbro
    Commented Jan 24, 2022 at 16:11
  • $\begingroup$ @EdoardoGuerriero uniform weight initialization in the last layer does not necessarily produce uniform policy, since final output also depends on the values coming from the previous layers, or am I thinking somewhere wrong? (e.g. if we would have same weights in last layer but use ReLU in the penultimate layer then final probabilities will be scaled by the input coming from the ReLU, won't they?) $\endgroup$
    – Druudik
    Commented Jan 24, 2022 at 17:18
  • $\begingroup$ @Druudik that is true, there will always be some unbalance due to the input values, but still uniform initialization would be the only one capable to minimize that variation. There's no way to generically account for input variation to output a constant value, or to say it differently, you would have to train the model on that task first. And that would be tricky as well, for example it's not hard to see that if using biases the model would probably converge immediately on a solution with zero weights an 1/n bias values. (n = amount f output nodes) $\endgroup$
    – Edoardo Guerriero
    Commented Jan 24, 2022 at 17:38
  • $\begingroup$ Thanks. The title was not really in the form of a question. Check it now. Is that your question? By the way, for completeness, it might be a good idea to specify also the RL algorithm you're using. $\endgroup$
    – nbro
    Commented Jan 24, 2022 at 17:51

1 Answer 1

Reset to default
3
$\begingroup$

You may be interested in section 3.2 of this paper What Matters In On-Policy Reinforcement Learning? A Large-Scale Empirical Study (2020) by Google Research.

They claim that the initialization of the policy is very important to performance, sometimes making a huge (66%) improvement, just from the initialization of the policy.

I'm assuming you already know that the kernel weights in neural networks are initialized using the glorot uniform distribution; in that paper, they suggest that for the output layer, those weights should additionally be divided by 100.

Improve this answer
$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .