0
$\begingroup$

I recently came across this SO question, wherein the poster was asked to normalize their weights while using a function approximator with SARSA. I don't remember having to normalize any weights while using a DQN and so therefore would like to when and why is this method needed.

$\endgroup$

1 Answer 1

3
$\begingroup$

The kind of divergence that the other question experienced is a common problem with deep RL and temporal difference methods (Q-learning, SARSA, or any Actor Critic).

The weight normalisation would not be needed if the asker of the question you linked had used some kind of protection against divergence, such as using a separate target network. The normalisation may have worked for them as an alternative.

DQN already has experience replay and separate target network to help with this. If you do get runaway feedback causing diverging values, you already have some hyperparameters you can change to try and fix it - size of minibatch, size of replay table, number of updates between copying to target network.

So, in short, you don't really need weight normalisation in Deep RL. You might want to use it for regularisation though.

$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .