2
$\begingroup$

I read the DDPG paper, in which the authors state that the actions are fed only later to their Q network:

Actions were not included until the 2nd hidden layer of Q. (Sec 7, Experiment Details)

So does that mean, that the input of the first hidden layer was simply the state and the input of the second hidden layer the output of the first hidden layer concatenated with the actions?

Why would you do that? To have the first layer focus on learning the state value independent of the selected action? How would that help?

Is this just a small little tweak or a more significant improvement?

Improve this question
$\endgroup$
1
  • 1
    $\begingroup$ > Is this just a small little tweak or a more significant improvement? This seems pretty major.. I tested this with and without this tweak and without it the networks just wouldn't learn at all. But it may depend on the environment too, I'm not sure. $\endgroup$
    – tache
    Dec 26, 2021 at 14:17

1 Answer 1

Reset to default
2
$\begingroup$

So does that mean, that the input of the first hidden layer was simply the state and the input of the second hidden layer the output of the first hidden layer concatenated with the actions?

Yes.

Why would you do that? To have the first layer focus on learning the state value independent of the selected action? How would that help?

Neural networks hidden layers learn representations that get progressively closer to a linear relationship with the target, layer by layer.

So the first layer would not be learning the state value per se, but some representation of the state that was better related to the action value at the output.

Neural network architectures are often established by experimentation, so I expect here they tried the idea and the performance was OK. If they do not give alternative architectures in the paper, then the precise reason is not clear.

I can try a few guesses:

  • Concatenating the actions with the states in the input layer would result in more parameters for the neural network to achieve the same accuracy, running slightly slower. That's because more weights would be required to link the input layer with the first hidden layer.

  • Separating the layers of state and action inputs is a form of regularisation, as the first layer has to produce features that are useful for all possible actions.

  • You don't want the neural network to construct a reverse map $\pi(a|s) \rightarrow Q(s,a)$, you want it to independently assess the action values to do its job as a critic. By having the states and actions presented in different laters, this reduces the chance of the neural network finding shortcuts due to "recognising the policy" to predict the values. That's because the state and action pairings may repeat (so be recognised), but the first layer activations change over time (so even with repeated state/action pairs, this is a new representation to learn from).

Is this just a small little tweak or a more significant improvement?

I don't know, and suggest looking through previous work by the same authors in case they describe the approach in more detail.

Improve this answer
$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .