2
$\begingroup$

I've been reading a lot of tutorials on DQNs for cartpole. In many of them, they have the funnel layer of the neural net be a linear activation. Why is this? Is it just a choice made by the implementer? Is this Choice specific to cartpole, or do most control task dqns use it? Thanks.

$\endgroup$
2
$\begingroup$

Q learning predicts the action value, $q(s, a)$ for taking action $a$ in state $s$. The action value is usually the discounted sum of all future rewards. In general it can take any scalar value.

DQN uses a neural network to approximate $q(s, a)$. Although you might use this to select an action (thus think of the problem as a classification), the NN has to perform regression to predict the action values.

It is most common to use a linear final layer, and mean squared error loss in DQN, to match this regression task. So yes, you will find most control DQNs will make the same choice as the cartpole example that you are looking at.

$\endgroup$
  • $\begingroup$ That makes sense. Thanks! $\endgroup$ – axon Apr 8 at 7:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.