1
$\begingroup$

Reading through the TensorFlow guide for Actor-Critic learning, I saw that the actor loss is multiplied by -1 when calculating:

2

The guide says this is to maximize the probabilities of actions with high rewards by minimizing its loss. My understanding may be flawed, but by taking multiplying the loss by -1, wouldn't the end gradients also be multiplied by -1 (and hence flipped), instead increasing the loss function?

Additionally, in Reinforcement Learning: An Introduction (Sutton and Barto), you see this instead, without the negative sign:

3

Specifically, this update here for the actor, where the gradient is not multiplied by -1:

4

Is there anything different between these two updates? I feel like there's an important concept or piece of calculus I might be missing here. Thanks!

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

The usual assumption in deep learning is that you want to minimize a loss function, say $\mathcal L(\theta)$. Follows that the SGD update rule is $$\theta \leftarrow \theta - \eta\nabla_\theta\mathcal L(\theta),$$ where $\eta$ is the learning rate. As you can notice we usually subtract the gradient to the parameters in order to minimize: since we take the negative direction. In the Sutton & Barto's book they assume a maximization: in fact there is a "+" sign in the weights update rule, and so you don't have to change the loss fn.

Said that, when the optimizer's default assumption is to minimize, you multiply the loss by $-1$ to maximize, i.e. to perform gradient ascent. Indeed, this would flip the gradient's direction which is then flipped again when taking the negative direction (i.e., $-\eta$) in the update rule.

Improve this answer
$\endgroup$
2
  • 2
    $\begingroup$ Great answer, thanks! Follow up question: the update for w in the pseudocode also seems to be gradient ascent then, since it has a "+", but the guide doesn't multiply critic loss by -1; why is this? $\endgroup$
    – jasooney23
    Commented Aug 23, 2023 at 16:07
  • $\begingroup$ @jasooney23 The critic loss is usually a minimization of a squared difference, like $\min_w (G-V_w)^2$: this is shown in equation 9.4 of the book (chapter 9.3 - page 201). Now, if you look at eq 9.5 (which is the one in the algo) you notice a rearrangement of the terms where $+\alpha$ appears. I think this is because they multiply by the negative derivative of the loss, which is $-2 (v_\pi-\hat v)$. But in practice you don't need to do this since autodiff handles that, you only need to care about the sign. $\endgroup$
    – Luca Anzalone
    Commented Aug 24, 2023 at 13:50

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .