0
$\begingroup$

This PyTorch implementation of the actor-critic algorithm calculates the losses like so:

actor_loss = -log_prob * discounted_reward
policy_loss = F.smooth_l1_loss(value, torch.tensor([discounted_reward]))

Both are different from the regular formulas which are, in the case of the actor loss (parameterized by $\theta$):

$$log[\pi_\theta(s_t,a_t)]Q_w(s_t,a_t)$$

and, in the case of the critic loss (parameterized by $w$):

$$r(s_t,a_t) + \gamma Q_w(s_{t+1},a_{t+1}) - Q_w(s_{t},a_{t}),$$

where $r(s_t,a_t)$ is the immediate reward following taking the action.

For the actor, "the immediate critic evaluation of the transition" was replaced with "the discounted reward". For the critic, the discounted evaluation of the value from the next state $r(s_t,a_t) + \gamma Q_w(s_{t+1},a_{t+1})$ was replaced by "the discounted reward". The $L_1$ loss is then calculated, effectively discarding the sign of the (equation) loss.

Questions:

  1. Why did they make these changes?

  2. Why is the sign discarded for the critic loss?

$\endgroup$
1
$\begingroup$

There is no 'regular' formula for calculating policy loss, the regular thing when calculating policy gradient is to multiply gradient with advantage function which can be many things. Look at section 2 of this paper for coverage on basic advantage functions. Also, the expected discounted reward is the same thing as the state-action value function (Q value).

$$Q^\pi(s_t, a_t) = \mathbb{E}_{s_{t+1:\infty}, a_{t+1:\infty}}[\sum_{l=0}^\infty \gamma^lr_{t+l}]$$

So, the variations you posted roughly calculate the same thing.

Regarding the negative sign, in policy gradient methods, we want to maximize our performance function which has the following form:

$$J(\theta) = \sum_\limits{\substack{a}} \pi(a \mid s, \theta)A(s, a)$$

So, the higher our performance the better it is. When people write code, they use minimizers that minimize the objective function, but, in this case, we want to maximize it, so maximizing the objective function is the same thing as minimizing the negative objective function therefore the negative sign.

| improve this answer | |
$\endgroup$
  • $\begingroup$ Thanks! However, there isn't actually a negative sign in the equations. There is an absolute value (l1_loss), which makes the sign, either positive or negative, lose its meaning. Why does it make sense to only care about the distance of the actual reward to the expected reward, but not about the direction? $\endgroup$ – Gulzar Feb 1 '19 at 18:15
  • $\begingroup$ The loss that you are refering to, judging by the code, is the value loss, that is critic loss, not actor (policy) loss. Taking the example you provided in the question, objective function of the critic is to minimize TD error which is the difference between $Q_w(s_{t},a_{t})$ and $r(s_t,a_t) + \gamma Q_w(s_{t+1},a_{t+1})$ . We don't care about which of those values is bigger we only care to minimize difference between them, therefore l1 loss. You could have used l2 loss as well . $\endgroup$ – Brale Feb 1 '19 at 19:00
  • $\begingroup$ I still don't see how this satisfies the equations (in the original question), in which clearly, there is significance to the sign. Is the absolute value missing from the equation? $\endgroup$ – Gulzar Feb 1 '19 at 19:05
  • $\begingroup$ Are you refering to the equation from the article where they update critic with this: $r(s_t,a_t) + \gamma Q_w(s_{t+1},a_{t+1}) - Q_w(s_{t},a_{t})$ ? If yes, then like I said we want to minimize that difference so the sign doesn't really matter, maybe sometimes we take a step in negative direction and sometimes in positive direction, but as the difference reduces we wouldn't be taking any steps, we would converge to parameter vector $w$ , it's same as basic tabular q-learning $\endgroup$ – Brale Feb 1 '19 at 19:36
  • $\begingroup$ Yes, I am referring to that equation. Please explain again why the sign doesn't matter. If we don't take sign into account (and the difference is not VERY small), then it could very easily cause divergence, as we could very likely change the weights in the opposite direction $\endgroup$ – Gulzar Feb 1 '19 at 20:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.