0
$\begingroup$

Open Ai's (working) actor critic code calculates the losses like so:

actor_loss = -log_prob * discounted_reward
policy_loss = F.smooth_l1_loss(value, torch.tensor([discounted_reward]))

Both are different from the regular formulas which are:

Actor_Loss (parameterized by $\theta$):

$log[\pi_\theta(s_t,a_t)]Q_w(s_t,a_t)$

Critic Loss (parameterized by $w$):

$r(s_t,a_t) + \gamma Q_w(s_{t+1},a_{t+1}) - Q_w(s_{t},a_{t})$

Where $r(s_t,a_t)$ is the immediate reward following taking the action.


Meaning,

For the actor, the immediate critic evaluation of the transaction was replaced with the discounted reward, and

For the critic, the discounted evaluation of the value from the next state

$r(s_t,a_t) + \gamma Q_w(s_{t+1},a_{t+1})$

was replaced by

the discounted reward

and

an l1 loss is then calculated, effectively discarding the sign of the (equation) loss.

Questions:

  1. Why are the changed equation elements?

  2. Why is the sign discarded for the critic loss?

$\endgroup$
1
$\begingroup$

There is no 'regular' formula for calculating policy loss, the regular thing when calculating policy gradient is to multiply gradient with advantage function which can be many things. Look at the section 2 of this paper for coverage on basic advantage functions. Also expected discounted reward is same thing as state-action value function (Q value)

$Q^\pi(s_t, a_t) = \mathbb{E}_{s_{t+1:\infty}, a_{t+1:\infty}}[\sum_{l=0}^\infty \gamma^lr_{t+l}]$

so the variations you posted roughly calculate the same thing.
Regarding the negative sign, in policy gradient methods we want to maximize our performance function which has the following form:

$J(\theta) = \sum_\limits{\substack{a}} \pi(a \mid s, \theta)A(s, a)$

so the higher our performance the better it is. When people write code they use minimizers that minimize objective function, but in this case we want to maximize it so maximizing objective function is same thing as minimizing negative objective function therefore the negative sign.
Btw, I don't think this is OpenAI's code but something people from Pytorch did.

$\endgroup$
  • $\begingroup$ Thanks! However, there isn't actually a negative sign in the equations. There is an absolute value (l1_loss), which makes the sign, either positive or negative, lose its meaning. Why does it make sense to only care about the distance of the actual reward to the expected reward, but not about the direction? $\endgroup$ – Gulzar Feb 1 at 18:15
  • $\begingroup$ The loss that you are refering to, judging by the code, is the value loss, that is critic loss, not actor (policy) loss. Taking the example you provided in the question, objective function of the critic is to minimize TD error which is the difference between $Q_w(s_{t},a_{t})$ and $r(s_t,a_t) + \gamma Q_w(s_{t+1},a_{t+1})$ . We don't care about which of those values is bigger we only care to minimize difference between them, therefore l1 loss. You could have used l2 loss as well . $\endgroup$ – Brale_ Feb 1 at 19:00
  • $\begingroup$ I still don't see how this satisfies the equations (in the original question), in which clearly, there is significance to the sign. Is the absolute value missing from the equation? $\endgroup$ – Gulzar Feb 1 at 19:05
  • $\begingroup$ Are you refering to the equation from the article where they update critic with this: $r(s_t,a_t) + \gamma Q_w(s_{t+1},a_{t+1}) - Q_w(s_{t},a_{t})$ ? If yes, then like I said we want to minimize that difference so the sign doesn't really matter, maybe sometimes we take a step in negative direction and sometimes in positive direction, but as the difference reduces we wouldn't be taking any steps, we would converge to parameter vector $w$ , it's same as basic tabular q-learning $\endgroup$ – Brale_ Feb 1 at 19:36
  • $\begingroup$ Yes, I am referring to that equation. Please explain again why the sign doesn't matter. If we don't take sign into account (and the difference is not VERY small), then it could very easily cause divergence, as we could very likely change the weights in the opposite direction $\endgroup$ – Gulzar Feb 1 at 20:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.