Why is this PyTorch implementation of the actor-critic algorithm inconsistent with the mathematical formulas?

Question

This PyTorch implementation of the actor-critic algorithm calculates the losses like so:

actor_loss = -log_prob * discounted_reward
policy_loss = F.smooth_l1_loss(value, torch.tensor([discounted_reward]))

Both are different from the regular formulas which are, in the case of the actor loss (parameterized by $\theta$):

$$log[\pi_\theta(s_t,a_t)]Q_w(s_t,a_t)$$

and, in the case of the critic loss (parameterized by $w$):

$$r(s_t,a_t) + \gamma Q_w(s_{t+1},a_{t+1}) - Q_w(s_{t},a_{t}),$$

where $r(s_t,a_t)$ is the immediate reward following taking the action.

For the actor, "the immediate critic evaluation of the transition" was replaced with "the discounted reward". For the critic, the discounted evaluation of the value from the next state $r(s_t,a_t) + \gamma Q_w(s_{t+1},a_{t+1})$ was replaced by "the discounted reward". The $L_1$ loss is then calculated, effectively discarding the sign of the (equation) loss.

Questions:

1. Why did they make these changes?

2. Why is the sign discarded for the critic loss?

Answer 1

There is no 'regular' formula for calculating policy loss, the regular thing when calculating policy gradient is to multiply gradient with advantage function which can be many things. Look at section 2 of this paper for coverage on basic advantage functions. Also, the expected discounted reward is the same thing as the state-action value function (Q value).

$$Q^\pi(s_t, a_t) = \mathbb{E}_{s_{t+1:\infty}, a_{t+1:\infty}}[\sum_{l=0}^\infty \gamma^lr_{t+l}]$$

So, the variations you posted roughly calculate the same thing.

Regarding the negative sign, in policy gradient methods, we want to maximize our performance function which has the following form:

$$J(\theta) = \sum_\limits{\substack{a}} \pi(a \mid s, \theta)A(s, a)$$

So, the higher our performance the better it is. When people write code, they use minimizers that minimize the objective function, but, in this case, we want to maximize it, so maximizing the objective function is the same thing as minimizing the negative objective function therefore the negative sign.