A2C loss is usually defined as advantage * (-log(actor_predictions)) * target where target is a one-hot vector (with some clipping/noise/etc...) with the selected target.

Does this mean that we get larger losses for smaller mistakes?

If for example the agent has predicted $\pi(a|s)=0.9$ but the advantage is negative, this would mean a larger mistake than if the agent predicted that $\pi(a|s)=0.1$, however, putting the numbers in the formula means a larger loss for the 0.1 prediction.

Assuming advantage=-1, advantage * (-log(actor_predictions)) * target would mean:

$$ -1 * (-log(0.9)) * 1 = log(0.9)=-0.045 $$ $$ -1 * (-log(0.1)) * 1 = log(0.1)=-1 $$

Is my understanding correct?


So this question is due to a misunderstanding I had with how the loss function works in A2C.

Usually loss functions are always positives, but in A2C it can be negative as well. I thought that minimizing the loss function means moving it closer to zero, but in fact reducing the loss is indifferent to the sign of the loss value and is about making the loss value smaller.

In other words, the loss for 0.1 is smaller than the loss for 0.9 prediction for a disadvantageous action.

When the advantage is positive, $-log (p) * advantage$ will be smaller if $p$ is growing, and when the advantage is negative, $-log (p) * advantage$ will be smaller if $p$ is decreasing.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.