# Does Value Loss in Actor Critic not decrease at all?

I am coding a problem with the Actor-Critic Method. The final loss is a summation of PolicyLoss and ValueLoss. The calculation of the PolicyLoss for each step is given at Equation Number 5 of https://arxiv.org/pdf/1707.06347.pdf. And for Valueloss:

$$V_t = \gamma * next\_Value + reward$$

$$Valueloss= L1\_Loss(Vt, Value)$$

I checked that the PolicyLoss depends on this 'Value' as well. And PolicyLoss is decreasing, and so the parts of my AI model depending on the 'Value' are also updating. But I am seeing that the ValueLoss is not decreasing, however, the Total Loss which is the sum(ValueLoss, PolicyLoss) is decreasing.

I don't understand how to make the ValueLoss decrease, or if it should decrease at all. Because, the equation of ValueLoss suggests that it is like a 'chasing the tail' situation, so it may fluctuate.

• Please, provide the correct link to the paper you're referring to. I believe that's https://arxiv.org/abs/1707.06347.
– nbro
Feb 26, 2022 at 10:05
• Yes, sorry, I edited now Feb 26, 2022 at 10:08