Does Value Loss in Actor Critic not decrease at all?

Question

I am coding a problem with the Actor-Critic Method. The final loss is a summation of PolicyLoss and ValueLoss. The calculation of the PolicyLoss for each step is given at Equation Number 5 of https://arxiv.org/pdf/1707.06347.pdf. And for Valueloss:

$V_t = \gamma \times V_{t+1} + r_t$

$\mbox{Valueloss} = \mbox{L}1\_\mbox{Loss}(Vt, Value)$

I checked that the PolicyLoss depends on this 'Value' as well. And PolicyLoss is decreasing, and so the parts of my AI model depending on the 'Value' are also updating. But I am seeing that the ValueLoss is not decreasing, however, the Total Loss which is the sum(ValueLoss, PolicyLoss) is decreasing.

I don't understand how to make the ValueLoss decrease, or if it should decrease at all. Because, the equation of ValueLoss suggests that it is like a 'chasing the tail' situation, so it may fluctuate.

Answer 1

So it seems I was right. The ValueLoss does not decrease as it IS a 'chasing the tail situation'. If I run the code with ValueLoss = a constant - Value instead of NextValue - Value, I see that the loss decreases. A way out of it can be to calculate NextValue not for every run but for some runs.