A2C Critic Loss Interpretation

Question

I'm working on an Advantage A2C implementation, and I just finished creating the value network $\hat{V_{\phi}}$. I train this network with the standard MSE loss of discounted rewards-to-go:$$\|\hat{V_{\phi}}(s_{t'}) - \sum_{t=t'}^{T}\gamma^{t-t'} r(s_t, a_t)\|^2$$

I would like to be able to evaluate and assess the performance and ability of the value network as I train, especially to see how that relates and interacts with the changes and improvements in the policy, however I'm not sure how to do this.

My first instinct was to track the loss after each batch of experiences, but this doesn't work. As the policy improves, episodes last longer, and the value loss increases. I understand why this happens, as it is harder to predict the rewards-to-go when the length of the future is more undetermined.

To fix this, I tried dividing the loss that I'm computing by the total number of steps in the batch of episodes. However, the loss is still increasing as the policy improves (as the episodes get longer). Why is this still happening? Then, is there anything I can do to get a better assessment of the quality of the value network?

Answer 1

Unfortunately no, the way to go is track the total reward and see if it's increasing and converging eventually. Value loss isn't a useful metric as the loss can be 0 when the value network always predicts 0 and the agent doesn't collect any reward, meaning very poor behavior.