# How to handle actor-critic when actor consistently overestimates one action?

Following multiple guides for actor-critic, I have attempted to implement a single agent one-step version to see how well it would perform on Pong.

My understanding is that the actor provides probabilities on what action to perform out of the action pool. The critic estimates the value of a given state.

The actor is trained by utilizing the advantage of the action taken at a given state. This advantage value is calculated as $$R + G * V(S) - V(S^{+1})$$.

The critic, however, is trained as $$R + G * V(S^{+1})$$.

For me, this seems to regularly result in overestimating a single action after 10-20 episodes. Once an action is overestimated this way and is chosen rather consistently, the advantage dwindles down close to zero over time as the critic is only obtaining the value of the single action taken for that sample.

I assume this is a flaw in my implementation or a misunderstanding of the algorithm because if the value for the critic of a given state matches $$R + G * V(S^{+1})$$ then the advantage calculation becomes zero when only a single action is consistently taken.

So, have I misunderstood how it is to be implemented? If not, what does it sound like I am missing?