Suppose we're training two agents to play an asymmetric game from scratch using self play (like Zerg vs. Protoss in Starcraft). During training one of the agents can become stronger (discover a good broad strategy for example) and start winning most of the time, which causes big portion of the state values (or Q(s,a) values) become very high for this agent and low for another, just because the first is generally stronger and receives most of the rewards. Some training time later the other one finds a weakness in the first's play (in many states too) and starts dominating and the reward stream shift the other way.
The problem is, we have to retrain function approximator (deep neural net) to wildly different value/Q states, this slows and destabilizes learning. For each of the agents this is similar to highly nonstationary environment (the opponent), that can be harsh or easy at times.
What do people usually do in such a case? I think what is needed is some kind of slowly changing baseline (similar to advantage in A2C), but applied to the reward values themselves.