If single-agent RL algorithms such as Q-learning is applied to multi-agent systems (e.g. Markov games), the environment from the perspective of the agent is non-stationary, and the agent is faced with a moving target problem, that is, the optimal policy changes as other agents' policies changes. I understand this as the optimal Q-function, $Q^*(s,a)$, being the moving target, since the optimal Q-function depends on other agents' policies.
If the agent learns in the space of the joint action space, then according to several references, the environment is stationary from the perspective of any agent, even though the agents' policies may change over time. Suppose the goal of the agent is to learn a joint optimal policy defined as a Nash equilibrium, from which it bases its actions on. Then, the agent tries to find an optimal Q-function defined as $Q_{{\pi^*}}(s,\textbf{a})$, where a denotes the joint action and $\pi^*$ denotes the joint optimal policy. I would then conclude that $Q_{{\pi^*}}(s,\textbf{a})$ is no longer a moving target, since we are in the space of joint policies, and thereby account for other agents' policies explicitly.
So as I understand it, non-stationarity and the moving target problem are two sides of the same coin. Is this correctly understood? Or can the environment be stationary while the problem is still a moving-target?
