1
$\begingroup$

If single-agent RL algorithms such as Q-learning is applied to multi-agent systems (e.g. Markov games), the environment from the perspective of the agent is non-stationary, and the agent is faced with a moving target problem, that is, the optimal policy changes as other agents' policies changes. I understand this as the optimal Q-function, $Q^*(s,a)$, being the moving target, since the optimal Q-function depends on other agents' policies.

If the agent learns in the space of the joint action space, then according to several references, the environment is stationary from the perspective of any agent, even though the agents' policies may change over time. Suppose the goal of the agent is to learn a joint optimal policy defined as a Nash equilibrium, from which it bases its actions on. Then, the agent tries to find an optimal Q-function defined as $Q_{{\pi^*}}(s,\textbf{a})$, where a denotes the joint action and $\pi^*$ denotes the joint optimal policy. I would then conclude that $Q_{{\pi^*}}(s,\textbf{a})$ is no longer a moving target, since we are in the space of joint policies, and thereby account for other agents' policies explicitly.

So as I understand it, non-stationarity and the moving target problem are two sides of the same coin. Is this correctly understood? Or can the environment be stationary while the problem is still a moving-target?

$\endgroup$
1
  • 2
    $\begingroup$ it seems as though you're defining the moving target problem and the non-stationarity problem to be the same thing. personally, when I heard the moving target problem, I would think immediately of the fact that the function approximates in DRL is being trained towards a "moving target", in the sense that it bootstraps from itself (i.e. when you update your network, then you're also updating the target, since the target comes in part from the network). $\endgroup$
    – David
    Jun 13, 2023 at 18:13

1 Answer 1

0
$\begingroup$

The definition of non-stationarity in multi-agent learning is indeed correct.

The 'moving target problem' you are referring to is investigated here: Transient Non-stationarity and Generalisation in Deep Reinforcement Learning. Both of them together have been discussed in section 5.10.1 in this book: Multi-Agent Reinforcement Learning: Foundations and Modern Approaches

$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .