# Can a Reinforcement Learning problem with multiple simultaneous actions be formalized as a Multiagent Partially Observable Markov Decision Process?

Consider the following decision making problem. We have a controller that selects locations from a grid of coordinates and captures an image (observation $$o_t$$) with a camera at each location (action $$a_t$$). We try to find an optimal sequence of locations for a specific goal. This decision making problem can be formalized as a Partially Observable Markov Decision Process (POMDP). Here, we seek an optimal stochastic policy $$\pi^{*}_{\theta}(a_t|h_t)$$ that maps the history $$h_t= \langle o_1, a_1, ..., o_{t-1},a_{t-1},o_t \rangle$$ of actions and observations up to the current time $$t$$ to action probabilities. The history $$h_t$$ can be summarized by the hidden state of a RNN and we can use a policy gradient method, e.g. REINFORCE, to update the policy parameters $$\theta$$.

Suppose now that we want to select multiple locations, i.e. actions, simultaneously. According to my understanding, we could formalize the problem as a Mutliagent POMDP (MPOMDP) [1]. In this formalism, we would replace the single action of the previous problem by joint actions $$\vec{a}_t = \langle a^1_t, ..., a^N_t \rangle$$, the single observation by joint observations $$\vec{o}_t = \langle o^1_t, ..., o^N_t \rangle$$ and the history by $$h_t= \langle \vec{o}_1, \vec{a}_1, ..., \vec{o}_{t-1},\vec{a}_{t-1},\vec{o}_t \rangle$$, where $$N$$ is the number of agents. We would now try to find an optimal joint policy $$\vec{\pi}^{*} = \langle \pi^{1*}, ...,\pi^{N*} \rangle$$ consisting of sub-policies $$\pi_{\theta_n}(a^n_t|h_t)$$ that map the history $$h_t$$ to the action probability of each agent $$n$$. This would mean that the RNN would have $$N$$ output nodes and each sub-policy $$\pi^n$$ would be parametrized by $$\theta_n$$, a sub-set of weights of the output layer [2]. Would it be correct to assume that an optimal or near-optimal joint policy $$\vec{\pi}^{*}$$ can be obtained by simply applying the policy gradient method used above to each sub-policy $$\pi^n$$?

I would be curious to hear what you think about the MPOMDP formalism applied to the latter decision making problem or whether you would suggest something else.

[1] Oliehoek, Frans A., et al. "A concise introduction to decentralized POMDPs." Springer, 2016.

[2] Gupta, Jayesh K., et al. "Cooperative multi-agent control using deep reinforcement learning." International Conference on Autonomous Agents and Multiagent Systems. Springer, Cham, 2017.

## 1 Answer

I guess it depends on what the goal is. If the goal is a general reward function, this formulation as an MPOMDP could make sense. One way to think about this, is as a way of modeling a general (centralized) POMDP with factored actions and observation spaces.

However, it seems that what you are describing might be an active perception problem, where the goal is to minimize uncertainty? In that case, there could be better formulations. I personally did some work in this direction with Yash Satsangi and others. E.g., these papers might be of interest:

Yash Satsangi, Shimon Whiteson, Frans A. Oliehoek, and Henri Bouma. Real-Time Resource Allocation for Tracking Systems. In Proceedings of the Thirty-Third Conference on Uncertainty in Artificial Intelligence (UAI), August 2017.

Yash Satsangi, Shimon Whiteson, Frans A. Oliehoek, and Matthijs T. J. Spaan. Exploiting submodular value functions for scaling up active perception. Autonomous Robots, 42(2):209–233, February 2018.

(and there is a large literature on this topic as well - look at the references in the papers).

Hope that is useful.