# Can a Reinforcement Learning problem with multiple simultaneous actions be formalized as a Multiagent Partially Observable Markov Decision Process?

Consider the following decision making problem. We have a controller that selects locations from a grid of coordinates and captures an image (observation $$o_t$$) with a camera at each location (action $$a_t$$). We try to find an optimal sequence of locations for a specific goal. This decision making problem can be formalized as a Partially Observable Markov Decision Process (POMDP). Here, we seek an optimal stochastic policy $$\pi^{*}_{\theta}(a_t|h_t)$$ that maps the history $$h_t= \langle o_1, a_1, ..., o_{t-1},a_{t-1},o_t \rangle$$ of actions and observations up to the current time $$t$$ to action probabilities. The history $$h_t$$ can be summarized by the hidden state of a RNN and we can use a policy gradient method, e.g. REINFORCE, to update the policy parameters $$\theta$$.

Suppose now that we want to select multiple locations, i.e. actions, simultaneously. According to my understanding, we could formalize the problem as a Mutliagent POMDP (MPOMDP) . In this formalism, we would replace the single action of the previous problem by joint actions $$\vec{a}_t = \langle a^1_t, ..., a^N_t \rangle$$, the single observation by joint observations $$\vec{o}_t = \langle o^1_t, ..., o^N_t \rangle$$ and the history by $$h_t= \langle \vec{o}_1, \vec{a}_1, ..., \vec{o}_{t-1},\vec{a}_{t-1},\vec{o}_t \rangle$$, where $$N$$ is the number of agents. We would now try to find an optimal joint policy $$\vec{\pi}^{*} = \langle \pi^{1*}, ...,\pi^{N*} \rangle$$ consisting of sub-policies $$\pi_{\theta_n}(a^n_t|h_t)$$ that map the history $$h_t$$ to the action probability of each agent $$n$$. This would mean that the RNN would have $$N$$ output nodes and each sub-policy $$\pi^n$$ would be parametrized by $$\theta_n$$, a sub-set of weights of the output layer . Would it be correct to assume that an optimal or near-optimal joint policy $$\vec{\pi}^{*}$$ can be obtained by simply applying the policy gradient method used above to each sub-policy $$\pi^n$$?

I would be curious to hear what you think about the MPOMDP formalism applied to the latter decision making problem or whether you would suggest something else.

 Oliehoek, Frans A., et al. "A concise introduction to decentralized POMDPs." Springer, 2016.

 Gupta, Jayesh K., et al. "Cooperative multi-agent control using deep reinforcement learning." International Conference on Autonomous Agents and Multiagent Systems. Springer, Cham, 2017.