# Can a Reinforcement Learning problem with multiple simultaneous actions be formalized as a Multiagent Partially Observable Markov Decision Process?

Consider the following decision making problem. We have a controller that selects locations from a grid of coordinates and captures an image (observation $$o_t$$) with a camera at each location (action $$a_t$$). We try to find an optimal sequence of locations for a specific goal. This decision making problem can be formalized as a Partially Observable Markov Decision Process (POMDP). Here, we seek an optimal stochastic policy $$\pi^{*}_{\theta}(a_t|h_t)$$ that maps the history $$h_t= \langle o_1, a_1, ..., o_{t-1},a_{t-1},o_t \rangle$$ of actions and observations up to the current time $$t$$ to action probabilities. The history $$h_t$$ can be summarized by the hidden state of a RNN and we can use a policy gradient method, e.g. REINFORCE, to update the policy parameters $$\theta$$.

Suppose now that we want to select multiple locations, i.e. actions, simultaneously. According to my understanding, we could formalize the problem as a Mutliagent POMDP (MPOMDP) [1]. In this formalism, we would replace the single action of the previous problem by joint actions $$\vec{a}_t = \langle a^1_t, ..., a^N_t \rangle$$, the single observation by joint observations $$\vec{o}_t = \langle o^1_t, ..., o^N_t \rangle$$ and the history by $$h_t= \langle \vec{o}_1, \vec{a}_1, ..., \vec{o}_{t-1},\vec{a}_{t-1},\vec{o}_t \rangle$$, where $$N$$ is the number of agents. We would now try to find an optimal joint policy $$\vec{\pi}^{*} = \langle \pi^{1*}, ...,\pi^{N*} \rangle$$ consisting of sub-policies $$\pi_{\theta_n}(a^n_t|h_t)$$ that map the history $$h_t$$ to the action probability of each agent $$n$$. This would mean that the RNN would have $$N$$ output nodes and each sub-policy $$\pi^n$$ would be parametrized by $$\theta_n$$, a sub-set of weights of the output layer [2]. Would it be correct to assume that an optimal or near-optimal joint policy $$\vec{\pi}^{*}$$ can be obtained by simply applying the policy gradient method used above to each sub-policy $$\pi^n$$?

I would be curious to hear what you think about the MPOMDP formalism applied to the latter decision making problem or whether you would suggest something else.

[1] Oliehoek, Frans A., et al. "A concise introduction to decentralized POMDPs." Springer, 2016.

[2] Gupta, Jayesh K., et al. "Cooperative multi-agent control using deep reinforcement learning." International Conference on Autonomous Agents and Multiagent Systems. Springer, Cham, 2017.