Consider the following decision making problem. We have a controller that selects locations from a grid of coordinates and captures an image (observation $o_t$) with a camera at each location (action $a_t$). We try to find an optimal sequence of locations for a specific goal. This decision making problem can be formalized as a Partially Observable Markov Decision Process (POMDP). Here, we seek an optimal stochastic policy $\pi^{*}_{\theta}(a_t|h_t)$ that maps the history $h_t= \langle o_1, a_1, ..., o_{t-1},a_{t-1},o_t \rangle$ of actions and observations up to the current time $t$ to action probabilities. The history $h_t$ can be summarized by the hidden state of a RNN and we can use a policy gradient method, e.g. REINFORCE, to update the policy parameters $\theta$.
Suppose now that we want to select multiple locations, i.e. actions, simultaneously. According to my understanding, we could formalize the problem as a Mutliagent POMDP (MPOMDP) [1]. In this formalism, we would replace the single action of the previous problem by joint actions $\vec{a}_t = \langle a^1_t, ..., a^N_t \rangle$, the single observation by joint observations $\vec{o}_t = \langle o^1_t, ..., o^N_t \rangle$ and the history by $h_t= \langle \vec{o}_1, \vec{a}_1, ..., \vec{o}_{t-1},\vec{a}_{t-1},\vec{o}_t \rangle$, where $N$ is the number of agents. We would now try to find an optimal joint policy $\vec{\pi}^{*} = \langle \pi^{1*}, ...,\pi^{N*} \rangle$ consisting of sub-policies $\pi_{\theta_n}(a^n_t|h_t)$ that map the history $h_t$ to the action probability of each agent $n$. This would mean that the RNN would have $N$ output nodes and each sub-policy $\pi^n$ would be parametrized by $\theta_n$, a sub-set of weights of the output layer [2]. Would it be correct to assume that an optimal or near-optimal joint policy $\vec{\pi}^{*}$ can be obtained by simply applying the policy gradient method used above to each sub-policy $\pi^n$?
I would be curious to hear what you think about the MPOMDP formalism applied to the latter decision making problem or whether you would suggest something else.
[1] Oliehoek, Frans A., et al. "A concise introduction to decentralized POMDPs." Springer, 2016.
[2] Gupta, Jayesh K., et al. "Cooperative multi-agent control using deep reinforcement learning." International Conference on Autonomous Agents and Multiagent Systems. Springer, Cham, 2017.
