I am new in reinforcement learning and not sure I have the right understanding of multi-agent policy gradient.

1, in my question, each agent has its own action space. When doing the sampling, for each agent, we can get a trajectory $\{s(state)_1, a(action)_1, s_2, a_2, ..., s_T, a_T\}$, so there is no reward after doing a single action, but a total reward after sampling for all agents. Then, by

\begin{equation} \frac{\partial \mathcal{J}\left(\theta_{i}\right)}{\partial \theta_{i}} \approx \frac{1}{N} \sum_{n=1}^{N}\left(\sum_{t=0}^{T-1} \frac{\partial}{\partial \theta_{i}} \log \pi_{\theta_{i}}\left(a_{t}^{(n)} \mid s_{t}^{(n)}\right) \gamma^{t} G_{\tau_{t: T}^{(n)}}\right) \end{equation} we can calculate the gradient of each agent and update the policy of each agent.

But in some examples from GitHub, there always exists a reward after an action. So I am confused.

2, is it possible that the action space for each agent is not fixed? For each agent, when turning to a new state, the action space is also new.

  • $\begingroup$ Hello. Welcome to AI SE. Here, we you can use latex/mathjax to format your math symbols, so please edit your post to do that. $\endgroup$
    – nbro
    Jul 5 '21 at 12:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.