Multi-agent policy gradient, 1 total reward instead of reward in each step, 2 changing action space

I am new in reinforcement learning and not sure I have the right understanding of multi-agent policy gradient.

1, in my question, each agent has its own action space. When doing the sampling, for each agent, we can get a trajectory $$\{s(state)_1, a(action)_1, s_2, a_2, ..., s_T, a_T\}$$, so there is no reward after doing a single action, but a total reward after sampling for all agents. Then, by

$$\begin{equation} \frac{\partial \mathcal{J}\left(\theta_{i}\right)}{\partial \theta_{i}} \approx \frac{1}{N} \sum_{n=1}^{N}\left(\sum_{t=0}^{T-1} \frac{\partial}{\partial \theta_{i}} \log \pi_{\theta_{i}}\left(a_{t}^{(n)} \mid s_{t}^{(n)}\right) \gamma^{t} G_{\tau_{t: T}^{(n)}}\right) \end{equation}$$ we can calculate the gradient of each agent and update the policy of each agent.

But in some examples from GitHub, there always exists a reward after an action. So I am confused.

2, is it possible that the action space for each agent is not fixed? For each agent, when turning to a new state, the action space is also new.

• Hello. Welcome to AI SE. Here, we you can use latex/mathjax to format your math symbols, so please edit your post to do that.
– nbro
Jul 5 '21 at 12:26