Suppose there are two players in my zero-sum game and they play in a row like chess. And I want to learn the policy function using the REINFORCE algorithm.
I have doubts about passing reward values in the episode trajectory. If there is a single agent, then it is straightforward to pass the reward values based on the action performed by the agent in a particular state. But, in the case of multiple players, action by any player $k$ may need the updation of reward for any other player $\ell$. In such cases, I have doubts about passing rewards for player $\ell$.
Suppose, in the state $s_0$, player 1 did an action $a_0$ and causes the reward of $r_1$ to player 1 and -1 to player 2 and leads to state $s_1$. Then while it's turn, player 2 performs some action $a_1$ on state $s_1$ which caused a reward of $r_2$ to it. Then which trajectory should I need to pass among the following?
- $s_0a_0r_1s_1a_1r_2 \cdots$
- $s_0a_0r_1s_1a_1(r_2-1) \cdots$
I am guessing the second one. But my doubt is that the reward $r_2$ is due to the action of player 2 and $r_2-1$ cannot be viewed as an immediate reward since $-1$ is due to the action of player 1 in the previous time step. How to pass the reward values in this case?