Suppose there are two players in my zero-sum game and they play in a row like chess. And I want to learn the policy function using the REINFORCE algorithm.

I have doubts about passing reward values in the episode trajectory. If there is a single agent, then it is straightforward to pass the reward values based on the action performed by the agent in a particular state. But, in the case of multiple players, action by any player $k$ may need the updation of reward for any other player $\ell$. In such cases, I have doubts about passing rewards for player $\ell$.

Suppose, in the state $s_0$, player 1 did an action $a_0$ and causes the reward of $r_1$ to player 1 and -1 to player 2 and leads to state $s_1$. Then while it's turn, player 2 performs some action $a_1$ on state $s_1$ which caused a reward of $r_2$ to it. Then which trajectory should I need to pass among the following?

  1. $s_0a_0r_1s_1a_1r_2 \cdots$
  2. $s_0a_0r_1s_1a_1(r_2-1) \cdots$

I am guessing the second one. But my doubt is that the reward $r_2$ is due to the action of player 2 and $r_2-1$ cannot be viewed as an immediate reward since $-1$ is due to the action of player 1 in the previous time step. How to pass the reward values in this case?

  • $\begingroup$ Let me try to understand if I understood correctly what you're trying to do. You're trying to apply REINFORCE to a two-player game. Both players don't take an action at each step, but you make one player take an action at step $t$ and the next player at step $t+1$, and so on. From your attempt to define the reward $r_2 - 1$, it seems that you're also trying to penalise the second player based on the past actions of player 1. But why don't you also penalise player $1$? Are you trying to find a policy only for player 1? It's not clear what kind of policy you're trying to find here... $\endgroup$
    – nbro
    Jan 10 at 17:24
  • $\begingroup$ ... because it seems that you're taking trajectories that take into account the moves of both players in order to update the policy, but then you only seem to penalise one player. I am not sure if this approach makes sense. So, maybe you should try to clarify what your policy is supposed to do. $\endgroup$
    – nbro
    Jan 10 at 17:27
  • $\begingroup$ But why don't you also penalise player 1? It is just an example. Player 1 may also get penalized in later steps due to the action of player 2. Are you trying to find a policy only for player 1?. No, I am not trying to find the policy for a single player. I want to update the same NN for all players. @nbro $\endgroup$
    – hanugm
    Jan 10 at 22:35
  • $\begingroup$ Important: Is the game a zero-sum game? Chess is, for example, it has simple "I win, you lose" combination. If it is a zero-sum game, then any collection of "points" along the way is secondary. You can treat gaing game assets (such as cash in monopoly) as a reward, but it may make more sense to treat it as state information, so you can use the zero-sum end result to drive the rewards, if all that matters is who wins at the end. $\endgroup$ Jan 10 at 22:45
  • 1
    $\begingroup$ That's not what I said, I was asking a for clarification about your scenario. However, that may be the best answer if you really do have a zero-sum game, and don't need to do any reward shaping. $\endgroup$ Jan 10 at 22:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.