# Can the opponent's turn affect the reward for a DQN agent action?

I made an engine for a 2 players card game and now I am trying to make an environment similar to OpenAI Gym envs, to ease out the training.

I fail to understand this thing however:

1. If I use step(agentAction), I play the agent's turn in the game, calculate the reward.
2. Play the opponent's turn (which will be either a random AI or a rule-based one).

Question:
Does the opponent's turn affect the calculated rewards? As far as I know, the reward should only be the result of the agent's action right?

Thank you.

Does the opponent's turn affect the calculated rewards?

Yes, in general it can. Obvious case, in a two player game where the opponent could win or lose on their turn, but has other options.

As far as I know, the reward should only be the result of the agent's action right?

In a well-defined MDP, the reward should be a stochastic function of the current state and the agent's action. The stochastic part can include any changes due to an opponent player.

If the opponent player is random, or follows a well-defined and fixed policy, then you consider them part of the environment. So this requirement is met technically. The reward does only depend on current state and the agent's action. The actual result may happen on the opponent's turn, but that does not matter.

In a card game where the opponent's cards are hidden and affect their strategy, this may not strictly be the case, because the visible state will not determine the opponent's behaviour. The problem stops being an MDP, and starts being a POMDP. Whether or not that impacts the agent will depend on how much strategy relies on the hidden nature of these cards. In blackjack, there is little impact to not knowing an opponent's cards before they are played out - there is little difference between hidden cards and cards that are randomly determined after the agent plays. So you can get away with pretending it is a normal MDP. In poker, the knowledge of hidden cards is almost everything about the game, so a POMDP or other approach that tracks possible hidden state is required.

Note that learning to defeat a random or expert player is usually not the same as learning to play optimally (unless your expert player is already optimal). For that you may need self-play and an agent which learns both players' policies.

• So if for example the agent's action results in him capturing 0 cards on the table but when the opponent plays, he captures 2. Does this mean the agent failed? Would the reward be -2? Jul 15, 2018 at 16:27
• Another scenario would be the end of the game, where my agent captures 0 cards and the cards left (e.g. 3) on the table go to his opponent (assuming he's the last capturing player), I believe in this case the reward would be -3 to teach the agent that he needs to capture last in the last round? Jul 15, 2018 at 16:28
• @Haytam: If the goal is definitely to capture cards, and not let the opponent do so (i.e. that is all that matters in the game), then you can have rewards that count cards. Otherwise, just +1 for a win, 0 for a draw and -1 for losing at the end of the game should be fine, and is a standard approach for RL playing classic 2-player games. Jul 15, 2018 at 16:32
• The goal is to capture cards, but efficiently. If the agent has the choice to capture 1 card or 3 sequential cards, I want him to choose the 3 sequential cards, that's why I want my reward to be the number of cards eaten, but I want to be sure about the opponent's turn, because I don't think the opponent's turn has anything to do with my agent's action right? Jul 15, 2018 at 16:34
• @Haytam: The opponent's turn is definitely important to the agent, and you should find a way to include its effects. I don't know the card game you are modelling, so it might be reasonable to ignore how many cards the opponent captures. However, my gut instinct says that is not the case. You mention "sequential cards" as a desirable outcome - this implies there is more to this game than just total number of captures (that's how I understand the term "sequential", unless you just mean "total"). So using captures as a reward signal would be inaccurate. Winning at the end always matters though. Jul 15, 2018 at 19:04