2
$\begingroup$

I'm using the DQN algorithm to train my agent to play a turn-based game. The winner of the game can be known before the game is over. Once the winning condition is satisfied, it cannot be reverted. For example, the game might last 100 turns, but it's possible to know that one of the players won at move 80, because some winning condition was satisfied. The last 20 moves don't change the outcome of the game. If people were playing this game, they, would play it to the very end, but the agent doesn't have to.

The agent will be using memory replay to learn from the experience. I wonder, is it helpful for the agent to have the experiences after the winning condition was satisfied for a more complete picture? Or is it better to terminate the game immediately, and why? How would this affect agent's learning?

Improve this question
$\endgroup$
4
  • 2
    $\begingroup$ What is your reward function? Is this a zero-sum game where the reward is -1 for a loss, 0 for a draw, and 1 for a win? $\endgroup$
    – DeepQZero
    Jun 29, 2020 at 16:41
  • 1
    $\begingroup$ Do you want the agent to ever continue playing after the winning condition (perhaps in production against human players)? If so, is it necessary to play out the game in a particular way in order to "secure" a logically obvious win? Can a winning condition actually be dropped through deliberately bad play? Would playing randomly or "badly" in some way be considered bad game etiquette for other players if e.g. they are playing to see who gets second place? $\endgroup$
    – Neil Slater
    Jun 29, 2020 at 16:58
  • 1
    $\begingroup$ It is a zero sum game $\endgroup$
    – mark mark
    Jun 29, 2020 at 17:50
  • 1
    $\begingroup$ Once the winning condition is satisfied, it cannot be reverted (random moves would suffice) $\endgroup$
    – mark mark
    Jun 29, 2020 at 17:52

1 Answer 1

Reset to default
1
$\begingroup$

You should probably grant reward at the point that the game is logically won. This will help the agent learn more efficiently, by reducing the number of timesteps over which return values need to be backed up.

Stopping the episode at that point should also be fine, and may add some efficiency too, in that there will be more focused relevant data in the experience replay. It seems like on the surface that there is no benefit to exploring or discovering any policy after the game is won, and from the comments no expectation from you as agent developer that the agent has any kind of behaviour - random actions would be fine.

It is still possible that the agent could learn more from play after a winning state. It would require certain things to be true about the environment and additional work from you as developer.

For example, if the game has an end phase where a certain kind of action is more common and it gains something within the game ("victory points", "gold" or some other numbered token that is part of the game mechanics and could be measured), then additional play where this happened could be of interest. Especially if the moves that gained this measure could also be part of winning moves in the earlier game. To allow the agent to learn this though, it would have to be something that it predicted in addition to winning or losing.

One way to achieve this is to have a secondary learning system as part of the agent, that learns to predict gains (or totals) of this resource. Such a prediction could either be learned separately (but very similarly to the action value) and fed into the q function as an input, or it could be a neural network that shares early layers with the q function (or policy function) but with a different head. Adding this kind of secondary function to the neural network can also have a regularising effect on the network, because the interim features have to be good for two types of prediction.

You definitley do not need to consider such an addition. It could be a lot more work. However, for some games it is possible that it helps. Knowing the game, and understanding whether there is any learning experience to be had as a human player to play on beyond winning or losing, might help you decide whether looking into trying to replicate this additional experience for a bot. Even if it works, the effect may be minimal and not worth the difference it makes. For instance running a more basic learning agent for more episodes may still result in a very good agent for the end game. That only costs you more run time for training, not coding effort.

Improve this answer
$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .