2
$\begingroup$

I'm using RL to train a Network on the game Connect4. It learns quickly that 4 connected pieces is good. It gets a reward of 1 for this. A zero is rewarded for all other moves.

It takes quite a time until the AI tries to stop the opponent from winning.

Is there a way this could be further reinforced?

I thought about giving a negative reward for the move played before the winning move. Thinking about this, I came to the conclusion that this is a bad idea. There'll be always a looser (except for ties), therefore there always be a last move from the losing player. This one hasn't to be a bad one. Mistakes could have been made much earlier.

Is there a way to improve this awareness of opponents? Or does it just have to train more?

I'm not perfectly sure if the rewards will propagate back in a way that encourages this behavior with my setup.

Improve this question
$\endgroup$

1 Answer 1

Reset to default
2
$\begingroup$

The classic working reward scheme for two player zero sum games (i.e. if I win, you lose and vice versa) is simply:

  • +1 for a win

  • 0 for a draw

  • -1 for a loss

These rewards should be associated with the last move made by the player before the game is resolved.

I thought about giving a negativ reward for the move played before the winning move.

That is what is normally done.

Thinking about this I came to the conclusion that this is a bad idea. There'll be always a looser (except for ties), therefor there always be a last move from the loosing player. This one hasn't to be a bad one. Mistakes could have been made much earlier.

It's OK, what the agent is learning is the value of a combination of state and action. It will, eventually, correctly associate the negative reward with a chain of board states that decrease in value and that it will then try to avoid.

This scenario of delayed rewards, not necessarily related to the choice of the current action, is handled correctly by reinforcement learning algorithms. It is still a tough problem though - a more delayed and difficult to predict reward makes for a tougher RL problem. However, even basic learners, such as Q-learning, can eventually solve this.

To get better performance, you might want to look into adding planning algorithms to the RL part.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Currently the reward plus 1 is given immediately by the Connect4 env. So I basically wait with adding my [state, action, reward, next state] to the experience replay for one move and choose the reward from [+1,0,-1]? Could you give some more explanation what you mean which adding planning? There are perfect min max algorithms for Connect 4, but this is like cheating. What kind of extra information would you hand the RL network? $\endgroup$
    – Mr.Sh4nnon
    Dec 18, 2018 at 15:50
  • $\begingroup$ For deriving correct [+1, 0, -1] it depends on how your agent and environment are structured for multi-players. I don't know what you have set up, so cannot tell you. How to combine RL learning and planning is a whole new question, my comment at the end was a suggestion of what you could research next. Typically add the results of a look-ahead search or rollout - e.g. MCTS, to guide the algorithm and learn faster. This is pretty much what AlphaZero does. It doesn't need to run to perfection, you can decide how to split effort between learning and planning to suit what you are studying. $\endgroup$
    – Neil Slater
    Dec 18, 2018 at 16:46

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .