I am wondering how to correctly implement the DQN algorithm for two-player games such as Tic Tac Toe and Connect 4. While my algorithm is mastering Tic Tac Toe relatively quickly, I cannot get great results for Connect 4. The agent is learning to win quickly, if it gets the chance, but it only plays in the centre. It is unable to detect threats in the first and last columns. I am using DDQN with memory replay. Also teacher and student refer to two agents at different strengths, while the teacher is frequently replaced by a new student. My algorithm looks simplified as follows:

for i in range(episodes):
    observation = env.reset()
    done = False
    while not done:
        if env.turn == 1:
            action = student.choose_action(observation)
            observation_, reward, done, info = env.step(action)
            loss = student.learn(observation, action, reward, observation_, done))
            observation = observation_
            action = teacher.choose_action(-observation)
            observation_, reward, done, info = env.step(action)
            observation = observation_

The observation is -1 for player "o", 1 for player "x" and 0 for empty. The agent learns to play as player "x" and through action = teacher.choose_action(-observation) it should find the best move for player "o". I hope that is clear.

The update rule looks as follows:

# Get predicted q values for the actions that were taken
q_pred = Q_eval.forward(state, action)
# Get Q value for opponent's next move
state_ *= -1.
q_next = Q_target.forward(state_, max_action)
# Update rule
q_target = reward_batch - gamma * q_next * terminal
loss = Q_eval.loss(q_pred, q_target)

I am using -gamma * q_next * terminal, because the reward is negative, if the opponent wins in the next move. Am I missing anything important or is it just a question of hyperparameter tuning?

  • $\begingroup$ What does terminal represent, and what values can it take? $\endgroup$ May 15, 2021 at 8:42
  • $\begingroup$ it is 0 if the next state is terminal (done == True) and 1 otherwise. This is, because all terminal states have Q-Value of zero in Q-Learning. $\endgroup$
    – spadel
    May 15, 2021 at 9:08
  • $\begingroup$ You also asked this question here. $\endgroup$
    – nbro
    May 17, 2021 at 0:25
  • $\begingroup$ yeah I did, and? $\endgroup$
    – spadel
    May 17, 2021 at 9:59


You must log in to answer this question.