1
$\begingroup$

This is my first question on this forum and I would like to welcome everyone. I am trying to implement DDQN Agent playing Othello (Reversi) game. I have tried multiple things but the agent which seems to be properly initialized does not learn against random opponent. Actually the score is about 50-60% won games out of nearly 500. Generally if it gets some score after first 20-50 episodes it stays on the same level. I have doubts on the process of learning and how to decide when the agent is trained. Current flow is as follows:

  1. Initialize game state.
  2. With epsilon greedy policy choose the action to make based on currently available actions depending on game state
  3. Get opponent to make his action
  4. Get the reward as number of flipped places that remain after opponent move.
  5. Save the observation to replay buffer
  6. If number of elements in replay buffer equals or higher than batch size do the training.

What I do not know is when do I know when to stop the training. Previously this agent trained against MinMax algorithm learned how to win 100% games because MinMax played exactly the same every time. I would like the agent to generalize the game. Right now I save the network weights after the game is won but I think it does not matter. I can't see that this agent find some policy and improve over time. Whole code for the environment, agent and training loop can be found here: https://github.com/MikolajMichalski/RL_othello_mgr I would appreciate any help. I would like to understand how the RL works :)

$\endgroup$
8
  • $\begingroup$ Hi Mikołaj and welcome! The best strategy for the game of reversi is somewhat counterintuitive. It's actually preferable to have little pieces of your color on the board at the beginning or midgame, and only in the very end flip the situation in your favor. The way you currently define your reward function encourages the agent to be greedy in the beginning, which may lead to losing the control of the corners (which are important). You want the agent to learn to win by itself, and your reward function should only award points for winning/drawing and losing. $\endgroup$ – mark mark Dec 23 '20 at 16:07
  • $\begingroup$ Hi, thank you for the answer. I changed reward to be only given after the episode is finished. Now it's counted as learning_agent_score - opponent_score. I can see a bit of improvement but is is not satisfying enough. I may experiment with hyper parameters. I have other question... Is it a proper way to train agent against random opponent? I thought about the policy of setting opponents weights to learning agent weights every time the win percentage gets higher so the trained agent plays against better opponent over time. $\endgroup$ – Mikołaj Michalski Dec 28 '20 at 15:57
  • $\begingroup$ To best of my knowledge, self-play would preferred over playing versus random opponent in terms of learning efficiency, since the state space is huge. Training network against itself using some minmax variation should be able to pick up some strategies. You can then evaluate the performance of the network against random opponent. Though, it might take a good while before agent learns anything the game is very complex (less than, but comparable to chess). $\endgroup$ – mark mark Dec 28 '20 at 17:06
  • $\begingroup$ As I did some experiments with hyperparameters and did not see much progress... I am coming back with more questions :) 1. What would be the best network architecture in this case? Mine is 1st layer - input 64 of size with ReLU, 2nd layer - hidden with ReLU- 32 of size, 3rd layer - 64 of size linear and 4th with softmax. I am not sure if this is proper approach but I can't find good source about building good network arch. 2. Currently I am getting minibatches with random samples for training. Is it good as the states sequentions for each games are highly dependent? $\endgroup$ – Mikołaj Michalski Jan 10 at 13:57
  • $\begingroup$ And another one: 3. Should the training be done after each episode or each step? $\endgroup$ – Mikołaj Michalski Jan 10 at 14:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.