Reinforcement learning to play snake - network seems to not get trained at all

Question

I am trying to build a network able to play snake game. This is my very first attempt to do such stuff. Unfortunately, I've stuck and even have no idea how to reason about the problem.

I use reinforcement neural network approach (q-leaning). My network is built on top of Keras. I use 6 input neurons for my snake:

• 1 - is any collision directly behind
• 2 - is any collision directly on the right
• 3 - is any collision directly on the left
• 4 - is snack up front (no matter how far)
• 5 - is a snack on the right side (no matter how far)
• 6 - is a snack on the left side (no matter how far)

the output has 3 neurons:

• 1 - do nothing (go ahead)
• 2 - turn right
• 3 - turn left

I believe this is a sufficient set of information to make proper decisions. But the snake seems to not even grasp the concept of not hitting the wall - which results with instant death.

I use the following rewards table:

• 100 for getting the snack
• -100 for hitting wall/tail
• 1 for staying alive (each step)

Snake tends to run randomly no matter how many training iterations it gets.

The code is available on my github: https://github.com/ayeo/snake/blob/master/main.py

Answer 1

I cannot comment much on your setup for inputs and outputs. It seems adequate to get some control, but does not cover the fully Markov state for the game, so I would expect that will limit the agent from ever being truly optimal. I would expect it to learn to play the game though, if you were implementing Q learning with a neural network correctly.

In your code, you are implementing a basic Q learning loop. It seems correct. However, this combination of Q learning and neural networks is known not to work - or more accurately, it rarely works this simply. The problem is mainly to do with the network receiving its own initially biased outputs back as new targets, plus receiving updates in correlated form (data on each time step is strongly correlated with data from previous time step). These biases are too large for the Q learning process to overcome, and typically the result is an agent that fixates on a single default action, because it has learned an inflated action value for it.

The problem is well known in RL research and called "The Deadly Triad" by Sutton & Barto.

The usual solution to this with Q learing is called DQN or "Deep" Q Learning ("Deep" is in quotes because this should be applied even if you just have a single hidden layer).

In basic DQN, you need to add the following features:

• An experience replay table. Instead of training directly on experience as it is received, instead store $s, a, r, s'$ in memory. When it is time later in the loop to train the NN for a step, take a random sample of some M items (e.g. 32 items) as a mini-batch, calculate a latest target for them, and train once on the mini batch. You will need logic to only start this training process once you have some minimal amount of experience from behaving randomly (e.g. 500 random steps).

• A "target network". When generating target Q values, use a cloned copy of the learning network, and only update this clone every N steps (with N typically set at 1000 or 10000).

These two additions are not really optional, even for really basic environments. You will need to add them to your script.