# Problem over DQN Algorithm not converging on snake

I'm using a DQN Algorithm to play Snake.

The input of the neural network is a stack of 4 images taken from the games 80x80.

The output is an array of 4 values, one for every direction.

The problem is that the program does not converge and I've a lot of doubts in the replay function, where I train the neural network over a batch of 32 events.

That's the snippet:

def replay(self, batch_size):

minibatch = random.sample(self.memory, batch_size)

for state, action, reward, next_state, done in minibatch:

target = reward

if not done:
target = (reward + self.gamma *
np.amax(self.model.predict(next_state)[0]))
target_f = self.model.predict(state)
target_f[0][action] = target
self.model.fit(state, target_f, epochs=1, verbose=0)

if self.epsilon > self.epsilon_min:
self.epsilon *= self.epsilon_decay


Targets are:

• +1 for eating an apple
• 0 for doing a movement without dying
• -1000 for hitting a wall or the snake hitting himself
• Hi! questions about implementation are off-topic here, you may want to try other SE. Anyway, isn't -1000 a bit too much? Commented Jun 22, 2019 at 18:10
• Sorry. I was trying to convince the snake to not die but I can't get around the Q approximation with the neural network Commented Jun 22, 2019 at 18:15
• @RobertoAureli Did you make some progress? I have the same problem that the snake is not converging Commented Sep 10, 2021 at 19:31
• @greedsin Unfortunately not, I stopped the project Commented Sep 10, 2021 at 20:31
• @greedsin -1000 is a really bad scaling for the loss. it will explode the gradient and not lead to good results. you should give something like +1/10 for eat and -1 for die. also deepq learning is notorious hard on pixels. you might want to look into Data regularized Q Learning. Commented Sep 10, 2021 at 22:06

I think the main issue here is that you are trying to train the snake` (network) on images. This will create a lot of issues a there are no set parameters that the model can learn from.

From images, there is no logical way to define the boundary, directions and objects on the board. It will be much easier to write a simple computer vision script of game API to provide actual meaningful inputs to the model.

Here is a great article on building a model to play the snake game. The author also provides the game API for input along with example code to train the snake game.

Final results from the model

• Thanks for you answer. The problem is that I need to do it in that way for academical pouposes I'm taking inspiration from cs.toronto.edu/~vmnih/docs/dqn.pdf where the state is only the image and the reward is where I have more freedom. This is another example medium.com/@hugo.sjoberg88/… Commented Jun 22, 2019 at 19:04
• Is snake game harder for dqn than any atari game google use to train their dqn? Commented Feb 18, 2023 at 11:34