0
$\begingroup$

I'm using a DQN Algorithm to play Snake.

The input of the neural network is a stack of 4 images taken from the games 80x80.

The output is an array of 4 values, one for every direction.

The problem is that the program does not converge and I've a lot of doubts in the replay function, where I train the neural network over a batch of 32 events.

That's the snippet:

def replay(self, batch_size):

    minibatch = random.sample(self.memory, batch_size)

    for state, action, reward, next_state, done in minibatch:

        target = reward

        if not done:
            target = (reward + self.gamma *
                      np.amax(self.model.predict(next_state)[0]))
        target_f = self.model.predict(state)
        target_f[0][action] = target
        self.model.fit(state, target_f, epochs=1, verbose=0)

    if self.epsilon > self.epsilon_min:
        self.epsilon *= self.epsilon_decay`

Targets are:

  • +1 for eating an apple
  • 0 for doing a movement without dying
  • -1000 for hitting a wall or the snake hitting himself
$\endgroup$
5
  • $\begingroup$ Hi! questions about implementation are off-topic here, you may want to try other SE. Anyway, isn't -1000 a bit too much? $\endgroup$
    – olinarr
    Jun 22 '19 at 18:10
  • $\begingroup$ Sorry. I was trying to convince the snake to not die but I can't get around the Q approximation with the neural network $\endgroup$ Jun 22 '19 at 18:15
  • $\begingroup$ @RobertoAureli Did you make some progress? I have the same problem that the snake is not converging $\endgroup$
    – greedsin
    Sep 10 at 19:31
  • $\begingroup$ @greedsin Unfortunately not, I stopped the project $\endgroup$ Sep 10 at 20:31
  • $\begingroup$ @greedsin -1000 is a really bad scaling for the loss. it will explode the gradient and not lead to good results. you should give something like +1/10 for eat and -1 for die. also deepq learning is notorious hard on pixels. you might want to look into Data regularized Q Learning. $\endgroup$
    – tnfru
    Sep 10 at 22:06
0
$\begingroup$

I think the main issue here is that you are trying to train the snake (network) on images. This will create a lot of issues a there are no set parameters that the model can learn from.

From images, there is no logical way to define the boundary, directions and objects on the board. It will be much easier to write a simple computer vision script of game API to provide actual meaningful inputs to the model.

Here is a great article on building a model to play the snake game. The author also provides the game API for input along with example code to train the snake game.

Final results from the model

Snake agent results

$\endgroup$
1

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.