This is a q-learning snake using a neural network as a q function aproximator and I'm losing my mind here the current model it's worst than the initial one.

The current model uses a 32x32x32 MLPRegressor from scikit-learn using relu as activation function and the adam solver.

The reward function is like following:

  • death reward = -100.0
  • alive reward = -10.0
  • apple reward = 100.0

The features extracted from each state are the following:

  1. what is in front of the snake's head(apple, empty, snake)
  2. what is in the left of the snake's head
  3. what is in the right of the snake's head
  4. euclidian distance between head and apple
  5. the direction from head to the apple measured in radians
  6. length of the snake

One episode consists of the snake playing until it dies, I'm also using in training a probability epsilon that represent the probability that the snake will take a random action if this isn't satisfied the snake will take the action for which the neural network gives the biggest score, this epsilon probability gradually decrements after each iteration.

The episode is learned by the regressor in reverse order one statet-action at a time.

However the neural network fails too aproximate the q function, no matter how many iterations the snake takes the same action for any state.

Things I tried:

  • changing the structure of the neural network
  • changing the reward function
  • changing the features extracted, I even tried passing the whole map to the network

Code (python): https://pastebin.com/57qLbjQZ

  • Can you provide a snippet of your code. – quintumnia Sep 12 at 21:10
  • the pastebin link to the code its in the post – oren revenge Sep 12 at 21:18
  • @orenrevenge I've updated my answer to address the code you posted too. – John Doucette Sep 13 at 19:44
up vote 2 down vote accepted

There are two problems here.

  1. The code you posted doesn't incrimentally train your multilayer perceptron. Instead, it effectively re-randomizes the weights, and then re-fits the model each time you call .fit() at lines 35 & 54. Using SKLearn's _fit() function with Incremental=true might solve this, or you can package up the data into a larger batch, and train on that offline instead after several episodes.

  2. Your reward function makes it painful to be alive, and doesn't give enough benefits through the Apples to make up for this. There are 100 squares that could contain the apple. On average, the apple will spawn about 5 squares from the snake in each direction. Since the snake can't move diagonally, that's 10 moves (5 left/right, 5 up/down). That means that if the snake plays perfectly, then on average, it might be able to get zero reward total. In practice, the snake will not play perfectly. This means living gives negative expected reward.

In contrast, if the snake can kill itself, it will stop getting negative rewards. The reward function you've used is maximized by getting big enough to run into your own tail as fast as possible. The snake should be able to do this after eating 3 apples I think. There is some incentive to hunt for food well, but not much compared with hitting your own tail as soon as possible.

If you want the snake to learn to hunt for the food, reduce the penalty for being alive to -1, or even -0.1. The snake will be much more responsive to signals from the food.

  • there are no walls in the game, the moment the snake goes on the edge of the map it reappears on the other end so basically it learns to never die which seems a little contradictory since he gets negative rewards for being alive – oren revenge Sep 13 at 7:20
  • @orenrevenge, you say that an episode lasts until the snake dies. How can it die? Only by hitting itself? – John Doucette Sep 13 at 7:21
  • yes thats correct – oren revenge Sep 13 at 7:23
  • @orenrevenge This is really more of an implementation question I think, and you might have better luck on the main StackOverflow site. Looking at your code, I think your mistake is in fit_episode, on lines 35 & 53. MLPRegressor.fit() promises to produce a trained model for the data you give it, but makes no promises about preserving the weights learned in past fits. See the SKLearn sourcecode: github.com/scikit-learn/scikit-learn/blob/f0ab589f/sklearn/… By default, the fit is not incremental. Using _fit with incremental=true might fix this. – John Doucette Sep 13 at 19:43
  • @orenrevenge Even after you fix that though, the reward function you've picked is optimized by having the snake try to kill itself as quickly as possible. The rest of my answer still applies. – John Doucette Sep 13 at 19:43

Your Answer

 

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.