3
$\begingroup$

This is a q-learning snake using a neural network as a q function aproximator and I'm losing my mind here the current model it's worst than the initial one.

The current model uses a 32x32x32 MLPRegressor from scikit-learn using relu as activation function and the adam solver.

The reward function is like following:

  • death reward = -100.0
  • alive reward = -10.0
  • apple reward = 100.0

The features extracted from each state are the following:

  1. what is in front of the snake's head(apple, empty, snake)
  2. what is in the left of the snake's head
  3. what is in the right of the snake's head
  4. euclidian distance between head and apple
  5. the direction from head to the apple measured in radians
  6. length of the snake

One episode consists of the snake playing until it dies, I'm also using in training a probability epsilon that represent the probability that the snake will take a random action if this isn't satisfied the snake will take the action for which the neural network gives the biggest score, this epsilon probability gradually decrements after each iteration.

The episode is learned by the regressor in reverse order one statet-action at a time.

However the neural network fails too aproximate the q function, no matter how many iterations the snake takes the same action for any state.

Things I tried:

  • changing the structure of the neural network
  • changing the reward function
  • changing the features extracted, I even tried passing the whole map to the network

Code (python): https://pastebin.com/57qLbjQZ

| improve this question | |
$\endgroup$
  • $\begingroup$ Can you provide a snippet of your code. $\endgroup$ – quintumnia Sep 12 '18 at 21:10
  • $\begingroup$ the pastebin link to the code its in the post $\endgroup$ – oren revenge Sep 12 '18 at 21:18
  • $\begingroup$ @orenrevenge I've updated my answer to address the code you posted too. $\endgroup$ – John Doucette♦ Sep 13 '18 at 19:44
2
$\begingroup$

There are two problems here.

  1. The code you posted doesn't incrimentally train your multilayer perceptron. Instead, it effectively re-randomizes the weights, and then re-fits the model each time you call .fit() at lines 35 & 54. Using SKLearn's _fit() function with Incremental=true might solve this, or you can package up the data into a larger batch, and train on that offline instead after several episodes.

  2. Your reward function makes it painful to be alive, and doesn't give enough benefits through the Apples to make up for this. There are 100 squares that could contain the apple. On average, the apple will spawn about 5 squares from the snake in each direction. Since the snake can't move diagonally, that's 10 moves (5 left/right, 5 up/down). That means that if the snake plays perfectly, then on average, it might be able to get zero reward total. In practice, the snake will not play perfectly. This means living gives negative expected reward.

In contrast, if the snake can kill itself, it will stop getting negative rewards. The reward function you've used is maximized by getting big enough to run into your own tail as fast as possible. The snake should be able to do this after eating 3 apples I think. There is some incentive to hunt for food well, but not much compared with hitting your own tail as soon as possible.

If you want the snake to learn to hunt for the food, reduce the penalty for being alive to -1, or even -0.1. The snake will be much more responsive to signals from the food.

| improve this answer | |
$\endgroup$
  • $\begingroup$ there are no walls in the game, the moment the snake goes on the edge of the map it reappears on the other end so basically it learns to never die which seems a little contradictory since he gets negative rewards for being alive $\endgroup$ – oren revenge Sep 13 '18 at 7:20
  • $\begingroup$ @orenrevenge, you say that an episode lasts until the snake dies. How can it die? Only by hitting itself? $\endgroup$ – John Doucette♦ Sep 13 '18 at 7:21
  • $\begingroup$ yes thats correct $\endgroup$ – oren revenge Sep 13 '18 at 7:23
  • $\begingroup$ @orenrevenge This is really more of an implementation question I think, and you might have better luck on the main StackOverflow site. Looking at your code, I think your mistake is in fit_episode, on lines 35 & 53. MLPRegressor.fit() promises to produce a trained model for the data you give it, but makes no promises about preserving the weights learned in past fits. See the SKLearn sourcecode: github.com/scikit-learn/scikit-learn/blob/f0ab589f/sklearn/… By default, the fit is not incremental. Using _fit with incremental=true might fix this. $\endgroup$ – John Doucette♦ Sep 13 '18 at 19:43
  • $\begingroup$ @orenrevenge Even after you fix that though, the reward function you've picked is optimized by having the snake try to kill itself as quickly as possible. The rest of my answer still applies. $\endgroup$ – John Doucette♦ Sep 13 '18 at 19:43
1
$\begingroup$

Assume you are the snake.

In front of you is empty. Left of you is empty. Right of you is empty. The distance to the apple is 4. The apple straight in front of you. Your length is 20.

Can you make a good decision with this input? In which direction would you go to achieve maximum score?

From the given input, you could go straight forward to the apple. But that might be a failure and lead to death.

IMHO, the input state is simply not enough to make a good decision, because

a) the snake doesn't even know in which direction it's currently moving.

b) the snake has no idea about where its body is

The situation could look like this:

Situation for the snake

The only way for the snake to move out of this trap is as indicated by the arrow, so that the tail frees the way out just in time. Your neural network does not have the necessary input to make that decision.

| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.