1
$\begingroup$

I'm training a Deep Q-learning model on a snake game and I would like some ideas on how to improve the model and maybe also efficiency of training it.

The game is currently set to a 12x12 grid, a blue snake with a green head and a red apple. The network is fed with 3x12x12 input parameters (RGB, width, height) and gets a positive reward when an apple is eaten and a negative reward when it collides with something.

It does learn, but plateaus around 12-13 apples per round (on average) after 3 million steps: enter image description here

What I have tried: Giving a partly reward on the steps before a "real" reward. For example:

Step    Action        Reward
N       Go straight      100
N-1     Go straight       50
N-2     Go straight       33
N-3     Go left           25

That was just an idea but it does not seem to work as I hoped.

What else can I try? What I don't want to do is tinker with the game, I just want the visual input and nothing else.

$\endgroup$
1
  • 1
    $\begingroup$ Hi @DanneP and welcome to AI Stack Exchange! If possible, could you supply a few more details, such as the specific variant of the DQN algorithm you are using? That would help us determine what improvements could be made. Thanks! $\endgroup$
    – DeepQZero
    Commented Nov 16, 2022 at 17:51

2 Answers 2

1
$\begingroup$

Here are your options:

  1. Try different starting points in the path.
  2. Try creating more paths to train your snake.
  3. Try using genetic algorithms to "evolve" your algorithm more systematically.
  4. I don't think that snakes eat apples, so try changing the reward to a mouse if you are serious about getting good performance out of your snake.
  5. Some snakes are smarter than others, so try using a cobras, pythons, etc. Do not use titanaboas because they are extinct, probably for a reason.
$\endgroup$
1
$\begingroup$

I would suggest looking at various optimisations used by Deep Mind when they originally developed DQN. You can check their "rainbow" paper for ideas that improved performance.

In addition, I think you are missing the idea of frame stacking, which should help the agent better understand velocity which is currently not in your state representation, although the position of the head will often be a clue. To frame stack, change the representation to include the last few turns (3 turns total might be enough for you) and concatenate the channels. You could also take the opportunity to flatten the representation of a single frame*, since each grid point only ever contains a single entity (e.g. you could use 0 for empty, 1 for apple, -1 for snake body and -0.5 for snake head). That should reduce the number of parameters required for the estimator, which in turn may improve model performance.

Reward values like +100 might be a problem in that they could cause large gradients in a neural network whilst learning them. It is worth checking to see what happens when you normalise your reward values to just +1 for eating an apple. Termination of an episode when the snake crashes into itself, and largish discount factor, $\gamma = 0.99$ should be enough to discourage the snake from hitting itself or walls.


* In DQN they turned the game inputs into greyscale, but if you do that with your current colour choices, all the entities would have the same value. However using greyscale could work for you as well, if you had all the entities display with different colour intensities.

$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .