0
$\begingroup$

I have made a Deep Q Network for the game snake but unfortunately, the snake exhibits some unwanted behavior. It generally does quite well but sometimes it gets stuck in an infinite loop that it can't escape and at the start of the game it takes a very long route to the apple rather than taking a more direct route.

The discount factor per time step is 0.99. The Snake gets a reward of +9 for getting an Apple and -1 for dying. Does anybody have any recommendations on how I should tune the hyperparameters/reward function to minimize this unwanted behavior?

I was thinking that reducing the discount factor may be a good idea?

Improve this question
$\endgroup$
1
$\begingroup$

Your network finds the infinite loop and notices that this has the best reward (0). This is probably because it hasn't found a path to eating the apple (through exploration).

Reducing the discount factor will only make the long term rewards less valuable. So it will learn eating the apple even slower.

I don't know what you are using as inputs for your network, but maybe changing your reward system could help. For example, you could give your network a reward if it advances in the direction of the apple. This way, your network will be encouraged to find the apple more than it is now.

Improve this answer
$\endgroup$
5
  • $\begingroup$ I'm using a convolutional neural network so I don't think that approach would be easily implemened I think. $\endgroup$ – KaneM Mar 9 '20 at 14:14
  • $\begingroup$ My thinking was that reducing the discount factor would incentivise the snake to get the reward sooner rather than later because if it waits a while to get it then it will recieve less reward $\endgroup$ – KaneM Mar 9 '20 at 14:15
  • $\begingroup$ How is this not easily implemented? After each decision you measure distance to the apple, if it is smaller than last distance you get reward +1. $\endgroup$ – Lustwelpintje Mar 9 '20 at 19:42
  • $\begingroup$ You are right. I feel that may cause some unwanted behavior though. It might keep going back and forth between the apple and not actually getting it as it can keep getting reward this way. I will try it out and see if it works $\endgroup$ – KaneM Mar 10 '20 at 10:54
  • 1
    $\begingroup$ This is of course the problem with reward shaping - spare rewards. Try this out, using different rewards for moving away form the apple, towards the apple, eating the apple, and dying. you can even give a negative reward for not eating the apple within a certain time period $\endgroup$ – Lustwelpintje Mar 10 '20 at 11:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.