0
$\begingroup$

I have made a Deep Q Network for the game snake but unfortunately, the snake exhibits some unwanted behavior. It generally does quite well but sometimes it gets stuck in an infinite loop that it can't escape and at the start of the game it takes a very long route to the apple rather than taking a more direct route.

The discount factor per time step is 0.99. The Snake gets a reward of +9 for getting an Apple and -1 for dying. Does anybody have any recommendations on how I should tune the hyperparameters/reward function to minimize this unwanted behavior?

I was thinking that reducing the discount factor may be a good idea?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

Your network finds the infinite loop and notices that this has the best reward (0). This is probably because it hasn't found a path to eating the apple (through exploration).

Reducing the discount factor will only make the long term rewards less valuable. So it will learn eating the apple even slower.

I don't know what you are using as inputs for your network, but maybe changing your reward system could help. For example, you could give your network a reward if it advances in the direction of the apple. This way, your network will be encouraged to find the apple more than it is now.

Improve this answer
$\endgroup$
5
  • $\begingroup$ I'm using a convolutional neural network so I don't think that approach would be easily implemened I think. $\endgroup$
    – KaneM
    Mar 9, 2020 at 14:14
  • $\begingroup$ My thinking was that reducing the discount factor would incentivise the snake to get the reward sooner rather than later because if it waits a while to get it then it will recieve less reward $\endgroup$
    – KaneM
    Mar 9, 2020 at 14:15
  • $\begingroup$ How is this not easily implemented? After each decision you measure distance to the apple, if it is smaller than last distance you get reward +1. $\endgroup$
    – Lustwelpintje
    Mar 9, 2020 at 19:42
  • $\begingroup$ You are right. I feel that may cause some unwanted behavior though. It might keep going back and forth between the apple and not actually getting it as it can keep getting reward this way. I will try it out and see if it works $\endgroup$
    – KaneM
    Mar 10, 2020 at 10:54
  • 1
    $\begingroup$ This is of course the problem with reward shaping - spare rewards. Try this out, using different rewards for moving away form the apple, towards the apple, eating the apple, and dying. you can even give a negative reward for not eating the apple within a certain time period $\endgroup$
    – Lustwelpintje
    Mar 10, 2020 at 11:18

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .