I built a simple AI system that tries to solve the 8 puzzle using DQN. The problem is, if the agent gets only a reward greater than zero when winning, the training will take a long time, so I made a smooth reward function instead: $R=(n/9)^3$, where $n$ is the number of pieces that are in the right position.

The training became quicker but the AI chose to match 7 pieces out of 9 to get a reward of $(7/9)^3/(1-\gamma) = 0.47/(1-\gamma) = 4.7$, for $\gamma=0.9$, choosing to win and getting reward of 1 doesn't make sense to the AI, lowering $\gamma$ will result in the AI to choose instant reward instead of long-term reward, so that will not be very helpful; lowering rewards of non-winning stats will make the training very slow.

So, how do I choose a good reward function?

  • $\begingroup$ Try using Manhattan distance between current position and supposed position for all pieces, use that as negative reward. $\endgroup$ – Brale Dec 9 '19 at 19:03
  • $\begingroup$ Increase the reward for winning. The reward for winning should normally dwarf all other rewards. $\endgroup$ – Recessive Dec 10 '19 at 3:05
  • $\begingroup$ Changed the reward of winning to 5, that didn't improve the results a lot somehow. i like the idea of negative rewards i will give it a go. $\endgroup$ – F0urAt Dec 10 '19 at 16:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.