Reward does not increase for a maze escaping problem with DQN

Question

I am using deep reinforcement learning to solve a classic maze escaping task, similar to the implementation provided here, except the following three key differences:

1. instead of using a numpy array as the input of a standard maze escaping task, I am feeding the model with an image at each step; the image is a 1300 * 900 RGB image, so it is not too small.

2. reward:

   • each valid move has a small negative reward (penalize long move)
   • each invalid move has a big negative reward (run into other objects or boundaries)
   • Each blocked move has the minimal reward (not common)
   • Find the remote detectors’ defect has a positive reward (5)

3. I tweaked the parameters of replay memory, reduced the size of the replay memory buffer.

Regarding the implementation, I basically do not change the agent setup except the above items, and I implemented my env to wrap my customized maze.

But the problem is that, the accumulated reward (first 200 rounds of successful escaping) does not increase:

And the number of steps it takes to escape one maze is also stable somewhat:

Here are my question, on which aspect I could start to look at to optimize my problem? Or is it still too early and I will need to train more time?

Answer 1

You should use an algorithm to try doing a solve for the maze optimally, maybe A* algorithm. If the optimal steps is also in the range of your network, your network may have reached it's best. If the optimal step is much less, you can try increasing the step penalty and increasing the reward for reaching the end. Hope you can succeed in this problem.