2
$\begingroup$

I am using deep reinforcement learning to solve a classic maze escaping task, similar to the implementation provided here, except the following three key differences:

  1. instead of using a numpy array as the input of a standard maze escaping task, I am feeding the model with an image at each step; the image is a 1300 * 900 RGB image, so it is not too small.

  2. reward:

    • each valid move has a small negative reward (penalize long move)
    • each invalid move has a big negative reward (run into other objects or boundaries)
    • Each blocked move has the minimal reward (not common)
    • Find the remote detectors’ defect has a positive reward (5)

  3. I tweaked the parameters of replay memory, reduced the size of the replay memory buffer.

Regarding the implementation, I basically do not change the agent setup except the above items, and I implemented my env to wrap my customized maze.

But the problem is that, the accumulated reward (first 200 rounds of successful escaping) does not increase:

enter image description here

And the number of steps it takes to escape one maze is also stable somewhat:

enter image description here

Here are my question, on which aspect I could start to look at to optimize my problem? Or is it still too early and I will need to train more time?

Improve this question
$\endgroup$
1
$\begingroup$

You should use an algorithm to try doing a solve for the maze optimally, maybe A* algorithm. If the optimal steps is also in the range of your network, your network may have reached it's best. If the optimal step is much less, you can try increasing the step penalty and increasing the reward for reaching the end. Hope you can succeed in this problem.

Improve this answer
$\endgroup$
  • $\begingroup$ Hello, thank you for the answer. I am thinking to use a DQN because 1) I am expecting to learn from, say, 1000 images, and has a model which can help to quickly solve the maze on other 5000 images. 2) my input are images of raw pixels. Therefore, I do need certain convolutional layers anyway, right? $\endgroup$ – lllllllllllll Jun 3 '19 at 11:39
  • $\begingroup$ if you use images, you should use CNN. $\endgroup$ – Clement Hui Jun 3 '19 at 13:03
  • $\begingroup$ Yes, that's right. The DQN is stacked with three convolutional layers followed by two fully connected layer. $\endgroup$ – lllllllllllll Jun 3 '19 at 13:22
  • $\begingroup$ If the pathways are 1 pixels apart, the network is optimized already. But if it is wider apart, you may have to change the rewards and penalties. $\endgroup$ – Clement Hui Jun 4 '19 at 6:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.