0
$\begingroup$

I built a simple X*Y grid world environment to learn and then trained my agent over it. All worked fine and the agent learned as well. Let me give some detail about the environment.

Environment:

  • A 4x4 grid world with episode starting at (0,0) and terminal state (3,3)
  • Four actions: Left, Up, Right, Down
  • The reward of -1 for moving into new state from the previous state to a new state. The reward of 0 when reaching the terminal state. The reward of -2 for bouncing off of the boundary
  • Epsilon-greedy scheme for action selection.

All works fine, and the following are the learning results of the agent.

Mean reward of QL-agent after 50 simulation runs

Later I ran a test run of my TRAINED QL-agent where I used greedy action selection. All I could see in every episode was that my agent start from (0,0), take right to move to (1,0), and then take left to move back to (0,0) again and this goes on and on and on... I check the Q table and it makes sense because the Q-values for these actions justifies such behaviour. But this is not a practical agent should be doing.

Improve this question
$\endgroup$
6
  • 1
    $\begingroup$ I don't think you will get an answer due to the data you have provided. The simple conclusion is that you have a mistake somewhere in the implementation. I would suspect something in how you have stored/interpreted the Q table or re-implemented the environment when moving from training to testing. $\endgroup$ – Neil Slater Jul 30 '20 at 8:15
  • 1
    $\begingroup$ Agreed with Neil Slater, and I also have another question. The return that you converge to is around -20, which seems very low to me. Shouldn't your agent be able to solve this in like 6 steps, achieving a return of -5? $\endgroup$ – harwiltz Jul 30 '20 at 13:41
  • $\begingroup$ @harwiltz I checked and found it reach min. a score of -9, with a Toral reward of -8. But mostly the number of steps in 10s or 20s for a trained agent. $\endgroup$ – Kashan Jul 30 '20 at 14:25
  • $\begingroup$ So is it possible that in your evaluation episodes the agent goes back and forth around 15 times before finally going to the goal? $\endgroup$ – harwiltz Jul 30 '20 at 17:48
  • 1
    $\begingroup$ Still, this is very strange behavior, especially since there's a negative reward at each timestep. Sounds like your learning algorithm might not be working correctly, since it shouldn't be too hard for it to converge to an optimal (or at least nearly optimal) policy. $\endgroup$ – harwiltz Jul 30 '20 at 17:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.