I built a simple X*Y grid world environment to learn and then trained my agent over it. All worked fine and the agent learned as well. Let me give some detail about the environment.
- A 4x4 grid world with episode starting at (0,0) and terminal state (3,3)
- Four actions: Left, Up, Right, Down
- The reward of -1 for moving into new state from the previous state to a new state. The reward of 0 when reaching the terminal state. The reward of -2 for bouncing off of the boundary
- Epsilon-greedy scheme for action selection.
All works fine, and the following are the learning results of the agent.
Later I ran a test run of my TRAINED QL-agent where I used greedy action selection. All I could see in every episode was that my agent start from (0,0), take right to move to (1,0), and then take left to move back to (0,0) again and this goes on and on and on... I check the Q table and it makes sense because the Q-values for these actions justifies such behaviour. But this is not a practical agent should be doing.