3
$\begingroup$

I'm building an RL agent using SARSA and Q-Learning for testing its capabilities.

The environment is a 10x10 grid, where it gets a reward of 1 if he reaches the goal while he takes -1 every time he takes a step out of the grid. So, it can freely move out and every time it takes a step outside of the grid it gets -1.

After tuning the main parameters

  • alpha_val: 0.25
  • discount: 0.99
  • episode_length: 50
  • eps_val: 0.5

I get the following plot for 10000 episodes (The plot is sampled every 100 episodes):

enter image description here

But when I look at the plots online I see usually plots like this one:

enter image description here

Since I'm new at RL, I'm asking some comments about my outcome or any type of suggestion if anyone of you think that I'm doing something wrong.

$\endgroup$
1
2
$\begingroup$

Well, the way to know that the agent is actually learning is by looking at its behavior while it performs the task, and by comparing against a known optimal performance.

So, does your agent reaches the goal quickly? Does it step out of the grid frequently? What is the maximum possible sum of rewards / minimum number of steps attainable? Is the agent close to that limit? From your graphic, and if I understood correctly your RL problem, the maximum average reward per step should be close to 1 (depending on the specific environment you are using), so I guess you are not so far from the optimal solution.

Also, probably if you keep training for a longer period, your agent will reach a stable solution that might or might not be optimal. If you keep training after that, your curves surely will look like the ones you found online.

$\endgroup$
2
  • $\begingroup$ Thank you for your answer. Yes, the maximum average reward per episode is 1 and yes, the agent a the end achieve a good average reward. My doubt is that it takes so much time and for more than 5000 episodes (which have length 50 each one) it reach the goal less than 40% of the time. The grid is 10x10 and only the goal gives a reward of +1. I wonder if this problem is caused by the fact that in reality the space of movement is infinite since it can move freely out of the grid taking a -1 of penalty for each step outside. The question is why it takes so many episodes to get a nice reward? $\endgroup$
    – FraMan
    Jan 13 '20 at 9:02
  • $\begingroup$ I also tried for 100k episodes by it is unstable. I noticed that it stay many time stuck just outside the grid and it stay there. Since its actions are (Up,Down,Left,Right,Stay) $\endgroup$
    – FraMan
    Jan 13 '20 at 13:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.