3
$\begingroup$

I'm building an RL agent using SARSA and Q-Learning for testing its capabilities.

The environment is a 10x10 grid, where it gets a reward of 1 if he reaches the goal while he takes -1 every time he takes a step out of the grid. So, it can freely move out and every time it takes a step outside of the grid it gets -1.

After tuning the main parameters

  • alpha_val: 0.25
  • discount: 0.99
  • episode_length: 50
  • eps_val: 0.5

I get the following plot for 10000 episodes (The plot is sampled every 100 episodes):

enter image description here

But when I look at the plots online I see usually plots like this one:

enter image description here

Since I'm new at RL, I'm asking some comments about my outcome or any type of suggestion if anyone of you think that I'm doing something wrong.

Improve this question
$\endgroup$
1

2 Answers 2

Reset to default
2
$\begingroup$

Well, the way to know that the agent is actually learning is by looking at its behavior while it performs the task, and by comparing against a known optimal performance.

So, does your agent reaches the goal quickly? Does it step out of the grid frequently? What is the maximum possible sum of rewards / minimum number of steps attainable? Is the agent close to that limit? From your graphic, and if I understood correctly your RL problem, the maximum average reward per step should be close to 1 (depending on the specific environment you are using), so I guess you are not so far from the optimal solution.

Also, probably if you keep training for a longer period, your agent will reach a stable solution that might or might not be optimal. If you keep training after that, your curves surely will look like the ones you found online.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Thank you for your answer. Yes, the maximum average reward per episode is 1 and yes, the agent a the end achieve a good average reward. My doubt is that it takes so much time and for more than 5000 episodes (which have length 50 each one) it reach the goal less than 40% of the time. The grid is 10x10 and only the goal gives a reward of +1. I wonder if this problem is caused by the fact that in reality the space of movement is infinite since it can move freely out of the grid taking a -1 of penalty for each step outside. The question is why it takes so many episodes to get a nice reward? $\endgroup$
    – FraMan
    Commented Jan 13, 2020 at 9:02
  • $\begingroup$ I also tried for 100k episodes by it is unstable. I noticed that it stay many time stuck just outside the grid and it stay there. Since its actions are (Up,Down,Left,Right,Stay) $\endgroup$
    – FraMan
    Commented Jan 13, 2020 at 13:11
0
$\begingroup$

This could mostly be depending on your exploration rate. Consider this exploration threshold:

exploration_threshold = EPS_END + (EPS_START - EPS_END) * math.exp(-1. * episodes / EPS_DECAY)
sample = random.random()
if sample > exploration_threshold:
      # take action using the policy
    else:
      # take action at random

Here the EPS_END=0.9, EPS_START=0.05 are starting and ending exploration probabilities. This will force the agent explore the action space with a decaying high probability and gain experiences. After a while it will choose actions from the policy with a high probability, and as the policy converges, it yeilds the maximum expected utility. This will help learn faster and cause the decaying exponential shape of the convergence plot.

Improve this answer
$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .