1
$\begingroup$

My environment is set up so that my self-driving agent can take maximum of 400 steps (which is the end goal) before it resets with a completion reward. Despite attaining the end goal during the $\epsilon$-greedy stage, it still kills/crashes itself in subsequent episodes.

I would like to know if this common in RL (D3QN) scenarios.

A graph showing episodes vs steps has been placed below.

enter image description here

As one can see, the agent reaches 400 steps in episode 1000. But, in the subsequent episode, it falls down below 50 steps.

$\endgroup$
10
  • $\begingroup$ Does your graph show steps during training? $\endgroup$ – Neil Slater Dec 3 '20 at 21:08
  • $\begingroup$ Right, although the epsilon is gradually decreasing and therefore it is also exploiting instead of just exploring. $\endgroup$ – desert_ranger Dec 3 '20 at 22:12
  • $\begingroup$ For the graph, can you say what epsilon starts at for epsilon starts at for episode 1, and ends at for episode ~2200? $\endgroup$ – Neil Slater Dec 4 '20 at 7:12
  • $\begingroup$ What agent are you using? Which environment (and maybe library)? Which hyper-parameters? I think you should provide also these details, because they may be useful. $\endgroup$ – nbro Dec 4 '20 at 11:24
  • 1
    $\begingroup$ Thanks for accepting the answer. I have extended the answer a little with another key point that you may want to consider. $\endgroup$ – Neil Slater Dec 5 '20 at 16:47
1
$\begingroup$

Your graph looks to me like a typical learning curve plotted for training process in reinforcement learning.

Looking at it in detail I can say:

  • There is clearly some learning occurring.

  • There is a strong random element throughout. As you say the epsilon is reduced to near zero by episode 2000, and I would assume a driving simulation is mostly deterministic but with a lot of state hidden from the agent, this mainly implies that episodes are started in different states.

  • There may be an effect from continued learning that causes the agent performance to vary so much. However, it is more likely towards the end that the agent is encountering new never-seen-before states and making poor decisions in them.

  • The resulting graph matches to my experience of training DQN agents where some of the hyperparameters are off. Probably you need to do some kind of exploration or search through those parameters.

From your comments:

Perhaps the model is subsequently overfitting? Or maybe there is some catastrophic forgetting?

These are possibilities that I cannot rule out by looking at the graph.

However, I think it is more likely that you have some agent design or hyperparameters that are not a good fit to the problem. Those hyperparameters could be almost anything - not enough episodes to cover variation, neural network too simple, neural network too complicated, epsilon decay too fast, poor choice of regularisation, experience replay memory too small, poor choice of optimiser, etc etc.

It is also possible you have limited the state model to the point that learning is hard or even impossible. A common beginner's mistake here is to use static sensor information such as a single current screenshot on each timestep, so the agent has no way to assess how fast it is going, or which way it is already turning etc (if your input includes direct knowledge of current speed, turning etc then this is probably not a problem for you, I am just raising one of many possibilities).

One thing that may help you understand the agent's learning performance better is instead of plotting the learning episode graph, plot a test graph, perhaps once every ~100 episodes, where you run some number of episodes without learning and with $\epsilon = 0$, and take average of the metrics you are interested in. This will remove some of the randomness from the values you are plotting and give you a much better sense of training progress than your current plot.

With Q learning, you may also get significant benefit by setting a non-zero minimum $\epsilon$, e.g. $\epsilon = \text{max}(\text{eps_decay}(n), 0.01)$ where $\text{eps_decay}(n)$ is your current epsilon decay function for step $n$. That is because the off-policy updates with exploration are still helpful at all stages of learning.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.