# Should my agent be taking varying number of steps?

My environment is set up so that my self-driving agent can take maximum of 400 steps (which is the end goal) before it resets with a completion reward. Despite attaining the end goal during the $$\epsilon$$-greedy stage, it still kills/crashes itself in subsequent episodes.

I would like to know if this common in RL (D3QN) scenarios.

A graph showing episodes vs steps has been placed below. As one can see, the agent reaches 400 steps in episode 1000. But, in the subsequent episode, it falls down below 50 steps.

• Does your graph show steps during training? – Neil Slater Dec 3 '20 at 21:08
• Right, although the epsilon is gradually decreasing and therefore it is also exploiting instead of just exploring. – desert_ranger Dec 3 '20 at 22:12
• For the graph, can you say what epsilon starts at for epsilon starts at for episode 1, and ends at for episode ~2200? – Neil Slater Dec 4 '20 at 7:12
• What agent are you using? Which environment (and maybe library)? Which hyper-parameters? I think you should provide also these details, because they may be useful. – nbro Dec 4 '20 at 11:24
• Thanks for accepting the answer. I have extended the answer a little with another key point that you may want to consider. – Neil Slater Dec 5 '20 at 16:47

Your graph looks to me like a typical learning curve plotted for training process in reinforcement learning.

Looking at it in detail I can say:

• There is clearly some learning occurring.

• There is a strong random element throughout. As you say the epsilon is reduced to near zero by episode 2000, and I would assume a driving simulation is mostly deterministic but with a lot of state hidden from the agent, this mainly implies that episodes are started in different states.

• There may be an effect from continued learning that causes the agent performance to vary so much. However, it is more likely towards the end that the agent is encountering new never-seen-before states and making poor decisions in them.

• The resulting graph matches to my experience of training DQN agents where some of the hyperparameters are off. Probably you need to do some kind of exploration or search through those parameters.

One thing that may help you understand the agent's learning performance better is instead of plotting the learning episode graph, plot a test graph, perhaps once every ~100 episodes, where you run some number of episodes without learning and with $$\epsilon = 0$$, and take average of the metrics you are interested in. This will remove some of the randomness from the values you are plotting and give you a much better sense of training progress than your current plot.
With Q learning, you may also get significant benefit by setting a non-zero minimum $$\epsilon$$, e.g. $$\epsilon = \text{max}(\text{eps_decay}(n), 0.01)$$ where $$\text{eps_decay}(n)$$ is your current epsilon decay function for step $$n$$. That is because the off-policy updates with exploration are still helpful at all stages of learning.