Your graph looks to me like a typical learning curve plotted for training process in reinforcement learning.
Looking at it in detail I can say:
There is clearly some learning occurring.
There is a strong random element throughout. As you say the epsilon is reduced to near zero by episode 2000, and I would assume a driving simulation is mostly deterministic but with a lot of state hidden from the agent, this mainly implies that episodes are started in different states.
There may be an effect from continued learning that causes the agent performance to vary so much. However, it is more likely towards the end that the agent is encountering new never-seen-before states and making poor decisions in them.
The resulting graph matches to my experience of training DQN agents where some of the hyperparameters are off. Probably you need to do some kind of exploration or search through those parameters.
From your comments:
Perhaps the model is subsequently overfitting? Or maybe there is some catastrophic forgetting?
These are possibilities that I cannot rule out by looking at the graph.
However, I think it is more likely that you have some agent design or hyperparameters that are not a good fit to the problem. Those hyperparameters could be almost anything - not enough episodes to cover variation, neural network too simple, neural network too complicated, epsilon decay too fast, poor choice of regularisation, experience replay memory too small, poor choice of optimiser, etc etc.
It is also possible you have limited the state model to the point that learning is hard or even impossible. A common beginner's mistake here is to use static sensor information such as a single current screenshot on each timestep, so the agent has no way to assess how fast it is going, or which way it is already turning etc (if your input includes direct knowledge of current speed, turning etc then this is probably not a problem for you, I am just raising one of many possibilities).
One thing that may help you understand the agent's learning performance better is instead of plotting the learning episode graph, plot a test graph, perhaps once every ~100 episodes, where you run some number of episodes without learning and with $\epsilon = 0$, and take average of the metrics you are interested in. This will remove some of the randomness from the values you are plotting and give you a much better sense of training progress than your current plot.
With Q learning, you may also get significant benefit by setting a non-zero minimum $\epsilon$, e.g. $\epsilon = \text{max}(\text{eps_decay}(n), 0.01)$ where $\text{eps_decay}(n)$ is your current epsilon decay function for step $n$. That is because the off-policy updates with exploration are still helpful at all stages of learning.
