This is a common problem and does have a name. It is called "catastrophic forgetting" (link is just to a paper I found randomly when searching for the term).
What could be causing this undesirable behavior?
It happens only when using function approximation for value functions (e.g. a neural network trained to learn Q values), and is caused by the agent's own success. After a while the only samples that you will train with will be near-optimal high return cases, and a neural network will optimise the Q value approximation only for that recent data in order to get the best loss. This will usually mean much poorer predictions on unseen but different data, as the network overfits to states and actions only seen in the optimal policy. Eventually the agent will explore into an area where its predictions are way off. Then, because Q learning also uses its own predictions to bootstrap new Q values, this can start a runaway feedback process where the agent starts to choose a suboptimal path through the environment.
Inside the hidden layers and weights, the neural network may have lost the ability to differentiate well between the states on the old, worse path and the newer better ones. It didn't need to differentiate, because it stopped needing to reduce loss on any data about the old states. So it will also start incorrectly associating the now poor results with the more optimal paths. It will behave at least partially as if the correct policy was set by the overfit Q predictions, but the values need adjusting - so as well as (correctly) reducing its value predictions of the suboptimal paths it has just encountered, it will also (incorrectly) reduce the value predictions of the optimal paths.
Sometimes, the swing back to receiving lower returns during this process is so strong and/or incorrectly associated with the high return states along with the low ones, that the agent never properly recovers. Other times, the agent can re-learn fully and you get random oscillations over time between optimal and non-optimal agent behaviour.
How can I combat it?
This is still an ongoing area of research. Here are a couple of things that have worked for me:
Low learning rates, and defences against sudden large gradients (e.g. gradient clipping).
Regularisation. Sadly dropout seems not to work in RL, but weight decay is still useful to prevent over-fitting, and it also helps combat catastrophic forgetting because it prevents bootstrap estimates of long-unseen state/action combinations from returning radically different Q values to the rest of the system.
Keep some early experience around from when the agent was still performing badly - this allows the agent to still train with some bad cases and prevents the Q function predicting that "everything is awesome" because it still has examples to learn from where this is not the case.
For simple environments, such as inverted pendulum, just keeping some very early fully random behaviour in the experience replay table is enough. For instance if you have a table with 10000 observations (of $s, a, r, s'$), keep 1000 of the first experiences in that table and don't discard them when the table is full. For more complex environments, this is not so useful, as the early random behaviour is too far removed from what the agent learns.
The DQN "rainbow" paper uses prioritized experience replay to focus on areas where Q value predictions from the NN are not matching the observations.
