# DQN + HER, TD Error spiked then success rate plummets. What went wrong?

TL;DR: I trained a DQN + HER model using stable-baselines library for a custom environment. I noticed that in most runs, sometimes the TD-Error will spike and then the success rate of my model plummets (which can be seen in the attached tensorboard logs). What is the problem here and what can I do to solve it?

Note: I log the values to tensorboard every 10000 steps for memory reasons, so each point in the graph is an average of those 10000 steps. I cannot log the network loss, but I hope the TD-Error can substitute for that.

Long version: I am currently trying to implement a variant of this paper: PCGRL (Paper TL;DR: they use deep reinforcement learning to perform procedural level generation. They use a game level as state, a command to modify the level as action, and the quality of the level as reward. They use PPO as the algorithm). In their version, the goal of the agent is to produce the most complex level possible (the complexity is calculated using a certain formula). In my version, I make the agent produce a level based on user's specification. For example, the user can now specify how many enemies appear in the map and the minimum distance between the starting position to the level goal. This converts the problem into a multi-goal reinforcement learning (hence the HER + DQN).

Currently, this is my setup:

• Stable-baselines library to train the model
• DQN + HER to train the model. I use DQN because it is the only algorithm in stable-baselines that is off-policy (required for HER) and can handle the environment's discrete action spaces (so no DDPG, SAC, or TD3).
• DQN: using dueling deep Q network architecture with gamma = 0.99, learning rate = 5e-4, batch size = 32, buffer size = 100000, target network updated every 100 steps, exploration fraction = 0.1
• HER: uses future goal selection strategy and 4 sampled goals
• Network input: an 11 x 7 "image" where each "pixel" is a one-hot encoded representation of the tile in that position. Plus, a vector of user specification (desired goal) and the level's specification (achieved goal), which is scaled so that each element is in range [0, 1].
• Reward: 0 if the agent achieves the desired goal, -1 otherwise.
• The environment terminates if the desired goal is achieved or if it fails to achieve the desired goal after 154 steps.

I have tried to make a toy version of the zelda problem mentioned in the PCGRL paper. In this version, the level only consists of empty and wall tiles. The user specifies the number of wall tiles that the level should have. Basically now it's like a 2D bit flipping environment and you determine the number of on bits.

When I trained my model I find this phenomenon where the success rate and the TD-error of my model starts to rise, but suddenly the TD-error spikes and then the success rate of my model drops significantly (it even performs worse than random chance). One thing that I have tried that sort of works is by making the target network update every 10000 steps. It kind of delays when the spike happened and my model managed to hit 0.75 success rate before the TD-Error spiked and the success rate plummets again. I don't think increasing the target network update again is a good idea.

Does anyone know why this happened? Is there anything that I can do to solve it? I am pretty new to Deep RL so any help is very appreciated.