I was trying to implement a DQN without experience reply memory and the agent is not learning anything at all. I know from readings; experience reply is used for stabilizing gradients. But how important is experience reply in DQN and similar rl algorithms. If the model needs to learn from memory why dont we use a recurrent network which has inbuilt memory to it? What is the advantage of experience reply over a recurrent memory?