1
$\begingroup$

I'm trying to solve the OpenAI's CarRacing-v0 environment with the DDPG algorithm. I've observed that after a period of learning, the agent's performance starts to deteriorate slowly. For some hyperparameter configurations this is followed by a rebound and again a slump. Here's what a typical reward plot looks like:1

Since I'm new to reinforcement learning (this is my first shot at it), I don't know if this a common phenomenon. I know of catastrophic forgetting, but I believe that's not the case here, since this is more akin to a "languishing dementia". As far as I understand, "catastrophic forgetting" is an abrupt event, which contrasts with a gradual change I've been seeing in my attempts.

Is this some kind of general phenomenon with coverage in the existing literature or is this rather a quirk of my specific setup (algorithm + hyperparameters) for which the solution would be "change the setup"?

For reference, the implementation I'm using: https://github.com/hirekk/pytorch-rl

Improve this question
$\endgroup$
2
  • $\begingroup$ This could well be catastrophic forgetting. A few weeks ago, I came across a paper (I don't remember the name now, but if I find it I will post this comment as a answer below) that actually states that similar patterns of the performance can be due to catastrophic forgetting within the same environment, which is actually a multi-task environment. $\endgroup$
    – nbro
    Jan 28, 2021 at 12:27
  • $\begingroup$ @nbro Thank you very much, I wasn't aware that catastrophic forgetting can manifest that way. If this paper you mention managed to find its way back into your hands, I'd appreciate greatly you pointing me to it. $\endgroup$
    – Hirek Kubica
    Jan 28, 2021 at 20:08

0

Reset to default

You must log in to answer this question.