1
$\begingroup$

I am trying to create a Car Following model, for which i am using DDPG. My action is acceleration bounded in a range of [-3,3] m/s2. While training the model, for every state it gives a single acceleration value i.e. 3 (or sometimes -3). Actor and critic loss

It can be clearly seen that my actor is performing really bad.

What can be done to resolve this issue?

$\endgroup$
3
  • 1
    $\begingroup$ "It can be clearly seen that my actor is performing really bad." how? $\endgroup$
    – Alberto
    Commented Jun 11 at 14:53
  • $\begingroup$ Because the actor loss is increasing. $\endgroup$ Commented Jun 11 at 16:25
  • 1
    $\begingroup$ That is not a valid sign of "not learning", it might just be due to the maximization to rewards, which leads to higher temporal difference reiforcement $\endgroup$
    – Alberto
    Commented Jun 11 at 17:04

1 Answer 1

0
$\begingroup$

Since I cannot yet comment, this is in the Form of an answer, even though it is mostly speculation.

What you're observing can be a lot of different things. It might be helpful to look into the return(discounted sum of rewards) of your model or manually inspect the learnt behavior to verify it is not performing / learning better than expected. It might also helpful to think about what amount of return you expect an optimal policy to achieve.

To improve learning, there are several steps you can take. First test your Learner on a different environment, e.g. from Gymnasium, to see if it is actually learning properly. Another step might be to test the environment by using a pre-built implementation of a Learning algorithm like CleanRL's DDPG single file implementation .

If both of these show that your Learner and your Environment function as expected, another step to take is normalizing the observations. A third idea is to look into the way you predict your action. You can directly use the output of the NN as the action, and just clip it to the desired range if the prediction ends up being outside of the interval. However, it tends to be tricky to get a neural network to properly predict values of a desired interval in my experience. Depending on how your output layer is structured, it might be a better idea to have your model predict a value between -1 and 1 by applying a tanh-Activation and rescaling it to [-3,3] afterwards. You can also interpret the output of you NN as the mu of a distribution, from which you draw the actual action. And lastly a simple alternative is discretizing your action space, e.g. to -3, 0, 3, which avoids the whole problem of a continuous bounded actionspace entirely.

Of course, there is always a chance that you're just of on your hyperparameters as well, so you can change them around or use a tool like Optuna to automatically identify good hyperparameters.

OpenAI Tutorial on DDPG provides a nice mix between theoretical background of RL Algorithms and thoughts on implementing them pragmatically.

$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .