1
$\begingroup$

Currently I'm working on a continuous control problem using DDPG as my RL algorithm. All in all, things are working out quite well, but the algorithm does not show any tendencies to eliminate the steady state control deviation towards the far end of the episode.

In the graphs you can see what happens:

In the first graph we see the setpoint in yellow and the controlled continuous parameter in purple. In the beginning, the algorithm brings the controlled parameter close to the setpoint fast, but then it ceases its further efforts and does not try to eliminate the remaining steady state error. This control deviation even increases over time.

steady state error not disappearing

In the second graph, the actual reward is depicted in yellow. (Just ignore the other colors.) I use the normalized control deviation to calculate the reward: $r = \frac{\frac{|dev|}{k}}{1+\frac{|dev|}{k}}$.

This gives me a reward that lies within the interval $]0, 1]$ and has a value of $0.5$ when the deviation $dev$ equals the parameter $k$. (That is the parameter $k$ kind of indicates when half of the work is done)

This reward function is relatively steep for the last fraction of the deviation from $k$ to $0$. So it would definitely be worth the effort for the agent to eliminate the residual deviation.

However, it looks like the agent is happy with the existing state and the control deviation never gets eliminated. Eventhough the reward is stuck at ~0.85 instead of the maximum achievable 1. Yellow is the actual reward in each step

Any ideas how to push the agent into some more effort to eliminate the steady state error? (A PID controller would exactly do this by using its I-term. How can I translate this to the RL-algo?)

The state presented to the algo consists of the current deviation and the speed of change (derivatve) of the controlled value. The deviation is not included in the calculation of the reward function, but in the end we wat a flat line with no steady state deviation of course.

Any ideas welcome!

Regards, Felix

Improve this question
$\endgroup$

2 Answers 2

Reset to default
0
$\begingroup$

Could you figure out a workaround for this problem of the steady-state error using DDPG?

I'm currently facing the same problem. My application is Satellite attitude control, and no matter which cost function variation I use, the resulting controlled system maintains a constant steady-state error.

It seems to be a limitation of the algorithm. But I can't understand why this happens. Hence, I've decided to use PPO for now. But would like to know if you could solve this problem using DDPG.

All the best,

Wilson

Improve this answer
$\endgroup$
2
  • $\begingroup$ Hey Wilson, yes, I could eliminate the steady state error with DDPG sucessfully by adding an integral component to the reward. I'll post a link to my master's thesis next week, where this is described in detail. You stated, that this is a DDPG-inherent problem which my not affect e. g. PPO. Do you have any references where to read about this? $\endgroup$
    – opt12
    Commented May 28, 2020 at 7:07
  • $\begingroup$ Hi Felix! That's great news! Yes, I will definitely want to read your master's thesis and understand better how you did it. Yes, a reference I read recently is this article arxiv.org/abs/1804.04154, it is about the attitude control for a UAV. The author compares different RL algorithms with a conventional PID, and you can see there that the DDPG indeed presented a steady-state error and the PPO was the one that performed the best. However, in my own experiments, the one that worked better for me so far was the improved version of DDPG, called TD3. $\endgroup$
    – wilsonsamarques
    Commented Jun 1, 2020 at 14:31
0
$\begingroup$

After some research on the subject, I found a solution to my problem of steady state errors in continuous control using DDPG:

I added a reward component based on the integral of the error. This yields maximum reward if the error integral is zero and yields lower rewards down to 0 if the integral has some value in it.

This integral component in the reward was found to be very effective, but introduces some overshoot after set-point changes. By limiting the integral to quite small quantities, this behavior could be overcome.

All these findings are detailed out in the "Reward Engineering" section of my master's thesis. Please have a look into https://github.com/opt12/Markov-Pilot/tree/master/thesis

I'll be glad to get feedback on it.

Regards, Felix

Improve this answer
$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .