4
$\begingroup$

In section 3 of the paper Continuous control with deep reinforcement learning, the authors write

As detailed in the supplementary materials we used an Ornstein-Uhlenbeck process (Uhlenbeck & Ornstein, 1930) to generate temporally correlated exploration for exploration efficiency in physical control problems with inertia (similar use of autocorrelated noise was introduced in (Wawrzynski, 2015)).

In section 7, they write

For the exploration noise process we used temporally correlated noise in order to explore well in physical environments that have momentum. We used an Ornstein-Uhlenbeck process (Uhlenbeck & Ornstein, 1930) with θ = 0.15 and σ = 0.2. The Ornstein-Uhlenbeck process models the velocity of a Brownian particle with friction, which results in temporally correlated values centered around 0.

In a few words, what is the Ornstein-Uhlenbeck process? How does it work? How exactly is it used in DDPG?

I want to implement the Deep Deterministic Policy Gradient algorithm, and, in the initial actions, noise has to be added. However, I cannot understand how this Ornstein-Uhlenbeck process works. I have searched the internet, but I have not understood the information that I found.

$\endgroup$
  • 1
    $\begingroup$ I have edited your post to clarify your question. Please, next time, provide some context, as I did with your post. $\endgroup$ – nbro Aug 21 at 22:25
  • $\begingroup$ You are amazing! Thanks a lot!!! $\endgroup$ – dani Aug 21 at 22:41
2
$\begingroup$

The Ornstein Ulhenebck Process is defined as (in the continuous setting) :

$$dX_t = -\beta(X_t - \alpha)dt + \sigma dW_t$$

The analogue for this process in the discrete time case which I assume will be applicable in the RL case will be: $$X_{t+1} = X_t -\beta(X_t - \alpha) + \sigma \{W_{t+1}-W_t\}=$$ $$X_{t+1} = (1 -\beta)X_t - \alpha + \sigma \{W_{t+1}-W_t\}$$

In the RL seting the terms in the equation probably means:

  • $X_t$ will stand for a state in RL i.e. the state is the number $\in \mathbb R$ where the particle moves to at time $t$.
  • $\beta$ and $\alpha$ are just constants which decide certain movement characteristics of the particle. Check here for graphs plotted for various $\beta$.
  • $W_t$ is a Weiner process which starts at $W_0 = 0$ and then adds independent increments of $\mathcal N(\mu,\sigma)$ as $W_{t+1} = W_t+\mathcal N(\mu,\sigma)$ which is basically a radom walk. More generally we use $\mathcal N(0,1)$. This is formulated as $W_t-W_s = \sqrt{t-s} \mathcal N(0,1)$. This is because of the fact, $W_t$ can be written recursively as $W_t = \mathcal N(0,1)+W_{t-1} = \mathcal N(0,1) + \mathcal N(0,1) + ...W_s$ and since the samplings are independent at each step the mean get added as $\mu_t+\mu_{t-1}...$ and the variances as $\sigma_t^2 + \sigma_{t-1}^2...$. SInce her the means and variances are $0$ and $1$ respectively, the final mean $\mu = 0$ and variance $\sigma^2 = (t-s)$. And hence, byproperties of Gaussian random variables you can write (it is easy to show this via variable tranformation) $W_t-W_s = \sqrt{t-s} \mathcal N(0,1)$. Here, is the formulation of standard Weiner process.
  • $\sigma$ will be the weighting factor of the Weiner process, which just means the amount of noise being added to the process.

Another useful resource on discrete Ornstein Ulhenbeck process, much less generalized. I think now you can extend this to whatever scenario you are intereted in RL setting.

| improve this answer | |
$\endgroup$
  • 1
    $\begingroup$ good answer ! the RL side is actually used for the continuous setting too, IIRC it is used to add exploration noise to the action selected in the DDPG algorithm, which deals with continuous action space. $\endgroup$ – David Ireland Aug 23 at 21:53
  • $\begingroup$ Thank you very much!!! $\endgroup$ – dani Aug 25 at 20:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.