0
$\begingroup$

This question comes from trying to build a SAC model. The action space is derived from a log normal distribution.

If in the appendix c of the original paper the equation for the log policy is:

$\log \pi(a|s) = \log \mu(u|s) - \sum^D_{i = 1} \log (1 - \tanh^2(u_i))$

where $u \in R^D$ is a random variable, $\mu(u|s)$ is the corresponding density wit infinite support (in this case being a Gaussian distribution.)

So, my question is, according to the code, the log probabilities are calculated using the following:

$\log \mu(u|s) = -0.5 \times \log(2 \times \pi) \times n + \sum (-0.5 \times \epsilon^2 - \log \sigma^2) $

where $\epsilon \sim \mathcal{N}(0,1)$

n = size of log_std vector

The implementation on stable baseline is more complicated.

I want to know where this formula come from because doesn't look normal pdf or log normal pdf.

\begin{equation} \frac{1}{\sigma \sqrt{2 \pi}} e^{\frac{1}{2}(\frac{x-\mu}{\sigma})^2} \\ \frac{1}{x \sigma \sqrt{2 \pi}} e^{ -( \frac{\log(x-\mu)^2}{2\sigma^2})} \end{equation}

Further elaborations on implementations that i'm trying to understand:

Example 1: Look up log_proba here

The log probability is defined as:

$\log \pi_\theta = -0.5 \times (n \times \log(2 \pi) + logdetvar + quadratic)$

where logdetvar is the log determinant of the variance or $\log(det(M)) = trace(\log(M))$

where quadratic seems to be the row sum of ($x - \mu$) and the negative variance.

and finally $n$ is the number of rows in the log variance.

So example exmaple 1 doesn't look like the either pdf.

Example 2: Stable baselines 3 normal distribution

Stable baselines defines the log probability of the normal distribution as:

$-((X - \mu)^2) / (2 \times \sigma^2) - \log(\sigma) - \log((2\pi)^2) $

Again nothing like the pdf of the normal. Can someone explain how they are deriving these different implementations, and where do they come from?

$\endgroup$
18
  • 1
    $\begingroup$ I am saying, if you are doing RL you will almost certainly not be wanting the CDF, you are wanting the PDF. In SAC, the log of the policy (note, this is not a log-normal distribution) is simply the log of the PDF (in a continuous case) evaluated at the given action. $\endgroup$
    – David
    May 8, 2023 at 23:58
  • 3
    $\begingroup$ Please, don't provide screenshots. Provide the code, if that's really necessary. We can't directly copy from screenshots and that could be a problem. $\endgroup$
    – nbro
    May 9, 2023 at 8:58
  • 1
    $\begingroup$ @DavidIreland is completely right. In SAC you only need the log-probability (not CDF) of actions; the policy is still Gaussian (actually is a SquashedGaussian by tanh to bound actions in $[-1, 1]$), you have the policy net to predict its mean and log-variance (which is later properly exponentiated.) In papers, if you see $\log \pi(a\mid s)$ it's a way to say "the log probability of predicted action $a$ from the policy $\pi$ in state $s$". $\endgroup$ May 9, 2023 at 15:33
  • 1
    $\begingroup$ I'm new to this community, I read the guidelines, I didn't see any violations that I made, if I need to make an edit or clarify I'm happy to. Please be nice, tell me what you aren't liking and I'll do better on future posts. $\endgroup$
    – chadmc
    May 9, 2023 at 17:52
  • 1
    $\begingroup$ @chadmc You're free to answer or ignore my comments. Mine was a recommendation for you. I was not referring to any specific thing in the on-topic. It was a general suggestion for you to get familiar with the site. Having said that, I think in the past I've answered a similar question (although I didn't even fully read your post), but now I don't want to waste my time searching for it. You can search for it. I wasn't planning to answer your question. I'm here to make this community a better place not just by answering questions, which I already did a lot. $\endgroup$
    – nbro
    May 20, 2023 at 1:54

1 Answer 1

1
$\begingroup$

The SAC algorithm was designed for control and robotics tasks in mind, i.e. environments with continuous states and actions spaces.

In practice, SAC implements a policy that is called a Squashed Gaussian (read here): is a Gaussian distribution that is bounded (or squashed) to output actions within the $[-1, 1]$ interval by a $\tanh$ operation. Intuitively, you can think of it as $a \sim \tanh(\mathcal{N}(\mu, \Sigma))$. This is because the Gaussian is unbounded having support in $[-\infty, \infty]$, and so limiting the actions was done by truncating the distribution which is not much clever, and in fact usually results in lower performance: I guess because you have no gradients when actions fall outside the desired range, instead the tanh provides a continuous yet bounded interval. (An alternative to the squashed gaussian could be a Beta distribution that is in $[0, 1]$, instead, as motivated here - but I think almost nobody uses that.)

Now, the term $\mu(u\mid s)$ corresponds to the underlying Gaussian distribution that outputs/samples actions $u$, providing they probability to occur. But since the $\tanh$ is applied you have to correct such probability, since you want to evaluate $\log \pi(a\mid a)$ instead of $\log \mu(u\mid s)$. In few words, the correction factor is equal to the inverse of the Jacobian determinant of the tanh transformation. So you compute the probability of the Gaussian policy $\mu(u\mid s)$ then multiply by the Jacobian, and finally take the log to obtain the log-prob of the squashed actions $a$.

If you're interested in how this is done in practice check the official and these (1 and 2) other implementations. As a side note, if you use tools like tensorflow-probability you can implement this quite easily by applying a Tanh bijector to a Normal distribution.


Update: log of the Normal pdf. The Normal $\mathcal{N}(\mu,\sigma^2)$ pdf of a sample $x$ (e.g. actions) is defined as follows:

$$\frac{1}{\sigma\sqrt{2\pi}}e^{-\frac12 \big(\frac{x-\mu}{\sigma}\big)}$$

Now, in RL you care about the logarithm of such probability density. So:

$$\log\Big(\frac{1}{\sigma\sqrt{2\pi}}e^{-\frac12 \big(\frac{x-\mu}{\sigma}\big)}\Big) \\ = \log\Big(\frac{1}{\sigma\sqrt{2\pi}}\Big) + \log\Big(e^{-\frac12 \big(\frac{x-\mu}{\sigma}\big)}\Big) \\ = -\log\sigma -\log\sqrt{2\pi} - \frac12\big(\frac{x-\mu}{\sigma}\big)^2$$

Which is the same formula SB and pytorch implements.

$\endgroup$
17
  • 1
    $\begingroup$ I would say that I think the tanh transformation makes sense for an action space that is symmetric, like the MuJoCo tasks which were the benchmark environments in the original paper. $\endgroup$
    – David
    May 9, 2023 at 20:07
  • 1
    $\begingroup$ Sure, I guess that one (good at math) can extend this idea to a sigmoid, or in general to even affine transformations (scaling and biasing) to handle whatever action space. $\endgroup$ May 9, 2023 at 20:11
  • 1
    $\begingroup$ Yes that's right, you can use the pdf transformation method for arbitrary transformations (assuming the function is one-to-one). I think the best is some combination of tanh or sigmoid followed by an affine transformation and you can satisfy most action space bounds. $\endgroup$
    – David
    May 9, 2023 at 20:13
  • 1
    $\begingroup$ If the code does .log_prob(action) of the distribution object, then it is evaluating the log pdf (this is how it works in PyTorch at least). $\endgroup$
    – David
    May 10, 2023 at 21:00
  • 1
    $\begingroup$ @chadmc I've updated my answer to include a derivation of the log-pdf of the Gaussian, have a look $\endgroup$ May 20, 2023 at 10:14

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .