# Where does the term $\log \mu(u \mid s)$ come from?

This question comes from trying to build a SAC model. The action space is derived from a log normal distribution.

If in the appendix c of the original paper the equation for the log policy is:

$$\log \pi(a|s) = \log \mu(u|s) - \sum^D_{i = 1} \log (1 - \tanh^2(u_i))$$

where $$u \in R^D$$ is a random variable, $$\mu(u|s)$$ is the corresponding density wit infinite support (in this case being a Gaussian distribution.)

So, my question is, according to the code, the log probabilities are calculated using the following:

$$\log \mu(u|s) = -0.5 \times \log(2 \times \pi) \times n + \sum (-0.5 \times \epsilon^2 - \log \sigma^2)$$

where $$\epsilon \sim \mathcal{N}(0,1)$$

n = size of log_std vector

The implementation on stable baseline is more complicated.

I want to know where this formula come from because doesn't look normal pdf or log normal pdf.

$$$$\frac{1}{\sigma \sqrt{2 \pi}} e^{\frac{1}{2}(\frac{x-\mu}{\sigma})^2} \\ \frac{1}{x \sigma \sqrt{2 \pi}} e^{ -( \frac{\log(x-\mu)^2}{2\sigma^2})}$$$$

Further elaborations on implementations that i'm trying to understand:

Example 1: Look up log_proba here

The log probability is defined as:

$$\log \pi_\theta = -0.5 \times (n \times \log(2 \pi) + logdetvar + quadratic)$$

where logdetvar is the log determinant of the variance or $$\log(det(M)) = trace(\log(M))$$

where quadratic seems to be the row sum of ($$x - \mu$$) and the negative variance.

and finally $$n$$ is the number of rows in the log variance.

So example exmaple 1 doesn't look like the either pdf.

Example 2: Stable baselines 3 normal distribution

Stable baselines defines the log probability of the normal distribution as:

$$-((X - \mu)^2) / (2 \times \sigma^2) - \log(\sigma) - \log((2\pi)^2)$$

Again nothing like the pdf of the normal. Can someone explain how they are deriving these different implementations, and where do they come from?

• I am saying, if you are doing RL you will almost certainly not be wanting the CDF, you are wanting the PDF. In SAC, the log of the policy (note, this is not a log-normal distribution) is simply the log of the PDF (in a continuous case) evaluated at the given action. May 8, 2023 at 23:58
• Please, don't provide screenshots. Provide the code, if that's really necessary. We can't directly copy from screenshots and that could be a problem.
– nbro
May 9, 2023 at 8:58
• @DavidIreland is completely right. In SAC you only need the log-probability (not CDF) of actions; the policy is still Gaussian (actually is a SquashedGaussian by tanh to bound actions in $[-1, 1]$), you have the policy net to predict its mean and log-variance (which is later properly exponentiated.) In papers, if you see $\log \pi(a\mid s)$ it's a way to say "the log probability of predicted action $a$ from the policy $\pi$ in state $s$". May 9, 2023 at 15:33
• I'm new to this community, I read the guidelines, I didn't see any violations that I made, if I need to make an edit or clarify I'm happy to. Please be nice, tell me what you aren't liking and I'll do better on future posts. May 9, 2023 at 17:52
• @chadmc You're free to answer or ignore my comments. Mine was a recommendation for you. I was not referring to any specific thing in the on-topic. It was a general suggestion for you to get familiar with the site. Having said that, I think in the past I've answered a similar question (although I didn't even fully read your post), but now I don't want to waste my time searching for it. You can search for it. I wasn't planning to answer your question. I'm here to make this community a better place not just by answering questions, which I already did a lot.
– nbro
May 20, 2023 at 1:54

The SAC algorithm was designed for control and robotics tasks in mind, i.e. environments with continuous states and actions spaces.

In practice, SAC implements a policy that is called a Squashed Gaussian (read here): is a Gaussian distribution that is bounded (or squashed) to output actions within the $$[-1, 1]$$ interval by a $$\tanh$$ operation. Intuitively, you can think of it as $$a \sim \tanh(\mathcal{N}(\mu, \Sigma))$$. This is because the Gaussian is unbounded having support in $$[-\infty, \infty]$$, and so limiting the actions was done by truncating the distribution which is not much clever, and in fact usually results in lower performance: I guess because you have no gradients when actions fall outside the desired range, instead the tanh provides a continuous yet bounded interval. (An alternative to the squashed gaussian could be a Beta distribution that is in $$[0, 1]$$, instead, as motivated here - but I think almost nobody uses that.)

Now, the term $$\mu(u\mid s)$$ corresponds to the underlying Gaussian distribution that outputs/samples actions $$u$$, providing they probability to occur. But since the $$\tanh$$ is applied you have to correct such probability, since you want to evaluate $$\log \pi(a\mid a)$$ instead of $$\log \mu(u\mid s)$$. In few words, the correction factor is equal to the inverse of the Jacobian determinant of the tanh transformation. So you compute the probability of the Gaussian policy $$\mu(u\mid s)$$ then multiply by the Jacobian, and finally take the log to obtain the log-prob of the squashed actions $$a$$.

If you're interested in how this is done in practice check the official and these (1 and 2) other implementations. As a side note, if you use tools like tensorflow-probability you can implement this quite easily by applying a Tanh bijector to a Normal distribution.

Update: log of the Normal pdf. The Normal $$\mathcal{N}(\mu,\sigma^2)$$ pdf of a sample $$x$$ (e.g. actions) is defined as follows:

$$\frac{1}{\sigma\sqrt{2\pi}}e^{-\frac12 \big(\frac{x-\mu}{\sigma}\big)}$$

Now, in RL you care about the logarithm of such probability density. So:

$$\log\Big(\frac{1}{\sigma\sqrt{2\pi}}e^{-\frac12 \big(\frac{x-\mu}{\sigma}\big)}\Big) \\ = \log\Big(\frac{1}{\sigma\sqrt{2\pi}}\Big) + \log\Big(e^{-\frac12 \big(\frac{x-\mu}{\sigma}\big)}\Big) \\ = -\log\sigma -\log\sqrt{2\pi} - \frac12\big(\frac{x-\mu}{\sigma}\big)^2$$

Which is the same formula SB and pytorch implements.

• I would say that I think the tanh transformation makes sense for an action space that is symmetric, like the MuJoCo tasks which were the benchmark environments in the original paper. May 9, 2023 at 20:07
• Sure, I guess that one (good at math) can extend this idea to a sigmoid, or in general to even affine transformations (scaling and biasing) to handle whatever action space. May 9, 2023 at 20:11
• Yes that's right, you can use the pdf transformation method for arbitrary transformations (assuming the function is one-to-one). I think the best is some combination of tanh or sigmoid followed by an affine transformation and you can satisfy most action space bounds. May 9, 2023 at 20:13
• If the code does .log_prob(action) of the distribution object, then it is evaluating the log pdf (this is how it works in PyTorch at least). May 10, 2023 at 21:00
• @chadmc I've updated my answer to include a derivation of the log-pdf of the Gaussian, have a look May 20, 2023 at 10:14