0
$\begingroup$

In the Soft Actor Critic Paper (found here https://arxiv.org/pdf/1801.01290.pdf), they use a neural network to approximate a diagonal gaussian distribution. In the sample function you can see that it has a function called reparameterize. As you can see in the reparameterize function, we use a tanh function to squash the action bound between -1 and 1. So why is it that we clamp the standard deviation between -20 and 2. I have read both Soft Actor Critic papers and can't find why we do this. Is there something about the normal distribution that would make that range of clamping of the std more desirable? Does bounding the range of the std help with convergence. Is there a paper you my recommend that would help with this answer? This is my first post here so if you need more information please let me know.

enter image description here

enter image description here

Link to code Repo

Improve this question
$\endgroup$
1
  • $\begingroup$ Can you please provide the code instead of a screenshot? Although note that programming issues are off-topic here. $\endgroup$
    – nbro
    Commented May 3, 2023 at 23:43

1 Answer 1

Reset to default
1
$\begingroup$

This is a common trick done in practice: this helps to stabilize training, and prevent large values that can blow up in NaNs.

The reason is that the std of the Gaussian is learned in log-space (because it's easier to learn with neural-nets) that is unbounded ($-\infty, +\infty$), and so you want to bound to something like $[-20, 2]$ because such log-std will be exponentiated later: you have $\exp(-20)\approx 0$, and $\exp(2) = 7.389$ that is reasonably small; a similar trick is also employed for log-probabilities.

Practical tricks like this one are often not documented in the paper (unfortunately), but can be found in code implementations.

Improve this answer
$\endgroup$
3
  • $\begingroup$ Thank you, I tried to look at other methods papers to see if it was documented and could not find any. I really appreciate your answer. One final question if you have a second, so the bound are truly arbitrary and I could make them anything as long as it it reasonably small? $\endgroup$
    – chadmc
    Commented May 4, 2023 at 1:10
  • $\begingroup$ Yes, the bounds are arbitrary but just consider that you're are in log-space and that the $\exp$ can easily underflow or overflow going to infinite. If you want to change the bounds, think about the range of values you want and take the log of that, finally clamp: say you want the std to be in $[1,3]$, you take the log to determine the bounds i.e. $[\log(1), \log(3)] = [0, 1.0986]$ for example. $\endgroup$
    – Luca Anzalone
    Commented May 4, 2023 at 7:04
  • 1
    $\begingroup$ I finally found a reference to your answer, the stable baselines, github.com/DLR-RM/stable-baselines3/blob/master/…, say that they clamp to avoid NaN values in the comments of the code. You can find it in the TanhBisector function. $\endgroup$
    – chadmc
    Commented May 8, 2023 at 21:48

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .