Skip to main content
4 votes
Accepted

What is the gradient of the objective function in the Soft Actor-Critic paper?

I'll give it a go here and try to answer your question, I'm not sure if this is entirely correct, so if someone thinks that it isn't please correct me. I'll disregard expectation here to make things ...
Brale's user avatar
  • 2,406
4 votes
Accepted

Where does entropy enter in Soft Actor-Critic?

In the answer I'll be using notation similar to the one from the SAC paper. If we look at the standard objective function for policy gradient methods we have \begin{align} J_\pi &= V_\pi(s_t)\\ &...
Brale's user avatar
  • 2,406
2 votes
Accepted

Can off-policy algorithms benefit from the parallelization?

From the point of view of someone developing an in-house DRL lib and working on extremely CPU-intensive environments (usually large finite element-based simulations that can require several hours to ...
Scrimbibete's user avatar
2 votes
Accepted

Why is soft actor critic an off policy scheme?

SAC is an off-policy method because it learns from a replay buffer, which contains experiences collected by the agent over time from potentially different versions of the policy. This means the agent ...
Hans-Peter Schrei's user avatar
1 vote

Can action be dominated by state features in actor-critic algorithms?

It is certainly possible for the state features to dominate the action features in the critic. There are several strategies you can use: Replace the action features with a high dimensional learned ...
chessprogrammer's user avatar
1 vote
Accepted

Does SAC perform better than PPO in sample-expensive tasks with discrete action spaces?

First, both SAC and PPO are usable for continuous and discrete action spaces. However, in the case of discrete action spaces, SAC cost functions must be previously adapted. As explained in this Stable ...
Antonio's user avatar
  • 26
1 vote
Accepted

Why soft actor critic uses exponential of Q when updating policy? and what is a partition function?

Regarding "we update the policy towards the exponential of the new Q-function", it's correct that the Q-function $Q(s_t,·)$ being exponentiated in the policy improvement step is new (latest) ...
cinch's user avatar
  • 2,277
1 vote

Why do we limit the standard deviation in Actor architectures in Reinforcement Learning?

Bounding the log-std is a common practice to avoid/reduce instabilities during training and also avoid numerical issues and NaNs since the log-std is unbounded, being in $(-\infty,\infty)$, when ...
Luca Anzalone's user avatar
1 vote
Accepted

Where does the term $\log \mu(u \mid s)$ come from?

The SAC algorithm was designed for control and robotics tasks in mind, i.e. environments with continuous states and actions spaces. In practice, SAC implements a policy that is called a Squashed ...
Luca Anzalone's user avatar
1 vote
Accepted

Why clamp std for reparameterization trick between -20 and 2?

This is a common trick done in practice: this helps to stabilize training, and prevent large values that can blow up in NaNs. The reason is that the std of the Gaussian is learned in log-space (...
Luca Anzalone's user avatar
1 vote

How to enforce action bounds between 0 & 1 in soft actor-critic algorithm?

Yes you can map the output onto [0,1] as you indicate. You should treat this as a modification to the environment. I.e. imagine that the environment takes actions in [-1, 1] instead of [0,1]. No you ...
Taw's user avatar
  • 1,281
1 vote

What is the gradient of the objective function in the Soft Actor-Critic paper?

This is more meant like a comment to the previous answer. I also originally thought that $$ \nabla_{\theta}\log \pi_{\theta}(f_{\theta}(\varepsilon, s)\mid s) = \nabla_{a}\log\pi_{\theta}(a\mid s)\...
matorbi's user avatar
  • 141

Only top scored, non community-wiki answers of a minimum length are eligible