Where does entropy enter in Soft Actor-Critic?

Question

I am currently trying to understand SAC (Soft Actor-Critic), and I am thinking of it as a basic actor-critic with the entropy included. However, I expected the entropy to appear in the Q-function. From SpinningUp-SAC, it looks like the entropy is entering through the value-function, so I'm thinking it enters by the $\log \pi_{\phi}(a_t \mid s_t)$ in the value function?

I'm a little stuck on understanding SAC, can anyone confirm/explain this to me?

Also, side-note question: is being a soft agent equivalent to including entropy in one of the object functions?

Answer 1

In the answer I'll be using notation similar to the one from the SAC paper. If we look at the standard objective function for policy gradient methods we have \begin{align} J_\pi &= V_\pi(s_t)\\ &= \mathbb E_{a_t \sim \pi(a|s_t)}[Q(s_t, a_t)]\\ &= \mathbb E_{a_t \sim \pi(a|s_t)}[ \mathbb E_{s_{t+1} \sim p(s|s_t, a_t)} [r(s_t, a_t) + V(s_{t+1})]]\\ &= \mathbb E_{a_t \sim \pi(a|s_t)}[ \mathbb E_{s_{t+1} \sim p(s|s_t, a_t)} [r(s_t, a_t) + \mathbb E_{a_{t+1} \sim \pi(a|s_{t+1})}[ \mathbb E_{s_{t+2} \sim p(s|s_{t+1}, a_{t+1})} [r(s_{t+1}, a_{t+1}) + V(s_{t+2})]]]]\\ &\cdots\\ &= \sum_t \mathbb E_{(a_t, s_t) \sim \rho_\pi} [r( s_t, a_t)] \end{align} If you keep unwinding this $V(s_{t+i})$ you will get expected sum of rewards. We can define soft state value as \begin{align} V(s_t) &= \mathbb E_{a_t \sim \pi(a|s_t)}[Q(s_t, a_t) + \mathcal H(\cdot|s_t)]\\ &= \mathbb E_{a_t \sim \pi(a|s_t)}[Q(s_t, a_t) + \mathbb E_{a \sim \pi(a|s_t)}[-\log(\pi(a|s_t))]]\\ &= \mathbb E_{a_t \sim \pi(a|s_t)}[Q(s_t, a_t) - \log(\pi(a_t|s_t))] \end{align} third equality comes from the fact that $\mathbb E_{a \sim \pi(a|s_t)}[-\log(\pi(a|s_t))]$ is nonrandom so its the same thing as if we are sampling over $\pi$ only once.

In maximum entropy framework objective function would then be \begin{align} J_\pi &= V_\pi(s_t)\\ &= \mathbb E_{a_t \sim \pi(a|s_t)}[Q(s_t, a_t) - \log(\pi(a_t|s_t))]\\ &= \mathbb E_{a_t \sim \pi(a|s_t)}[ \mathbb E_{s_{t+1} \sim p(s|s_t, a_t)} [r(s_t, a_t) - \log(\pi(a_t|s_t)) + V(s_{t+1})]]\\ & \cdots\\ &= \sum_t \mathbb E_{(a_t, s_t) \sim \rho_\pi} [r(s_t, a_t) -\log(\pi(a_t|s_t))]\\ &= \sum_t \mathbb E_{(a_t, s_t) \sim \rho_\pi} [r(s_t, a_t) + \mathcal H(\cdot|s_t)] \end{align}