3
$\begingroup$

While reading the original paper of Soft Actor Critic, I came across on page number 5, under equation (5) and (6)

$$ J_{V}(\psi)=\mathbb{E}_{\mathbf{s}_{t} \sim \mathcal{D}}\left[\frac{1}{2}\left(V_{\psi}\left(\mathbf{s}_{t}\right)-\mathbb{E}_{\mathbf{a}_{t} \sim \pi_{\phi}}\left[Q_{\theta}\left(\mathbf{s}_{t}, \mathbf{a}_{t}\right)-\log \pi_{\phi}\left(\mathbf{a}_{t} \mid \mathbf{s}_{t}\right)\right]\right)^{2}\right] \tag{5}\label{5} $$

$$ \hat{\nabla}_{\psi} J_{V}(\psi)=\nabla_{\psi} V_{\psi}\left(\mathbf{s}_{t}\right)\left(V_{\psi}\left(\mathbf{s}_{t}\right)-Q_{\theta}\left(\mathbf{s}_{t}, \mathbf{a}_{t}\right)+\log \pi_{\phi}\left(\mathbf{a}_{t} \mid \mathbf{s}_{t}\right)\right) \tag{6}\label{6} $$

The following quote:

where the actions are sampled according to the current policy, instead of the replay buffer

In the context of deriving the formulation of the (estimated) gradient for the value function square residual error (Equation 5 in the paper)

I'm having a hard time understanding why they use the action sampled from the current policy instead of the replay buffer. My intuition tells me that this is because SAC is an off policy Reinforcement Learning algorithm, and Q-learning uses $\max Q$ in one-step Q-value function update (to keep it off-policy), but why would sampling one action from the current policy still make it off-policy?

I first asked a friend of mine (researcher in RL) and the answer I got was

"If the action is sampled with the current policy given any state the update is on-policy."

I've checked SpinningUpRL by OpenAI's explanation of SAC but they only make it more clear which action is sampled from the current policy, and which one is from the replay buffer, but does not specify why.

Does this have anything to do with the stochastic policy? Or the entropy term in the update equation?

So I'm still quite confused. Link/references to explanation are also appreciated!

Improve this question
$\endgroup$
9
  • $\begingroup$ I was also not able to find an answer on the internet. I think the reason for using the current policy starts with the entropy term in the Q and policy updates. Reading very closely through this section of the Spinning Up explanation helped me. Maybe the reason is that the entropy is supposed to be minimized and it would not work to use entropy from the current policy but actions from a different policy. And it does not seem like using entropy from a past policy would make sense. $\endgroup$
    – S2673
    Nov 11, 2020 at 14:17
  • $\begingroup$ If I get the time, I might try to see if SAC can still learn with using a past policy for the entropy, next actions, or both. $\endgroup$
    – S2673
    Nov 11, 2020 at 14:18
  • $\begingroup$ Interestingly, SAC learned the Pendulum environment the same with and without a target actor giving the entropy term and the next actions, though it might have learned faster without the target actor. It even learned with the target actor not updating at all because I forgot to add the code in. Maybe there is no target actor just because it isn’t needed. $\endgroup$
    – S2673
    Nov 11, 2020 at 16:51
  • $\begingroup$ Hello, thank you for the comment (and also even trying it! :D ) S2673! Just to be sure we are on the same page, by "target actor" you mean the stored action in the replay buffer (from another policy, most often previous one), the a in $(s_t, a_t, r_t, s_{t+1}) \in D$ right? $\endgroup$
    – DannyBoi
    Nov 12, 2020 at 5:52
  • $\begingroup$ $a_t$ was from whatever policy was operating when the experience batch was collected. I used the target actor to get the next-state actions($a_{t+1}$) because that is how DDPG and TD3 work. SAC usually gets $a_{t+1}$ from the current policy. $\endgroup$
    – S2673
    Nov 12, 2020 at 12:48

0

Reset to default

You must log in to answer this question.