Why is soft actor critic an off policy scheme?

Question

I am struggling to understand what makes a scheme on-policy or off-policy. From what I have read, we can say that deep Q-learning is off-policy because we use a different policy like $\epsilon$-greedy technique for exploration while training our DNN, so this makes the behavior policy ($\epsilon$-greedy) and the target policy (from DNN) to be different.

However, in the case of soft actor-critic (SAC) we use the same policy DNN to generate the data stored in the memory buffer. Later we use this data to update the policy DNN. There is only one DNN that is being used for generating the data for exploration, while this data is used for updating the policy network (same DNN). When the DNN is the same, how can we say that the behavior policy and target policy are different?

Answer 1

SAC is an off-policy method because it learns from a replay buffer, which contains experiences collected by the agent over time from potentially different versions of the policy. This means the agent can learn from past experiences generated by older versions of the policy, which might not be the same as the current policy.