1
$\begingroup$

Soft Actor-Critic paper proposes $\pi_{new}=\arg\min_{\pi'\in \prod}D_{KL}\left(\pi'(\cdot|s_t)\big|\big| \frac{\exp(Q^{\pi_{old}}(s_t,\cdot))}{Z^{\pi_{old}}(s_t)}\right)$

Paper says, we update the policy towards the exponential of the new Q-function(is it really a new Q function when it is $Q^{\pi_{old}}$..?) because it guarantees to result in an imporved policy in terms of its soft value.

  1. Does it mean, Q is exponential by just a modeling/analytical reason, as it allows us to proof the soft policy improvement theorem? There is no intuitive explanation about it?

  2. It says $Z^{\pi_{old}}(s_t)$ is a partition function which normalizes the distribution. I understand that its usage is to normalize the distribution. However, what does it mean by partition function? Can someone provide a descriptive explanation/example about it to better understand what is partition function and how it is used to normalize the distribution of $exp(Q^{\pi_{old}}(s_t,\cdot))$?

Improve this question
$\endgroup$
2
  • $\begingroup$ looking at energy based models might answer some of your questions. $\endgroup$
    – David
    Commented Mar 29 at 16:46
  • $\begingroup$ The idea comes from physics. The exp(Q)/Z term is from the Boltzmann equation. Large language models do the same thing, though they call this the "softmax" (which is likely where the name "soft actor-critic" comes from). I'd do some reading up on statistical mechanics, information theory, and evolutionary game theory if you want to understand this on a deeper level. $\endgroup$
    – programjames
    Commented Mar 30 at 1:10

1 Answer 1

Reset to default
1
$\begingroup$

Regarding "we update the policy towards the exponential of the new Q-function", it's correct that the Q-function $Q(s_t,·)$ being exponentiated in the policy improvement step is new (latest) at timestep $t$, though under the old policy $\pi_{old}$. And it's the consideration of energy based ML models rooted in statistical mechanics to motivate taking exponential of the Q-function since intuitively the soft (stochastic) policy as a probability distribution can be regarded as the Boltzmann distribution and the learnable Q-function interpreted as energy can be flexibly arbitrary so long as you have a normalizing partition function.

A quintessential ML model using partition function is the restricted Boltzmann machine (RBM) which is a special case of Markov random field whose graphical model is undirected, inducing unnormalized joint density of the input in the form of a product of factors aka factor analysis in statistics.

In all references you could see the partition function involves summation or integral of many or infinite inputs, thus it's usually intractable. And here for SAC policy improvement with continuous action space defined earlier, the inputs are just all actions in the action space $\mathcal{A}$, therefore the partition function is $Z^{\pi_{old}}(s_t)=\int_\mathcal{A} \exp Q^{\pi_{old}}(s_t,a)da$.

Improve this answer
$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .