1
$\begingroup$

In section 5 of the paper Soft Actor-Critic Algorithms and Applications, it is proposed an optimization problem to obtain an optimal temperature parameter $\alpha^*_t$. First, one uses the original evaluation and improvement steps to estimate $Q_t^*$ and $\pi_t^*$, and then one somehow solves the optimization problem:

$$\alpha_t^* = \arg\min_{\alpha_t} \mathbb E _{a_t\sim\pi^*}\left[\alpha_t(-\log\pi_t^*(a_t|s_t;\alpha_t)-H)\right]\text .$$

As far as I understand, we should use our current estimate of $\pi_t^*$ to solve that problem. Since it was obtained from a previous $\alpha_{t-1}^*$, in practice it is not dependent on $\alpha_t$ and so the optimization problem becomes a linear problem with the only restriction being $\alpha_t\geq0$.

Here comes my problem: under this rationale, if $\alpha_t$ is a scalar independent of both state $s_t$ and action $a_t$, the value of the cost function is just proportional to $\alpha_t$ and so the solutions are either $0$ or $\infty$, depending on the sign of the expected value (something similar happens if $\alpha_t^*=\alpha_t^*(s_t,a_t)$). However, the whole idea of introducing this parameter is to account optimally for the exploration of the policy.

What is the correct way to solve this optimization problem along with the evaluation and improvement steps? I am particularly interested in the tabular case. Also, is there any explanation why they use a negative minimum entropy $H$, when the entropy is always positive?

By the way, in the approximate case the current official implementation seems to be doing just that: moving $\alpha_t^*$ up or down a little bit (closer to $\infty$ or 0, respectively), depending on the magnitude of the expected value. I guess one could do the same for the tabular case, modifying the $\alpha_t^*$ only a little bit in each step, but this seems rather suboptimal.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.