In the soft q-learning paper, they provide an expression for the maximum entropy objective that takes discounting into account.

My main question is: can someone explain how they incorporated discounting into the objective?

I've also got a few other questions related to the form of the discounted objective as well.

The first one being is: they first define the objective in way of obtaining $\pi_{\text{MaxEnt}}^*$.

In this first expression,

$$ \pi_{\mathrm{MaxEnt}}^{*}=\arg \max _{\pi} \sum_{t} \mathbb{E}_{\left(\mathbf{s}_{t}, \mathbf{a}_{t}\right) \sim \rho_{\pi}}\left[\sum_{l=t}^{\infty} \gamma^{l-t} \mathbb{E}_{\left(\mathbf{s}_{l}, \mathbf{a}_{l}\right)}\left[r\left(\mathbf{s}_{t}, \mathbf{a}_{t}\right)+\alpha \mathcal{H}\left(\pi\left(\cdot \mid \mathbf{s}_{t}\right)\right) \mid \mathbf{s}_{t}, \mathbf{a}_{t}\right]\right], $$

I don't really understand the purpose of the inner expectation. If it's an expectation over $(s_l,a_l)$, the terms within the expectation are constants, so they can be taken out of the expectation and even the inner sum too. So, I think the subscript might be wrong, but was hoping someone could confirm this.

My second issue is: they rewrite the maximum entropy objective using $Q_{soft}$ in (16)

$$J(\pi) \triangleq \sum_{t} \mathbb{E}_{\left(\mathbf{s}_{t}, \mathbf{a}_{t}\right) \sim \rho_{\pi}}\left[Q_{\mathrm{soft}}^{\pi}\left(\mathbf{s}_{t}, \mathbf{a}_{t}\right)+\alpha \mathcal{H}\left(\pi\left(\cdot \mid \mathbf{s}_{t}\right)\right)\right]$$

I'm not sure how they do this. If someone could provide a proof of this connection, that would be much appreciated.

  • 3
    $\begingroup$ Edit your question to only contain one question. Make separate questions for your others. $\endgroup$ – David Ireland Mar 20 at 10:01
  • $\begingroup$ Please, do as suggested in the previous comment. Split this post into 3: one for each of your 3 questions. $\endgroup$ – nbro Mar 21 at 9:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.