How is the discounted maximum entropy objective obtained for soft-q-learning and SAC

In the soft q-learning paper, they provide an expression for the maximum entropy objective that takes discounting into account.

My main question is: can someone explain how they incorporated discounting into the objective?

I've also got a few other questions related to the form of the discounted objective as well.

The first one being is: they first define the objective in way of obtaining $$\pi_{\text{MaxEnt}}^*$$.

In this first expression,

$$\pi_{\mathrm{MaxEnt}}^{*}=\arg \max _{\pi} \sum_{t} \mathbb{E}_{\left(\mathbf{s}_{t}, \mathbf{a}_{t}\right) \sim \rho_{\pi}}\left[\sum_{l=t}^{\infty} \gamma^{l-t} \mathbb{E}_{\left(\mathbf{s}_{l}, \mathbf{a}_{l}\right)}\left[r\left(\mathbf{s}_{t}, \mathbf{a}_{t}\right)+\alpha \mathcal{H}\left(\pi\left(\cdot \mid \mathbf{s}_{t}\right)\right) \mid \mathbf{s}_{t}, \mathbf{a}_{t}\right]\right],$$

I don't really understand the purpose of the inner expectation. If it's an expectation over $$(s_l,a_l)$$, the terms within the expectation are constants, so they can be taken out of the expectation and even the inner sum too. So, I think the subscript might be wrong, but was hoping someone could confirm this.

My second issue is: they rewrite the maximum entropy objective using $$Q_{soft}$$ in (16)

$$J(\pi) \triangleq \sum_{t} \mathbb{E}_{\left(\mathbf{s}_{t}, \mathbf{a}_{t}\right) \sim \rho_{\pi}}\left[Q_{\mathrm{soft}}^{\pi}\left(\mathbf{s}_{t}, \mathbf{a}_{t}\right)+\alpha \mathcal{H}\left(\pi\left(\cdot \mid \mathbf{s}_{t}\right)\right)\right]$$

I'm not sure how they do this. If someone could provide a proof of this connection, that would be much appreciated.

• Edit your question to only contain one question. Make separate questions for your others. Mar 20 at 10:01
• Please, do as suggested in the previous comment. Split this post into 3: one for each of your 3 questions.
– nbro
Mar 21 at 9:59