# What is the meaning of $(I - \gamma P^{\pi})^{-1} \left[\frac{\mu(a|s)}{\hat \pi_{\beta}(a|s)} \right](s, a)$?

In Theorem 3.1 of the conservative q-learning paper, what is the meaning of

$$(I - \gamma P^{\pi})^{-1} \left[\frac{\mu(a|s)}{\hat \pi_{\beta}(a|s)} \right](s, a)$$?

I thought $$(I - \gamma P^{\pi})^{-1}$$ is to be interpreted as an operator on function space over state and action space, but it is unclear to me how it is being applied to a $$\frac{\mu(a|s)}{\hat \pi_{\beta}(a|s)}$$, which is a scalar.

Also, it is unclear what $$(I - \gamma P^{\pi})^{-1} \frac{C_{r, T, \delta}R_{\max}}{1 - \gamma}$$ means?

We have for $$\gamma < 1$$:

$$(I-\gamma P^{\pi})^{-1}=I+\gamma P^{\pi} + \gamma^2 (P^{\pi})^2 + \cdots$$

What they mean by $$(I-\gamma P^{\pi})^{-1}\frac{\mu(a\mid s)}{\hat{\pi}_{\beta}(a\mid s)}(s,a)$$ is the function from $$\mathcal{S}\times\mathcal{A}\longrightarrow \mathbb{R}$$, sending $$(s,a)\mapsto (I-\gamma P^{\pi})^{-1}\frac{\mu(a\mid s)}{\hat{\pi}_{\beta}(a\mid s)}$$.

Now, we have

$$P^{\pi}\frac{\mu(a\mid s)}{\hat{\pi}_{\beta}(a\mid s)}=\mathbb{E}_{s'\sim p(\cdot\mid s,a), a'\sim \pi(\cdot\mid s')}\left[ \frac{\mu(a'\mid s')}{\hat{\pi}_{\beta}(a'\mid s')} \right].$$ Similarily,

$$(P^{\pi})^2\frac{\mu(a\mid s)}{\hat{\pi}_{\beta}(a\mid s)}=\mathbb{E}_{s'\sim p(\cdot\mid s,a), a'\sim \pi(\cdot\mid s'), s''\sim p(\cdot\mid s',a'), a''\sim \pi(\cdot\mid s'')}\left[ \frac{\mu(a''\mid s'')}{\hat{\pi}_{\beta}(a''\mid s'')} \right].$$

$$(I-\gamma P^{\pi})^{-1}\frac{\mu(a\mid s)}{\hat{\pi}_{\beta}(a\mid s)}(s,a)=\mathbb{E}_{\pi}\left[\sum_{t\geq 0} \gamma^t\frac{\mu(a_t\mid s_t)}{\hat{\pi}_{\beta}(a_t\mid s_t)}\bigg\rvert s_0=s, a_0=a \right].$$
The last quantity is easier, as it does not depend on $$(s,a)$$, so:
$$(I-\gamma P^{\pi})^{-1}\frac{C_{r,T,\delta}R_{\max}}{1-\gamma}=\sum_{t\geq 0}\gamma^t\frac{C_{r,T,\delta}R_{\max}}{1-\gamma}=C_{r,T,\delta}R_{\max}$$