We have for $\gamma < 1$:
$$ (I-\gamma P^{\pi})^{-1}=I+\gamma P^{\pi} + \gamma^2 (P^{\pi})^2 + \cdots $$
What they mean by $(I-\gamma P^{\pi})^{-1}\frac{\mu(a\mid s)}{\hat{\pi}_{\beta}(a\mid s)}(s,a) $ is the function from $\mathcal{S}\times\mathcal{A}\longrightarrow \mathbb{R}$, sending $(s,a)\mapsto (I-\gamma P^{\pi})^{-1}\frac{\mu(a\mid s)}{\hat{\pi}_{\beta}(a\mid s)}$.
Now, we have
$$P^{\pi}\frac{\mu(a\mid s)}{\hat{\pi}_{\beta}(a\mid s)}=\mathbb{E}_{s'\sim p(\cdot\mid s,a), a'\sim \pi(\cdot\mid s')}\left[ \frac{\mu(a'\mid s')}{\hat{\pi}_{\beta}(a'\mid s')} \right]. $$
Similarily,
$$(P^{\pi})^2\frac{\mu(a\mid s)}{\hat{\pi}_{\beta}(a\mid s)}=\mathbb{E}_{s'\sim p(\cdot\mid s,a), a'\sim \pi(\cdot\mid s'), s''\sim p(\cdot\mid s',a'), a''\sim \pi(\cdot\mid s'')}\left[ \frac{\mu(a''\mid s'')}{\hat{\pi}_{\beta}(a''\mid s'')} \right]. $$
Adding this together, we have
$$(I-\gamma P^{\pi})^{-1}\frac{\mu(a\mid s)}{\hat{\pi}_{\beta}(a\mid s)}(s,a)=\mathbb{E}_{\pi}\left[\sum_{t\geq 0} \gamma^t\frac{\mu(a_t\mid s_t)}{\hat{\pi}_{\beta}(a_t\mid s_t)}\bigg\rvert s_0=s, a_0=a \right]. $$
The last quantity is easier, as it does not depend on $(s,a)$, so:
$$(I-\gamma P^{\pi})^{-1}\frac{C_{r,T,\delta}R_{\max}}{1-\gamma}=\sum_{t\geq 0}\gamma^t\frac{C_{r,T,\delta}R_{\max}}{1-\gamma}=C_{r,T,\delta}R_{\max} $$
