In Theorem 3.1 of the conservative q-learning paper, what is the meaning of

$$ (I - \gamma P^{\pi})^{-1} \left[\frac{\mu(a|s)}{\hat \pi_{\beta}(a|s)} \right](s, a)$$?

I thought $(I - \gamma P^{\pi})^{-1}$ is to be interpreted as an operator on function space over state and action space, but it is unclear to me how it is being applied to a $\frac{\mu(a|s)}{\hat \pi_{\beta}(a|s)}$, which is a scalar.

Also, it is unclear what $(I - \gamma P^{\pi})^{-1} \frac{C_{r, T, \delta}R_{\max}}{1 - \gamma}$ means?


1 Answer 1


We have for $\gamma < 1$:

$$ (I-\gamma P^{\pi})^{-1}=I+\gamma P^{\pi} + \gamma^2 (P^{\pi})^2 + \cdots $$

What they mean by $(I-\gamma P^{\pi})^{-1}\frac{\mu(a\mid s)}{\hat{\pi}_{\beta}(a\mid s)}(s,a) $ is the function from $\mathcal{S}\times\mathcal{A}\longrightarrow \mathbb{R}$, sending $(s,a)\mapsto (I-\gamma P^{\pi})^{-1}\frac{\mu(a\mid s)}{\hat{\pi}_{\beta}(a\mid s)}$.

Now, we have

$$P^{\pi}\frac{\mu(a\mid s)}{\hat{\pi}_{\beta}(a\mid s)}=\mathbb{E}_{s'\sim p(\cdot\mid s,a), a'\sim \pi(\cdot\mid s')}\left[ \frac{\mu(a'\mid s')}{\hat{\pi}_{\beta}(a'\mid s')} \right]. $$ Similarily,

$$(P^{\pi})^2\frac{\mu(a\mid s)}{\hat{\pi}_{\beta}(a\mid s)}=\mathbb{E}_{s'\sim p(\cdot\mid s,a), a'\sim \pi(\cdot\mid s'), s''\sim p(\cdot\mid s',a'), a''\sim \pi(\cdot\mid s'')}\left[ \frac{\mu(a''\mid s'')}{\hat{\pi}_{\beta}(a''\mid s'')} \right]. $$

Adding this together, we have

$$(I-\gamma P^{\pi})^{-1}\frac{\mu(a\mid s)}{\hat{\pi}_{\beta}(a\mid s)}(s,a)=\mathbb{E}_{\pi}\left[\sum_{t\geq 0} \gamma^t\frac{\mu(a_t\mid s_t)}{\hat{\pi}_{\beta}(a_t\mid s_t)}\bigg\rvert s_0=s, a_0=a \right]. $$

The last quantity is easier, as it does not depend on $(s,a)$, so:

$$(I-\gamma P^{\pi})^{-1}\frac{C_{r,T,\delta}R_{\max}}{1-\gamma}=\sum_{t\geq 0}\gamma^t\frac{C_{r,T,\delta}R_{\max}}{1-\gamma}=C_{r,T,\delta}R_{\max} $$

  • $\begingroup$ This makes a lot of sense, thanks @matorbi $\endgroup$ Nov 6, 2022 at 21:50

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .