2
$\begingroup$

I am looking at the paper Conservative Q-Learning for Offline Reinforcement Learning, but I'm not sure how they proved theorem 3.1.

Here is a screenshot of theorem 3.1.

enter image description here

In the proof of theorem 3.1

enter image description here

they say

By setting the derivative of Equation 1 to 0, we obtain the following expression

...

$$\forall \mathbf{s}, \mathbf{a} \in \mathcal{D}, k, \quad \hat{Q}^{k+1}(\mathbf{s}, \mathbf{a})=\hat{\mathcal{B}}^{\pi} \hat{Q}^{k}(\mathbf{s}, \mathbf{a})-\alpha \frac{\mu(\mathbf{a} \mid \mathbf{s})}{\hat{\pi}_{\beta}(\mathbf{a} \mid \mathbf{s})} \tag{11}\label{11}$$

Here's equation 1 from the paper.

$$\hat{Q}^{k+1} \leftarrow \arg \min _{Q} \alpha \mathbb{E}_{\mathbf{s} \sim \mathcal{D}, \mathbf{a} \sim \mu(\mathbf{a} \mid \mathbf{s})}[Q(\mathbf{s}, \mathbf{a})]+\frac{1}{2} \mathbb{E}_{\mathbf{s}, \mathbf{a} \sim \mathcal{D}}\left[\left(Q(\mathbf{s}, \mathbf{a})-\hat{\mathcal{B}}^{\pi} \hat{Q}^{k}(\mathbf{s}, \mathbf{a})\right)^{2}\right] \tag{1}\label{1}$$

My question is: what exactly is the derivative of equation (1)? And how does that result in equation (11)?

The $\hat{B}^\pi$ is the empirical Bellman operator and is defined as $\hat{B}^\pi \hat{Q}^k (s,a) = r + \gamma \sum_{s'} \hat{T}(s' \mid s,a) \mathbb{E}_{a'\sim \pi(a' \mid s')}\hat{Q}_k(s', a')$. Since in offline reinforcement learning, the dataset $\mathcal{D}$ typically does not contain all possible transitions $(s, a, s')$, the policy evaluation step actually uses an empirical Bellman operator that only backs up a single sample.

$\endgroup$
3
  • 1
    $\begingroup$ Can you actually format the equations as text, rather than an image? That would make it a lot more accessible. $\endgroup$ Sep 15, 2021 at 13:10
  • 1
    $\begingroup$ Yes, please, next time, rather than providing the screenshots, for future reproducibility and easy accessibility, copy and quote the relevant parts of text from the paper, at least, as I've just done to your post. To get the equations in latex automatically, you can use a tool like "Mathpix Snipping Tool" (which is what I used). $\endgroup$
    – nbro
    Sep 15, 2021 at 13:12
  • $\begingroup$ I apologize, I will try to copy the relevant parts. And thanks for recommending mathpix snipping tools! $\endgroup$
    – MoneyBall
    Sep 15, 2021 at 13:31

1 Answer 1

0
$\begingroup$

first we need to transform the distribution of the first term:

$\mathrm{argmin} \space \alpha \mathrm{E}_{s\sim D, a \sim D}[\frac{\mu}{\hat\pi_{\beta}}Q(s,a)] + \frac{1}{2}\mathrm{E}_{s,a \sim D}(Q(s,a)-\hat\beta^{\pi}\hat Q^k(s,a))^2 $

then its derivate respect to Q is:

$\alpha\frac{\mu}{\hat \pi ^{\beta}} + Q(s,a)-\hat\beta^{\pi}\hat Q^k(s,a) = 0$

then we have

$ Q(s,a) =\hat\beta^{\pi}\hat Q^k(s,a) -\alpha\frac{\mu}{\hat \pi ^{\beta}}$

$\endgroup$
1
  • $\begingroup$ how did you find the derivative with respect to Q? $\endgroup$
    – MoneyBall
    Dec 15, 2021 at 23:53

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .