# What is the derivative of equation 1 in the paper "Conservative Q-Learning for Offline Reinforcement Learning"?

I am looking at the paper Conservative Q-Learning for Offline Reinforcement Learning, but I'm not sure how they proved theorem 3.1.

Here is a screenshot of theorem 3.1. In the proof of theorem 3.1 they say

By setting the derivative of Equation 1 to 0, we obtain the following expression

...

$$\forall \mathbf{s}, \mathbf{a} \in \mathcal{D}, k, \quad \hat{Q}^{k+1}(\mathbf{s}, \mathbf{a})=\hat{\mathcal{B}}^{\pi} \hat{Q}^{k}(\mathbf{s}, \mathbf{a})-\alpha \frac{\mu(\mathbf{a} \mid \mathbf{s})}{\hat{\pi}_{\beta}(\mathbf{a} \mid \mathbf{s})} \tag{11}\label{11}$$

Here's equation 1 from the paper.

$$\hat{Q}^{k+1} \leftarrow \arg \min _{Q} \alpha \mathbb{E}_{\mathbf{s} \sim \mathcal{D}, \mathbf{a} \sim \mu(\mathbf{a} \mid \mathbf{s})}[Q(\mathbf{s}, \mathbf{a})]+\frac{1}{2} \mathbb{E}_{\mathbf{s}, \mathbf{a} \sim \mathcal{D}}\left[\left(Q(\mathbf{s}, \mathbf{a})-\hat{\mathcal{B}}^{\pi} \hat{Q}^{k}(\mathbf{s}, \mathbf{a})\right)^{2}\right] \tag{1}\label{1}$$

My question is: what exactly is the derivative of equation (1)? And how does that result in equation (11)?

The $$\hat{B}^\pi$$ is the empirical Bellman operator and is defined as $$\hat{B}^\pi \hat{Q}^k (s,a) = r + \gamma \sum_{s'} \hat{T}(s' \mid s,a) \mathbb{E}_{a'\sim \pi(a' \mid s')}\hat{Q}_k(s', a')$$. Since in offline reinforcement learning, the dataset $$\mathcal{D}$$ typically does not contain all possible transitions $$(s, a, s')$$, the policy evaluation step actually uses an empirical Bellman operator that only backs up a single sample.

• Can you actually format the equations as text, rather than an image? That would make it a lot more accessible. Sep 15, 2021 at 13:10
• Yes, please, next time, rather than providing the screenshots, for future reproducibility and easy accessibility, copy and quote the relevant parts of text from the paper, at least, as I've just done to your post. To get the equations in latex automatically, you can use a tool like "Mathpix Snipping Tool" (which is what I used).
– nbro
Sep 15, 2021 at 13:12
• I apologize, I will try to copy the relevant parts. And thanks for recommending mathpix snipping tools! Sep 15, 2021 at 13:31

first we need to transform the distribution of the first term:

$$\mathrm{argmin} \space \alpha \mathrm{E}_{s\sim D, a \sim D}[\frac{\mu}{\hat\pi_{\beta}}Q(s,a)] + \frac{1}{2}\mathrm{E}_{s,a \sim D}(Q(s,a)-\hat\beta^{\pi}\hat Q^k(s,a))^2$$

then its derivate respect to Q is:

$$\alpha\frac{\mu}{\hat \pi ^{\beta}} + Q(s,a)-\hat\beta^{\pi}\hat Q^k(s,a) = 0$$

then we have

$$Q(s,a) =\hat\beta^{\pi}\hat Q^k(s,a) -\alpha\frac{\mu}{\hat \pi ^{\beta}}$$

• how did you find the derivative with respect to Q? Dec 15, 2021 at 23:53