What is the derivative of equation 1 in the paper "Conservative Q-Learning for Offline Reinforcement Learning"?

Question

I am looking at the paper Conservative Q-Learning for Offline Reinforcement Learning, but I'm not sure how they proved theorem 3.1.

Here is a screenshot of theorem 3.1.

In the proof of theorem 3.1

they say

By setting the derivative of Equation 1 to 0, we obtain the following expression

...

$$\forall \mathbf{s}, \mathbf{a} \in \mathcal{D}, k, \quad \hat{Q}^{k+1}(\mathbf{s}, \mathbf{a})=\hat{\mathcal{B}}^{\pi} \hat{Q}^{k}(\mathbf{s}, \mathbf{a})-\alpha \frac{\mu(\mathbf{a} \mid \mathbf{s})}{\hat{\pi}_{\beta}(\mathbf{a} \mid \mathbf{s})} \tag{11}\label{11}$$

Here's equation 1 from the paper.

$$\hat{Q}^{k+1} \leftarrow \arg \min _{Q} \alpha \mathbb{E}_{\mathbf{s} \sim \mathcal{D}, \mathbf{a} \sim \mu(\mathbf{a} \mid \mathbf{s})}[Q(\mathbf{s}, \mathbf{a})]+\frac{1}{2} \mathbb{E}_{\mathbf{s}, \mathbf{a} \sim \mathcal{D}}\left[\left(Q(\mathbf{s}, \mathbf{a})-\hat{\mathcal{B}}^{\pi} \hat{Q}^{k}(\mathbf{s}, \mathbf{a})\right)^{2}\right] \tag{1}\label{1}$$

My question is: what exactly is the derivative of equation (1)? And how does that result in equation (11)?

The $\hat{B}^\pi$ is the empirical Bellman operator and is defined as $\hat{B}^\pi \hat{Q}^k (s,a) = r + \gamma \sum_{s'} \hat{T}(s' \mid s,a) \mathbb{E}_{a'\sim \pi(a' \mid s')}\hat{Q}_k(s', a')$. Since in offline reinforcement learning, the dataset $\mathcal{D}$ typically does not contain all possible transitions $(s, a, s')$, the policy evaluation step actually uses an empirical Bellman operator that only backs up a single sample.

Answer 1

first we need to transform the distribution of the first term:

$\mathrm{argmin} \space \alpha \mathrm{E}_{s\sim D, a \sim D}[\frac{\mu}{\hat\pi_{\beta}}Q(s,a)] + \frac{1}{2}\mathrm{E}_{s,a \sim D}(Q(s,a)-\hat\beta^{\pi}\hat Q^k(s,a))^2 $

then its derivate respect to Q is:

$\alpha\frac{\mu}{\hat \pi ^{\beta}} + Q(s,a)-\hat\beta^{\pi}\hat Q^k(s,a) = 0$

then we have

$ Q(s,a) =\hat\beta^{\pi}\hat Q^k(s,a) -\alpha\frac{\mu}{\hat \pi ^{\beta}}$