1
$\begingroup$

I am looking at the paper Conservative Q-Learning for Offline Reinforcement Learning, but I'm not sure how they proved theorem 3.1.

Here is a screenshot of theorem 3.1.

enter image description here

In the proof of theorem 3.1

enter image description here

they say

By setting the derivative of Equation 1 to 0, we obtain the following expression

...

$$\forall \mathbf{s}, \mathbf{a} \in \mathcal{D}, k, \quad \hat{Q}^{k+1}(\mathbf{s}, \mathbf{a})=\hat{\mathcal{B}}^{\pi} \hat{Q}^{k}(\mathbf{s}, \mathbf{a})-\alpha \frac{\mu(\mathbf{a} \mid \mathbf{s})}{\hat{\pi}_{\beta}(\mathbf{a} \mid \mathbf{s})} \tag{11}\label{11}$$

Here's equation 1 from the paper.

$$\hat{Q}^{k+1} \leftarrow \arg \min _{Q} \alpha \mathbb{E}_{\mathbf{s} \sim \mathcal{D}, \mathbf{a} \sim \mu(\mathbf{a} \mid \mathbf{s})}[Q(\mathbf{s}, \mathbf{a})]+\frac{1}{2} \mathbb{E}_{\mathbf{s}, \mathbf{a} \sim \mathcal{D}}\left[\left(Q(\mathbf{s}, \mathbf{a})-\hat{\mathcal{B}}^{\pi} \hat{Q}^{k}(\mathbf{s}, \mathbf{a})\right)^{2}\right] \tag{1}\label{1}$$

My question is: what exactly is the derivative of equation (1)? And how does that result in equation (11)?

The $\hat{B}^\pi$ is the empirical Bellman operator and is defined as $\hat{B}^\pi \hat{Q}^k (s,a) = r + \gamma \sum_{s'} \hat{T}(s' \mid s,a) \mathbb{E}_{a'\sim \pi(a' \mid s')}\hat{Q}_k(s', a')$. Since in offline reinforcement learning, the dataset $\mathcal{D}$ typically does not contain all possible transitions $(s, a, s')$, the policy evaluation step actually uses an empirical Bellman operator that only backs up a single sample.

$\endgroup$
3
  • 1
    $\begingroup$ Can you actually format the equations as text, rather than an image? That would make it a lot more accessible. $\endgroup$ Sep 15 at 13:10
  • 1
    $\begingroup$ Yes, please, next time, rather than providing the screenshots, for future reproducibility and easy accessibility, copy and quote the relevant parts of text from the paper, at least, as I've just done to your post. To get the equations in latex automatically, you can use a tool like "Mathpix Snipping Tool" (which is what I used). $\endgroup$
    – nbro
    Sep 15 at 13:12
  • $\begingroup$ I apologize, I will try to copy the relevant parts. And thanks for recommending mathpix snipping tools! $\endgroup$
    – MoneyBall
    Sep 15 at 13:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.