# Fit Q Evaluation in offline reinforcement learning

I am working on a PyTorch implementation of Implicit Q-Learning (IQL) (paper), given a dataset $$\mathcal D = \left\{ (\mathbf s_i, \mathbf a_i, \mathbf s_i', r_i ) \right\}$$ of transitions. I think I have implemented IQL correctly, so now I have a learned policy $$\pi$$ that takes in an element of the state space $$\mathcal S$$, and outputs a mean vector and covariance diagonals for a multivariate normal distribution over the action space $$\mathcal A = \mathbb R^d$$. I would like to evaluate this learned policy $$\pi$$.

This leads me to investigate off-policy evaluation, specifically Fit Q Evaluation (FQE), given a test dataset of transitions $$\mathcal D_e$$ as above. I have been reading the offline reinforcement learning paper by Prudencio et al. (link). My problem is that I do not understand how to implement FQE as described in the paper, so I would appreciate guidance for this. Below I quote the relevant part of the paper and ask some questions.

C. Fit Q Evaluation

In fit Q evaluation (FQE), we first train a $$Q$$-function $$Q_{\phi}^{\pi}$$ by minimizing the Bellman error under the policy $$\pi$$. Then, we evaluate the policy by computing the average expected return over the states and actions from $$\mathcal D_e$$, such that \begin{align*} \hat J(\pi) = \mathbb E_{\mathbf s, \mathbf a \sim \mathcal D_e} \left[ Q_{\phi}^{\pi}(\mathbf s, \mathbf a) \right]. \end{align*}

Questions:

1. Presumably I should use a neural network for $$Q_{\phi}^{\pi}$$, right?
2. What is the Bellman error in this context? I have some course notes that give a definition of the Bellman error involving $$\max_{\mathbf a'}$$, which is not workable since my action space $$\mathcal A$$ is $$\mathbb R^d$$ for some $$d \in \mathbb Z$$.

I have looked at the off-policy evaluation paper by Voloshin et al. (link), which seems to describe FQE in a different way. Specifically,

$$\hat Q(\cdot, \theta) = \lim_{k \to \infty} \hat Q_k$$, where \begin{align*} \hat Q_k &= \min_{\theta} \frac{1}{N} \sum_{i = 1}^N \sum_{t = 0}^{\tilde T} \left( \hat Q_{k - 1}(x_t^i, a_t^i ; \theta) - y_t^i \right)^2\\ y_t^i &\equiv r_t^i + \gamma \mathbb E_{\pi_e} \hat Q_{k - 1}(x_{t + 1}^i, \cdot ; \theta) \end{align*}

If I understand this correctly, we fit a sequence of neural network $$Q$$ functions $$\hat Q_1, \hat Q_2, \dots$$ here, trying to minimize the trajectory-average squared distance between $$\hat Q_{k - 1}(x_t^i, a_t^i ; \theta)$$ and $$y_t^i$$. But how do I calculate $$y_t^i$$ here? I don't know what $$\mathbb E_{\pi_e} \hat Q_{k - 1}(x_{t + 1}^i, \cdot ; \theta)$$ means.

I appreciate any help.