2
$\begingroup$

I am working on a PyTorch implementation of Implicit Q-Learning (IQL) (paper), given a dataset $\mathcal D = \left\{ (\mathbf s_i, \mathbf a_i, \mathbf s_i', r_i ) \right\}$ of transitions. I think I have implemented IQL correctly, so now I have a learned policy $\pi$ that takes in an element of the state space $\mathcal S$, and outputs a mean vector and covariance diagonals for a multivariate normal distribution over the action space $\mathcal A = \mathbb R^d$. I would like to evaluate this learned policy $\pi$.

This leads me to investigate off-policy evaluation, specifically Fit Q Evaluation (FQE), given a test dataset of transitions $\mathcal D_e$ as above. I have been reading the offline reinforcement learning paper by Prudencio et al. (link). My problem is that I do not understand how to implement FQE as described in the paper, so I would appreciate guidance for this. Below I quote the relevant part of the paper and ask some questions.

C. Fit Q Evaluation

In fit Q evaluation (FQE), we first train a $Q$-function $Q_{\phi}^{\pi}$ by minimizing the Bellman error under the policy $\pi$. Then, we evaluate the policy by computing the average expected return over the states and actions from $\mathcal D_e$, such that \begin{align*} \hat J(\pi) = \mathbb E_{\mathbf s, \mathbf a \sim \mathcal D_e} \left[ Q_{\phi}^{\pi}(\mathbf s, \mathbf a) \right]. \end{align*}

Questions:

  1. Presumably I should use a neural network for $Q_{\phi}^{\pi}$, right?
  2. What is the Bellman error in this context? I have some course notes that give a definition of the Bellman error involving $\max_{\mathbf a'}$, which is not workable since my action space $\mathcal A$ is $\mathbb R^d$ for some $d \in \mathbb Z$.

I have looked at the off-policy evaluation paper by Voloshin et al. (link), which seems to describe FQE in a different way. Specifically,

$\hat Q(\cdot, \theta) = \lim_{k \to \infty} \hat Q_k$, where \begin{align*} \hat Q_k &= \min_{\theta} \frac{1}{N} \sum_{i = 1}^N \sum_{t = 0}^{\tilde T} \left( \hat Q_{k - 1}(x_t^i, a_t^i ; \theta) - y_t^i \right)^2\\ y_t^i &\equiv r_t^i + \gamma \mathbb E_{\pi_e} \hat Q_{k - 1}(x_{t + 1}^i, \cdot ; \theta) \end{align*}

If I understand this correctly, we fit a sequence of neural network $Q$ functions $\hat Q_1, \hat Q_2, \dots$ here, trying to minimize the trajectory-average squared distance between $\hat Q_{k - 1}(x_t^i, a_t^i ; \theta)$ and $y_t^i$. But how do I calculate $y_t^i$ here? I don't know what $\mathbb E_{\pi_e} \hat Q_{k - 1}(x_{t + 1}^i, \cdot ; \theta)$ means.

I appreciate any help.

$\endgroup$
1
  • $\begingroup$ Bellman error in general is the MSE error between your estimate of $Q(s, \pi(s))$ and your target, e.g. $r + \gamma Q(s', \pi(s'))$ (keeping notation simple here for how we obtain actions from policy). Your notes probably use a max because it will have been introduced in the context of Q-learning. $\endgroup$
    – David
    Commented Feb 29 at 10:07

1 Answer 1

0
$\begingroup$

Did you check the original paper? See algorithm 3.

Batch Policy Learning under Constraints

Here is an implementation I found: https://github.com/clvoloshin/COBS/blob/master/ope/algos/fqe.py See their OPE paper for details: http://arxiv.org/abs/1911.06854

$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .