I am implementing the A3C algorithm and I want to add off-policy training using Retrace but I am having some trouble understanding how to compute the retrace target. Retrace is used in combination with A3C for example in the Reactor.

I often see the retrace update written as

\begin{equation} \Delta Q(s, a) = \sum_{t' = t}^{T} \gamma^{t'-t}\left(\prod_{j=t+1}^{t'}c_j\right) \delta_{t'} \end{equation}

with $\delta_{t'} = r(s_{t'}, a_{t'}) + \gamma \mathbb{E}[Q(s_{t'+1}, a_{t'+1})] - Q(s_{t'}, a_{t'})$ and $c_j$ being the Retrace factors $c_j = \lambda \min(c, \frac{\pi(a_j|s_j)}{b(a_j|s_j)})$.

Now, when employing neural networks to approximate $Q_{\theta}(s, a)$ it is often easier to define a loss \begin{equation} \mathcal{L}_{\theta} = \left(G_t - Q(s, a)\right)^2 \end{equation} and let the backward function and the optimizer do the update. How can I write the Retrace target $G_t$ to use in such a setup?

Is it correct to write it as follows? \begin{equation} G_t = \sum_{t'=t}^T \gamma^{t'-t} \left(\prod_{j=t+1}^{t'}c_j\right) (r_{t'} + \gamma Q(s_{t'+1}, a_{t'+1}) - Q(s_{t'}, a_{t'})) \end{equation}

and then compute $\mathcal{L}$ as above, take the gradient $\nabla\mathcal{L}_{\theta}$ and perform the update step $Q(s_t, a_t) = Q(s_t, a_t) + \alpha \nabla\mathcal{L}_{\theta}$ ?


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.