# What is the double sample problem in reinforcement learning?

According to the SBEED: Convergent Reinforcement Learning with Nonlinear Function Approximation for convergent reinforcement learning, the Smoothed Bellman operator is a way to dodge the double sample problem? Can someone explain to me what the double sample problem is and how SBEED solves it?

$$w_{t+1} = w_t + \alpha[\mathbb{E}_b[\rho_t[R_{t+1} + \gamma\hat{v}(S_{t+1},\mathbf{w})] - \hat{v}(S_{t},\mathbf{w})][\nabla\hat{v}(S_{t},\mathbf{w})- \gamma\mathbb{E}_b[\rho_t\nabla\hat{v}(S_{t+1},\mathbf{w})]]$$
[...] But this is naive, because the equation above involves the next state, $$S_{t+1}$$, appearing in two expectations that are multiplied together. To get an unbiased sample of the product, two independent samples of the next state are required, but during normal interaction with an external environment only one is obtained. One expectation or the other can be sampled, but not both.
Basically, unless you have an environment that you can routinely re-wind and re-sample to get two independent estimates (for $$\hat{v}(S_{t+1},\mathbf{w})$$ and $$\nabla\hat{v}(S_{t+1},\mathbf{w})$$) then the update rule that naturally arises from gradient descent on the Bellman error does will work any better than other approaches, such as semi-gradient methods. If you can do this rewind process on every step, then it may be worth it because of the guarantees of convergence, even in off-policy with non-linear approximators.