In Deep Q Learning algorithm the convergence is generally achieved using smart tricks like the target network and the replay buffer.

However there is one thing which is not clear to me. When the Q network is trained through SGD, the loss function is an expectation over all possible states, which is approximated stochastically using samples from the replay buffer. But the replay buffer itself is not constructed from a unique stationary distribution: the behaviour policy used to collect state transitions typically changes during the overall training procedure (it typically becomes more greedy as new data are collected).

As a result, we draw sample transitions from the replay buffer hoping to obtain a useful stochastic approximation of some stationary distribution, but those data were not actually drawn from a stationary distribution.

Why isn't this an issue for the SGD procedure?


1 Answer 1


DQN is a off-policy method, this means that you are fine (apart from some variance factor) with sampling data from another distribution

However, Replay Buffer have a state distribution that is different from the new policy one, which might create some problem

SGD on the other hand, with such method, has the problem of the fact that its convergence guarantees assumes that the samples are IID, which is not true in case of replay buffer (and generally, in most RL scenarios)


You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .