1
$\begingroup$

The concept of experience replay is saving our experiences in our replay buffer. We select at random to break the correlation between consecutive samples, right?

What would happen if we calculate our loss using just one experience instead of a mini-batch of experiences?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
0
$\begingroup$

The concept of experience replay is saving our experiences in our replay buffer. We select at random to break the correlation between consecutive samples, right?

Yes that is a major benefit of using a replay buffer.

A secondary benefit is the ability to use the same sample more than once. This can lead to beter sample efficiency, although that is not guaranteed.

What would happen if we calculate our loss using just one experience instead of a mini-batch of experiences?

The algorithm is still valid, but the gradient estimate for the update step would be based on a single record of [state, action, reward, next state]. This would be a high variance update process, with many steps in wrong directions, but in expectation over many steps you should still see a correct gradient. You would probably need to compensate for the high variance per sample by reducing the learning rate.

In addition, assuming the standard approach of collecting one time step then making one update to DQN neural network, each piece of experience would only be used once on average before being discarded.

These two effects will likely combine such that the learning process would not be very sample efficient.

The size of the minibatch is one of many hyperparameters you can change in DQN. It might be the case for some problems that choosing a low minibatch size is helpful, provided other adjustments (such as a lower learning rate) are made along with it. If you are not sure, you mostly have to try and see.

In my experience on a small range of problems, a moderate size of minibatch - ranging from 10 to 100 - has worked the best in terms of end results of high scoring agents. However, I have not spent a long time trying to make low batch sizes work.

Improve this answer
$\endgroup$
5
  • $\begingroup$ Can you simplify what you mean by a high variance update ?, does this mean it would give a higher divergence than when a batch is used? $\endgroup$
    – Chukwudi
    Jun 22, 2020 at 20:40
  • $\begingroup$ @Chukwudi: High variance = "varies a lot". Statistcally in the sense of mean and variance. If you take more samples and take the mean (which is what a batch does), then the variance of the expected mean is divided by the number of samples. $\endgroup$
    – Neil Slater
    Jun 22, 2020 at 21:01
  • $\begingroup$ Oh so the higher number of samples the more accurate the measurements, so the lower the variance right $\endgroup$
    – Chukwudi
    Jun 22, 2020 at 21:03
  • $\begingroup$ @Chukwudi: Yes. But of course if takes more time and effort to collect those samples. Meaning you take longer to make updates to your NN. So it is not always clear whether to take small or large batches. $\endgroup$
    – Neil Slater
    Jun 22, 2020 at 21:05
  • $\begingroup$ I understand now, thank you so much $\endgroup$
    – Chukwudi
    Jun 22, 2020 at 21:06

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .