3
$\begingroup$

When implementing batch sampling in RL learning, I saw that we are sampling with repetition, we use np.random.choice(record_range, batch_size) in Python for example, that can return the same element multiple times. Why one would sample the element more than once? Isn't this useless?

$\endgroup$
1
  • 1
    $\begingroup$ Example: let's say you have this distribution: 10 As, 1 billion Bs, 1 C. You sample without replacement 3 times. What distribution do you have? Is that representative of the actual one? $\endgroup$
    – nbro
    Aug 26, 2023 at 1:50

1 Answer 1

5
$\begingroup$

Pythons random.choice may return the same sample within a single batch. Although you could view this as redundant data, it is still a fair sample statistically, so does not adversely affect results. In addition, the agent may already have sampled the same thing multiple times, resulting in repeated entries in memory. For the best approximation, this is often desirable - approximation quality is usually considered in terms of loss weighted by frequency of data (or sample density with continuous inputs).*

You could easily code a sample without replacement approach if you want - the difference in performance will be undetectable. So most people implementing this don't worry about this detail.


* In RL, there is a dilemma closely related to the exploration vs exploitation problem. Training a value approximator only on a target optimal policy's state/action distribution will produce the best approximations for action values for the real state distribution and on-policy decisions. However, it will tend to forget off-policy details, and may perform much worse as a result (even though it would have a low cost metric if treated as a supervised learning problem on predicting values for the optimal policy).

Hence training with at least a little off-policy data, compromising accuracy for resilience, is often preferred. Taking it further are approaches like prioritised sampling which ignore experienced distribution of states anyway. So using "fair" samples is not some gold standard in RL, often we deliberately choose alternatives.

$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .