How to sample the tuples during the initial time steps of the DDPG algorithm?

Question

I am facing an issue in understanding the following line from the pseudocode of the DDPG algorithm

Sample a random minibatch of $N$ transitions $(s_i, a_i, r_i, s_{i+1})$ from $R$

Here $N$ is a hyperparameter that is equal to the number of transitions or samples that need to be present in the minibatch.

Traditionally, we take a minibatch of samples from a dataset, which contains all samples, and pass them to a neural network, but if we observe the DDPG pseudocode, we are storing transition after transition into the buffer $R$. So, I think, it needs several steps before sampling from the buffer $R$. But, if we observe the pseudocode, it says that we need to sample from the first timestep of the first episode.

How is it possible? Where am I missing?

Answer 1

You have a free choice to either:

• Wait until the replay buffer hits a minimum size for sampling.

• Take smaller samples from the buffer initially, until the buffer is large enough. On the first time step the minibatch size will be 1.

Both these approaches are valid, and neither changes the DDPG algorithm significantly. Plenty of other hyperparameters for DDPG will have bigger effects on the end result.

The Keras implementation of DDPG here https://keras.io/examples/rl/ddpg_pendulum/ uses the second approach.

The pseudocode in the paper is not 100% clear, but I would expect they did the same.