0
$\begingroup$

Consider the following paragraph, taken from OPTIMIZING BATCHES of the textbook named Deep Learning with PyTorch by Eli Stevens et al., regarding the reasons for processing data into batches

The reason we want to do this batching is multifaceted. One big motivation is to make sure the computation we’re asking for is big enough to saturate the computing resources we’re using to perform the computation. GPUs in particular are highly parallelized, so a single input on a small model will leave most of the computing units idle. By providing batches of inputs, the calculation can be spread across the otherwise-idle units, which means the batched results come back just as quickly as a single result would. Another benefit is that some advanced models use statistical information from the entire batch, and those statistics get better with larger batch sizes.

The bolded portion says a statistical reason for batch processing. I know only batch normalization that satisfies the portion as mean and standard deviation are calculated over the batch of inputs.

Is there any such example that needs statistics of batches that are apt to the description?

$\endgroup$

1 Answer 1

1
$\begingroup$

Here's some examples:

Notice how in general different normalization techniques are implemented using as a backbone the same idea of batch normalization, i.e. computing mean and standard deviation, but along a different dimension rather than the batch one, or on a subset of points (image below taken from the switchable normalization paper)

enter image description here

The alternative, as performed by the attentive and spectral normalization authors, is to rethink the normalization as something different from the classic linear transformation $\mu x + \sigma$, or to add extra parameters to learn, to make the normalization more non linear (image taken from the attentive normalization paper).

enter image description here

$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .