What is batch / batch size in neural networks?

Question

I have some problems with understanding of the batch concept and batch size. I messed something up. First I start it consider based on convolutional neural network I heard two versions:

1. When the batch size is set to 50, the first network is fed with 50 images and then learned / recalculated (it doesn't make sense to me, because in this case the network learns one of 50 images).
2. When the batch size is set to 50, one of 50 neurons is recalculated in the learning process on a single image.

Both of these explanations seem to be wrong to me, so I assume, that I completely don't understand this. What is batch / batch size in RNN? Could you show any example?

I can tell you how I would teach a recurrent neural network. Let's say, that I would like to teach a neural network to predict the weather the next day:

1. I would take weather data from an expected area from the last 30,000 days.
2. I would assume that my prediction would be based on measurements from the last 365 days.
3. I would take data from day 1 to 365 - feed RNN with it and learn.
4. Then I would take data from day 2 to 366 => feed + learn
5. Then day 3 to 367 => feed + learn
6. And so on.

Is this 365 measurement concept a batch size?

Answer 1

You are confusing two concepts, batch size and window size.

Batch Size

The concept of a batch is related to your data and not the number of neurons. A batch is just a subset of your training data. The size of the batch affects computational speed, quality, and accuracy. Large batch sizes will train faster than smaller ones but the model's accuracy can suffer.

There is a rule of thumb that a batch size should be a power of two (e.g. 32, 64, 128, etc.).

Generally speaking larger batch sizes do not generalize as well as smaller batch sizes. You will need to experiment with the batch size to achieve optimal performance. Batch size is considered a hyperparameter.

The following video, Batch Size in a Neural Network explained should help clarify things.

Window Size

Window size applies to time series and sequences when using recurrent style networks. For your case, 365 is the window size. You did not mentioned how far in the future you are trying to predict. If we assume that you are predicting one day in the future your output would be the 366th measurement.

Your Application

A batch for you would be a number of training sets. For example, if we used 32 as a batch size, you would have 32 sets of input windows and outputs. The first two batches might look like the following:

Batch 1, size 32
1 Data from days 1 to 365 as the input, day 366 as the output
2 Data from days 2 to 366 as the input, day 367 as the output
...
32 Data from days 32 to 396 as the input, day 397 as the output

Batch 2, size 32
1 Data from days 33 to 397 as the input, day 398 as the output
2 Data from days 34 to 398 as the input, day 399 as the output
...