Questions tagged [batch-size]

For questions related to the notion of batch size in machine learning.

Filter by
Sorted by
Tagged with
3 votes
1 answer
310 views

Isn't it useless to batch with repetition in RL?

When implementing batch sampling in RL learning, I saw that we are sampling with repetition, we use np.random.choice(record_range, batch_size) in Python for example,...
zdm's user avatar
  • 301
1 vote
0 answers
88 views

Why is it said transformers are more parallelizable than RNN's?

The parallelization of transformers and RNNs (Recurrent Neural Networks) is often discussed. It's commonly said that transformers are more parallelizable than RNNs. However, this is a rather vague ...
Erick Macias's user avatar
0 votes
0 answers
9 views

What are the main contributors to memory overload in NNs

In this article they talk about computational considerations for convnets. I used to think that the main bottleneck were the weights. They name: activations and gradients input data (like images) ...
Mah Neh's user avatar
  • 79
0 votes
0 answers
45 views

Progressive GAN breaks at higher resolution because of batch size?

I'm trying to reproduce the paper of progressive GANs, by following a tutorial available on github. All is well up until a resolution of 64: Then at the resolution of 128 all is going downhill pretty ...
Tim Kuipers's user avatar
0 votes
2 answers
65 views

Why does linearly decreasing batch sizes result in exponentially increasing training times?

I'm quite new to machine learning and wanted to ask a question regarding why reducing batch sizes cause exponentially increasing training times. An example of this relationship between batch size and ...
Alpy's user avatar
  • 3
0 votes
1 answer
87 views

Training a neural network in full batch training

It is a trend in deep learning to train models using multi-batches, i.e., to show the model a subset of the entire dataset for each weight update. In some cases, as in continual learning, we see that ...
Alfred's user avatar
  • 165
0 votes
1 answer
198 views

Why should data batches in a neural network have an equal size?

Why should data batches in a neural network have an equal size? I have seen some recent research works on making the batch size dynamic, but still, I can't find an answer to my question.
Minions's user avatar
  • 123
0 votes
1 answer
32 views

What can be an example other than batch normalization that uses statistics of batches?

Consider the following paragraph, taken from OPTIMIZING BATCHES of the textbook named Deep Learning with PyTorch by Eli Stevens et al., regarding the reasons for processing data into batches The ...
hanugm's user avatar
  • 3,581
0 votes
2 answers
199 views

Is it true that batch size of form $2^k$ gives better results?

I am confused among the following in selecting the batch size for my model. #1: powers of 2 I generally see that batch sizes are in powers of two: 32, 64, 128, 256. #2: maximum GPU Suppose my GPU ...
hanugm's user avatar
  • 3,581
0 votes
2 answers
1k views

Is there any relationship between the batch size and the number of epochs?

I am currently running a program with a batch size of 17 instead of batch size 32. The benchmark results are obtained at a batch size of 32 with the number of epochs 700. Now I am running with batch ...
hanugm's user avatar
  • 3,581
0 votes
1 answer
42 views

Will there be any changes in the model's performance due to the usage of very small batch sizes?

I am trying to run a code that has a batch size around 28. I can run the program on my GPU with this batch size. But, when I modify the code for my requirements and try to run, it is showing an run-...
hanugm's user avatar
  • 3,581
4 votes
1 answer
1k views

Why would a VAE train much better with batch sizes closer to 1 over batch size of 100+?

I've been training a VAE to reconstruct human names and when I train it on a batch size of 100+ after about 5 hours of training it tends to just output the same thing regardless of the input and I'm ...
user8714896's user avatar
2 votes
1 answer
829 views

Are there any rules for choosing batch size? [duplicate]

I am training a CNN with a batch size of 128, but I have some fluctuations in the validation loss, which are greater than one. I want to increase my batch size to 150 or 200, but, in the code examples ...
SahaTib's user avatar
  • 150
10 votes
1 answer
10k views

What is the relationship between gradient accumulation and batch size?

I am currently training some models using gradient accumulation since the model batches do not fit in GPU memory. Since I am using gradient accumulation, I had to tweak the training configuration a ...
JVGD's user avatar
  • 1,088
5 votes
2 answers
1k views

What is the purpose of the batch size in neural networks?

Why is a batch size needed to update the weights of a neural network? According to that Youtube Video from 3B1B, the weights are updated by calculating the error between expectation and outcome of ...
MScott's user avatar
  • 445
8 votes
2 answers
8k views

Effect of batch size and number of GPUs on model accuracy

I have a data set that was split using a fixed random seed and I am going to use 80% of the data for training and the rest for validation. Here are my GPU and batch size configurations use ...
bit_scientist's user avatar
2 votes
1 answer
422 views

What is the difference between validation percentage and batch size?

I'm doing transfer learning using Inception on Tensorflow. The code that I used for training is https://raw.githubusercontent.com/tensorflow/hub/master/examples/image_retraining/retrain.py If you take ...
gameon67's user avatar
  • 215
3 votes
1 answer
2k views

What is the reason behind using a test batch size?

If one examines the SSD: Single Shot MultiBox Detector code from this GitHub repository, it can be seen that, for a testing phase (evaluating network on test data set), there is a parameter ...
carobnodrvo's user avatar
24 votes
3 answers
76k views

How do I choose the optimal batch size?

Batch size is a term used in machine learning and refers to the number of training examples utilised in one iteration. The batch size can be one of three options: batch mode: where the batch size is ...
Sebastian Nielsen's user avatar
4 votes
2 answers
3k views

Is there a way to translate the concept of batch size into reinforcement learning?

I am using a neural network as my function approximator for reinforcement learning. In order to get it to train well, I need to choose a good learning rate. Hand-picking one is difficult, so I read up ...
user10011538's user avatar