Questions tagged [batch-size]
For questions related to the notion of batch size in machine learning.
20
questions
3
votes
1
answer
310
views
Isn't it useless to batch with repetition in RL?
When implementing batch sampling in RL learning, I saw that we are sampling with repetition, we use np.random.choice(record_range, batch_size) in Python for example,...
- 301
1
vote
0
answers
88
views
Why is it said transformers are more parallelizable than RNN's?
The parallelization of transformers and RNNs (Recurrent Neural Networks) is often discussed. It's commonly said that transformers are more parallelizable than RNNs. However, this is a rather vague ...
- 11
0
votes
0
answers
9
views
What are the main contributors to memory overload in NNs
In this article they talk about computational considerations for convnets.
I used to think that the main bottleneck were the weights. They name:
activations and gradients
input data (like images)
...
- 79
0
votes
0
answers
45
views
Progressive GAN breaks at higher resolution because of batch size?
I'm trying to reproduce the paper of progressive GANs, by following a tutorial available on github.
All is well up until a resolution of 64:
Then at the resolution of 128 all is going downhill pretty ...
- 111
0
votes
2
answers
65
views
Why does linearly decreasing batch sizes result in exponentially increasing training times?
I'm quite new to machine learning and wanted to ask a question regarding why reducing batch sizes cause exponentially increasing training times. An example of this relationship between batch size and ...
- 3
0
votes
1
answer
87
views
Training a neural network in full batch training
It is a trend in deep learning to train models using multi-batches, i.e., to show the model a subset of the entire dataset for each weight update. In some cases, as in continual learning, we see that ...
- 165
0
votes
1
answer
198
views
Why should data batches in a neural network have an equal size?
Why should data batches in a neural network have an equal size?
I have seen some recent research works on making the batch size dynamic, but still, I can't find an answer to my question.
- 123
0
votes
1
answer
32
views
What can be an example other than batch normalization that uses statistics of batches?
Consider the following paragraph, taken from OPTIMIZING BATCHES of the textbook named Deep Learning with PyTorch by Eli Stevens et al., regarding the reasons for processing data into batches
The ...
- 3,581
0
votes
2
answers
199
views
Is it true that batch size of form $2^k$ gives better results?
I am confused among the following in selecting the batch size for my model.
#1: powers of 2
I generally see that batch sizes are in powers of two: 32, 64, 128, 256.
#2: maximum GPU
Suppose my GPU ...
- 3,581
0
votes
2
answers
1k
views
Is there any relationship between the batch size and the number of epochs?
I am currently running a program with a batch size of 17 instead of batch size 32. The benchmark results are obtained at a batch size of 32 with the number of epochs 700.
Now I am running with batch ...
- 3,581
0
votes
1
answer
42
views
Will there be any changes in the model's performance due to the usage of very small batch sizes?
I am trying to run a code that has a batch size around 28. I can run the program on my GPU with this batch size.
But, when I modify the code for my requirements and try to run, it is showing an run-...
- 3,581
4
votes
1
answer
1k
views
Why would a VAE train much better with batch sizes closer to 1 over batch size of 100+?
I've been training a VAE to reconstruct human names and when I train it on a batch size of 100+ after about 5 hours of training it tends to just output the same thing regardless of the input and I'm ...
- 717
2
votes
1
answer
829
views
Are there any rules for choosing batch size? [duplicate]
I am training a CNN with a batch size of 128, but I have some fluctuations in the validation loss, which are greater than one. I want to increase my batch size to 150 or 200, but, in the code examples ...
- 150
10
votes
1
answer
10k
views
What is the relationship between gradient accumulation and batch size?
I am currently training some models using gradient accumulation since the model batches do not fit in GPU memory. Since I am using gradient accumulation, I had to tweak the training configuration a ...
- 1,088
5
votes
2
answers
1k
views
What is the purpose of the batch size in neural networks?
Why is a batch size needed to update the weights of a neural network?
According to that Youtube Video from 3B1B, the weights are updated by calculating the error between expectation and outcome of ...
- 445
8
votes
2
answers
8k
views
Effect of batch size and number of GPUs on model accuracy
I have a data set that was split using a fixed random seed and I am going to use 80% of the data for training and the rest for validation.
Here are my GPU and batch size configurations
use ...
- 241
2
votes
1
answer
422
views
What is the difference between validation percentage and batch size?
I'm doing transfer learning using Inception on Tensorflow. The code that I used for training is https://raw.githubusercontent.com/tensorflow/hub/master/examples/image_retraining/retrain.py
If you take ...
- 215
3
votes
1
answer
2k
views
What is the reason behind using a test batch size?
If one examines the SSD: Single Shot MultiBox Detector code from this GitHub repository, it can be seen that, for a testing phase (evaluating network on test data set), there is a parameter ...
- 143
24
votes
3
answers
76k
views
How do I choose the optimal batch size?
Batch size is a term used in machine learning and refers to the number of training examples utilised in one iteration. The batch size
can be one of three options:
batch mode: where the batch size is ...
4
votes
2
answers
3k
views
Is there a way to translate the concept of batch size into reinforcement learning?
I am using a neural network as my function approximator for reinforcement learning. In order to get it to train well, I need to choose a good learning rate. Hand-picking one is difficult, so I read up ...
- 41