Does more data increase training accuracy in neural networks?

Question

I was wondering what is the performance benefit of feeding more data to a machine learning model like a neural network? Like I know one of the benefits is that it increases generalization - testing accuracy, but I was wondering if does it affect training accuracy of the model?

Answer 1

The question use specific terms in a vague way so let me set some very basic ground definitions first. It might sounds trivial but please bear with me cause it's easy to give reasonable answers that in reality make no sense.

data: any unprocessed fact, value, text, sound, or picture that is not being interpreted and analyzed. It can be real, i.e. gathered trough real observations/experiments or synthetic, i.e. artificially constructed based on some hand crafted distribution.

training/test data: different subset of data, the difference being that testing are not used to train or tune the model parameters and hyperparameters. Important to note is that we don't always have knowledge about the real distribution of the data, meaning that the distribution of our training data many times do not match the distribution of testing data.

accuracy: it's a specific metric used to evaluate only a specific subset of machine learning tasks among which (and mostly) classification. Is also a pretty unreliable metric in some circumstances like multi class classification or unbalanced datasets.

generalization: the ability of a model to perform well on unseen data. In principle it has nothing to do with the metrics used to evaluate a model, even though metrics scores are the only tool we have to assess it.

You ask if using more data can increase training accuracy and you already pointed out that using more data is meant to increase generalization, which you consider equal to testing accuracy. You`re right when you say that adding more data serve the purpose of increasing generalization, but as I wrote in the definition of generalization, in principle we can't always expect a linear relation between a model generalization and its metrics scores, and in fact adding data might as well decrease model generalization in some situations.

As a basic example let's consider an imbalanced dataset with 90% instances belonging to class A and 10% instances belonging to class B. A model will easily learn to overfitt the data predicting only class A, still reaching 90% training accuracy. In test phase we might even observe a similar score if the distribution match the 90/10 ratio of training data. To prevent overfitting and increase generalization, we add instances of class B to make the dataset balanced, i.e. 50% instances class A 50% instances class B. Suddenly the model works perfectly, and reach 100% training accuracy. We see tough that in test phase the accuracy drops to 40%. How come? If the test indeed had the same 90/10% distribution among classes A and B, training the model on a 50/50% distribution teach the model to overpredict class B, so the model is now predicting class B, which is why we added more data, but it's now predicting too many times class B, leading again to poor generalization.

Of course this is a toy scenario, but be aware that classic data augmentation and synthetic data always introduce the problem of introducing biases in the training distribution. Also, note that using a different metric like f score alongside accuracy would let you catch immediately if the model is generalizing more or if it's only an artifact produced by the accuracy metric (like the initial 90% score when totally overfitting).

Answer 2

Using more training samples decreases the chance of over-fitting. However, I think, it may not occasionally result in a decrease in the training error, maybe the opposite (look at the loss function definition). for instance, if you have only very few samples a very deep network is capable of memorizing them all which makes training error = 0.