How general is generalization?

Question

I am sorry but I have to explain my question using an example, I do not know how to ask it in proper scientific terms.

Let's assume, I have trained a deep learning model on classifying hand gestures, but training and testing datasets' images are shot only in one lighting conditions and I achieved certain accuracy, let's assume 85%. As far as I understand, adding more data of the same hand gestures images but shot with different lightning should increase my model's "generalization" capabilities, right?

So the question is, if I take this model, trained in two lightning conditions, and test it only on the dataset of the first lightning conditions, would that increase it's accuracy (the 85%) or maybe this "generalization" would only mean that it can now also classify correctly images with different lightning, but not increase the accuracy on the first set?

Answer 1

I think there's a crucial point missed in the question, touched by jros answer but without further elaboration.

If you train a model on domain A: single lightning condition and test it on domain B: two lightning condition then you're not evaluating generalization but transfer learning capabilities. Or to phrase it differently you're evaluating how close domain A and B are for the model you trained.

The test set as you said is truly made of instances never seen by the model during training, but it should nevertheless be representative, i.e. correctly sampled, from the training domain, or from the same distribution as jros wrote. So the generalization of your model, trained on single lightning condition, should be evaluated on single lighting condition as well.

A final remark about the rest that have been said:

• everything holds only under the assumptions that the initial training dataset is not only unbiased but also balanced. In a real case scenario changing the training distribution from something specific (single light condition) to another distribution (multiple light conditions) might well be lead to a worse model, simply cause the problem is now inherently harder to solve.

So the answer to your question (regarding both, true generalization on same distribution and what you describe, transfer learning) is actually just empirical.

Answer 2

Generalization

In machine learning, generalization describes a model's ability to properly correct its algorithms to predict new data from the same distribution as the data used to train the model.

By providing additional training for your model (on data with varying lighting conditions), you are correct that you would be increasing the capabilities of your model.

Does better generalization equal better performance?

Consider two models and assume they have the same number of coefficients/ layers & nodes:

• Model 1, which was trained solely on a single lighting condition
• Model 2, which was trained on multiple lighting conditions

Let's say we have two test sets as well:

• Test Set 1 : data with the same lighting condition as the data used to train Model 1
• Test Set 2 : data with the multiple lighting conditions

Model 1 would be expected to have an equal or better performance than Model 2 on Test Set 1, due to "over" fitting on that lighting condition, but as you noted, Model 1 would not perform as well on Test Set 2 as on Test Set 1. We also would not expect Model 2 to perform better on Test Set 1 than Model 1 due to the generalizability achieved during training.

Simply put, you're probably sacrificing some lighting condition-specific accuracy for better accuracy across multiple lighting conditions.

However

By allowing your Model 2 to increase the layers & nodes, or coefficients (including interactions), Model 2 may well be capable of performing just as well. All this depends on the size of training sets as well. For instance, if Model 1 is trained on 1,000 data points from the single lighting condition, and Model 2 is trained on 500 data points, Model 1 is generally expected to perform better on Test Set 1.