3
$\begingroup$

I am sorry but I have to explain my question using an example, I do not know how to ask it in proper scientific terms.

Let's assume, I have trained a deep learning model on classifying hand gestures, but training and testing datasets' images are shot only in one lighting conditions and I achieved certain accuracy, let's assume 85%. As far as I understand, adding more data of the same hand gestures images but shot with different lightning should increase my model's "generalization" capabilities, right?

So the question is, if I take this model, trained in two lightning conditions, and test it only on the dataset of the first lightning conditions, would that increase it's accuracy (the 85%) or maybe this "generalization" would only mean that it can now also classify correctly images with different lightning, but not increase the accuracy on the first set?

$\endgroup$
1
$\begingroup$

I think there's a crucial point missed in the question, touched by jros answer but without further elaboration.

If you train a model on domain A: single lightning condition and test it on domain B: two lightning condition then you're not evaluating generalization but transfer learning capabilities. Or to phrase it differently you're evaluating how close domain A and B are for the model you trained.

The test set as you said is truly made of instances never seen by the model during training, but it should nevertheless be representative, i.e. correctly sampled, from the training domain, or from the same distribution as jros wrote. So the generalization of your model, trained on single lightning condition, should be evaluated on single lighting condition as well.

A final remark about the rest that have been said:

  • everything holds only under the assumptions that the initial training dataset is not only unbiased but also balanced. In a real case scenario changing the training distribution from something specific (single light condition) to another distribution (multiple light conditions) might well be lead to a worse model, simply cause the problem is now inherently harder to solve.

So the answer to your question (regarding both, true generalization on same distribution and what you describe, transfer learning) is actually just empirical.

$\endgroup$
3
  • $\begingroup$ Although you say this in other parts of your answer, when you write "So the generalization of your model, trained on single lightning condition, should be evaluated on single lighting condition as well.", I think you should also emphasize that this is true only if you're trying to learn a target function where the inputs are assumed to have the same lighting condition and you assume that your training data is representative of your target function or probability distribution (you say that in other parts of the answer, but that sentence read alone may be misleading). $\endgroup$
    – nbro
    Sep 29 at 1:10
  • $\begingroup$ I dont exactly need my model trained on domain A to test on domain B. My situation is that my model performs not as expected on domain A only images, I am waiting to get new data from other domains, and i wonder whether training model again on set with samples from additional domains would also help the performance on set A, or do I have to fix something within the model first. $\endgroup$
    – Makintosz
    Oct 2 at 4:32
  • $\begingroup$ Amd this has led me to a bit more general theoretical question that I asked, cus I was hoping that since most of the features from domain B and C and D would be same as in A (cus general shapes of gestures are always the same), maybe it will also help domain A. $\endgroup$
    – Makintosz
    Oct 2 at 4:36
0
$\begingroup$
Generalization

In machine learning, generalization describes a model's ability to properly correct its algorithms to predict new data from the same distribution as the data used to train the model.

By providing additional training for your model (on data with varying lighting conditions), you are correct that you would be increasing the capabilities of your model.

Does better generalization equal better performance?

Consider two models and assume they have the same number of coefficients/ layers & nodes:

  • Model 1, which was trained solely on a single lighting condition
  • Model 2, which was trained on multiple lighting conditions

Let's say we have two test sets as well:

  • Test Set 1 : data with the same lighting condition as the data used to train Model 1
  • Test Set 2 : data with the multiple lighting conditions

Model 1 would be expected to have an equal or better performance than Model 2 on Test Set 1, due to "over" fitting on that lighting condition, but as you noted, Model 1 would not perform as well on Test Set 2 as on Test Set 1. We also would not expect Model 2 to perform better on Test Set 1 than Model 1 due to the generalizability achieved during training.

Simply put, you're probably sacrificing some lighting condition-specific accuracy for better accuracy across multiple lighting conditions.

However

By allowing your Model 2 to increase the layers & nodes, or coefficients (including interactions), Model 2 may well be capable of performing just as well. All this depends on the size of training sets as well. For instance, if Model 1 is trained on 1,000 data points from the single lighting condition, and Model 2 is trained on 500 data points, Model 1 is generally expected to perform better on Test Set 1.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.