If I do supervised learning the model learns from the labeled input data. This seems to be quite often a small set of human annotated data.

Is it true to say this is the only 'learning' the model does?

It seems like the small data set has a huge influence on the model. Can it be made better using future unlabeled data?

  • $\begingroup$ Could you please explain what you mean saying that the labeled data is quite often a small set of human annotated data? Do you mean that the inputs are manually labeled? $\endgroup$ – gvgramazio Jul 10 '18 at 11:41
  • $\begingroup$ Also, without labels, how could you expect to train a NN? You cannot provide a cost function for these data. $\endgroup$ – gvgramazio Jul 10 '18 at 11:42
  • $\begingroup$ @gvgramazio yes manual labeling of a sample. $\endgroup$ – schoon Jul 11 '18 at 10:44
  • $\begingroup$ Then if you use supervised learning you cannot use those labeled data. Instead, if you use unsupervised learning you could use unlabeled data. $\endgroup$ – gvgramazio Jul 11 '18 at 12:49

This may not seem trivial but yes, the models we train can potentially learn a variety of things they weren't intended to learn. There are already some examples in computer vision. A typical convolutional network learns things like edge detection, various potentially useful masks etc. in the early layers while learns more high-level features like eyes, nose etc. in higher layers.

It is reasonable too. Given the dataset size is moderately high and the model is trained for long enough, a sufficiently deep network learns various kinds of hidden representations, which may not even be specific to the task at hand. This is the reason transfer learning works very well even on a host of different datasets.

This is limited since not all the learnable things can be described using mathematics. So, the answer is a surprising no. The model does learn some extra things other than the task at hand.

P.S.: There was also a case when a group of researchers trained a model to make a robot walk. It turned out the robot had learned to recognize faces too and reacted in different ways on seeing different faces. I saw the video on YouTube a while ago and couldn't find the exact video to post the link here, anyways.

  • 1
    $\begingroup$ Good answer, thanks. But for simple regression, say, there is nothing new unless we label new data, is that right? $\endgroup$ – schoon Jul 10 '18 at 10:51
  • 1
    $\begingroup$ Yeah @schoon, I can't think of any way a simple regression model can learn something interesting. But, since the structure of data in classification and regression problems is more or less the same, it may still learn something which might not be perceivable by us. This gave me an idea though, we can try to do transfer learning in case of regression tasks and see if it yields any useful results. If it does, we can say that the model has indeed learned something else other than the task at hand! $\endgroup$ – Suyog Jadhav Jul 10 '18 at 11:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.