What's the purpose of "feature extraction using a pretrained model"?

Question

I'm sorry this is such an elementary question because I'm an AI beginner.

In this link, it says

One thing that is commonly done in computer vision is to take a model trained on a very large dataset, run it on your own, smaller dataset, and extract the intermediate representations (features) that the model generates. These representations are frequently informative for your own computer vision task, even though the task may be quite different from the problem that the original model was trained on.

There are few things I think I need to understand to grasp this, but I have had trouble finding appropriate information to understand them.

1. It's not clear to me what the "features" represent for "features extracted via Inception v3". I'm pretty new to the idea of feature extraction in itself, but it almost seems like a "feature" can be anything you define it to be. For example, in this article, the features are simply the RGB values of each pixel. But for Inception v3, I'm having trouble finding what the features represent.

2. What does it mean to "extract the intermediate representation"? So if you have your own smaller dataset, does it mean it's getting the features from that dataset (such as RGB of each pixel)? Or does it mean that Inception v3 creates new images and then extracts features from them? What kind of rules are used to generate these new images? Why would we use these new images for classification rather than only using your dataset?

Answer 1

A feature extractor is just the first part of a CNN (convolutional neural network), which is a neural-network model that can process images and so solve computer vision tasks.

As shown in the image the feature extractor learns (extracts) a representation of the data (often a vector) that is suitable for the next classification part of the CNN. The classifier part (also called classification head) takes these vector representation of the input image and apply dense layers, the last of them has an output dimensionality that matches the number of classes the images are supposed to belong to.

Now, in this context the term features refers to such learned vector representation, and not the features given as input to the CNN. These are also called the intermediate representation of the network: intermediate because they are kind in the middle, before the classification head.

In the context of transfer learning, you take a pre-trained model (e.g. Inception-v3) trained on a very large and varied dataset, you drop the classification part (by appending your own layers) and then fine-tune the new (classification) layers on your target dataset. In this way you reuse the learned intermediate representations/features, making training more efficient: faster to converge with way less data.

Answer 2

A neural network typically processes an image by taking all of the pixel-level data as input, and passing that through hidden feature layers which extract useful but more complex aspects of the image. These "extracted features" can be easily interpretable combinations of pixels representing things like lines, curves, or corners, or could be more "black box" features that aren't as easily interpretable. You can then continue to aggregate more layers of features, eventually getting nodes that recognize more complex items like shapes, faces, etc.

The intermediate features in many image recognition tasks tend to be useful across different image processing tasks - that first step of getting to lines, corners, and other basic features is useful in many domains. You can take an already-trained network for some arbitrary task, put in a new image, and extract not the final output (which is a classification you don't care about), but the intermediate layer features which represent some level of processing of your image. You can use this intermediate layer as a "head start" on your new network, as it will be able to leverage the existing low-level features rather than starting all over from pixel-level data.