3
$\begingroup$

Feature selection is a process of selecting a subset of features that contribute the most.

Feature extraction allows to get new features that are not actually present in the given set of features.

Representation learning is the process of learning a new representation that contributes the most.

I can see no difference between the feature extraction and representation learning.

Is feature extraction the same as representation learning? If no, where do they differ? Do they differ at the application level only?

$\endgroup$
4
$\begingroup$

Feature extraction (FE) is not the same as representation learning (RL), but they are similar and related.

You describe accurately what feature extraction typically refers to, i.e. the process of extracting (new) features from existing ones or raw data (e.g. images). For example, let's say you have a dataset associated with a car. You have only two features in your dataset: distance and velocity. However, from these two, you can extract a third feature, e.g. the acceleration. So, feature extraction can be performed with a fixed algorithm (e.g. PCA for dimensionality reduction or SIFT) or manually.

Representation learning is the collection of all techniques that extract features automatically from the data (i.e. they learn the features or representations, hence the name representation/feature learning). So, for example, a convolutional neural network trained on ImageNet can (and/or needs to) learn general features in order to solve the corresponding classification task. (Chapter 9 of the book Deep Learning by Goodfellow et al. talks more about this topic.) These features are learned from the data (and that's why CNNs are data-driven), and they can later be exploited for transfer learning (TL), i.e. TL is based on the idea that neural networks learn general representations (which is thus a synonym for features) of data that can be exploited to solve other tasks (sometimes known as downstream tasks, especially in the context of self-supervised learning).

Yoshua Bengio et al. define representation learning as follows

learning representations of the data that make it easier to extract useful information when building classifiers or other predictors

So, RL is a subset of FE, given that RL also extracts features, but RL emphasizes the extraction of features automatically.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.