Feature extraction is a concept concerning the translation of raw data into the inputs that a particular machine learning algorithm requires. These derived features from the raw data that are actually relevant to tackle the underlying problem. On the other hand, word embeddings are basically distributed representations of text in an n-dimensional space.

As far as I understand, word embedding is a somehow feature extraction technique. Am I wrong ? I had an argument with a friend who believes the two topics are totally separate. Is he right? What are the similarities and dissimilarities between word embedding and feature extraction?

| improve this question | | | | |

Though word-embedding is primarily a language modeling tool, it also acts as a feature extraction method because it helps transform raw data (characters in text documents) to a meaningful alignment of word vectors in the embedding space that the model can work with more effectively (than other traditional methods such as TF-IDF, Bag of Words, etc, on a large corpus). Word Embedding techniques help extract information from the pattern and occurence of words and goes further than other traditional token representation methods to decode/identify the meaning/context of the words, thereby providing more relevant and important features to the model to tackle the underlying problem.

However, from another standpoint, word-embedding models were not developed aiming to solve a particular feature extraction problem, but rather, to generalize and model the language used in a corpus to gain a semantic understanding of the words and the relationships between them. Such that, all the various corpus-specific tasks can then employ the same "library" of information which was collectively & exhaustively learnt by the embedding model. Meaning, the word embedding model learns a language model that is task-agnostic for all tasks on that corpus unlike feature extraction methods which are specifically task-oriented.

Hence, the similarity is - word-embeddings can effectively aid in feature extraction; the dissimilarity is - they're not primarily meant to extract features more than they are for modeling a language which might be an "overkill" for a particular feature extraction task on a dataset.

| improve this answer | | | | |
  • 1
    $\begingroup$ Welcome to AI.SE. Great answer! $\endgroup$ – DuttaA Dec 27 '19 at 17:41
  • $\begingroup$ One thing that should be mentioned is that when two semantically similar words are mapped into the embedding space, the defined metric within the space (for word2vec it is cosine distance) is proportional to the word similarity. For example 'hog' and 'pig' will have a cosine distance(distance of 1 means same) closer to 1 than would 'hog' and 'cow'. So neighbours in the embedding space are similar in the language. I consider this property to be the most important aspect of embedding. $\endgroup$ – user1269942 Dec 27 '19 at 20:12

I think you guys are playing on semantics.

If you consider feature extraction to be an unlearned preprocessing step to get inputs for your model, then no, word embeddings are not a feature extraction technique (examples here would be BoW counts, n-gram features, etc)

If you consider feature extraction to be any form of conversion from text to a set of features, then yes, word embeddings should be considered a form of feature extraction, given that they've learned in the process (or stolen from another model's training). Note though if you do include this, you would probably include most pretrained models as a whole as feature extraction techniques (like BERT).

So the whole conversation you had can go either way depending on the definitions you set.

| improve this answer | | | | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.