When reading about NLP, I saw it said that "input embeddings" are a main element of encoder-decoder learning frameworks for sequence modelling. What is an "input embedding" in the context of NLP?
$\begingroup$
$\endgroup$
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
2
An embedding is a vector that semantically represents an object/input, which, in the context of NLP, can be a character, word, or sentence, depending on the task. The main property of embeddings is that they are close/similar to each other (for some notion of similarity, e.g. the cosine similarity) if the corresponding original objects/inputs also have similar meanings or are contextually related. So, for example, the words "boy" and "man" are expected to be mapped to vectors/embeddings that are close to each other. There are different ways to create these embeddings. An example is word2vec. Note that the notion of an embedding also applies to other contexts, e.g. geometric deep learning. You can read more about this topic in chapter 6 of this book.
-
1$\begingroup$ Ahh, yes, I'm in the early chapters of the Jurafsky and Martin textbook. Thanks for the clarification. $\endgroup$ Commented May 30, 2021 at 21:21
-
2$\begingroup$ It's worth adding where the similarity comes from: usually embeddings are based on the surrounding context of a word, and both 'man' and 'boy' tend to be used in similar environments, at least more than 'man' and 'quickly'. The more the usage of two words overlaps, the more similar they tend to be. $\endgroup$ Commented May 31, 2021 at 9:08