Artificial Intelligence Stack Exchange is a question and answer site for people interested in conceptual questions about life and challenges in a world where "cognitive" functions can be mimicked in purely digital environment. It only takes a minute to sign up.
Sign up to join this community
Word embedding refers to the techniques in which a word is represented by a vector. There are also integer encoding and one-hot encoding, which I will collectively call categorical encoding.
I can see no fundamental difference between the categorical encoding and word embedding at a fundamental level. They may be different at an application level.
Is it true that categorical encoding is a type of word embedding? And are different names solely due to the task in which apply the technique?
One-hot encoding is different than the concept of a word embedding, although both approaches use vectors to represent the objects (e.g. words).
A one-hot vector contains one element that is 1 and all other elements are 0. So, for example, the vector $[0, 0, 1, 0]$ is a one-hot vector, while the vector $[0, 2, 0.2, 0]$ is not. (Given that the sum of all elements in the one-hot vector is equal to 1 and all elements are in the range $[0, 1]$, a one-hot vector is a probability vector, although this may be irrelevant.) To represent your objects as one-hot vectors, you first need to know how many objects you have (in the context of NLP, this is often the vocabulary size). Let's say you have $N$ distinct elements (e.g. words), then, in order for two objects not to be mapped to the same one-hot vector, you need to have one-hot vectors of $N$ dimensions. So, if $N$ is very large, this can be a disadvantage of one-hot encoding, although, in principle, you only need to store the index of the $1$ for each one-hot vector.
As opposed to one-hot vectors, which do not encode the meaning of the objects they represent, word embeddings semantically represent the objects, i.e. similar objects (for some notion of similarity) are typically mapped to similar word embeddings. Here's a picture that illustrates this concept (taken from the book that I cite below).
Moreover, word embeddings can easily be smaller than one-hot vectors and do not necessarily contain only 0s and 1s. More importantly, word embeddings are derived from data (often learned).
A similar argument can be said for integer encoding, i.e. they are not learned from data and they do not carry any meaning. Chapter 6 of this book discusses word embeddings more in detail.
To conclude, the common definition of a word embedding usually implies that word embeddings semantically represent objects and are derived/learned from data, so, generally, one-hot encoding is not a form of word embedding, but it's a way to embed objects into a vector space.
