What is the difference between latent and embedding spaces?

Question

In general, the word "latent" means "hidden" and "to embed" means "to incorporate". In machine learning, the expressions "hidden (or latent) space" and "embedding space" occur in several contexts. More specifically, an embedding can refer to a vector representation of a word. An embedding space can refer to a subspace of a bigger space, so we say that the subspace is embedded in the bigger space. The word "latent" comes up in contexts like hidden Markov models (HMMs) or auto-encoders.

What is the difference between these spaces? In some contexts, do these two expressions refer to the same concept?

Answer 1

Embedding vs Latent Space

Due to Machine Learning's recent and rapid renaissance, and the fact that it draws from many distinct areas of mathematics, statistics, and computer science, it often has a number of different terms for the same or similar concepts.

"Latent space" and "embedding" both refer to an (often lower-dimensional) representation of high-dimensional data:

• Latent space refers specifically to the space from which the low-dimensional representation is drawn.
• Embedding refers to the way the low-dimensional data is mapped to ("embedded in") the original higher dimensional space.

For example, in this "Swiss roll" data, the 3d data on the left is sensibly modelled as a 2d manifold 'embedded' in 3d space. The function mapping the 'latent' 2d data to its 3d representation is the embedding, and the underlying 2d space itself is the latent space (or embedded space):

Synonyms

Depending on the specific impression you wish to give, "embedding" often goes by different terms:

Term	Context
dimensionality reduction	combating the "curse of dimensionality"
feature extraction feature projection feature embedding feature learning representation learning	extracting 'meaningful' features from raw data
embedding manifold learning latent feature representation	understanding the underlying topology of the data

However this is not a hard-and-fast rule, and they are often completely interchangeable.

Answer 2

When it comes to normal layman terms "latent space" means it cannot be accessed, thus we have no direct control over it. We can only manipulate it indirectly, while "Embeddings" can be obtained directly. We can use deterministic operations or transformations to convert an object into its corresponding embedding space.

There is no marked difference between these 2 terms as far as Machine Learning is concerned. If we look at this famous paper on Variational Autoencoders, we can see the words has been used interchangeably.

More specifically, I would consider the word (in the context of Machine Learning only) latent as a more general term than Embedding. Embeddings will refer to a more specific object (in context of ML), for example the embedding of $word_1$ is $embedding_1$. Whereas, we can use the term latent to describe broader terms like latent space, latent representation, latent variables (latent variables of a word is same as an embedding of a word).

After digging some more I found some what of a formal definition of Latent Variables in Deep Learning by Goodfellow:

• Latent Variables - A latent variable is a random variable that we cannot observe directly. The component identity variable $c$ of the mixture model provides an example. Latent variables may be related to $x$ through the joint distribution, in this case, $P(x, c) = P(x | c)P(c)$. The distribution $P(c)$ over the latent variable and the distribution $P(x | c)$ relating the latent variables to the visible variables determines the shape of the distribution $P(x)$, even though it is possible to describe $P(x)$ without reference to the latent variable.

Also a paper cited by Goodfellow while discussing embeddings has the following excerpt:

Following the success of user/item clustering or matrix factorization techniques in collaborative filtering to represent non-trivial similarities between the connectivity patterns of entities in single relational data, most existing methods for multi-relational data have been designed within the framework of relational learning from latent attributes, as pointed out by; that is, by learning and operating on latent representations (or embeddings) of the constituents (entities and relationships).

So clearly these are somewhat interchangeable terms.

But my interpretation would be that embeddings are helpful more explicitly (more visible, latent variables are meant to be hidden), that is we can construct a new data-set from it and use various ML methods on it, whereas latent variables are something not useful explicitly (it is a part of a bigger problem we are trying to solve).

EDIT: In the context of HMM's the term better suitable is hidden state and not latent space. Thus, in a HMM (from Wiki) The adjective hidden refers to the state sequence through which the model passes, not to the parameters of the model; the model is still referred to as a hidden Markov model even if these parameters are known exactly.

Answer 3

The expression "latent space" explicitly indicates that the space is associated with the mathematical concept of an hidden (or latent) variable, which cannot be observed directly, but only indirectly.

The expression "embedding space" refers to a vector space that represents an original space of inputs (e.g. images or words). For example, in the case of "word embeddings", which are vector representations of words. It can also refer to a latent space because a latent space can also be a space of vectors. However, an embedding space is not necessarily an hidden space. It is just another (vector) representation of another space.

These two expressions can be used interchangeably, also because the expression "embedding space" is often not formally defined.

Answer 4

To give a statistician's answer, the distinction is empirical (embedding) versus theoretical (latent positions). You define a statistical model which has latent positions that you could then try to estimate, given data. Or, given data, you might simply find a vector representation of each object of interest in a way that makes sense for the applications considered - and you would call that set of representations an embedding. There's a lot of work going on trying to re-interpret popular embeddings as estimating a compatible model's latent positions, but that's not always straightforward.

Answer 5

The words latent space and embedding space are often used interchangeably. However, latent space can more specifically refer to the sample space of a stochastic representation, whereas embedding space more often refers to the space of a deterministic representation.

This comes from latent referring to an unobserved random variable, for which we can infer a belief distribution over its plausible values, for example using an encoder network. You can then draw samples of the predicted distribution for further processing. To learn more, you can look into VAEs.