5
$\begingroup$

In general, the word "latent" means "hidden" and "to embed" means "to incorporate". In machine learning, the expressions "hidden (or latent) space" and "embedding space" occur in several contexts. More specifically, an embedding can refer to a vector representation of a word. An embedding space can refer to a subspace of a bigger space, so we say that the subspace is "embedded" in the bigger space. The word "latent" comes up in contexts like hidden Markov models (HMMs) or auto-encoders.

What is the difference between these spaces? In some contexts, do these two expressions refer to the same concept?

$\endgroup$
3
$\begingroup$

When it comes to normal layman terms "latent space" means it cannot be accessed, thus we have no direct control over it. We can only manipulate it indirectly, while "Embeddings" can be obtained directly. We can use deterministic operations or transformations to convert an object into its corresponding embedding space.

There is no marked difference between these 2 terms as far as Machine Learning is concerned. If we look at this famous paper on Variational Autoencoders, we can see the words has been used interchangeably.

More specifically, I would consider the word (in the context of Machine Learning only) latent as a more general term than Embedding. Embeddings will refer to a more specific object (in context of ML), for example the embedding of $word_1$ is $embedding_1$. Whereas, we can use the term latent to describe broader terms like latent space, latent representation, latent variables (latent variables of a word is same as an embedding of a word).

After digging some more I found some what of a formal definition of Latent Variables in Deep Learning by Goodfellow:

  • Latent Variables - A latent variable is a random variable that we cannot observe directly. The component identity variable $c$ of the mixture model provides an example. Latent variables may be related to $x$ through the joint distribution, in this case, $P(x, c) = P(x | c)P(c)$. The distribution $P(c)$ over the latent variable and the distribution $P(x | c)$ relating the latent variables to the visible variables determines the shape of the distribution $P(x)$, even though it is possible to describe $P(x)$ without reference to the latent variable.

Also a paper cited by Goodfellow while discussing embeddings has the following excerpt:

Following the success of user/item clustering or matrix factorization techniques in collaborative filtering to represent non-trivial similarities between the connectivity patterns of entities in single relational data, most existing methods for multi-relational data have been designed within the framework of relational learning from latent attributes, as pointed out by; that is, by learning and operating on latent representations (or embeddings) of the constituents (entities and relationships).

So clearly these are somewhat interchangeable terms.

But my interpretation would be that embeddings are helpful more explicitly (more visible, latent variables are meant to be hidden), that is we can construct a new data-set from it and use various ML methods on it, whereas latent variables are something not useful explicitly (it is a part of a bigger problem we are trying to solve).

EDIT: In the context of HMM's the term better suitable is hidden state and not latent space. Thus, in a HMM (from Wiki) The adjective hidden refers to the state sequence through which the model passes, not to the parameters of the model; the model is still referred to as a hidden Markov model even if these parameters are known exactly.

$\endgroup$
  • $\begingroup$ In some contexts, these terms might be used interchangeably, but this is not always the case. An embedding space is not necessarily an "hidden" space (e.g. in the sense of HMMs). So, I don't agree with your "a latent space is more general than an embedding space". $\endgroup$ – nbro Mar 18 at 20:18
  • $\begingroup$ The concepts are the same. They are sometimes called state, but they are just a random variable. It is just a slightly different terminology. HMMs are just a particular Bayesian network. $\endgroup$ – nbro May 24 at 12:03
0
$\begingroup$

The expression "latent space" explicitly indicates that the space is associated with the mathematical concept of an hidden (or latent) variable, which cannot be observed directly, but only indirectly.

The expression "embedding space" refers to a vector space that represents an original space of inputs (e.g. images or words). For example, in the case of "word embeddings", which are vector representations of words. It can also refer to a latent space because a latent space can also be a space of vectors. However, an embedding space is not necessarily an hidden space. It is just another (vector) representation of another space.

These two expressions can be used interchangeably, also because the expression "embedding space" is often not formally defined.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.