Can I always use "encoding" and "embedding" interchangeably?

Question

This question is restricted to the text domain only.

The meaning of the word "encode" is Convert (information or instruction) into a particular form. One which performs encoding is called an encoder.

In deep learning, an encoder can also be the first part of a neural network (autoencoder) that simulates identity function, which governs the English meaning of encoder since it encodes the input.

Embeddings are encodings where the intention is to preserve semantics. You can observe the following excerpt from the chapter Vector Semantics and Embeddings

In this chapter we introduce vector semantics, which instantiates this linguistic hypothesis by learning representations of the meaning of words, called embeddings, directly from their distributions in texts.

But all encodings may not be the embeddings since encodings might not always preserve semantics (?). I have doubt in this statement which I inferred based on my current knowledge.

Many times, I came across the terms text encoding and text embedding interchangeably. But failing to catch whether they are the same or we need to be choosy while using them.

Consider the following usages of encoding and embedding in the paper titled Generative Adversarial Text to Image Synthesis by Scott Reed et al.

#1: The intuition here is that a text encoding should have a higher compatibility score with images of the correspondong class compared to any other class and vice-versa.

#2: Text encoding $\phi(t)$ is used by both generator and discriminator.

#3: ...where $T$ is the dimension of the text description embedding.

#4: ... we encode the text query $t$ using text encoder $\phi$. The description embedding $\phi(t)$ is first compressed ...

I think they are used interchangeably. Is it true? Can I use any word if I am confident enough that my encoding is semantic preserving? Or is there any strong reason for choosing the words?

If you observe the last point, the word "encoder" is used. Can I use embedder instead of it?

Answer 1

Caveat: I am not a native English speaker (but French). And mostly interested in symbolic artificial intelligence (the topic of my PhD thesis defended in 1990; see books by Jacques Pitrat)

Encoding is related to decoding. Most of the time, if you encode something A into some other thing B, you can "decode" B to get back A.

Otherwise, you would just say "parsing".

Embedding means mapping something into a "greater" set (or category, e.g. differential manifolds).

Feel free to email me for more explanations.

Answer 2

From my experience with reading papers and books, I think these two terms are sometimes used interchangeably.

As you also point out, an encoder (in an auto-encoder) also may also learn some "semantics" of the inputs in order to produce the latent space. However, the way encoders are trained may not produce embeddings, with similar properties to e.g. word embeddings. For example, an image of a cat may be mapped to a latent vector that is closer to the latent vector of a person than to the latent vector of a dog (the usual way deterministic autoencoders are trained doesn't enforce these properties).

So, in my head, an encoding may not have any semantics (one-hot encoding is the typical example), but an embedding has. However, again, it depends on the context, so you should take context into account. So, I expect people to use the term encoding to refer to an embedding.