What is the meaning of "continuous" in a continuous bag-of-words model?

Question

The word continuous in mathematics is a property of either a set or a function that says that the underlying object has no discontinuity in the range mentioned. If the object is a set, then $[-1,1]$ is a continuous one while $\{-1, +1\}$ is not. Similarly, a function is said to be continuous if the actual value and the limiting value at every point in the domain are equal.

Now, coming to CBOW. I read the following statement from p:334 of Natural Language Processing by Jacob Eisenstein

Thus, CBOW is a bag-of-words model, because the order of the context words does not matter; it is continuous, because rather than conditioning on the words themselves, we condition on a continuous vector constructed from the word embeddings.

What is meant by continuous in this case? Does continuous vector stand for a vector of real numbers?

Answer 1

A bag-of-words-model (BOW) is usually used to represent a text: you throw all the words together (as if in a bag), without keeping track of their sequence. This is a gross simplification over a text, as word sequencing plays an important role in creating the meaning of a text. But on the positive side it's easier to handle, eg in information retrieval tasks, where you might not need the precise meaning anyway.

So the BOW is discrete and symbolic, as it represents each of its elements by a set of words that are contained in it. Nothing numeric in there. You'd calculate the similarity of two items by comparing the two sets, how big is their intersection, and the difference between the two.

A CBOW is a slight modification: instead of the words, we use vector representations of them; and instead of having $n$ vectors for the $n$ surrounding words, they're all added up (formula 14.14) It's still a BOW, as the set of words used to represent an element is now the set of words surrounding it within a certain distance ($h$). What makes it continuous is the switch from a set of words (ie symbols) to a vector.

He contrasts this with a recurrent neural network, where words are represented by a state vector which gets updated after every new word, going back to the very beginning of the text. This would give different representations for the same word occurring in the same localised context, whereas the CBOW would return the same representation.

For example, for $h$ being 1 (to keep it simple):

when a word has a meaning, then a word has a purpose.

Now imagine we're interested in the encoding of word: in the recurrent case the first one is when + a + word, whereas the second one is when + a + word + has + a + meaning + , + then + a +word — the sequences here represent the updated state of the network after the respective words have been added.

In the CBOW case, both occurrences of word are encoded by a + word + has (the word plus/minus one word either side, as $h$ is 1). So they will be identical.

To answer your question, continuous here is in contrast to discrete or symbolic, and indeed refers to a numerical vector.