In the Manifold Hypothesis applied to LLMs, are text sequences points or paths on the manifold?

Question

The Manifold Hypothesis makes a ton of sense to me for images. Images are points in high dimensional space, where each dimension corresponds to the intensity value of a single pixel. For example, we can think about the 28x28 pixel grayscale MNIST images as points lying on some manifold in 28x28=784 dimensional space.

When we jump to language modeling however, I know that each word is mapped to a vector using various embedding approaches. In this sense, each word is a point in a high dimensional space, that I imagine sweep out some manifold in that space. However, the examples LLMs learn from are of course sequences of words, and as far as I know that when transformers are applied to images, it's done by making images into pixel sequences. From this perspective I'm wondering if sequences of text are better thought of as points on a high dimensional manifold?

Answer 1

When transformers process sequences, they learn to generate contextual embeddings for each word. However, often for downstream tasks, these word-level embeddings are combined (e.g., using pooling or attention mechanisms) into a single fixed-length vector that represents the entire sequence. Thus a sequence of text can indeed be thought of as a point in an even higher-dimensional space than each word's semantic embedding space where each dimension corresponds to one of the input features across all words or more general tokens. Just like in images where the nearby pixels are related and constrained by the rules of natural scenes, in text words within sequences are contextually related. Not every possible sequence of words makes sense in a given language which follows specific syntactic and semantic rules, so only a subset lower-dimensional manifold of the high-dimensional space is occupied by meaningful sequences.

Of course there's nothing wrong to alternatively view text sequences as paths sweeping out some manifold in the word embedding space, but since word embeddings of a generic text sequence are not continuously varying geometrically speaking it's hard to leverage this view to make any substantive conclusion.