BERT encodes a piece of text such that each token (usually words) in the input text map to a vector in the encoding of the text. However, this makes the length of the encoding vary as a function of the input length of the text, which makes it more cumbersome to use as input to downstream neural networks that take only fixed-size inputs.

Are there any transformer-based neural network architectures that can encode a piece of text into a fixed-size feature vector more suitable for downstream tasks?

Edit: To illustrate my question, I’m wondering whether there is some framework that allows the input to be either a sentence, a paragraph, an article, or a book, and produce an output encoding on the same, fixed-sized format for all of them.

  • $\begingroup$ This might help and partially answers your question. You can only try to reduce the size of the obtained state with some convolution, but I don't think this is being done yet. $\endgroup$
    – N. Kiefer
    Commented Sep 18, 2020 at 13:12
  • 2
    $\begingroup$ BERT does provide a fixed-size output. The encoding of the special [CLS] token, which is always prepended to every input example, is meant to encode the entire sentence. Since it is a token, its encoding is always a fixed-length vector of length H, e.g. H=768 for BERT-Base. Specifically, the [CLS] encoding is passed through a "pooling layer", which is just a HxH fully-connected layer, with linear activation. This is called pooled_output in the TF Hub module. $\endgroup$ Commented Sep 21, 2020 at 16:14
  • $\begingroup$ As mentioned, use the CLS token. Else you could just define some max_length, and pad to it when its to low and then use the mean of the tokens. But, look at sentence-bert for e.g sbert.net/#usage - it generates fixed sized sentence embeddings with varying input sizes :) $\endgroup$
    – Isbister
    Commented Sep 30, 2020 at 15:43

2 Answers 2


One way you could do it is by using SentenceTransformers.

SentenceTransformers is a Python framework for state-of-the-art sentence, text and image embeddings. The initial work is described in our paper Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks.

To install it via pip

pip install -U sentence-transformers

To generate sentence embedding

from sentence_transformers import SentenceTransformer

# We are using "paraphrase-MiniLM-L6-v2" model here, You can find list of model [here][2]
model = SentenceTransformer('paraphrase-MiniLM-L6-v2')

# Our sentences we like to encode
sentences = ['This framework generates embeddings for each input sentence',
    'Sentences are passed as a list of string.',
    'The quick brown fox jumps over the lazy dog.']

# Sentences are encoded by calling model.encode()
embeddings = model.encode(sentences)

#Print the embeddings
for sentence, embedding in zip(sentences, embeddings):
    print("Sentence:", sentence)
    print("Embedding:", embedding)

But remember, SentenceTransformers models have an input limit as well, 512 tokens usually. If your text is more than that, then it might not be a suitable method.

  • $\begingroup$ Thanks for you answer. Well, if it has an input limit, the input isn’t truly of arbitrary length (in practice I guess the input also becomes of fixed length, and that you just use a padding token to fill it out after the text has ended). I was wondering whether there is some framework that allows the input to be either a sentence, a paragraph, an article, or a book, and produce an output encoding on the same, fixed-sized format for all of them. $\endgroup$ Commented Feb 11, 2022 at 21:23
  • $\begingroup$ Not till today. $\endgroup$ Commented Feb 13, 2022 at 4:34

Building transformer models, in general, for arbitrary context retrieval is difficult because the architecture and training simply aren't designed to handle this. This is even more of an issue if you want to use the model to embed documents, as you typically need to do some specialized training to get meaningful embeddings.

There has, however, been work on long-context embedding models more generally: e.g., M2-BERT with a maximum of 32k tokens. i.e., it can support documents with length <= 32k. This, however, isn't a transformer model. Comparing to other models in their paper, transformer-based embedding methods only seem to have a maximum context length of 8k.

  • $\begingroup$ Ok, so is M2-BERT a state space model, kind of like RWKV and Mamba? $\endgroup$ Commented Mar 6 at 23:33
  • $\begingroup$ I don't think RWKV is a state space model, but it would be more similar to Mamba. $\endgroup$ Commented Mar 7 at 2:00

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .