BERT encodes a piece of text such that each token (usually words) in the input text map to a vector in the encoding of the text. However, this makes the length of the encoding vary as a function of the input length of the text, which makes it more cumbersome to use as input to downstream neural networks that take only fixed-size inputs.

Are there any transformer-based neural network architectures that can encode a piece of text into a fixed-size feature vector more suitable for downstream tasks?

  • $\begingroup$ This might help and partially answers your question. You can only try to reduce the size of the obtained state with some convolution, but I don't think this is being done yet. $\endgroup$ – N. Kiefer Sep 18 '20 at 13:12
  • 1
    $\begingroup$ BERT does provide a fixed-size output. The encoding of the special [CLS] token, which is always prepended to every input example, is meant to encode the entire sentence. Since it is a token, its encoding is always a fixed-length vector of length H, e.g. H=768 for BERT-Base. Specifically, the [CLS] encoding is passed through a "pooling layer", which is just a HxH fully-connected layer, with linear activation. This is called pooled_output in the TF Hub module. $\endgroup$ – primussucks Sep 21 '20 at 16:14
  • $\begingroup$ As mentioned, use the CLS token. Else you could just define some max_length, and pad to it when its to low and then use the mean of the tokens. But, look at sentence-bert for e.g sbert.net/#usage - it generates fixed sized sentence embeddings with varying input sizes :) $\endgroup$ – Isbister Sep 30 '20 at 15:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.