1
$\begingroup$

Is it theoretically possible to use a transformer architecture to autoregressively generate a sequence of embedding vectors, instead of discrete tokens?

For example, if I were to provide an input of a stream of audio embeddings in the format (batch_size, seq_len, embed_dim), would it theoretically be model be able to train a transformer to predict the next audio embedding in the sequence, using a linear layer instead of creating token embeddings?

If this is possible, what loss function would best be used in this scenario? (for non-discrete data)

$\endgroup$

1 Answer 1

1
$\begingroup$

Great news -- transformers do this already. The way transformers generate discrete tokens is by using a linear layer to do classification over the vocabulary given the transformer outputs. The transformer outputs are just dense vectors in the same shape as the input token embeddings (i.e., of shape = (batch_size, seq_len, embed_dim)).

You could train the model with something like MSE loss. Transformers don't necessarily have to predict discrete tokens (e.g., the MAE paper use a transformer to predict pixel values directly, and trains with MSE). Cosine-distance could also work as a loss function for embedding prediction, as used in this paper which has a transformer model that produces an embedding output.

However, it's important to note that it's common for people to convert the data into some discrete format first. For example, for applications like image or audio generation, you often have a VQ-VAE create a "code-book" of embeddings, and you'd use a transformer to model those codes as a discrete "vocabulary". See this paper for an example. Finally, models like imageGPT instead quantize color values to create a discrete "vocabulary".

$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .