Search Results

It is clear that word positions are essential for the meaning of a sentence, and so are essential when feeding a sentence (= sequence of words) as a matrix of word embedding vectors into a transformer. …

In my understanding encoder-decoder transformers for translation are trained with sentence or text pairs. How can it be explained in simple (high-level) terms that decoder-only transformers (e.g. GPT) …

Or if there is only one transformer which handles all downstream tasks. How then can it be understood that ChatGPT performs so well in so many tasks - as if it used specialized transformers. … (Maybe it's easier to answer the question if there is a specific and specifically trained transformer for each supported language.) …

I believe to have understood that it's only after a transformer has done its deterministic work, suggesting some probable next words. … Can you confirm that the transformer works strictly deterministically and there is no randomness inside or between the attention layers? …

There is a high-level analogy between attention mechanisms (to be specific: in the transformer) and Google's PageRank algorithm: both claim and strive to calculate "relative importances" – of parts of …

In my understanding of transformer-based language models one attention head is responsible for one syntactic or semantic relation between any two words in the context. … Or viewed differently: How similar are the three matrices of an attention head in practice, e.g. when considering some 100$\times$100 attention heads of a large transformer model like ChatGPT? …

Assume you have a pretrained transformer language model (M1) which already underwent reinforcement learning by human feedback (M2). …