0
$\begingroup$

I can find a million explanations of the diagram in the original transformer paper:

enter image description here

But I know that modern GPT models have many millions of weights. Where are they? Or in other words, how does this thing scale?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
2
$\begingroup$

The diagram is a concise representation where the Nx refers to a collapsed array. This diagram can be expanded horizontally with multiple left columns chained together. This diagram can be expanded vertically with multiple yellow/orange rows chained together. If you do an image search you'll find some zoomed out color coded maps of various topologies.

Improve this answer
$\endgroup$
1
  • 2
    $\begingroup$ Can you please provide an example of a model based on the transformer that does what you say? $\endgroup$
    – nbro
    Commented Feb 11, 2023 at 11:18

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .