Skip to main content
Artificial Intelligence

Return to Answer

Note axis over which softmax is applied (in image)
Source Link
orome
orome
  • 103
  • 5

As I understand it (and I'm not an AI researcher, so any helpful comments from folks who know the topic better will be illuminating) the output of layer $l \in 1 ...\bf{L}$, $\bf{X}^l$, is

enter image description hereenter image description here

where $a\in 1...A$ is the head number, and $f$ is some function like RELU or whatever and the $\bf{b}$s are biases ($M$ is the attention mask and $d_E$ is the size of the embedding). The first bit corresponds to @Soltius's correction (and the second bit is the FFN). (And $\text{softmax}\left(\bf{X}^L\bf{W}_E^{-1}\right)$$\underset{\mathsf{vocab}}{\mathsf{softmax}}\left(\bf{X}^L\bf{W}_E^{-1}\right)$ is what's used in calculating cost).

As I understand it (and I'm not an AI researcher, so any helpful comments from folks who know the topic better will be illuminating) the output of layer $l \in 1 ...\bf{L}$, $\bf{X}^l$, is

enter image description here

where $a\in 1...A$ is the head number, and $f$ is some function like RELU or whatever and the $\bf{b}$s are biases ($M$ is the attention mask and $d_E$ is the size of the embedding). The first bit corresponds to @Soltius's correction (and the second bit is the FFN). (And $\text{softmax}\left(\bf{X}^L\bf{W}_E^{-1}\right)$ is what's used in calculating cost).

As I understand it (and I'm not an AI researcher, so any helpful comments from folks who know the topic better will be illuminating) the output of layer $l \in 1 ...\bf{L}$, $\bf{X}^l$, is

enter image description here

where $a\in 1...A$ is the head number, and $f$ is some function like RELU or whatever and the $\bf{b}$s are biases ($M$ is the attention mask and $d_E$ is the size of the embedding). The first bit corresponds to @Soltius's correction (and the second bit is the FFN). (And $\underset{\mathsf{vocab}}{\mathsf{softmax}}\left(\bf{X}^L\bf{W}_E^{-1}\right)$ is what's used in calculating cost).

added 68 characters in body
Source Link
orome
orome
  • 103
  • 5

As I understand it (and I'm not an AI researcher, so any helpful comments from folks who know the topic better will be illuminating) the output of layer $l \in 1 ...\bf{L}$, $\bf{X}^l$, is

enter image description hereenter image description here

where $a\in 1...A$ is the head number, and $f$ is some function like RELU or whatever and the $\bf{b}$s are biases ($M$ is the attention mask and $d_E$ is the size of the embedding). The first bit corresponds to @Soltius's correction (and the second bit is the FFN). (And $\text{softmax}\left(\bf{X}^L\bf{W}_E^{-1}\right)$ is what's used in calculating cost).

As I understand it (and I'm not an AI researcher, so any helpful comments from folks who know the topic better will be illuminating) the output of layer $l \in 1 ...\bf{L}$, $\bf{X}^l$, is

enter image description here

where $a\in 1...A$ is the head number, and $f$ is some function like RELU or whatever and the $\bf{b}$s are biases. The first bit corresponds to @Soltius's correction (and the second bit is the FFN). (And $\text{softmax}\left(\bf{X}^L\bf{W}_E^{-1}\right)$ is what's used in calculating cost).

As I understand it (and I'm not an AI researcher, so any helpful comments from folks who know the topic better will be illuminating) the output of layer $l \in 1 ...\bf{L}$, $\bf{X}^l$, is

enter image description here

where $a\in 1...A$ is the head number, and $f$ is some function like RELU or whatever and the $\bf{b}$s are biases ($M$ is the attention mask and $d_E$ is the size of the embedding). The first bit corresponds to @Soltius's correction (and the second bit is the FFN). (And $\text{softmax}\left(\bf{X}^L\bf{W}_E^{-1}\right)$ is what's used in calculating cost).

Source Link
orome
orome
  • 103
  • 5

As I understand it (and I'm not an AI researcher, so any helpful comments from folks who know the topic better will be illuminating) the output of layer $l \in 1 ...\bf{L}$, $\bf{X}^l$, is

enter image description here

where $a\in 1...A$ is the head number, and $f$ is some function like RELU or whatever and the $\bf{b}$s are biases. The first bit corresponds to @Soltius's correction (and the second bit is the FFN). (And $\text{softmax}\left(\bf{X}^L\bf{W}_E^{-1}\right)$ is what's used in calculating cost).