As I understand it (and I'm not an AI researcher, so any helpful comments from folks who know the topic better will be illuminating) the output of layer $l \in 1 ...\bf{L}$, $\bf{X}^l$,  is

[![enter image description here][1]][1] 

where $a\in 1...A$ is the head number, and $f$ is some function like RELU or whatever and the $\bf{b}$s are biases. The first bit corresponds to [@Soltius's correction][2] (and the second bit is the FFN). (And $\text{softmax}\left(\bf{X}^L\bf{W}_E^{-1}\right)$
is what's used in calculating cost).

  [1]: https://i.sstatic.net/VuK7J.png
  [2]: https://ai.stackexchange.com/a/39195/10649