I was looking into the loss function in t5x here and see there is a z-loss added to the typical log loss definition.

The only paper I could surface on this was https://arxiv.org/abs/1604.08859, but I don't see the correlation. In the code, it is logsumexp of the logits squared.

But in the paper it is calculating mean/variance for the z-loss you can see the author's implementation here.

I am asking for help either connecting the two or looking for resources on the current implementation if they are in fact unrelated.

  • $\begingroup$ Could you please reformulate the title as a question? Like "How is this z-loss implementation in t5x related to this paper's loss X?" $\endgroup$
    – nbro
    Commented Jul 10 at 9:34
  • $\begingroup$ Sure, I changed it. $\endgroup$
    – Jacob B
    Commented Jul 10 at 17:04


You must log in to answer this question.

Browse other questions tagged .