I'm sampling multiple generations from a transformer model and I would like to have a confidence score for each generation. The Hugging Face library's generate() method can return per-token logits if output_scores=True is set. What is the right way to combine these per-token logits to get an overall score?

My first thought was to simply add them up. Since they are negative numbers, this causes the confidence score to favor shorter outputs. Is there a score that is not biased by output length?

  • 1
    $\begingroup$ probably softmax and then perplexity? $\endgroup$
    – Alberto
    Commented Jul 14, 2023 at 18:22


You must log in to answer this question.

Browse other questions tagged .