What is a 'degenerate run' in evaluating model performance?

Question

I've recently come across a paper that uses the term "degenerate run", but I'm not sure if I understand what it means. The idea is that when they report the average performance of running fine-tuned models using multiple random seeds (e.g., a deep learning model where we need to initialize model parameters using multiple seeds to ensure results are robust,) they exclude the degenerate runs in some of their analyses.

As this paper mentions, a degenerate run is "where fine-tuned models fail to outperform the random baseline." But I wonder if this is a standard practice to eliminate such results when reporting the average performance? Or is the definition they give the correct meaning of a degenerate run?

Answer 1

The authors explain their use of the term in the paper:

Without the bias correction we observe many degenerate runs, where fine-tuned models fail to outperform the random baseline

Specifically, as it pertains to fine tuning, degenerate has this dictionary meaning in technical contexts: Lacking some usual or expected property or quality. In the paper's case the retrained classifier lacks the property of being able to classify inputs to any accuracy.

That is backed up by their use of the term to apply to failed training runs.

There is also a mathematics use of the term, which sometimes appears in computer science, and is about object classes e.g. a triangle with one angle 0 degrees is degenerate - it is just a line, not meaningfully a triangle. I don't think it applies here, although it could - the failed refined classifier has all the technical build qualities of a classifier, but does not classify.

I do not think degenerate runs is a commonly used phrase in machine learning, nor does it have any additional technical meaning when applied to supervised learning, beyond what they explain in the paper.

A more common term related to trained systems losing capabilities when trained further on new data is catastrophic forgetting. This may be subtly different to the effect that the authors are correcting for though, perhaps that is the reason for them using a different term.