Why do the authors of the T5 paper say that the "architectural changes are orthogonal to the experimental factors"?

Question

Here's a quote from the T5 paper (T5 stands for "Text-to-Text Transfer Transformer") titled Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer by Colin Raffel et al.:

To summarize, our model is roughly equivalent to the original Transformer proposed by Vaswani et al. (2017) with the exception of removing the Layer Norm bias, placing the layer normalization outside the residual path, and using a different position embedding scheme. Since these architectural changes are orthogonal to the experimental factors we consider in our empirical survey of transfer learning, we leave the ablation of their impact for future work.

What exactly does 'orthogonal' mean in this context? Also, is it just me or have I seen the word used in a similar way before, but can't remember where?

Answer 1

"Orthogonal" is often used to mean "independent", as in "independent variable which does not correlate with the other variables". I believe this terminology originates from principal component analysis, where uncorrelated variation would be along orthogonal axes.

Or, in the words of the Wikipedia article on orthogonality applied to computer science:

Orthogonality is a system design property which guarantees that modifying the technical effect produced by a component of a system neither creates nor propagates side effects to other components of the system.

So in this excerpt they state that their changes do not affect anything else (because they are independent/uncorrelated), and can hence be discussed elsewhere.

Answer 2

Looking at the paper, it seems to me that they are not using orthogonal in a literal, mathematics (or geometric) sense. Instead, I read that as two things (especially since the word "ablation" appears later in the sentence):

• They are attempting to use lots of fancy words
• They are simply indicating that these changes are separate from and have no impact on (at least they claim this to be so) the "experimental factors [they] consider in [their] empirical survey..."