Here's a quote from the
T5 paper (T5 stands for "Text-to-Text Transfer Transformer") titled Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer by Colin Raffel et al.:
To summarize, our model is roughly equivalent to the original Transformer proposed by Vaswani et al. (2017) with the exception of removing the Layer Norm bias, placing the layer normalization outside the residual path, and using a different position embedding scheme. Since these architectural changes are orthogonal to the experimental factors we consider in our empirical survey of transfer learning, we leave the ablation of their impact for future work.
What exactly does 'orthogonal' mean in this context? Also, is it just me or have I seen the word used in a similar way before, but can't remember where?