Should I use the same learning rate when I compare the performance for different language models?

Question

If I'm using the same datasets and using different language models, such as BERT, DistilBERT, Electra, etc., and my goal to compare the performance, such as the accuracy, macro f score, etc. should I used the same learning rate and number of epochs to make comparison fair, or is it allowed to have different learning rates for each? For example, if BERT provides the best result with a learning rate 2e-5 and Electra provides the best results with 1e-6, can I register both results and use them?

Answer 1

Since learning rate is a hyperparameter, using different learning rates or epochs for each model is not only allowed but generally encouraged as long as the hyperparameter tuning process such as grid search is consistent across all models. Different models have unique architectures, parameter counts, and training dynamics, which can affect how they respond to specific learning rates. Comparing models using the same fixed learning rate may disadvantage certain models, as they might not reach their optimal performance under those conditions.

Finally it's important to keep other factors constant such as the training dataset, evaluation metrics, data splits, preprocessing steps, etc. Of course, if resources are permitted, you can also run all models with the same learning rate and training setup to explore how they perform under identical conditions which could potentially provide additional insights.