Fine Tuning Transformer Model for Machine Translation

Question

I am working on the Transformer example demonstrated on TensorFlow's website. https://www.tensorflow.org/text/tutorials/transformer

In this example, Machine Translation model is trained to translate from Portuguese to English. The transformer is coded from scratch and other popular libraries like huggingface are not used.

Let's say I have another dataset which includes pairs of sentences of Portuguese and Finnish and let's say this dataset is fairly small. Since it is a small dataset, I want to use my model trained on Portuguese to English as a PreTrained model for creating the translation model for Portuguese to Finnish.

My question is, what are the key points to consider when using such a PreTrained model and changing ONLY its decoder output structure?

Answer 1

Transfer learning in machine translation is a relatively common technique in machine translation. Mostly, it means fine-tuning pre-trained self-supervised sequence-to-sequence models, such as mBART. It is also often used for low-resource languages to transfer from a related high-resource language, as e.g. most of the participants in the WMT21 low-resource competition did.

What you suggest is very close to a 2018 paper by Kocmi and Bojar. You might use their setup as a starting point. The main challenges addressed in the paper are:

• Carefully setting the learning rate schedule to avoid catastrophic forgetting.
• Do something about the vocabulary mismatch (Finnish uses a different vocabulary than English).