Does fine-tuning a multilingual transformer model allow it to generalize to languages unseen in the fine-tuning dataset?

Question

Example: https://huggingface.co/google/umt5-base

Note: UMT5 was only pre-trained on mC4 excluding any supervised training. Therefore, this model has to be fine-tuned before it is useable on a downstream task.

The model was pre-trained on a whole lot of languages. Let's suppose I devise a fine-tuning dataset to classify sentences (like user feedback). But due to resource constraints, only a few languages will be included. In my fine-tuned task, will the model be able to generalize to languages it was pre-trained, but not fine-tuned on?

Answer 1

The short answer: Very unlikely.

The extended answer: If you fine-tune a model, it becomes specialized for the type of data you fine-tune it on but you trade in some of its generalization capabilities. In your particular example, that means that your model becomes very attuned to user feedback and performs very highly on user feedback in the languages it has seen during training. It looses both its capabilities for more general texts and other languages from pre-training. It will perform lower on general texts and significantly lower on user feedback in different languages from your training set.

To address this problem, you need something called domain adaptation or transfer learning. Domain adaptation means you train your model on a domain dataset, e.g. your user feedback dataset in a small range of languages, and then adapt/transfer it to your actual target dataset. See an example below:

There are different approaches for this. Some adapt the data, others prefer to transfer the model. There is a lot of research in this area. This is a good article that covers the basics and might provide you with a more specific solution you can try out.

Answer 2

The attention doesn't discriminate between what token it is producing as long as it is following the protocol/heuristic defined by the finetuning dataset,

So essentially, a finetuning dataset that contains

Hi, How are you ---> I am fine, thank you.

should be able to generalize it to French as well,

Salut, Comment allez-vous ---> Je vais bien, merci.

As long as the transformer understands both English and French.