It is known that multitask objectives in neural networks sometimes have the effect of improving the performance of the neural network for each of the tasks individually (versus training the same network for each task individually). To what extent is this true when fine tuning a pretrained, single-task neural network to deal with a multi-task objectives versus training from scratch?
$\begingroup$
$\endgroup$
asked Mar 15 at 10:27
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
Your statement is most likely true to the extent when all involved tasks are similar or strongly related and the amount of (labeled) data available for each individual objective task is limited. Therefore they can transfer the knowledge acquired from the pretrained single-task neural network to learn the new tasks with fewer available examples. Furthermore, fine-tuning a pretrained model acts as a form of regularization as it constrains the model to learn task-specific features while retaining the general knowledge captured by the pretrained layers to prevent overfitting, especially when the amount of data available for each task is limited.
In general multi-task learning encourages the model to efficiently learn shared representations that are useful across multiple relevant tasks, leading to better generalization and improved performance on each task individually. It's like the ordinary fact that jointly learning math and physics improves performance of each subject for most students as they're very related areas and most students time to study are limited.
