I'm solving a sequential prediction task that has multiple features attached to each timestamp, and my goal is to predict one of them. However, the feature's label is highly imbalanced: only about 4% of them are positive and the remaining are negative. I've tried multiple strategies (weighted loss, data augmentation, ...), and, surprisingly, the best approach was using multi-objective learning. More precisely, I trained the model with one more classifier attached which predicts another feature, and train the model with the (weighted) sum of BCE loss for each feature.
Also, it substantially reduced the training time. However, two tasks aren't perfectly correlated - as time goes, the model seems to fit into the latter task (not the original task) and the validation AUC for the original task decreases soon (although the performance is better than single objective training).
I wonder if there's a good approach to improve the performance in this case. Also, I want to ask if transfer learning (pre-training on the task 2 and fine-tuning on the task 1) would help. I think this would be better than baseline since multi-objective learning showed a non-trivial correlation between two tasks, but I'm not sure how much it would help. Especially, I wonder if it gives a better results than multi-objective training. (It would be good if there's a reference for comparing multi-objective training and transfer learning in detail.)