How to split data for meta-learning?

Question

I've been trying to understand the meta-learning paradigm, more precisely, the optimization-based models, such as MAML, but I have a hard time understanding how I should correctly split my data to train such models.

For example, let's consider we have 4 traffic datasets, and we would like to use 3 of them as source datasets (to train the model) and the remaining one as target (to fine-tune on it and then test the model performance). As far as I understood, for each source dataset, I need to split it into train and validation. During training, I would randomly select 2 batches of samples from the training datasets, use one batch to compute the task-specific parameters and the other one to compute the loss. Then repeat the same process with the validation dataset, such that I can select the best candidate model. After the training is done, I need to fine-tune the model on the target dataset. I assume the process is the same, but I need to use the target dataset instead.

During testing (after the model is fully learned and fine-tuned), how exactly do I test it? It is the same procedure as if I was training a supervised model? I would like to know if the setup I described is correct and it fits the meta-learning paradigm.

Answer 1

I assume in your case what you need to be doing is to collate your 3 datasets together - these would form the training dataset, and then leave the testing dataset aside.

During Meta-Training, the code will sample a batch of tasks in each iteration. This batch of tasks will be split into support and query, the algorithm will train on the support and update model parameters, then it will test on the query and then re-update initial theta.

This Meta-Training is a form of 'cross validation'. Now during meta testing, you must ensure that it samples from the testing dataset.

You can use this code as a start. In data_generator.py it has a place where it splits the images into training and validation. What you need to be doing is to do the same thing but for your structured data, there, you will specify which datasets to sample for training and which dataset to sample for testing.