Suppose I have a dataset with (X, Y) training samples where X is a 1 dimension, and Y is also 1 dimension. Example: if this is a housing price dataset, X would be an area in square feet, and Y would be the price of the house. I can train a linear regression model to fit a line through these data points and then the model can be used without any problems.

Now consider a harder problem, I have many datasets, each of which is a housing price dataset. See the attached image. You will notice that there are 3 datasets indicated by black, red, and blue markers. For example, black is a housing price dataset of a crowded major city (quite expensive), and blue is a dataset of a countryside area (quite cheap). I want to train a single model on all of these datasets. I expect that when I feed the model an X, it will give me the correct Y. This means that the model needs to figure out which dataset the X is coming from. multi-domain regression

If X is less than 20, the model will know for sure that X is from the black dataset, because in the training set, there was only the black dataset that contains X less than 20. If X is more than 80, the model also knows for sure that X is from the blue dataset, so it can predict correctly. The problem would occur if X is between 20 and 80, the model cannot know which dataset X is coming from, so it's logical to expect the prediction to be incorrect.

So my idea of helping the model figure out the dataset is to give it many data points to predict at once, instead of giving it only one X value. For example, I can guarantee you that the last 100 data points (100 X values) to predict as a batch, will come from the same dataset.

Ideally, the model can summarize the data points and figure out which dataset these data points are coming from. If I give you 100 points, and you see that every point's X lies between 20 and 80, then you know for sure it's the red dataset. Sophisticatedly, you can model the distribution of the data points to figure out which dataset you are looking at. After you know it's the red dataset, you output a number or a vector of numbers that represent the dataset.

In this case, the representation of the dataset could only be the "slope" of the dataset. The black dataset has higher slope than the red and blue. (A more accurate model might also model the "bias/intercept" instead of only the slope) The model can use this slope "representation" to refer to the dataset, and use it to predict all 100 points correctly. Let's say that to solve this, you might need 2 models, first model will figure out the representation (slope in this example) of the data points. And the 2nd model will receive the slope along with all 100 X values, and output 100 Y values correctly. Some of you might be able to merge these 2 models into a single model that expects 100 X values, and return 100 Y values directly. The merged model will internally figure out the slope of 100 X values by itself. In the end, this model will be trained with many datasets, e.g. 10 datasets, and it's expected to be able to handle dataset it has never seen before. It can handle the new dataset because it can model the slope representation of anything.

I'm interested to know what kinds of techniques can help me figure out the representation of data points in a batch, figuring out which dataset it's coming from, this is useful when you want your system to work in different data distributions. An application could be a sensory system trained to work with one person. When it's trained to work with both people, it fails because there are Y conflicts (one X value maps to 2 Y values). The only way for the system to work with both people is that it must figure out which person it's working with by looking at the time series of the sensor.


More generally, this can be defined as a problem where I can give you a batch of X values, (X could be a vector of numbers, instead of a scalar like in this example) that come from the same dataset, and I want you to model the "latent representation" of the dataset. (The representation could be a vector of numbers instead of a scalar) Finally, you would use the dataset representation along with the X values, to predict Y values. You are so smart you can work with the dataset you have never seen before because you can model the representation of any dataset, as long as the dataset has a similar tone to other datasets.

What kinds of techniques/topics can solve this general problem? If would be best if you can give a concrete example in code.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.