3
$\begingroup$

I am currently studying the paper Learning and Evaluating Classifiers under Sample Selection Bias by Bianca Zadrozny. In the introduction, the author says the following:

One of the most common assumptions in the design of learning algorithms is that the training data consist of examples drawn independently from the same underlying distribution as the examples about which the model is expected to make predictions. In many real-world applications, however, this assumption is violated because we do not have complete control over the data gathering process.

For example, suppose we are using a learning method to induce a model that predicts the side-effects of a treatment for a given patient. Because the treatment is not given randomly to individuals in the general population, the available examples are not a random sample from the population. Similarly, suppose we are learning a model to predict the presence/absence of an animal species given the characteristics of a geographical location. Since data gathering is easier in certain regions than others, we would expect to have more data about certain regions than others.

In both cases, even though the available examples are not a random sample from the true underlying distribution of examples, we would like to learn a predictor from the examples that is as accurate as possible for this distribution. Furthermore, we would like to be able to estimate its accuracy for the whole population using the available data.

It's this part that I am confused about:

In both cases, even though the available examples are not a random sample from the true underlying distribution of examples, we would like to learn a predictor from the examples that is as accurate as possible for this distribution.

What exactly is "this distribution"? Is it referring to the true underlying distribution, or the distribution of our sample (which, as was said, is not necessarily a "good" reflection of the underlying distribution, since it is not a random sample)?

$\endgroup$
0
$\begingroup$

In both cases, even though the available examples are not a random sample from the true underlying distribution of examples, we would like to learn a predictor from the examples that is as accurate as possible for this distribution.

"The true underlying distribution" is the closest "distribution" that is explicitly mentioned as such in the part of the text preceding the phrase "this distribution", so that's what it's referring to. For clarity, I've put the two things that are "the same" in bold in the above quote.

$\endgroup$
2
  • $\begingroup$ Thanks for the answer. Are you sure about this? The next sentence in the paper says the following: "Furthermore, we would like to be able to estimate its accuracy for the whole population using the available data." I think this implies that "this distribution" is referring to the biased distribution, no? $\endgroup$ – The Pointer Nov 27 '20 at 3:46
  • 1
    $\begingroup$ No so it's again referring to the "true underlying distribution" of "the whole population" in both cases: they have 2 different things they want to be able to accomplish for this. 1) described in the quote in my answer, they want to learn a predictor that can make good predictions for the true distribution despite being trained on a potentially non-representative sample. 2) described in the quote from your comment, they also want to be able to accurately estimate what performance the trained predictor would have if it were to be used for the true distribution $\endgroup$ – Dennis Soemers Nov 27 '20 at 9:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.