0
$\begingroup$

I have a database that contains healthy persons and lung cancer patients. I need to design a deep neural network for the binary classification problem (cancer/no cancer). I need to split the dataset into 70% train and 30% test.

How can I do the splitting? According to persons?

I think that splitting according to persons is correct since this will ensure that the same person will not exist simultaneously in both the training and the test subsets. This is reasonable since we are recognizing the disease, not the person. If images from the same person exist in both subsets, the problem will be easy, and not reasonable, from a practical point of view. Do you agree?

Improve this question
$\endgroup$
2
  • $\begingroup$ it is not clear to me what other alternatives you have. From the way you describe the data it seems you have information per each person, e.g. [age:25, weight:60, height: 1.80, blood_pressure:120, ..., cancer: False]. By splitting into train test, since you have one record per person, it would be impossible to have the same person into train and test sets at the same time. Am I missing something? $\endgroup$
    – Edoardo Guerriero
    Commented Nov 10, 2021 at 11:26
  • $\begingroup$ I have many samples for the same person (10 samples for examples)). The samples represent X rays images from the person. $\endgroup$
    – Noha
    Commented Nov 10, 2021 at 19:55

1 Answer 1

Reset to default
1
$\begingroup$

we are recognizing the disease, not the person.

If you're training a computer vision model with only images and no auxiliary information then a randomized sampling should be enough to prevent the model from over fitting on x-ray scans taken on the same person.

If images from the same person exist in both subsets, the problem will be easy, and not reasonable, from a practical point of view. Do you agree?

Only partially. The aim of a test set is to allow you to quantify how well your model will perform on a real use case. Which is close but not the same as making the problem as hard as possible in testing phase.

Also, it is not necessarily true that having 20 examples from the same person in the training set will lead to high accuracy on even 1 single test instance coming again from the same person. This is because those training instances might contain bias (like many tumors on the left side of the scan rather than the right side).

This is why I personally would go for the following splitting approach:

  • sample from the whole dataset a small group of people (for example top n people with fewest amount of scans). Let's call this set between groups test set.
  • sample per person (from the remaining data) 70% training instances and 30% test instances. Let's call it within groups test set.

During validation I would then perform the following checks:

  • metrics calculation (precision, recall, f-score, i.e the whole package) on the between group test performed as usual.
  • metrics calculation per person (or clusters of people, paying attention in keeping the clusters the same at every validation step) on the within groups test set.

The second check is the most interesting. Of course there will be some random variation in the performance, but if the model will start performing better on some specific clusters of people that might be a hint that the model is learning some confounding variables probably related to the presence of really similar scans or other bias. testing only on the between groups will not let you to draw that conclusion.

Improve this answer
$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .