6 votes
Accepted

What are "development test sets" used for?

In machine learning, you normally split your data into 3 parts (80-10-10%). The first part (80% of your initial data) is for the training of your ML model: this is known as the training dataset. The ...
user avatar
4 votes
Accepted

How do I select the (number of) negative cases, if I'm given a set of positive cases?

Short answer To select the proper dataset to construct, you should first figure out a metric to use to compare, and then select the dataset construction that gives the better metric. There is no ...
user avatar
  • 1,260
3 votes
Accepted

What are possible ways to combat overfitting or improve the test accuracy in my case?

There are a few issues you need to address first. Normalise your data. You should try and keep your values for each input in a good range, otherwise you're never going to train anything useful. A ...
user avatar
  • 1,246
3 votes

What does it mean by overfitting the test set?

Essentially, any data you use to train or develop the model shouldn't be used as test data. In principle, "unseen" data gives a good estimate for the generalisation performance of the model; ...
user avatar
  • 941
2 votes
Accepted

How to perform PCA in the validation/test set?

for the first point I'm very sorry that I cannot give you any literature on this, but I might be able to explain you, why you don't take PCA on both datasets independently. Principal components ...
user avatar
1 vote

Given a dataset of people with and without cancer, should I split it into training and test datasets such that the same person is not in both?

we are recognizing the disease, not the person. If you're training a computer vision model with only images and no auxiliary information then a randomized sampling should be enough to prevent the ...
user avatar
1 vote

how to decide the optimum model?

Testing each time on a test set is against the point of a train-val-test split. The reason test is important, is that you are only supposed to test on it when you think your model is good and ready ...
user avatar
  • 2,289
1 vote
Accepted

Are the held-out datasets used for testing, validation or both?

I think that these terms may be used inconsistently across sources. If someone says held-out dataset, I would immediately think of a dataset that is not used for training, but can be used for anything ...
user avatar
  • 34.5k
1 vote

How to build a test set for a model in industry?

I am not sure whether that solves your problem at hand, but one approach you could look into is k-fold Cross Validation (CV). In this approach, you split your combined train, development, and test ...
user avatar
  • 705
1 vote
Accepted

What is the reason behind using a test batch size?

I am not familiar with using batches during network evaluation. Can someone explain what is the reason behind using it and what are advantages and disadvantages? It is usually just for memory use ...
user avatar
  • 24.5k

Only top scored, non community-wiki answers of a minimum length are eligible