6 votes
Accepted

What are "development test sets" used for?

In machine learning, you normally split your data into 3 parts (80-10-10%). The first part (80% of your initial data) is for the training of your ML model: this is known as the training dataset. The ...
user avatar
2 votes

What is the theoretical basis for the use of a validation set?

I think Cross-Validation serves a completely different purpose. From your post, it looks like you think we would use CV to get a better estimate of the parameters of our model (i.e. the model ...
user avatar
2 votes
Accepted

What is the difference between validation percentage and batch size?

The percentages refer to the number of samples to use (out of full dataset) as the validation and test datasets. So if you pass a dataset that consists of 100,000 samples to the model and set the ...
user avatar
  • 426
2 votes
Accepted

How to perform PCA in the validation/test set?

for the first point I'm very sorry that I cannot give you any literature on this, but I might be able to explain you, why you don't take PCA on both datasets independently. Principal components ...
user avatar
1 vote

how to decide the optimum model?

Testing each time on a test set is against the point of a train-val-test split. The reason test is important, is that you are only supposed to test on it when you think your model is good and ready ...
user avatar
  • 2,249
1 vote

Why not make the training set and validation set one if their roles are similar?

I think this is best explained using an analogy. Also you seen to have the misconception that you don't tune hyper-parameters for training data. You want to increase the accuracy of the training set ...
user avatar
  • 1,344
1 vote

Why not make the training set and validation set one if their roles are similar?

Idea is to optimize with regards to unseen data in each step in order to avoid overfitting and data leakage so that the final network will be most generalizable to novel data. First, you initialize ...
user avatar
1 vote
Accepted

Are the held-out datasets used for testing, validation or both?

I think that these terms may be used inconsistently across sources. If someone says held-out dataset, I would immediately think of a dataset that is not used for training, but can be used for anything ...
user avatar
  • 33.8k

Only top scored, non community-wiki answers of a minimum length are eligible