Skip to main content
6 votes
Accepted

What are "development test sets" used for?

In machine learning, you normally split your data into 3 parts (80-10-10%). The first part (80% of your initial data) is for the training of your ML model: this is known as the training dataset. The ...
user3352632's user avatar
5 votes
Accepted

How is MNIST only providing the training and the test sets? What about the validation?

The test set should never be seen and ran once at the end of training. The validation set is used to help you select hyperparameters and it would be cheating to tune your model on the test set because ...
Winnie Xu's user avatar
2 votes

What is the theoretical basis for the use of a validation set?

I think Cross-Validation serves a completely different purpose. From your post, it looks like you think we would use CV to get a better estimate of the parameters of our model (i.e. the model ...
John Doucette's user avatar
2 votes
Accepted

What is the difference between validation percentage and batch size?

The percentages refer to the number of samples to use (out of full dataset) as the validation and test datasets. So if you pass a dataset that consists of 100,000 samples to the model and set the ...
Mark.F's user avatar
  • 446
2 votes
Accepted

How to perform PCA in the validation/test set?

for the first point I'm very sorry that I cannot give you any literature on this, but I might be able to explain you, why you don't take PCA on both datasets independently. Principal components ...
adriculteur's user avatar
1 vote

Is it legitimate to train a model on a benchmark dataset and use this model only for labeling another datasets

I'm not sure I understand the point. The secret model will invariably have some loss. This loss will be overlaid on the larger dataset as the secret model operates on it. Whatever model is trained ...
foreverska's user avatar
  • 1,298
1 vote

Is it legitimate to train a model on a benchmark dataset and use this model only for labeling another datasets

neural network are function approximators... What you are saying is that given a data distribution $D$, you train a model $f$ on it, and then you you se $f$ to label a new dataset $D'$ Now, you want ...
Alberto's user avatar
  • 2,273
1 vote

Is there validation data in K-fold cross-validation?

Before doing the $k$-fold cross-val you divide all your dataset into train and test splits (e.g., 80-20 in proportion), then the $k$-fold cross validation is performed on the training split as follows:...
Luca Anzalone's user avatar
1 vote

how to decide the optimum model?

Testing each time on a test set is against the point of a train-val-test split. The reason test is important, is that you are only supposed to test on it when you think your model is good and ready ...
mshlis's user avatar
  • 2,369
1 vote

Why not make the training set and validation set one if their roles are similar?

I think this is best explained using an analogy. Also you seen to have the misconception that you don't tune hyper-parameters for training data. You want to increase the accuracy of the training set ...
Recessive's user avatar
  • 1,396
1 vote

Why not make the training set and validation set one if their roles are similar?

Idea is to optimize with regards to unseen data in each step in order to avoid overfitting and data leakage so that the final network will be most generalizable to novel data. First, you initialize ...
meliksahturker's user avatar
1 vote
Accepted

Are the held-out datasets used for testing, validation or both?

I think that these terms may be used inconsistently across sources. If someone says held-out dataset, I would immediately think of a dataset that is not used for training, but can be used for anything ...
nbro's user avatar
  • 40.9k

Only top scored, non community-wiki answers of a minimum length are eligible