6
votes
Accepted
What are "development test sets" used for?
In machine learning, you normally split your data into 3 parts (80-10-10%).
The first part (80% of your initial data) is for the training of your ML model: this is known as the training dataset.
The ...
- 283
6
votes
Should I continue training if the neural network attains 100% training accuracy?
First of all, as mentioned by @Neil Slater in the comment - you need to have three splits into the train, validation and test set.
One sometimes disregards the difference between validation and test ...
- 2,754
5
votes
Accepted
Is k-fold cross-validation more effective than splitting the dataset into training and test datasets to prevent overfitting?
K-fold cross-validation is probably preferred in terms of completeness and generalization: you ensure that the system has seen the complete dataset for training. However, in deep learning this is ...
- 156
5
votes
Accepted
What is the best measure for detecting overfitting?
tl;dr
The safest method I've found is to use cross-validation for hyperparameter selection and a hold-out test set for a final evaluation.
Why this isn't working for you...
In your case, I suspect you'...
- 3,213
3
votes
Accepted
How to fairly conduct a model performance with 5-fold cross validation after augmentation?
If you used your five $X_{test}$ sets multiple times (to measure the average AUC) to decide on the best set of hyperparameters (i.e. optimizer, learning rate, batch size, dropout, activation) then yes,...
3
votes
How do you interpret this learning curve?
The validation loss settles exactly at an error of one. Probably means there's something off with either the kind of data validation set has or with something in the training. An exact validation loss ...
- 1,419
3
votes
How do you interpret this learning curve?
Depends on what does 1 represent in your task.
If you are trying to predict household prices and 1 represents \$1, I think the average validation loss is good. If 1 represents \$10000 in this case, ...
- 81
3
votes
Accepted
Should I choose the model with highest validation accuracy or the model with highest mean of training and validation accuracy?
Neither of the above mentioned methods could be a potent indicator of the performance of a model.
A simple way to train the model just enough so that it generalizes well on unknown datasets would be ...
- 370
3
votes
Is k-fold cross-validation more effective than splitting the dataset into training and test datasets to prevent overfitting?
Purely in terms of overfitting, and assuming you train both for equal amounts of time, 70/30 is probably better but performance is not going to be very good. Not training on %30 of data will make both ...
- 900
3
votes
Accepted
Why is the validation loss less than the training loss, and what can be said about the effect of the learning rate?
This is very difficult to tell with the information provided, but the phenomenon is something that I have encountered many times before. Sometimes this is not a bad thing, here are some possible ...
- 258
3
votes
Accepted
Weights initialization once the Neural Network is trained
Regarding your first code snippet, there is no weight storing or continuation of training between the different CV folds whatsoever: each model is trained anew with the respective training data of ...
- 1,040
2
votes
Accepted
What is the difference between validation percentage and batch size?
The percentages refer to the number of samples to use (out of full dataset) as the validation and test datasets. So if you pass a dataset that consists of 100,000 samples to the model and set the ...
- 446
2
votes
Should I use leave-one-out cross-validation for testing?
Concerning $k$-fold Cross Validation, I like to think of it by considering two extremes you can do: Leave-One-Out Cross-Validation where you leave one sample each time and train your model on the ...
2
votes
After having selected the best model with cross-validation, for how long should I train it?
Short answer: training "duration" or number of epochs/updates should be cross-validated too: you want to early-stop your training to prevent overfitting.
Longer answer:
Think of accuracy on the ...
- 276
2
votes
What is the theoretical basis for the use of a validation set?
I think Cross-Validation serves a completely different purpose.
From your post, it looks like you think we would use CV to get a better estimate of the parameters of our model (i.e. the model ...
- 9,392
2
votes
Accepted
How should I interpret this validation plot?
This is a sign of overfitting.
As you make your trees deeper, it becomes possible to "memorize" the data: each leaf of the tree is just a single point. The trees begin to learn patterns that do not ...
- 9,392
2
votes
Calculating accuracy for cross validation
I guess you could train your model with 10 different folds and in each fold calculate the average accuracy. So you would have 10 values - one corresponding to each fold. And then take the mean of all ...
- 343
2
votes
Accepted
How to decide a train-test split?
I don't think there is any rationale behind choosing 80/20 over 75/25 or others. But those are the numbers for rather small datasets. If your dataset is large enough (like hundreds of thousands of ...
- 900
1
vote
Accepted
What are non-held-out data or non-held-out classes?
Held-out simply means "not included" particularly in the sense of:
This part of the data was not included in this specific training run.
Depending on the context of all of these text non-held-out ...
- 880
1
vote
Accepted
Does adding a model complexity penalty to the loss function allow you to skip cross-validation?
It's my understanding that selecting for small models, i.e. having a multi-objective function where you're optimizing for both model accuracy and simplicity, automatically takes care of the danger of ...
- 33.3k
1
vote
How to fill NaNs in Cross-Validation?
I would do the exact same thing as you are describing! One of the main reasons that you would want to do cross-validation is to prevent that your model is unable to generalize later. Therefore, you ...
- 2,660
1
vote
How to avoid over-fitting using early stopping when using R cross validation package caret
You are handling a very small dataset. The only way to prevent overfitting then is to choose a very restrictive model search space. The simpler the better, and you should prefer models involving some ...
- 183
1
vote
Accepted
How exactly does nested cross-validation work?
"Selecting the model" in this case refers to selecting the hyperparameters of the model. The reason to use a nested CV is simply to avoid overfitting training data.
Consider the example in ...
- 628
1
vote
Is my 57% sports betting accuracy correct?
My question is, can I rely on my Accuracy (mean & standard deviation) for future games even though my Testing Accuracy is lower than 52.5%?
If by Accuracy you mean training accuracy, then ...
- 33.3k
1
vote
Accepted
While we split data in training and test data, why we have two pairs of each?
Are you talking about (X_train,y_train) and (X_test,y_test).
If yes, then X represents the data(features) and y represents the labels of that data. That's why you get a pair when you divide it into ...
- 26
1
vote
While we split data in training and test data, why we have two pairs of each?
For any Machine Learning model, the available data is usually split into three sets:
Training Set:
The part of data used to train the model and learn the parameters of the network.
The data that ...
- 370
1
vote
Should I choose the model with highest validation accuracy or the model with highest mean of training and validation accuracy?
The training accuracy tells you nothing about how good it is on other data than the ones it learned on, it could be better on this data because it memorized this examples.
On the other hand the ...
- 434
1
vote
Accepted
What is the relationship between the training accuracy and validation accuracy?
very interesting questions:
1. what exactly is happening when training and validation accuracy change during training
The accuracy change after every batch ...
- 1,168
1
vote
How do you interpret this learning curve?
The telltale signature of overfitting is when your validation loss starts increasing, while your training loss continues decreasing, i.e.:
(Image adapted from Wikipedia entry on overfitting)
It is ...
- 1,040
1
vote
Metrics for evaluating models that output probabilities
For a binary classifier, the cross-entropy loss is a natural measure of probability accuracy, if you care about relative probabilities. By that I mean if you care that the estimate $\hat{p}$ is within ...
- 33.3k
Only top scored, non community-wiki answers of a minimum length are eligible