6 votes

Should I continue training if the neural network attains 100% training accuracy?

First of all, as mentioned by @Neil Slater in the comment - you need to have three splits into the train, validation and test set. One sometimes disregards the difference between validation and test ...
user avatar
6 votes
Accepted

What are "development test sets" used for?

In machine learning, you normally split your data into 3 parts (80-10-10%). The first part (80% of your initial data) is for the training of your ML model: this is known as the training dataset. The ...
user avatar
5 votes
Accepted

What is the best measure for detecting overfitting?

tl;dr The safest method I've found is to use cross-validation for hyperparameter selection and a hold-out test set for a final evaluation. Why this isn't working for you... In your case, I suspect you'...
user avatar
  • 3,093
4 votes
Accepted

Is k-fold cross-validation more effective than splitting the dataset into training and test datasets to prevent overfitting?

K-fold cross-validation is probably preferred in terms of completeness and generalization: you ensure that the system has seen the complete dataset for training. However, in deep learning this is ...
user avatar
3 votes
Accepted

Why is the validation loss less than the training loss, and what can be said about the effect of the learning rate?

This is very difficult to tell with the information provided, but the phenomenon is something that I have encountered many times before. Sometimes this is not a bad thing, here are some possible ...
user avatar
  • 218
3 votes
Accepted

Should I choose the model with highest validation accuracy or the model with highest mean of training and validation accuracy?

Neither of the above mentioned methods could be a potent indicator of the performance of a model. A simple way to train the model just enough so that it generalizes well on unknown datasets would be ...
user avatar
  • 360
3 votes

Is k-fold cross-validation more effective than splitting the dataset into training and test datasets to prevent overfitting?

Purely in terms of overfitting, and assuming you train both for equal amounts of time, 70/30 is probably better but performance is not going to be very good. Not training on %30 of data will make both ...
user avatar
  • 870
3 votes

How do you interpret this learning curve?

The validation loss settles exactly at an error of one. Probably means there's something off with either the kind of data validation set has or with something in the training. An exact validation loss ...
user avatar
  • 1,359
3 votes

How do you interpret this learning curve?

Depends on what does 1 represent in your task. If you are trying to predict household prices and 1 represents \$1, I think the average validation loss is good. If 1 represents \$10000 in this case, ...
user avatar
  • 71
2 votes
Accepted

What is the difference between validation percentage and batch size?

The percentages refer to the number of samples to use (out of full dataset) as the validation and test datasets. So if you pass a dataset that consists of 100,000 samples to the model and set the ...
user avatar
  • 426
2 votes

After having selected the best model with cross-validation, for how long should I train it?

Short answer: training "duration" or number of epochs/updates should be cross-validated too: you want to early-stop your training to prevent overfitting. Longer answer: Think of accuracy on the ...
user avatar
  • 246
2 votes

What is the theoretical basis for the use of a validation set?

I think Cross-Validation serves a completely different purpose. From your post, it looks like you think we would use CV to get a better estimate of the parameters of our model (i.e. the model ...
user avatar
2 votes
Accepted

How to fairly conduct a model performance with 5-fold cross validation after augmentation?

If you used your five $X_{test}$ sets multiple times (to measure the average AUC) to decide on the best set of hyperparameters (i.e. optimizer, learning rate, batch size, dropout, activation) then yes,...
user avatar
2 votes
Accepted

How should I interpret this validation plot?

This is a sign of overfitting. As you make your trees deeper, it becomes possible to "memorize" the data: each leaf of the tree is just a single point. The trees begin to learn patterns that do not ...
user avatar
2 votes

Calculating accuracy for cross validation

I guess you could train your model with 10 different folds and in each fold calculate the average accuracy. So you would have 10 values - one corresponding to each fold. And then take the mean of all ...
user avatar
  • 313
2 votes
Accepted

How to decide a train-test split?

I don't think there is any rationale behind choosing 80/20 over 75/25 or others. But those are the numbers for rather small datasets. If your dataset is large enough (like hundreds of thousands of ...
user avatar
  • 870
2 votes
Accepted

Weights initialization once the Neural Network is trained

Regarding your first code snippet, there is no weight storing or continuation of training between the different CV folds whatsoever: each model is trained anew with the respective training data of ...
user avatar
1 vote
Accepted

Does adding a model complexity penalty to the loss function allow you to skip cross-validation?

It's my understanding that selecting for small models, i.e. having a multi-objective function where you're optimizing for both model accuracy and simplicity, automatically takes care of the danger of ...
user avatar
1 vote

How to fill NaNs in Cross-Validation?

I would do the exact same thing as you are describing! One of the main reasons that you would want to do cross-validation is to prevent that your model is unable to generalize later. Therefore, you ...
user avatar
1 vote
Accepted

How exactly does nested cross-validation work?

"Selecting the model" in this case refers to selecting the hyperparameters of the model. The reason to use a nested CV is simply to avoid overfitting training data. Consider the example in ...
user avatar
  • 600
1 vote

Is my 57% sports betting accuracy correct?

My question is, can I rely on my Accuracy (mean & standard deviation) for future games even though my Testing Accuracy is lower than 52.5%? If by Accuracy you mean training accuracy, then ...
user avatar
1 vote
Accepted

What are non-held-out data or non-held-out classes?

Held-out simply means "not included" particularly in the sense of: This part of the data was not included in this specific training run. Depending on the context of all of these text non-held-out ...
user avatar
  • 880
1 vote
Accepted

What is the relationship between the training accuracy and validation accuracy?

very interesting questions: 1. what exactly is happening when training and validation accuracy change during training The accuracy change after every batch ...
user avatar
  • 1,018
1 vote

How do you interpret this learning curve?

The telltale signature of overfitting is when your validation loss starts increasing, while your training loss continues decreasing, i.e.: (Image adapted from Wikipedia entry on overfitting) It is ...
user avatar
1 vote
Accepted

While we split data in training and test data, why we have two pairs of each?

Are you talking about (X_train,y_train) and (X_test,y_test). If yes, then X represents the data(features) and y represents the labels of that data. That's why you get a pair when you divide it into ...
user avatar
1 vote

While we split data in training and test data, why we have two pairs of each?

For any Machine Learning model, the available data is usually split into three sets: Training Set: The part of data used to train the model and learn the parameters of the network. The data that ...
user avatar
  • 360
1 vote
Accepted

Can we say: the more we increase the numbers of cross validation the less likely it is that we overfit?

In general, no. There is a tradeoff between making the validation set for each fold smaller, and having more folds in total. As an example, if you have $N$ folds for $N$ datapoints, each fold will ...
user avatar
1 vote
Accepted

How to interpret this learning curve plot

Note the X index is training set size. For the first and second case, teh training set size starts at 0(or 1). The model will overfit certainly at that data size. When data size increases, the model ...
user avatar
  • 1,715
1 vote

Metrics for evaluating models that output probabilities

For a binary classifier, the cross-entropy loss is a natural measure of probability accuracy, if you care about relative probabilities. By that I mean if you care that the estimate $\hat{p}$ is within ...
user avatar
1 vote
Accepted

Ideal score of a model on training and cross validation data

You should only look for the cross-validation score. If this set is large enough, it will give you an accurate prediction of how your model will act for unseen data. Your case is exceptional. The ...
user avatar

Only top scored, non community-wiki answers of a minimum length are eligible