# Tag Info

### Should I continue training if the neural network attains 100% training accuracy?

First of all, as mentioned by @Neil Slater in the comment - you need to have three splits into the train, validation and test set. One sometimes disregards the difference between validation and test ...
Accepted

### What are "development test sets" used for?

In machine learning, you normally split your data into 3 parts (80-10-10%). The first part (80% of your initial data) is for the training of your ML model: this is known as the training dataset. The ...
• 274
Accepted

### What is the best measure for detecting overfitting?

tl;dr The safest method I've found is to use cross-validation for hyperparameter selection and a hold-out test set for a final evaluation. Why this isn't working for you... In your case, I suspect you'...
• 3,093
Accepted

### Is k-fold cross-validation more effective than splitting the dataset into training and test datasets to prevent overfitting?

K-fold cross-validation is probably preferred in terms of completeness and generalization: you ensure that the system has seen the complete dataset for training. However, in deep learning this is ...
• 146
Accepted

### Why is the validation loss less than the training loss, and what can be said about the effect of the learning rate?

This is very difficult to tell with the information provided, but the phenomenon is something that I have encountered many times before. Sometimes this is not a bad thing, here are some possible ...
• 218
Accepted

### Should I choose the model with highest validation accuracy or the model with highest mean of training and validation accuracy?

Neither of the above mentioned methods could be a potent indicator of the performance of a model. A simple way to train the model just enough so that it generalizes well on unknown datasets would be ...
• 360

### Is k-fold cross-validation more effective than splitting the dataset into training and test datasets to prevent overfitting?

Purely in terms of overfitting, and assuming you train both for equal amounts of time, 70/30 is probably better but performance is not going to be very good. Not training on %30 of data will make both ...
• 870

### How do you interpret this learning curve?

The validation loss settles exactly at an error of one. Probably means there's something off with either the kind of data validation set has or with something in the training. An exact validation loss ...
• 1,359

### How do you interpret this learning curve?

Depends on what does 1 represent in your task. If you are trying to predict household prices and 1 represents \$1, I think the average validation loss is good. If 1 represents \$10000 in this case, ...
• 71
Accepted

### What is the difference between validation percentage and batch size?

The percentages refer to the number of samples to use (out of full dataset) as the validation and test datasets. So if you pass a dataset that consists of 100,000 samples to the model and set the ...
• 426

### After having selected the best model with cross-validation, for how long should I train it?

Short answer: training "duration" or number of epochs/updates should be cross-validated too: you want to early-stop your training to prevent overfitting. Longer answer: Think of accuracy on the ...
• 246

### What is the theoretical basis for the use of a validation set?

I think Cross-Validation serves a completely different purpose. From your post, it looks like you think we would use CV to get a better estimate of the parameters of our model (i.e. the model ...
• 8,877
Accepted

### How to fairly conduct a model performance with 5-fold cross validation after augmentation?

If you used your five $X_{test}$ sets multiple times (to measure the average AUC) to decide on the best set of hyperparameters (i.e. optimizer, learning rate, batch size, dropout, activation) then yes,...
Accepted

### How should I interpret this validation plot?

This is a sign of overfitting. As you make your trees deeper, it becomes possible to "memorize" the data: each leaf of the tree is just a single point. The trees begin to learn patterns that do not ...
• 8,877

### Calculating accuracy for cross validation

I guess you could train your model with 10 different folds and in each fold calculate the average accuracy. So you would have 10 values - one corresponding to each fold. And then take the mean of all ...
• 313
Accepted

### How to decide a train-test split?

I don't think there is any rationale behind choosing 80/20 over 75/25 or others. But those are the numbers for rather small datasets. If your dataset is large enough (like hundreds of thousands of ...
• 870
Accepted

### Weights initialization once the Neural Network is trained

Regarding your first code snippet, there is no weight storing or continuation of training between the different CV folds whatsoever: each model is trained anew with the respective training data of ...
• 944
1 vote
Accepted

### Does adding a model complexity penalty to the loss function allow you to skip cross-validation?

It's my understanding that selecting for small models, i.e. having a multi-objective function where you're optimizing for both model accuracy and simplicity, automatically takes care of the danger of ...
• 24k
1 vote

### How to fill NaNs in Cross-Validation?

I would do the exact same thing as you are describing! One of the main reasons that you would want to do cross-validation is to prevent that your model is unable to generalize later. Therefore, you ...
1 vote
Accepted

### How exactly does nested cross-validation work?

"Selecting the model" in this case refers to selecting the hyperparameters of the model. The reason to use a nested CV is simply to avoid overfitting training data. Consider the example in ...
• 600
1 vote

### Is my 57% sports betting accuracy correct?

My question is, can I rely on my Accuracy (mean & standard deviation) for future games even though my Testing Accuracy is lower than 52.5%? If by Accuracy you mean training accuracy, then ...
• 24k
1 vote
Accepted

### What are non-held-out data or non-held-out classes?

Held-out simply means "not included" particularly in the sense of: This part of the data was not included in this specific training run. Depending on the context of all of these text non-held-out ...
• 880
1 vote
Accepted

### What is the relationship between the training accuracy and validation accuracy?

very interesting questions: 1. what exactly is happening when training and validation accuracy change during training The accuracy change after every batch ...
• 1,018
1 vote

### How do you interpret this learning curve?

The telltale signature of overfitting is when your validation loss starts increasing, while your training loss continues decreasing, i.e.: (Image adapted from Wikipedia entry on overfitting) It is ...
• 944
1 vote
Accepted

### While we split data in training and test data, why we have two pairs of each?

Are you talking about (X_train,y_train) and (X_test,y_test). If yes, then X represents the data(features) and y represents the labels of that data. That's why you get a pair when you divide it into ...
1 vote

### While we split data in training and test data, why we have two pairs of each?

For any Machine Learning model, the available data is usually split into three sets: Training Set: The part of data used to train the model and learn the parameters of the network. The data that ...
• 360
1 vote
Accepted

### Can we say: the more we increase the numbers of cross validation the less likely it is that we overfit?

In general, no. There is a tradeoff between making the validation set for each fold smaller, and having more folds in total. As an example, if you have $N$ folds for $N$ datapoints, each fold will ...
• 8,877
1 vote
Accepted

### How to interpret this learning curve plot

Note the X index is training set size. For the first and second case, teh training set size starts at 0(or 1). The model will overfit certainly at that data size. When data size increases, the model ...
• 1,715
1 vote

### Metrics for evaluating models that output probabilities

For a binary classifier, the cross-entropy loss is a natural measure of probability accuracy, if you care about relative probabilities. By that I mean if you care that the estimate $\hat{p}$ is within ...
• 24k
1 vote
Accepted

### Ideal score of a model on training and cross validation data

You should only look for the cross-validation score. If this set is large enough, it will give you an accurate prediction of how your model will act for unseen data. Your case is exceptional. The ...

Only top scored, non community-wiki answers of a minimum length are eligible