Skip to main content
OverflowAI is here! AI power for your Stack Overflow for Teams knowledge community. Learn more

Questions tagged [cross-validation]

For questions related to the cross-validation techniques (e.g. k-fold cross-validation or leave-one-out cross-validation) used in machine learning to assess the quality (e.g. average accuracy) of the models.

Filter by
Sorted by
Tagged with
0 votes
0 answers
12 views

3-way hold-out for picking between different ML models

I have a dataset which is a time series. Before jumping on more heavy models such as LSTM, I wanted to test out the performance of linear models. I have currently a 80/20 split between training and ...
xingern's user avatar
1 vote
1 answer
32 views

Is there validation data in K-fold cross-validation?

We know that in machine learning the dataset is divided into 3 parts: training data, validation data and test data. On the other hand, K-fold cross-validation is defined as follows: the dataset is ...
DSPinfinity's user avatar
0 votes
0 answers
7 views

If not chosen all the data in the train partition, is it still k-fold cross validation?

I have a dataset of 900 images, distributed across 6 classes, with 150 images per class. To develop a classifier and assess its performance, I will utilize k-fold cross-validation. In this case, I ...
noone's user avatar
  • 123
0 votes
1 answer
110 views

cross_val_score of sklearn and LinearRegression scoring method

cross_val_score (https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.cross_val_score.html) uses the estimator’s default scorer (if available) and LinearRgression (the estimator I ...
FluidMechanics Potential Flows's user avatar
0 votes
1 answer
19 views

Training model after Cross Validation: train one of the *k* split models plus the remaining data, or train from scratch?

So after a, say, 5-fold CV, you are left with 5 models, each trained on 80% of the data. You now want to have the best model possible, i.e. train it on all data. In order to save computation time, can ...
BiomedicalEngineeringStudent's user avatar
1 vote
1 answer
62 views

What is a result of a cross validation process?

I am trying to determine the result of a cross validation process. Is it just a set of standalone models which is produced after each cross-validation round, or is there some kind of final model which ...
Igor's user avatar
  • 181
0 votes
0 answers
35 views

How can validation accuracy be more than test accuracy?

I have been trying to implement DenseNet on small dataset using k-fold cross validation. Training accuracy is 94% ,validation accuracy is 73% whereas test accuracy is 90%.I have taken 10% of my total ...
srij's user avatar
  • 13
0 votes
0 answers
20 views

How to speed up the learning process

I have built a network that performans pretty well on my data. The issue I have is that for a larger number of epochs at the start of the training process the val/train acc/loss are stagnating (for ...
Skobo Do's user avatar
0 votes
0 answers
93 views

uniform gap between training and validation metrics

I am training a neural network (Deep and cross network) for a multi-label classification task (~700 labels). I have around 2.5 million samples, splitted 8/1/1 for train/test/validation. I am seeing a ...
Ryan's user avatar
  • 121
0 votes
1 answer
78 views

Cross Validation and hyperparameter selection correct procedure

I am trying to run a regression supervised learning problem. The dataset is not very large and I wanted to do some cross-validation to avoid overfitting. As I have read it's important to do a ...
metc's user avatar
  • 1
1 vote
1 answer
569 views

Weights initialization once the Neural Network is trained

I am trying to understand how weights are initialized in a Neural Network using Keras deep learning framework and what happens if I train a Neural Network and then I want to train it again: are the ...
HelpNeederStudent's user avatar
0 votes
1 answer
68 views

Given a dataset of people with and without cancer, should I split it into training and test datasets such that the same person is not in both?

I have a database that contains healthy persons and lung cancer patients. I need to design a deep neural network for the binary classification problem (cancer/no cancer). I need to split the dataset ...
Noha's user avatar
  • 111
0 votes
1 answer
230 views

How to arrange test dataset distribution for an imbalanced classification problem?

I have a dataset that contains 560 datapoints, and I would like to do binary classification on it. 400 datapoints belong to class 1, and 160 points belong to class 2. In the case of an imbalanced ...
Clara's user avatar
  • 11
5 votes
1 answer
829 views

How to decide a train-test split?

In almost every ML model, a train-test (or train-test-val split) is critical to assess the model's performance. However, I have always wondered what the rationale is to decide a particular train-test ...
user avatar
2 votes
1 answer
2k views

Should I continue training if the neural network attains 100% training accuracy?

I have a neural network where there are two hidden layers. Each hidden layer has 128 neurons. The input layer has 20 inputs, and the output layer has 3 outputs. I have 1 million records of data. 80% ...
user366312's user avatar
0 votes
1 answer
1k views

Is it valid to implement hyper-parameter tuning and THEN cross-validation?

I have a multi-label classification task I am solving. I have done hyperparameter tuning (with Keras Tuner) to determine the best configuration for my neural network. Is it valid to do this (determine ...
user9317212's user avatar
1 vote
1 answer
268 views

Why is the validation loss less than the training loss, and what can be said about the effect of the learning rate?

I have the following results I am trying to make sense of. I have attached the loss curves here for reference. As you can see, the first issue is that the validation loss is lower than the training ...
chinmay's user avatar
  • 113
3 votes
1 answer
133 views

Does adding a model complexity penalty to the loss function allow you to skip cross-validation?

It's my understanding that selecting for small models, i.e. having a multi-objective function where you're optimizing for both model accuracy and simplicity, automatically takes care of the danger of ...
Redrock's user avatar
  • 33
1 vote
1 answer
361 views

How to fill NaNs in Cross-Validation?

I have been searching this but did not find the answer, so sorry if this is a duplicated question. I was working with cross-validation, where some doubts came to my mind, and I am not sure which is ...
Soutuyo's user avatar
  • 11
0 votes
2 answers
613 views

How to avoid over-fitting using early stopping when using R cross validation package caret

I have a data set with 36 rows and 9 columns. I am trying to make a model to predict the 9th column I have tried modeling the data using a range of models using caret to perform cross-validation and ...
user1573820's user avatar
2 votes
1 answer
71 views

How exactly does nested cross-validation work?

I have trouble understanding how nested cross-validation works - I understand the need for two loops (one for selecting the model, and another for training the selected model), but why are they nested?...
stoic-santiago's user avatar
1 vote
1 answer
481 views

Is my 57% sports betting accuracy correct?

I have been creating sports betting algorithms for many years using Microsoft access and I am transitioning to the ML world and trying to get a grasp on determining the success of my algorithms. I ...
Sports_Stats's user avatar
3 votes
2 answers
205 views

Calculating accuracy for cross validation

I'm struggling with calculating accuracy when I do cross-validation for a deep learning model. I have two candidates for doing this. 1. Train a model with 10 different folds and get the best accuracy ...
Juna's user avatar
  • 31
0 votes
1 answer
89 views

What are non-held-out data or non-held-out classes?

I'm Spanish and I don't understand the meaning of "non-held-out". I have tried Google Translator and online dictionaries like Longman but I can't find a suitable translation for this term. You can ...
VansFannel's user avatar
2 votes
1 answer
108 views

After having selected the best model with cross-validation, for how long should I train it?

When using k-fold cross-validation in a deep learning problem, after you have computed your hyper-parameters, how do you decide how long to train your final model? My understanding is that, after the ...
FourierFlux's user avatar
1 vote
1 answer
66 views

What is the theoretical basis for the use of a validation set?

Let's say we use an MLE estimator (implementation doesn't matter) and we have a training set. We assume that we have sampled the training set from a Gaussian distribution $\mathcal N(\mu, \sigma^2)$. ...
user avatar
1 vote
1 answer
104 views

How to fairly conduct a model performance with 5-fold cross validation after augmentation?

I have, say, a (balanced) data-set with 2k images for binary classification. What I have done is that randomly divided the data-set into 5 folds; copy-pasted all 5-fold data-set to have 5 exact ...
bit_scientist's user avatar
1 vote
3 answers
743 views

While we split data in training and test data, why we have two pairs of each?

Why do we split the data into two parts, and then split those segments into training and testing data? Why do we have two sets of data for each training and test data?
ana's user avatar
  • 131
2 votes
2 answers
1k views

Should I choose the model with highest validation accuracy or the model with highest mean of training and validation accuracy?

I'm training a deep network in Keras on some images for a binary classification (I have around 12K images). Once in a while, I collect some false positives and add ...
Mary's user avatar
  • 973
1 vote
0 answers
68 views

How can I split the data into training and validation sets such that entries with a certain value are kept together?

I have the following kind of data frame. These are just example: A 1 Normal A 2 Normal A 3 Stress B 1 Normal B 2 Stress B 3 Stress C 1 Normal C 2 Normal C 3 Normal ...
user1631306's user avatar
3 votes
1 answer
620 views

What is the relationship between the training accuracy and validation accuracy?

During model training, I noticed various behaviour in between training and validation accuracy. I understand that 'The training set is used to train the model, while the validation set is only used to ...
nilsinelabore's user avatar
2 votes
3 answers
252 views

How do you interpret this learning curve?

Loss is MSE; orange is validation loss, blue training loss. The task is NN regression (18 inputs, 2 outputs), one layer 300 hidden units. Tuning the lr, mom, l2 regularization parameters this is the ...
Andrea Favilli's user avatar
6 votes
4 answers
1k views

Is k-fold cross-validation more effective than splitting the dataset into training and test datasets to prevent overfitting?

I want to prevent my model from overfitting. I think that k-fold cross-validation (because it is doing this each time with different datasets) may be more effective than splitting the dataset into ...
jennifer ruurs's user avatar
2 votes
1 answer
29 views

Can we say: the more we increase the numbers of cross validation the less likely it is that we overfit?

Based on the answer of my previous question: How can I avoid overfitting when doing parameter tuning? Can we say: the more we increase the numbers K of cross validation the less likely it is that we ...
jennifer ruurs's user avatar
0 votes
1 answer
97 views

How to interpret this learning curve plot

Bellow I have a Learning Curve plot How should I interpret this plot for my random forrest algorithm (the second one the most complex one)? Which one is the best?
jennifer ruurs's user avatar
1 vote
1 answer
101 views

How should I interpret this validation plot?

Bellow I have a validation plot How should I interpret this validation plot? Is my data underfitting? What else can be seen from this? Which one is the best? What does it mean that the right line is ...
jennifer ruurs's user avatar
16 votes
1 answer
376 views

Will parameter sweeping on one split of data followed by cross validation discover the right hyperparameters?

Let's call our dataset splits train/test/evaluate. We're in a situation where we require months of data. So we prefer to use the evaluation dataset as infrequently as possible to avoid polluting our ...
Philipp Cannons's user avatar
7 votes
1 answer
439 views

What is the best measure for detecting overfitting?

I wanted to ask about the methodology of testing the ML models against overfitting. Please note that I don't mean any overfitting reducing methods like regularisation, just a measure to judge whether ...
GKozinski's user avatar
  • 1,260
2 votes
1 answer
62 views

Metrics for evaluating models that output probabilities

I'm aware of metrics like accuracy (correct predictions / total predictions) for models that classify things. However, I'm working on a model that outputs the probability of a datapoint belonging to ...
Shylock's user avatar
  • 21
2 votes
2 answers
66 views

Ideal score of a model on training and cross validation data

The question is little bit broad, but I could not find any concrete explanation anywhere, hence decided to ask the experts here. I have trained a classifier model for binary classification task. Now ...
Supratim Haldar's user avatar
3 votes
1 answer
1k views

Should I use leave-one-out cross-validation for testing?

I am currently working with a small dataset of 20x300. Since I have so few data points, I was wondering if I could use an approach similar to leave-one-out cross-validation but for testing. Here's ...
Diogo Bastos's user avatar
1 vote
2 answers
266 views

Which model is better given their training and validation errors?

Below you have the plots of the training and validation errors for two different models. Both plots show the RMSE values for the validation dataset versus the number of training epochs. It is observed ...
NaveganTeX's user avatar
0 votes
1 answer
1k views

Should I call the error "validation error" or "test error" during cross validation?

I'm using 10-fold cross validation on all models. Here you can see both plots: Since I am using k-fold cross validation, is it okay to name it "validation error vs training error" or "test error vs ...
NaveganTeX's user avatar
2 votes
1 answer
469 views

What is the difference between validation percentage and batch size?

I'm doing transfer learning using Inception on Tensorflow. The code that I used for training is https://raw.githubusercontent.com/tensorflow/hub/master/examples/image_retraining/retrain.py If you take ...
gameon67's user avatar
  • 215
4 votes
1 answer
222 views

What are "development test sets" used for?

This is a theoretical question. I am a newbie to artificial intelligence and machine learning, and the more I read the more I like this. So far, I have been reading about the evaluation of language ...
little_mice's user avatar