Questions tagged [cross-validation]

For questions related to the cross-validation technique used e.g. in machine learning or statistics.

Filter by
Sorted by
Tagged with
13
votes
1answer
164 views

Will parameter sweeping on one split of data followed by cross validation discover the right hyperparameters?

Let's call our dataset splits train/test/evaluate. We're in a situation where we require months of data. So we prefer to use the evaluation dataset as infrequently as possible to avoid polluting our ...
5
votes
3answers
459 views

Is k-fold cross-validation more effective than splitting the dataset into training and test datasets to prevent overfitting?

I want to prevent my model from overfitting. I think that k-fold cross-validation (because it is doing this each time with different datasets) may be more effective than splitting the dataset into ...
3
votes
1answer
64 views

What are “development test sets” used for?

This is a theoretical question. I am a newbie to artificial intelligence and machine learning, and the more I read the more I like this. So far, I have been reading about the evaluation of language ...
3
votes
2answers
57 views

Calculating accuracy for cross validation

I'm struggling with calculating accuracy when I do cross-validation for a deep learning model. I have two candidates for doing this. 1. Train a model with 10 different folds and get the best accuracy ...
3
votes
1answer
22 views

Does adding a model complexity penalty to the loss function allow you to skip cross-validation?

It's my understanding that selecting for small models, i.e. having a multi-objective function where you're optimizing for both model accuracy and simplicity, automatically takes care of the danger of ...
2
votes
3answers
144 views

How do you interpret this learning curve?

Loss is MSE; orange is validation loss, blue training loss. The task is NN regression (18 inputs, 2 outputs), one layer 300 hidden units. Tuning the lr, mom, l2 regularization parameters this is the ...
2
votes
1answer
53 views

After having selected the best model with cross-validation, for how long should I train it?

When using k-fold cross-validation in a deep learning problem, after you have computed your hyper-parameters, how do you decide how long to train your final model? My understanding is that, after the ...
2
votes
1answer
42 views

How exactly does nested cross-validation work?

I have trouble understanding how nested cross-validation works - I understand the need for two loops (one for selecting the model, and another for training the selected model), but why are they nested?...
2
votes
1answer
52 views

Metrics for evaluating models that output probabilities

I'm aware of metrics like accuracy (correct predictions / total predictions) for models that classify things. However, I'm working on a model that outputs the probability of a datapoint belonging to ...
2
votes
1answer
225 views

Leave One Out Testing

I am currently working with a small dataset of 20x300. Since I have so few datapoints, I was wondering if I could use an approach similar to leave-one-out cross-validation but for testing. Here's ...
2
votes
1answer
77 views

Relationship between training accuracy and validation accuracy

During model training, I noticed various behaviour in between training and validation accuracy. I understand that 'The training set is used to train the model, while the validation set is only used to ...
1
vote
1answer
78 views

How to fairly conduct a model performance with 5-fold cross validation after augmentation?

I have, say, a (balanced) data-set with 2k images for binary classification. What I have done is that randomly divided the data-set into 5 folds; copy-pasted all 5-fold data-set to have 5 exact ...
1
vote
3answers
60 views

While we split data in training and test data, why we have two pairs of each?

Why do we split the data into two parts, and then split those segments into training and testing data? Why do we have two sets of data for each training and test data?
1
vote
1answer
64 views

How should I interpret this validation plot?

Bellow I have a validation plot How should I interpret this validation plot? Is my data underfitting? What else can be seen from this? Which one is the best? What does it mean that the right line is ...
1
vote
1answer
126 views

Is my 57% sports betting accuracy correct?

I have been creating sports betting algorithms for many years using Microsoft access and I am transitioning to the ML world and trying to get a grasp on determining the success of my algorithms. I ...
1
vote
1answer
47 views

What is the theoretical basis for the use of Cross Validation set?

So let's follow this line of reasoning. We use a MLE estimator (implementation doesn't matter) and we have a train set. We assume that we have sampled training set from a Gaussian distribution $\...
1
vote
1answer
18 views

Can we say: the more we increase the numbers of cross validation the less likely it is that we overfit?

Based on the answer of my previous question: How can I avoid overfitting when doing parameter tuning? Can we say: the more we increase the numbers K of cross validation the less likely it is that we ...
1
vote
2answers
46 views

Ideal score of a model on training and cross validation data

The question is little bit broad, but I could not find any concrete explanation anywhere, hence decided to ask the experts here. I have trained a classifier model for binary classification task. Now ...
1
vote
2answers
158 views

Which model is better given their training and validation errors?

Below you have the plots of the training and validation errors for two different models. Both plots show the RMSE values for the validation dataset versus the number of training epochs. It is observed ...
1
vote
0answers
53 views

How can I split the data into training and validation sets such that entries with a certain value are kept together?

I have the following kind of data frame. These are just example: A 1 Normal A 2 Normal A 3 Stress B 1 Normal B 2 Stress B 3 Stress C 1 Normal C 2 Normal C 3 Normal ...
0
votes
1answer
43 views

What are non-held-out data or non-held-out classes?

I'm Spanish and I don't understand the meaning of "non-held-out". I have tried Google Translator and online dictionaries like Longman but I can't find a suitable translation for this term. You can ...
0
votes
1answer
65 views

How to interpret this learning curve plot

Bellow I have a Learning Curve plot How should I interpret this plot for my random forrest algorithm (the second one the most complex one)? Which one is the best?
0
votes
1answer
43 views

Should I call the error “validation error” or “test error” during cross validation?

I'm using 10-fold cross validation on all models. Here you can see both plots: Since I am using k-fold cross validation, is it okay to name it "validation error vs training error" or "test error vs ...
0
votes
1answer
108 views

How to fill NaNs in Cross-Validation?

I have been searching this but did not find the answer, so sorry if this is a duplicated question. I was working with cross-validation, where some doubts came to my mind, and I am not sure about which ...
0
votes
0answers
15 views

Why does 0.8:0.2 divided dataset have a much greater AUROC than its 5-fold cross validated counterpart's mean AUROC?

I trained a dataset with a 5-fold cross validation to search for hyper-parameters by an AUROC metric. For splitting I used ...
0
votes
0answers
18 views

Is it a fair evaluation if each model of k-fold cross validation is trained with different epochs and the mean AUC score gathered out of k folds?

Let's say I have a dataset for binary classification. And I am going to conduct 5-fold cross-validation and get AUC scores for each fold (mean AUC score too). However, if I set the training epoch to $...
0
votes
2answers
126 views

How to avoid over-fitting using early stopping when using R cross validation package caret

I have a data set with 36 rows and 9 columns. I am trying to make a model to predict the 9th column I have tried modeling the data using a range of models using caret to perform cross-validation and ...
0
votes
0answers
29 views

What would happen if validation set was the same as the training set?

Just to check if everything is working properly in my neural net, I set my validation data to be the same as my training data, expecting to achieve a better NRMS for validation data (since it uses ...
0
votes
0answers
23 views

Different result from k-cross validation model and Train-Validation-Test split model ? (AI fresher question)

I am starting to learn about Neural Network and I have come into one problem that I am still learning how to figure it out. I have a dataset with shape (105,96) (105 samples and 95 first column as ...