# Tag Info

## Hot answers tagged cross-validation

6

First of all, as mentioned by @Neil Slater in the comment - you need to have three splits into the train, validation and test set. One sometimes disregards the difference between validation and test set. However they serve for different purposes. Here I would like to cite https://machinelearningmastery.com/difference-test-validation-datasets/ : Validation ...

6

In machine learning, you normally split your data into 3 parts (80-10-10%). The first part (80% of your initial data) is for the training of your ML model: this is known as the training dataset. The second part (10%) is the development set (or dataset), aka validation set. This is used as measuring your performance with various hyperparameters (e.g. in ...

5

tl;dr The safest method I've found is to use cross-validation for hyperparameter selection and a hold-out test set for a final evaluation. Why this isn't working for you... In your case, I suspect you're either running a large number of iterations during for hyperparameter selection or you have a fairly small dataset (or even a combination of both). If you ...

4

K-fold cross-validation is probably preferred in terms of completeness and generalization: you ensure that the system has seen the complete dataset for training. However, in deep learning this is often not feasible due to time and power constraints. They can both be used, and there is not one better than the other. It really depends on the specific case, the ...

3

This is very difficult to tell with the information provided, but the phenomenon is something that I have encountered many times before. Sometimes this is not a bad thing, here are some possible considerations/explanations: Data from the training set could be identical or leaking in to the validation set. Using a high dropout rate can cause this as well as ...

3

Neither of the above mentioned methods could be a potent indicator of the performance of a model. A simple way to train the model just enough so that it generalizes well on unknown datasets would be to monitor the validation loss. Training should be stopped once the validation loss progressively starts increasing over multiple epochs. Beyond this point, ...

3

I don't think there is any rationale behind choosing 80/20 over 75/25 or others. But those are the numbers for rather small datasets. If your dataset is large enough (like hundreds of thousands of samples), you can even work with 98/1/1 percents for train/val/test as discussed by Andrew Ng in this video. Neural networks thrive with big data and it is always ...

3

Purely in terms of overfitting, and assuming you train both for equal amounts of time, 70/30 is probably better but performance is not going to be very good. Not training on %30 of data will make both training and test results equally bad (in my opinion). But it won't overfit, that is for sure. Cross validation (you have in mind 90/10, I assume) will take a ...

3

The validation loss settles exactly at an error of one. Probably means there's something off with either the kind of data validation set has or with something in the training. An exact validation loss of one almost definitely means there's something off. I'd recommend before doing anything thoroughly go through your data or see if there's anything to debug ...

3

Depends on what does 1 represent in your task. If you are trying to predict household prices and 1 represents \$1, I think the average validation loss is good. If 1 represents \$10000 in this case, probably something is not right. But remember that there are 2 parts contributing to the overall loss. The mse loss and the l2 penalty loss. (Also remember that ...

2

The percentages refer to the number of samples to use (out of full dataset) as the validation and test datasets. So if you pass a dataset that consists of 100,000 samples to the model and set the validation and testing percentages to be 10% each, your model will train on 80,000 samples, validate them on 10,000 and save additional 10,000 samples for the final ...

2

Short answer: training "duration" or number of epochs/updates should be cross-validated too: you want to early-stop your training to prevent overfitting. Longer answer: Think of accuracy on the validation set as an estimate of accuracy on future data, given the value of some hyperparameter. In this case, the hyperparameter of interest is the number of ...

2

I think Cross-Validation serves a completely different purpose. From your post, it looks like you think we would use CV to get a better estimate of the parameters of our model (i.e. the model parameters after cross validation are closer to the parameters of the test data). In fact, we use CV to get an estimate of generalization error while keeping our test ...

2

If you used your five $X_{test}$ sets multiple times (to measure the average AUC) to decide on the best set of hyperparameters (i.e. optimizer, learning rate, batch size, dropout, activation) then yes, you successfully conducted hyper-parameter optimization. However, the AUC you received for the best set of hyperparameters found (by manual tuning) is not ...

2

This is a sign of overfitting. As you make your trees deeper, it becomes possible to "memorize" the data: each leaf of the tree is just a single point. The trees begin to learn patterns that do not exist. When you try out these patterns on new data (which is what cross-validation is imitating), then the patterns do not work, and your model fails to ...

2

I guess you could train your model with 10 different folds and in each fold calculate the average accuracy. So you would have 10 values - one corresponding to each fold. And then take the mean of all of them to get the average accuracy of your model. Your first option doesn't seem great because you take the highest accuracy among folds. If for some reason, ...

2

Regarding your first code snippet, there is no weight storing or continuation of training between the different CV folds whatsoever: each model is trained anew with the respective training data of each fold and validated on the validation data. Notice that this is exactly the idea behind cross validation - models trained on different folds are completely ...

1

It's my understanding that selecting for small models, i.e. having a multi-objective function where you're optimizing for both model accuracy and simplicity, automatically takes care of the danger of overfitting the data. Sort of. A secondary objective function often works as a form of regularisation, and can work to reduce overfit. However, this ...

1

I would do the exact same thing as you are describing! One of the main reasons that you would want to do cross-validation is to prevent that your model is unable to generalize later. Therefore, you take out a random small subset which will be your new small validation set and do all the 'operations' on that which you are also doing on your training set. This ...

1

"Selecting the model" in this case refers to selecting the hyperparameters of the model. The reason to use a nested CV is simply to avoid overfitting training data. Consider the example in the link. First you like to select the best hyperparameters of your svm model by GridSearchCV(). This is done by 4-fold CV. Now the clf.best_score_ will be the ...

1

My question is, can I rely on my Accuracy (mean & standard deviation) for future games even though my Testing Accuracy is lower than 52.5%? If by Accuracy you mean training accuracy, then absolutely you should not trust those values. For almost all machine learning algorithms there is a problem with overfitting to training data, which will result in ...

1

Held-out simply means "not included" particularly in the sense of: This part of the data was not included in this specific training run. Depending on the context of all of these text non-held-out data/classes means the data that actually was included in a particular modeling exercise. Consider this excerpt from your first example: For instance, Owen ...

1

The telltale signature of overfitting is when your validation loss starts increasing, while your training loss continues decreasing, i.e.: (Image adapted from Wikipedia entry on overfitting) It is clear that this does not happen in your diagram, hence your model does not overfit. A difference between a training and a validation score by itself does not ...

1

Are you talking about (X_train,y_train) and (X_test,y_test). If yes, then X represents the data(features) and y represents the labels of that data. That's why you get a pair when you divide it into training and test data

1

For any Machine Learning model, the available data is usually split into three sets: Training Set: The part of data used to train the model and learn the parameters of the network. The data that remains after allocation of the Training Dataset, is split into the Validation and Test sets. Validation Set: This sample of data is used to provide an unbiased ...

1

In general, no. There is a tradeoff between making the validation set for each fold smaller, and having more folds in total. As an example, if you have $N$ folds for $N$ datapoints, each fold will have only a single datapoint in its validation set. The validation accuracy of a model on a single datapoint is not a reliable estimator for the test performance ...

1

Note the X index is training set size. For the first and second case, teh training set size starts at 0(or 1). The model will overfit certainly at that data size. When data size increases, the model overfits less and less and eventually the model have enough data samples that it won't overfit. The data size continue to increase and the model performance ...

1

For a binary classifier, the cross-entropy loss is a natural measure of probability accuracy, if you care about relative probabilities. By that I mean if you care that the estimate $\hat{p}$ is within some ratio of the true value. So an estimate of $\hat{p} = 0.1$ is a better estimate if the true value is $p = 0.2$ than if the true value is $p = 0.01$ (even ...

1

You should only look for the cross-validation score. If this set is large enough, it will give you an accurate prediction of how your model will act for unseen data. Your case is exceptional. The fitted model which is obviously overfitted actually performs better on the cross-validation set. This means in turn that your overfitted model will perform better ...

1

Concerning $k$-fold Cross Validation, I like to think of it by considering two extremes you can do: Leave-One-Out Cross-Validation where you leave one sample each time and train your model on the remaining $n-1$, and 2-fold Cross Validation at which you split your dataset in half and train (and validate) two models on two different halves. The important ...

Only top voted, non community-wiki answers of a minimum length are eligible