5

In machine learning, you normally split your data into 3 parts (80-10-10%). The first part (80% of your initial data) is for the training of your ML model: this is known as the training dataset. The second part (10%) is the development set (or dataset), aka validation set. This is used as measuring your performance with various hyperparameters (e.g. in ...


5

tl;dr The safest method I've found is to use cross-validation for hyperparameter selection and a hold-out test set for a final evaluation. Why this isn't working for you... In your case, I suspect you're either running a large number of iterations during for hyperparameter selection or you have a fairly small dataset (or even a combination of both). If you ...


4

K-fold cross-validation is probably preferred in terms of completeness and generalization: you ensure that the system has seen the complete dataset for training. However, in deep learning this is often not feasible due to time and power constraints. They can both be used, and there is not one better than the other. It really depends on the specific case, the ...


3

This is very difficult to tell with the information provided, but the phenomenon is something that I have encountered many times before. Sometimes this is not a bad thing, here are some possible considerations/explanations: Data from the training set could be identical or leaking in to the validation set. Using a high dropout rate can cause this as well as ...


3

Purely in terms of overfitting, and assuming you train both for equal amounts of time, 70/30 is probably better but performance is not going to be very good. Not training on %30 of data will make both training and test results equally bad (in my opinion). But it won't overfit, that is for sure. Cross validation (you have in mind 90/10, I assume) will take a ...


3

The validation loss settles exactly at an error of one. Probably means there's something off with either the kind of data validation set has or with something in the training. An exact validation loss of one almost definitely means there's something off. I'd recommend before doing anything thoroughly go through your data or see if there's anything to debug ...


3

Depends on what does 1 represent in your task. If you are trying to predict household prices and 1 represents \$1, I think the average validation loss is good. If 1 represents \$10000 in this case, probably something is not right. But remember that there are 2 parts contributing to the overall loss. The mse loss and the l2 penalty loss. (Also remember that ...


2

This is a sign of overfitting. As you make your trees deeper, it becomes possible to "memorize" the data: each leaf of the tree is just a single point. The trees begin to learn patterns that do not exist. When you try out these patterns on new data (which is what cross-validation is imitating), then the patterns do not work, and your model fails to ...


2

If you used your five $X_{test}$ sets multiple times (to measure the average AUC) to decide on the best set of hyperparameters (i.e. optimizer, learning rate, batch size, dropout, activation) then yes, you successfully conducted hyper-parameter optimization. However, the AUC you received for the best set of hyperparameters found (by manual tuning) is not ...


2

Short answer: training "duration" or number of epochs/updates should be cross-validated too: you want to early-stop your training to prevent overfitting. Longer answer: Think of accuracy on the validation set as an estimate of accuracy on future data, given the value of some hyperparameter. In this case, the hyperparameter of interest is the number of ...


2

I guess you could train your model with 10 different folds and in each fold calculate the average accuracy. So you would have 10 values - one corresponding to each fold. And then take the mean of all of them to get the average accuracy of your model. Your first option doesn't seem great because you take the highest accuracy among folds. If for some reason, ...


1

It's my understanding that selecting for small models, i.e. having a multi-objective function where you're optimizing for both model accuracy and simplicity, automatically takes care of the danger of overfitting the data. Sort of. A secondary objective function often works as a form of regularisation, and can work to reduce overfit. However, this ...


1

"Selecting the model" in this case refers to selecting the hyperparameters of the model. The reason to use a nested CV is simply to avoid overfitting training data. Consider the example in the link. First you like to select the best hyperparameters of your svm model by GridSearchCV(). This is done by 4-fold CV. Now the clf.best_score_ will be the ...


1

My question is, can I rely on my Accuracy (mean & standard deviation) for future games even though my Testing Accuracy is lower than 52.5%? If by Accuracy you mean training accuracy, then absolutely you should not trust those values. For almost all machine learning algorithms there is a problem with overfitting to training data, which will result in ...


1

Held-out simply means "not included" particularly in the sense of: This part of the data was not included in this specific training run. Depending on the context of all of these text non-held-out data/classes means the data that actually was included in a particular modeling exercise. Consider this excerpt from your first example: For instance, Owen ...


1

I think Cross-Validation serves a completely different purpose. From your post, it looks like you think we would use CV to get a better estimate of the parameters of our model (i.e. the model parameters after cross validation are closer to the parameters of the test data). In fact, we use CV to get an estimate of generalization error while keeping our test ...


1

Are you talking about (X_train,y_train) and (X_test,y_test). If yes, then X represents the data(features) and y represents the labels of that data. That's why you get a pair when you divide it into training and test data


1

For any Machine Learning model, the available data is usually split into three sets: Training Set: The part of data used to train the model and learn the parameters of the network. The data that remains after allocation of the Training Dataset, is split into the Validation and Test sets. Validation Set: This sample of data is used to provide an unbiased ...


1

The telltale signature of overfitting is when your validation loss starts increasing, while your training loss continues decreasing, i.e.: (Image adapted from Wikipedia entry on overfitting) It is clear that this does not happen in your diagram, hence your model does not overfit. A difference between a training and a validation score by itself does not ...


1

In general, no. There is a tradeoff between making the validation set for each fold smaller, and having more folds in total. As an example, if you have $N$ folds for $N$ datapoints, each fold will have only a single datapoint in its validation set. The validation accuracy of a model on a single datapoint is not a reliable estimator for the test performance ...


1

Note the X index is training set size. For the first and second case, teh training set size starts at 0(or 1). The model will overfit certainly at that data size. When data size increases, the model overfits less and less and eventually the model have enough data samples that it won't overfit. The data size continue to increase and the model performance ...


1

For a binary classifier, the cross-entropy loss is a natural measure of probability accuracy, if you care about relative probabilities. By that I mean if you care that the estimate $\hat{p}$ is within some ratio of the true value. So an estimate of $\hat{p} = 0.1$ is a better estimate if the true value is $p = 0.2$ than if the true value is $p = 0.01$ (even ...


1

You should only look for the cross-validation score. If this set is large enough, it will give you an accurate prediction of how your model will act for unseen data. Your case is exceptional. The fitted model which is obviously overfitted actually performs better on the cross-validation set. This means in turn that your overfitted model will perform better ...


1

Concerning $k$-fold Cross Validation, I like to think of it by considering two extremes you can do: Leave-One-Out Cross-Validation where you leave one sample each time and train your model on the remaining $n-1$, and 2-fold Cross Validation at which you split your dataset in half and train (and validate) two models on two different halves. The important ...


1

I would say that your intuition is correct: the model associated with the first plot is likely to generalise more than the one associated with the second plot. In both cases, it doesn't seem that your model has overfitted the training data. Overfitting often occurs when the training error keeps decreasing but the validation error starts to increase. In both ...


1

The expression "validation error vs training error" is likely more appropriate because the data you use during cross validation that is not the training data is often considered the validation data. The test data is the data you use to test your model after having performed e.g. cross-validation. The test dataset should be independent of both the training ...


Only top voted, non community-wiki answers of a minimum length are eligible