Questions tagged [validation]
For questions related to the validation of a machine learning model, which is different from testing a model, which is done after training. Validation is usually performed for early stopping (i.e. assess when the model is over-fitting) or hyper-parameter optimization during training. However, often people use (either correctly/intensionally or not) the terms "validation" and "testing" interchangeably, so context needs to be taken into account.
8
questions
0
votes
0
answers
7
views
If not chosen all the data in the train partition, is it still k-fold cross validation?
I have a dataset of 900 images, distributed across 6 classes, with 150 images per class. To develop a classifier and assess its performance, I will utilize k-fold cross-validation. In this case, I ...
- 123
1
vote
0
answers
21
views
Why is my validation accuracy fluctuating between two inverse values?
I am currently going through the FastAI course and to practise, I wanted to code a neural network that classifies the FashionMNIST dataset from scratch.
Lately, I've been running into an issue where I ...
- 13
0
votes
0
answers
18
views
Very high ACC (ca. 95%) with 1DConvNet for Time Series
Does this sound legit, for people working with CNN and Time Series?
I have a Framework that applies Dynamic Tim Warping (DTW) on time series, using the DTW distance matrix, I cluster my data and ...
- 51
-1
votes
1
answer
65
views
how to decide the optimum model?
I have split the database available into 70% training, 15% validation, and 15% test, using holdout validation. I have trained the model and got the following results: training accuracy 100%, ...
0
votes
0
answers
57
views
What are the standard ways to measure the quality of a set of numerical predictions that include uncertainties?
I have a radial basis function that supplies uncertainties (standard deviations) with its predictions, which are numerical values.
This function is computed for a particular point by computing its ...
- 101
0
votes
0
answers
151
views
Why would my neural network have either an accuracy of 90% or 10% on the validation data, given a random initialization?
I'm making a custom neural network framework (in C++, if that is of any help). When I train the model on MNIST, depending on how happy the network is feeling, it'll give me either 90%+ accuracy, or ...
- 115
10
votes
3
answers
4k
views
Should I choose a model with the smallest loss or highest accuracy?
I have two Machine Learning models (I use LSTM) that have a different result on the validation set (~100 samples data):
Model A: Accuracy: ~91%, Loss: ~0.01
Model B: Accuracy: ~83%, Loss: ~0.003
The ...
- 2,739
2
votes
0
answers
138
views
Which evaluation metrics should be used in training, validation and testing of a model?
Which specific performance evaluation metrics are used in training, validation, and testing, and why? I am thinking error metrics (RMSE, MAE, MSE) are used in validation, and testing should use a wide ...
- 53