Questions tagged [testing]

For questions related to the concept of testing (or evaluating) machine learning models and algorithms, e.g. in terms of some performance measure (such as accuracy or cumulative reward).

Filter by
Sorted by
Tagged with
7
votes
4answers
2k views

Why is my test error lower than the training error?

I am trying to train a CNN regression model using the ADAM optimizer, dropout and weight decay. My test accuracy is better than training accuracy. But, as far as I know, usually, the training accuracy ...
6
votes
1answer
119 views

How to decide a train-test split?

In almost every ML model, a train-test (or train-test-val split) is critical to assess the model's performance. However, I have always wondered what the rationale is to decide a particular train-test ...
4
votes
4answers
1k views

What is the difference between training and testing in reinforcement learning?

In reinforcement learning (RL), what is the difference between training and testing an algorithm/agent? If I understood correctly, testing is also referred to as evaluation. As I see it, both imply ...
4
votes
1answer
217 views

How to evaluate an RL algorithm when used in a game?

I'm planning to create a web-based RL board game, and I wondered how I would evaluate the performance of the RL agent. How would I be able to say, "Version X performed better than version Y, as ...
4
votes
1answer
58 views

What is the most statistically acceptable method for tuning neural network hyperparameters on very small datasets?

Neural networks are usually evaluated by dividing a dataset into three splits: training, validation, and test The idea is that critical hyperparameters of the network such as the number of epochs ...
3
votes
1answer
49 views

Is the test time the phase when the model's accuracy is calculated with test data set?

When papers talk about the "test time", does this mean the phase when the model is passed with new data instances to derive the accuracy of the test data set? Or is "test time" the phase when the ...
3
votes
2answers
98 views

How can I predict the true label for data with incomplete features based on the trained model with data with more features?

Suppose I have a model that was trained with a dataset that contains the features (f1, f2, f3, f4, f5, f6). However, my test dataset does not contain all features ...
2
votes
1answer
110 views

How do I check that the combination of these models is good?

I've selected more than 10 discriminative (classification) models, each wrapped with a BaggingClassifier object, optimized with a ...
2
votes
1answer
305 views

Should I use leave-one-out cross-validation for testing?

I am currently working with a small dataset of 20x300. Since I have so few data points, I was wondering if I could use an approach similar to leave-one-out cross-validation but for testing. Here's ...
2
votes
0answers
21 views

Are there standardized forms of the Turing Test?

Most computer science instructors will tell you that the Turing Test is more a theoretical or conceptual thought experiment than an actual exam that someone (or something!) can formally sit and ...
2
votes
1answer
118 views

PyTorch `torch.no_grad` vs `torch.inference_mode` [closed]

PyTorch has new functionality torch.inference_mode as of v1.9 which is "analogous to torch.no_grad... Code run under this ...
2
votes
0answers
118 views

Which evaluation metrics should be used in training, validation and testing of a model?

Which specific performance evaluation metrics are used in training, validation, and testing, and why? I am thinking error metrics (RMSE, MAE, MSE) are used in validation, and testing should use a wide ...
1
vote
1answer
890 views

Should we also shuffle the test dataset when training with SGD?

When training machine learning models (e.g. neural networks) with stochastic gradient descent, it is common practice to (uniformly) shuffle the training data into batches/sets of different samples ...
1
vote
1answer
42 views

How to build a test set for a model in industry?

Most of the tutorials only teach us to split the whole dataset into three parts: training set, develop set, and test set. But in the industry, we are kind of doing test-driven development, and what ...
1
vote
0answers
24 views

Is there a way, while training (with contrastive learning) the embedding network, to find the test accuracy?

I aim to do action recognition in videos on a private dataset. To compare with the existing state-of-the-art implementations, other guys published their code on Github, like the one here (for the ...
1
vote
0answers
59 views

What is the amount of test data needed to evaluate a CNN?

I have an image dataset of about 400 images. 70% of these data points were used for training, 15% for validation, and 15% for testing. I am using the 70% to train a CNN-based binary classifier. I ...
0
votes
2answers
76 views

How can I be sure that the final model, trained on all data, is correct?

The 'by the book' method of delivering final machine learning models is to include all data in the final training (including validation and test sets). To check robustness of my model I use randomly ...
0
votes
1answer
35 views

How to test the robustness of an agent in a custom reinforcement learning environment?

I have used the stable-baseline3 implementation of the SAC algorithm to train policies in a custom gym environment. So far the results look promising. However, I would like to test the robustness of ...
0
votes
1answer
43 views

What are the differences in testing between traditional software and artificial intelligence?

The testing problem in traditional software has been fully explored over the last decades, but it seems that testing in artificial intelligence/machine learning has not (see this question and this one)...
0
votes
1answer
12 views

What does a value of -1.000 mean in MS COCO Metrics for Object Detection

I am training some Object-Detection-Models from the TensorFlow Object Detection API and got from the evaluation with MS COCO metrics the following results for Average Precision: IoU = 0.5;0.9 maxDets =...
0
votes
0answers
17 views

How to do reference-free evaluation for speech?

The outliers in speech include three types: syntax errors, semantic errors, and errors in pragmatics. I wonder if we can borrow the techniques from computer vision (CV) to tackle that, such as ...
0
votes
0answers
66 views

Strange behavior of Q-learning agent after being trained

I built a simple X*Y grid world environment to learn and then trained my agent over it. All worked fine and the agent learned as well. Let me give some detail about the environment. Environment: A ...