Questions tagged [testing]

For questions related to the concept of testing (or evaluating) machine learning models and algorithms, e.g. in terms of some performance measure (such as accuracy or cumulative reward).

Filter by
Sorted by
Tagged with
-2 votes
0 answers
19 views

Evaluation measure [closed]

I have the following question on my mock exam: You want to estimate the probability that your classifier gives an arbitrarily selected positive example a higher (positiveness) score than an ...
user avatar
1 vote
0 answers
31 views

What considerations should I take to train my transformer model?

I want to train my vision transformer model on a benchmark for an image segmentation task: (LoveDA: A Remote Sensing Land-Cover Dataset for Domain Adaptive Semantic Segmentation) (GitHub), but I don't ...
user avatar
1 vote
1 answer
31 views

Is it mandatory to multiply every activation of a layer by droupout factor during testing?

Dropout is a regularization technique used in neural networks. It is useful in preventing overfitting by making a neural network as good as an ensemble system. In dropout, we switch off $p$ percent of ...
user avatar
  • 3,109
0 votes
1 answer
43 views

Does $S_{t+1}$ denote the future information in Q-learning?

In Q-learning, $Q(S_t,a)$ is updated by the Bellman equation. $Q(S_t,a) = r + \max_{a'}(Q(S_{t+1},a'))$ where $S_{t+1}$ is the future state. Let's say $S$ denotes the stock price, does it mean we are ...
user avatar
  • 1
2 votes
1 answer
59 views

Why doesn't dropout mislead results during evaluation?

I have seen that, usually, the dropout layer is used differently in training and evaluation modes, i.e. it is recommended to use during training but not in evaluation/testing. Dropout does remove a ...
user avatar
  • 23
1 vote
1 answer
432 views

What is the time complexity for testing a stacked LSTM model?

In the data preparation phase, we have to divide the dataset into two parts: the training dataset and the test dataset. I have seen this post regarding the time complexity for training a model. ...
user avatar
1 vote
1 answer
37 views

How to do testing for an RNN that was trained with teacher forcing only?

If an RNN is trained using only the teacher forcing, then the network takes the actual output from the previous time step as input to the hidden state the next time step. We know that the actual ...
user avatar
  • 3,109
2 votes
0 answers
25 views

Are there standardized forms of the Turing Test?

Most computer science instructors will tell you that the Turing Test is more a theoretical or conceptual thought experiment than an actual exam that someone (or something!) can formally sit and ...
user avatar
5 votes
1 answer
347 views

How to decide a train-test split?

In almost every ML model, a train-test (or train-test-val split) is critical to assess the model's performance. However, I have always wondered what the rationale is to decide a particular train-test ...
user avatar
0 votes
1 answer
42 views

What does a value of -1.000 mean in MS COCO Metrics for Object Detection

I am training some Object-Detection-Models from the TensorFlow Object Detection API and got from the evaluation with MS COCO metrics the following results for Average Precision: IoU = 0.5;0.9 maxDets =...
user avatar
0 votes
1 answer
66 views

How to test the robustness of an agent in a custom reinforcement learning environment?

I have used the stable-baseline3 implementation of the SAC algorithm to train policies in a custom gym environment. So far the results look promising. However, I would like to test the robustness of ...
user avatar
  • 35
3 votes
1 answer
66 views

What is the most statistically acceptable method for tuning neural network hyperparameters on very small datasets?

Neural networks are usually evaluated by dividing a dataset into three splits: training, validation, and test The idea is that critical hyperparameters of the network such as the number of epochs ...
user avatar
  • 391
1 vote
1 answer
86 views

What are the differences in testing between traditional software and artificial intelligence?

The testing problem in traditional software has been fully explored over the last decades, but it seems that testing in artificial intelligence/machine learning has not (see this question and this one)...
user avatar
1 vote
0 answers
27 views

Is there a way, while training (with contrastive learning) the embedding network, to find the test accuracy?

I aim to do action recognition in videos on a private dataset. To compare with the existing state-of-the-art implementations, other guys published their code on Github, like the one here (for the ...
user avatar
1 vote
1 answer
49 views

How to build a test set for a model in industry?

Most of the tutorials only teach us to split the whole dataset into three parts: training set, develop set, and test set. But in the industry, we are kind of doing test-driven development, and what ...
user avatar
2 votes
1 answer
2k views

Should we also shuffle the test dataset when training with SGD?

When training machine learning models (e.g. neural networks) with stochastic gradient descent, it is common practice to (uniformly) shuffle the training data into batches/sets of different samples ...
user avatar
0 votes
2 answers
133 views

How can I be sure that the final model, trained on all data, is correct?

The 'by the book' method of delivering final machine learning models is to include all data in the final training (including validation and test sets). To check robustness of my model I use randomly ...
user avatar
  • 1,039
0 votes
0 answers
72 views

Strange behavior of Q-learning agent after being trained

I built a simple X*Y grid world environment to learn and then trained my agent over it. All worked fine and the agent learned as well. Let me give some detail about the environment. Environment: A ...
user avatar
  • 331
3 votes
2 answers
120 views

How can I predict the true label for data with incomplete features based on the trained model with data with more features?

Suppose I have a model that was trained with a dataset that contains the features (f1, f2, f3, f4, f5, f6). However, my test dataset does not contain all features ...
user avatar
1 vote
0 answers
80 views

What is the amount of test data needed to evaluate a CNN?

I have an image dataset of about 400 images. 70% of these data points were used for training, 15% for validation, and 15% for testing. I am using the 70% to train a CNN-based binary classifier. I ...
user avatar
3 votes
1 answer
60 views

Is the test time the phase when the model's accuracy is calculated with test data set?

When papers talk about the "test time", does this mean the phase when the model is passed with new data instances to derive the accuracy of the test data set? Or is "test time" the phase when the ...
user avatar
4 votes
4 answers
2k views

What is the difference between training and testing in reinforcement learning?

In reinforcement learning (RL), what is the difference between training and testing an algorithm/agent? If I understood correctly, testing is also referred to as evaluation. As I see it, both imply ...
user avatar
1 vote
0 answers
39 views

How can we make sure how well the reinforcement learning agent works on a stock dataset?

I read a paper, which is about Deep Reinforcement Learning and it tries to use this method on stock data set. It has been shown that it reaches the maximum return (profit). It has been implemented in ...
user avatar
4 votes
1 answer
325 views

How to evaluate an RL algorithm when used in a game?

I'm planning to create a web-based RL board game, and I wondered how I would evaluate the performance of the RL agent. How would I be able to say, "Version X performed better than version Y, as ...
user avatar
  • 573
3 votes
1 answer
536 views

Should I use leave-one-out cross-validation for testing?

I am currently working with a small dataset of 20x300. Since I have so few data points, I was wondering if I could use an approach similar to leave-one-out cross-validation but for testing. Here's ...
user avatar
1 vote
1 answer
146 views

What is the reason behind using a test batch size?

If one examines the SSD: Single Shot MultiBox Detector code from this GitHub repository, it can be seen that, for a testing phase (evaluating network on test data set), there is a parameter ...
user avatar
1 vote
1 answer
114 views

How do I check that the combination of these models is good?

I've selected more than 10 discriminative (classification) models, each wrapped with a BaggingClassifier object, optimized with a ...
user avatar
  • 177
2 votes
0 answers
125 views

Which evaluation metrics should be used in training, validation and testing of a model?

Which specific performance evaluation metrics are used in training, validation, and testing, and why? I am thinking error metrics (RMSE, MAE, MSE) are used in validation, and testing should use a wide ...
user avatar
2 votes
1 answer
170 views

Is it a good idea to train a CNN to detect the hydration value (percentage) in skin images and evaluate it with the MSE?

I have a large dataset of skin images, each one associated with a hydration value (percentage). Now I'm looking into predicting the hydration value from an image. My thinking: train a CNN on the ...
user avatar
7 votes
4 answers
2k views

Why is my test error lower than the training error?

I am trying to train a CNN regression model using the ADAM optimizer, dropout and weight decay. My test accuracy is better than training accuracy. But, as far as I know, usually, the training accuracy ...
user avatar