Questions tagged [testing]

For questions related to the concept of testing (or evaluating) machine learning models and algorithms, e.g. in terms of some performance measure (such as accuracy or cumulative reward).

Filter by
Sorted by
Tagged with
0 votes
0 answers
12 views

3-way hold-out for picking between different ML models

I have a dataset which is a time series. Before jumping on more heavy models such as LSTM, I wanted to test out the performance of linear models. I have currently a 80/20 split between training and ...
xingern's user avatar
0 votes
0 answers
28 views

I dont understand this way of having a stable train/test split even after updating the dataset

...
samsamradas's user avatar
1 vote
1 answer
32 views

How to choose the metric value when evaluating the performance of a deep learning model?

When evaluating the performance of a deep learning model (for the purpose of publishing a research work), should we choose the optimal value from the metric curve (such as the accuracy curve) or the ...
Liuji's user avatar
  • 19
1 vote
1 answer
192 views

What are alternatives to Inception Score? Can it be used for non-photographic image types?

Most online sources recommend using versions of the Inception score to evaluate the synthetic images generated by a GAN. These scores are pre-trained on the InceptionV3 model. Does this mean that ...
ThreeOrangeOneRed's user avatar
0 votes
0 answers
18 views

Regression Model overestimates in train-mode

I have a Deep Learning Regression model to predict some values. The results are fine when I use the model in Evaluation Mode, but when I turn Training Mode on the model tends to overestimate the ...
nmb's user avatar
  • 1
1 vote
1 answer
5k views

High precision and low recall results. What does it mean?

I am working on a classification algorithm for brain rhythms. However, when I implemented the metrics for precision, accuracy, F1 score and recall. My results show that my algorithm has a high ...
GGChe's user avatar
  • 123
0 votes
1 answer
69 views

Can AUCROC curves be used to evaluate forecasting results for time series data?

I have built a time series forecasting model based on RNN. Can I calculate AUCROC on the test set and the actual predicted values? Example: ...
KYH's user avatar
  • 17
0 votes
0 answers
1k views

What is the "attack success rate" of an Adversarial Attack?

For a typical adversarial attack, a sample $x_{0}$ is chosen from a training set belonging to class $C_{i}$ and a transformation $A$ is applied such that the adversarial example $x=A(x_{0})$ would be ...
VirginieDlpts's user avatar
1 vote
0 answers
66 views

What considerations should I take to train my transformer model?

I want to train my vision transformer model on a benchmark for an image segmentation task: (LoveDA: A Remote Sensing Land-Cover Dataset for Domain Adaptive Semantic Segmentation) (GitHub), but I don't ...
sara yaghoobi's user avatar
1 vote
1 answer
56 views

Is it mandatory to multiply every activation of a layer by droupout factor during testing?

Dropout is a regularization technique used in neural networks. It is useful in preventing overfitting by making a neural network as good as an ensemble system. In dropout, we switch off $p$ percent of ...
hanugm's user avatar
  • 3,830
0 votes
1 answer
73 views

Does $S_{t+1}$ denote the future information in Q-learning?

In Q-learning, $Q(S_t,a)$ is updated by the Bellman equation. $Q(S_t,a) = r + \max_{a'}(Q(S_{t+1},a'))$ where $S_{t+1}$ is the future state. Let's say $S$ denotes the stock price, does it mean we are ...
L.Chau's user avatar
  • 1
2 votes
1 answer
421 views

Why doesn't dropout mislead results during evaluation?

I have seen that, usually, the dropout layer is used differently in training and evaluation modes, i.e. it is recommended to use during training but not in evaluation/testing. Dropout does remove a ...
prat__'s user avatar
  • 33
1 vote
1 answer
3k views

What is the time complexity for testing a stacked LSTM model?

In the data preparation phase, we have to divide the dataset into two parts: the training dataset and the test dataset. I have seen this post regarding the time complexity for training a model. ...
Anik Islam Abhi's user avatar
1 vote
1 answer
194 views

How to do testing for an RNN that was trained with teacher forcing only?

If an RNN is trained using only the teacher forcing, then the network takes the actual output from the previous time step as input to the hidden state the next time step. We know that the actual ...
hanugm's user avatar
  • 3,830
3 votes
0 answers
49 views

Are there standardized forms of the Turing Test?

Most computer science instructors will tell you that the Turing Test is more a theoretical or conceptual thought experiment than an actual exam that someone (or something!) can formally sit and ...
Robert Columbia's user avatar
5 votes
1 answer
826 views

How to decide a train-test split?

In almost every ML model, a train-test (or train-test-val split) is critical to assess the model's performance. However, I have always wondered what the rationale is to decide a particular train-test ...
user avatar
0 votes
2 answers
378 views

What does a value of -1.000 mean in MS COCO Metrics for Object Detection

I am training some Object-Detection-Models from the TensorFlow Object Detection API and got from the evaluation with MS COCO metrics the following results for Average Precision: IoU = 0.5;0.9 maxDets =...
Benjamin Stecker's user avatar
0 votes
1 answer
137 views

How to test the robustness of an agent in a custom reinforcement learning environment?

I have used the stable-baseline3 implementation of the SAC algorithm to train policies in a custom gym environment. So far the results look promising. However, I would like to test the robustness of ...
Manuel's user avatar
  • 45
3 votes
1 answer
113 views

What is the most statistically acceptable method for tuning neural network hyperparameters on very small datasets?

Neural networks are usually evaluated by dividing a dataset into three splits: training, validation, and test The idea is that critical hyperparameters of the network such as the number of epochs ...
Mike NZ's user avatar
  • 401
1 vote
1 answer
177 views

What are the differences in testing between traditional software and artificial intelligence?

The testing problem in traditional software has been fully explored over the last decades, but it seems that testing in artificial intelligence/machine learning has not (see this question and this one)...
Lerner Zhang's user avatar
1 vote
0 answers
91 views

Is there a way, while training (with contrastive learning) the embedding network, to find the test accuracy?

I aim to do action recognition in videos on a private dataset. To compare with the existing state-of-the-art implementations, other guys published their code on Github, like the one here (for the ...
krishna chaitanya's user avatar
1 vote
1 answer
133 views

How to build a test set for a model in industry?

Most of the tutorials only teach us to split the whole dataset into three parts: training set, develop set, and test set. But in the industry, we are kind of doing test-driven development, and what ...
Lerner Zhang's user avatar
4 votes
1 answer
6k views

Should we also shuffle the test dataset when training with SGD?

When training machine learning models (e.g. neural networks) with stochastic gradient descent, it is common practice to (uniformly) shuffle the training data into batches/sets of different samples ...
SpiderRico's user avatar
0 votes
2 answers
975 views

How can I be sure that the final model, trained on all data, is correct?

The 'by the book' method of delivering final machine learning models is to include all data in the final training (including validation and test sets). To check robustness of my model I use randomly ...
GKozinski's user avatar
  • 1,260
0 votes
0 answers
114 views

Strange behavior of Q-learning agent after being trained

I built a simple X*Y grid world environment to learn and then trained my agent over it. All worked fine and the agent learned as well. Let me give some detail about the environment. Environment: A ...
SJa's user avatar
  • 393
3 votes
2 answers
371 views

How can I predict the true label for data with incomplete features based on the trained model with data with more features?

Suppose I have a model that was trained with a dataset that contains the features (f1, f2, f3, f4, f5, f6). However, my test dataset does not contain all features ...
Dae-Young Park's user avatar
1 vote
0 answers
147 views

What is the amount of test data needed to evaluate a CNN?

I have an image dataset of about 400 images. 70% of these data points were used for training, 15% for validation, and 15% for testing. I am using the 70% to train a CNN-based binary classifier. I ...
user38639's user avatar
3 votes
1 answer
107 views

Is the test time the phase when the model's accuracy is calculated with test data set?

When papers talk about the "test time", does this mean the phase when the model is passed with new data instances to derive the accuracy of the test data set? Or is "test time" the phase when the ...
MJimitater's user avatar
4 votes
4 answers
5k views

What is the difference between training and testing in reinforcement learning?

In reinforcement learning (RL), what is the difference between training and testing an algorithm/agent? If I understood correctly, testing is also referred to as evaluation. As I see it, both imply ...
Cristian M's user avatar
1 vote
0 answers
44 views

How can we make sure how well the reinforcement learning agent works on a stock dataset?

I read a paper, which is about Deep Reinforcement Learning and it tries to use this method on stock data set. It has been shown that it reaches the maximum return (profit). It has been implemented in ...
Mahdi Amrollahi's user avatar
4 votes
1 answer
593 views

How to evaluate an RL algorithm when used in a game?

I'm planning to create a web-based RL board game, and I wondered how I would evaluate the performance of the RL agent. How would I be able to say, "Version X performed better than version Y, as ...
mason7663's user avatar
  • 613
3 votes
1 answer
959 views

Should I use leave-one-out cross-validation for testing?

I am currently working with a small dataset of 20x300. Since I have so few data points, I was wondering if I could use an approach similar to leave-one-out cross-validation but for testing. Here's ...
Diogo Bastos's user avatar
3 votes
1 answer
3k views

What is the reason behind using a test batch size?

If one examines the SSD: Single Shot MultiBox Detector code from this GitHub repository, it can be seen that, for a testing phase (evaluating network on test data set), there is a parameter ...
carobnodrvo's user avatar
1 vote
1 answer
145 views

How do I check that the combination of these models is good?

I've selected more than 10 discriminative (classification) models, each wrapped with a BaggingClassifier object, optimized with a ...
Miko Diko's user avatar
  • 177
2 votes
0 answers
139 views

Which evaluation metrics should be used in training, validation and testing of a model?

Which specific performance evaluation metrics are used in training, validation, and testing, and why? I am thinking error metrics (RMSE, MAE, MSE) are used in validation, and testing should use a wide ...
user9645302's user avatar
2 votes
1 answer
183 views

Is it a good idea to train a CNN to detect the hydration value (percentage) in skin images and evaluate it with the MSE?

I have a large dataset of skin images, each one associated with a hydration value (percentage). Now I'm looking into predicting the hydration value from an image. My thinking: train a CNN on the ...
Patrick Samy's user avatar
7 votes
4 answers
3k views

Why is my test error lower than the training error?

I am trying to train a CNN regression model using the ADAM optimizer, dropout and weight decay. My test accuracy is better than training accuracy. But, as far as I know, usually, the training accuracy ...
이희준's user avatar