Questions tagged [testing]
For questions related to the concept of testing (or evaluating) machine learning models and algorithms, e.g. in terms of some performance measure (such as accuracy or cumulative reward).
38
questions
0
votes
0
answers
17
views
Averaging classification result vs direct infer
Suppose I have tensor X_test with this shape:
(10,2,512) Where:
10 is num of ID
2 is num of Channel of every ID, let's say ...
0
votes
1
answer
40
views
Can software testers transfer their skills into adversarial testing for AI/LLMS?
My sincere apologies if I am more asking for advice, rather than a solution to a specific problem. However, I am trying to survive in this new world as a software tester. I have been doing test ...
- 101
0
votes
0
answers
20
views
3-way hold-out for picking between different ML models
I have a dataset which is a time series. Before jumping on more heavy models such as LSTM, I wanted to test out the performance of linear models.
I have currently a 80/20 split between training and ...
- 1
0
votes
0
answers
30
views
1
vote
1
answer
34
views
How to choose the metric value when evaluating the performance of a deep learning model?
When evaluating the performance of a deep learning model (for the purpose of publishing a research work), should we choose the optimal value from the metric curve (such as the accuracy curve) or the ...
- 19
1
vote
1
answer
332
views
What are alternatives to Inception Score? Can it be used for non-photographic image types?
Most online sources recommend using versions of the Inception score to evaluate the synthetic images generated by a GAN. These scores are pre-trained on the InceptionV3 model. Does this mean that ...
1
vote
1
answer
6k
views
High precision and low recall results. What does it mean?
I am working on a classification algorithm for brain rhythms. However, when I implemented the metrics for precision, accuracy, F1 score and recall. My results show that my algorithm has a high ...
- 123
0
votes
1
answer
80
views
Can AUCROC curves be used to evaluate forecasting results for time series data?
I have built a time series forecasting model based on RNN.
Can I calculate AUCROC on the test set and the actual predicted values?
Example:
...
- 17
0
votes
0
answers
1k
views
What is the "attack success rate" of an Adversarial Attack?
For a typical adversarial attack, a sample $x_{0}$ is chosen from a training set belonging to class $C_{i}$ and a transformation $A$ is applied such that the adversarial example $x=A(x_{0})$ would be ...
1
vote
0
answers
68
views
What considerations should I take to train my transformer model?
I want to train my vision transformer model on a benchmark for an image segmentation task: (LoveDA: A Remote Sensing Land-Cover Dataset for Domain Adaptive Semantic Segmentation) (GitHub), but I don't ...
1
vote
1
answer
57
views
Is it mandatory to multiply every activation of a layer by droupout factor during testing?
Dropout is a regularization technique used in neural networks. It is useful in preventing overfitting by making a neural network as good as an ensemble system.
In dropout, we switch off $p$ percent of ...
- 3,950
0
votes
1
answer
73
views
Does $S_{t+1}$ denote the future information in Q-learning?
In Q-learning, $Q(S_t,a)$ is updated by the Bellman equation. $Q(S_t,a) = r + \max_{a'}(Q(S_{t+1},a'))$ where $S_{t+1}$ is the future state.
Let's say $S$ denotes the stock price, does it mean we are ...
- 1
2
votes
1
answer
480
views
Why doesn't dropout mislead results during evaluation?
I have seen that, usually, the dropout layer is used differently in training and evaluation modes, i.e. it is recommended to use during training but not in evaluation/testing.
Dropout does remove a ...
- 33
1
vote
1
answer
4k
views
What is the time complexity for testing a stacked LSTM model?
In the data preparation phase, we have to divide the dataset into two parts: the training dataset and the test dataset.
I have seen this post regarding the time complexity for training a model.
...
- 113
1
vote
1
answer
229
views
How to do testing for an RNN that was trained with teacher forcing only?
If an RNN is trained using only the teacher forcing, then the network takes the actual output from the previous time step as input to the hidden state the next time step.
We know that the actual ...
- 3,950
3
votes
0
answers
51
views
Are there standardized forms of the Turing Test?
Most computer science instructors will tell you that the Turing Test is more a theoretical or conceptual thought experiment than an actual exam that someone (or something!) can formally sit and ...
- 131
5
votes
1
answer
850
views
How to decide a train-test split?
In almost every ML model, a train-test (or train-test-val split) is critical to assess the model's performance. However, I have always wondered what the rationale is to decide a particular train-test ...
user48670
0
votes
2
answers
431
views
What does a value of -1.000 mean in MS COCO Metrics for Object Detection
I am training some Object-Detection-Models from the TensorFlow Object Detection API and got from the evaluation with MS COCO metrics the following results for Average Precision:
IoU = 0.5;0.9
maxDets =...
0
votes
1
answer
152
views
How to test the robustness of an agent in a custom reinforcement learning environment?
I have used the stable-baseline3 implementation of the SAC algorithm to train policies in a custom gym environment. So far the results look promising. However, I would like to test the robustness of ...
- 45
3
votes
1
answer
113
views
What is the most statistically acceptable method for tuning neural network hyperparameters on very small datasets?
Neural networks are usually evaluated by dividing a dataset into three splits:
training,
validation, and
test
The idea is that critical hyperparameters of the network such as the number of epochs ...
- 411
1
vote
1
answer
186
views
What are the differences in testing between traditional software and artificial intelligence?
The testing problem in traditional software has been fully explored over the last decades, but it seems that testing in artificial intelligence/machine learning has not (see this question and this one)...
- 947
1
vote
0
answers
119
views
Is there a way, while training (with contrastive learning) the embedding network, to find the test accuracy?
I aim to do action recognition in videos on a private dataset.
To compare with the existing state-of-the-art implementations, other guys published their code on Github, like the one here (for the ...
1
vote
1
answer
148
views
How to build a test set for a model in industry?
Most of the tutorials only teach us to split the whole dataset into three parts: training set, develop set, and test set. But in the industry, we are kind of doing test-driven development, and what ...
- 947
4
votes
1
answer
6k
views
Should we also shuffle the test dataset when training with SGD?
When training machine learning models (e.g. neural networks) with stochastic gradient descent, it is common practice to (uniformly) shuffle the training data into batches/sets of different samples ...
- 1,020
0
votes
2
answers
1k
views
How can I be sure that the final model, trained on all data, is correct?
The 'by the book' method of delivering final machine learning models is to include all data in the final training (including validation and test sets). To check robustness of my model I use randomly ...
- 1,280
0
votes
0
answers
138
views
Strange behavior of Q-learning agent after being trained
I built a simple X*Y grid world environment to learn and then trained my agent over it. All worked fine and the agent learned as well. Let me give some detail about the environment.
Environment:
A ...
- 393
3
votes
2
answers
384
views
How can I predict the true label for data with incomplete features based on the trained model with data with more features?
Suppose I have a model that was trained with a dataset that contains the features (f1, f2, f3, f4, f5, f6). However, my test dataset does not contain all features ...
1
vote
0
answers
152
views
What is the amount of test data needed to evaluate a CNN?
I have an image dataset of about 400 images. 70% of these data points were used for training, 15% for validation, and 15% for testing. I am using the 70% to train a CNN-based binary classifier. I ...
- 11
3
votes
1
answer
111
views
Is the test time the phase when the model's accuracy is calculated with test data set?
When papers talk about the "test time", does this mean the phase when the model is passed with new data instances to derive the accuracy of the test data set? Or is "test time" the phase when the ...
- 133
4
votes
4
answers
5k
views
What is the difference between training and testing in reinforcement learning?
In reinforcement learning (RL), what is the difference between training and testing an algorithm/agent? If I understood correctly, testing is also referred to as evaluation.
As I see it, both imply ...
- 249
1
vote
0
answers
45
views
How can we make sure how well the reinforcement learning agent works on a stock dataset?
I read a paper, which is about Deep Reinforcement Learning and it tries to use this method on stock data set. It has been shown that it reaches the maximum return (profit). It has been implemented in ...
- 209
4
votes
1
answer
612
views
How to evaluate an RL algorithm when used in a game?
I'm planning to create a web-based RL board game, and I wondered how I would evaluate the performance of the RL agent. How would I be able to say, "Version X performed better than version Y, as ...
- 623
3
votes
1
answer
1k
views
Should I use leave-one-out cross-validation for testing?
I am currently working with a small dataset of 20x300. Since I have so few data points, I was wondering if I could use an approach similar to leave-one-out cross-validation but for testing.
Here's ...
- 39
3
votes
1
answer
3k
views
What is the reason behind using a test batch size?
If one examines the SSD: Single Shot MultiBox Detector code from this GitHub repository, it can be seen that, for a testing phase (evaluating network on test data set), there is a parameter ...
- 143
1
vote
1
answer
148
views
How do I check that the combination of these models is good?
I've selected more than 10 discriminative (classification) models, each wrapped with a BaggingClassifier object, optimized with a ...
- 177
2
votes
0
answers
140
views
Which evaluation metrics should be used in training, validation and testing of a model?
Which specific performance evaluation metrics are used in training, validation, and testing, and why? I am thinking error metrics (RMSE, MAE, MSE) are used in validation, and testing should use a wide ...
- 53
3
votes
1
answer
186
views
Is it a good idea to train a CNN to detect the hydration value (percentage) in skin images and evaluate it with the MSE?
I have a large dataset of skin images, each one associated with a hydration value (percentage).
Now I'm looking into predicting the hydration value from an image. My thinking: train a CNN on the ...
- 39
7
votes
4
answers
3k
views
Why is my test error lower than the training error?
I am trying to train a CNN regression model using the ADAM optimizer, dropout and weight decay.
My test accuracy is better than training accuracy. But, as far as I know, usually, the training accuracy ...
- 73