5
votes
How is the F1 score calculated in a question-answering system?
In QA, it's computed over the individual words in the prediction against those in the True Answer. The number of shared words between the prediction and the truth is the basis of the F1 score: ...
4
votes
Accepted
Why is there more than one way of calculating the accuracy?
In machine learning, the accuracy is usually defined as the number of correct predictions divided by the total number of predictions. The correct predictions are the true positives ($\mathrm {TP}$) ...
- 41.1k
3
votes
Accepted
How can we compare, in terms of similarity, two pieces of text?
There are more than 1 way of doing this:
You can compute the bleu score between them if you are looking at the quality of machine translation. Check this link.
You can convert them into 2 vectors ...
- 562
2
votes
What loss function will be correlated with classification metrics?
Different metrics measure different quantities, so there is no reason to expect different metrics to move together unless one is a function of the other (such as MSE and RMSE).
Further, metrics like ...
- 658
2
votes
Accepted
Do we need to know or verify properties of loss functions / metrics' implementations?
Why we would like a function to satisfy some properties?
If we're talking about a loss function, you need to prove at the very least that the function has a minimum, otherwise you can't expect it to ...
- 5,378
2
votes
Accepted
How to calculate a meaningful distance between multidimensional tensors
You could try an earth mover distance in 2d or 3d over the image? For example you could follow this example, but call it sequentially. The idea would be something like the following (untested and ...
- 206
2
votes
Accepted
Aside from dice score, what other good metrics are used to evaluate segmentation models?
Typical metrics used with segmentation problems are Recall, Precision and the F1 Score (similar or the same as the Dice score depending on the definition used). These can be evaluated per class or for ...
2
votes
Which metric should I use to assess the quality of the clusters?
You can compute Silhouette Coefficient for your aim. Its values mean:
1: Means clusters are well apart from each other and clearly distinguished.
0: Means clusters are indifferent, or we can say that ...
- 1,826
2
votes
How to choose the metric value when evaluating the performance of a deep learning model?
In short: you should provide the optimal (best) value, but this is not the whole story.
As you said:
I think sometimes the optimal value is a result of randomness.
This is perfectly true. Therefore, ...
- 3,111
1
vote
Accepted
How to use the tf.image.SSIM function
The filter_size parameter in the tf.image.ssim function determines the size of the Gaussian filter used to smooth the image before calculating the SSIM.
Option 1: Adjust Filter Parameters
When using ...
- 409
1
vote
Accepted
Why does the pass@k metric not "behave like" probability?
Very late to this question, but pass@k doesn't behave like you're describing because each single pass@k sample is itself a union of k independent events, so averaging them doesn't behave like a union ...
- 26
1
vote
How to determine alignment / correlation with comparison based judgments?
I ended up with converting the compares to an absolute scale using something like ELO scoring from chess (Wikipedia, a paper), than you can just use correlation as usual.
1
vote
What are the disadvantages to using a distance metric in character recognition prediction
As I see it, the question boils down to the comparison between distance (function/metric) based Optical Character Recognition (OCR) and (for example) OCR done by means of Convolutional Neural Networks ...
- 825
1
vote
Accepted
How do I know what a good mean absolute error value is?
Mean Absolute Error is nothing but the mean of absolute errors.
If your model gave $n$ predictions $\{\hat{y}_i\}_{i = 1}^{n}$ against $n$ ground truths $\{y_i\}_{i = 1}^{n}$, then MAE is defines as ...
- 3,960
1
vote
Is there any metric for calculating how natural a single image is given a dataset of the same class images?
Evaluating synthetically generated images is challenging and an active area of research. The problem is that the "how natural is an image"-task is not well-defined and subjective.
To ...
- 967
1
vote
Which metric should I use to assess the quality of the clusters?
One more popular metric for this is the Davies Bouldin Score.
You can also take a look at the clustering metrics in scikit documentation.
- 868
1
vote
Should I need to interpret the word "metric" in "performance metric" rigorously?
"Metric" should be understood as "a function of the trained model and of a dataset which returns a number".
For example, in reinforcement learning, one can use as an evaluation ...
1
vote
What inherent quality of a function makes it treated as either loss or evaluation metric?
Common loss functions, like the cross-entropy or mean squared error, are chosen because, if you minimize them, you are actually maximizing the likelihood of the parameters given the observed data. In ...
- 41.1k
1
vote
Compare the efficiency of a trained ML model with a non-learning-based method for solving the same problem
The most generic answer to this question is:
the same metrics you use to evaluate the quality of your model during training or in test phase. (Plus the timing of inference if you're referring to ...
- 5,378
1
vote
Accepted
Does it make sense to use BLEU or ROUGE for any machine translation task?
Yes - and no. The important distinction is whether your data contains proper word boundaries and rigorous translation references.
BLEU and ROGUE both work by comparing a candidate (ie, model output) ...
- 1,406
1
vote
Aside from dice score, what other good metrics are used to evaluate segmentation models?
I agree fully with @a crazy Minon's answer. I will just slightly expand on it and provide a couple of additional references.
While Dice is a popular metric for evaluating segmentation, it is ...
- 987
1
vote
How should we interpret all the different metrics in reinforcement learning?
As you said, generally the most important one is reward per episode. If this isn't increasing overall, there's a problem (of course this metric can fluctuate, I mean to say that macroscopically it ...
- 1,136
1
vote
What is meant by the expected BLEU cost when training with BLEU and SIMILE?
It looks like the method they use for training takes a set of candidate hypotheses $\mathcal{U}(x)$, along with associated probabilities, and then minimizes the expected value of the cost function ...
- 346
1
vote
How can we compare, in terms of similarity, two pieces of text?
It depends what you mean by "comparison", but in general I would think not really.
Neural networks operate on the sub-symbolic level, ie instead of handling discrete symbols (such as letters) they ...
- 5,397
1
vote
Why is the perplexity a good evaluation metric for chatbots?
With perplexity you are trying to evaluate the similarity between the token (in your case probably sentences) distribution generated by the model and the one in the test data.
For instance, assuming ...
- 146
1
vote
What evaluation metric are used for sequence-to-sequence prediction problems?
This really depends on your data. MSE and its variant, the RMSE, are good for regression problems. In other words, when you want to produce a real number as an output, for example, in a time series ...
- 236
1
vote
Using True Positive as a Cost Function
The vector functions for true positive, false positive etc all make use of the "magic" numbers $0$ and $1$ used to represent Boolean values. They are convenience methods that you can use in a ...
- 32.9k
1
vote
Metrics for evaluating models that output probabilities
For a binary classifier, the cross-entropy loss is a natural measure of probability accuracy, if you care about relative probabilities. By that I mean if you care that the estimate $\hat{p}$ is within ...
- 32.9k
Only top scored, non community-wiki answers of a minimum length are eligible