# Tag Info

### How is the F1 score calculated in a question-answering system?

In QA, it's computed over the individual words in the prediction against those in the True Answer. The number of shared words between the prediction and the truth is the basis of the F1 score: ...
Accepted

### Why is there more than one way of calculating the accuracy?

In machine learning, the accuracy is usually defined as the number of correct predictions divided by the total number of predictions. The correct predictions are the true positives ($\mathrm {TP}$) ...
• 41.1k
Accepted

### How can we compare, in terms of similarity, two pieces of text?

There are more than 1 way of doing this: You can compute the bleu score between them if you are looking at the quality of machine translation. Check this link. You can convert them into 2 vectors ...
• 562

### What loss function will be correlated with classification metrics?

Different metrics measure different quantities, so there is no reason to expect different metrics to move together unless one is a function of the other (such as MSE and RMSE). Further, metrics like ...
• 658
Accepted

### Do we need to know or verify properties of loss functions / metrics' implementations?

Why we would like a function to satisfy some properties? If we're talking about a loss function, you need to prove at the very least that the function has a minimum, otherwise you can't expect it to ...
• 5,378
Accepted

### How to calculate a meaningful distance between multidimensional tensors

You could try an earth mover distance in 2d or 3d over the image? For example you could follow this example, but call it sequentially. The idea would be something like the following (untested and ...
Accepted

### Aside from dice score, what other good metrics are used to evaluate segmentation models?

Typical metrics used with segmentation problems are Recall, Precision and the F1 Score (similar or the same as the Dice score depending on the definition used). These can be evaluated per class or for ...

### Which metric should I use to assess the quality of the clusters?

You can compute Silhouette Coefficient for your aim. Its values mean: 1: Means clusters are well apart from each other and clearly distinguished. 0: Means clusters are indifferent, or we can say that ...
• 1,826

### How to choose the metric value when evaluating the performance of a deep learning model?

In short: you should provide the optimal (best) value, but this is not the whole story. As you said: I think sometimes the optimal value is a result of randomness. This is perfectly true. Therefore, ...
• 3,111
1 vote
Accepted

### How to use the tf.image.SSIM function

The filter_size parameter in the tf.image.ssim function determines the size of the Gaussian filter used to smooth the image before calculating the SSIM. Option 1: Adjust Filter Parameters When using ...
• 409
1 vote
Accepted

### Why does the pass@k metric not "behave like" probability?

Very late to this question, but pass@k doesn't behave like you're describing because each single pass@k sample is itself a union of k independent events, so averaging them doesn't behave like a union ...
1 vote

### How to determine alignment / correlation with comparison based judgments?

I ended up with converting the compares to an absolute scale using something like ELO scoring from chess (Wikipedia, a paper), than you can just use correlation as usual.
1 vote

### What are the disadvantages to using a distance metric in character recognition prediction

As I see it, the question boils down to the comparison between distance (function/metric) based Optical Character Recognition (OCR) and (for example) OCR done by means of Convolutional Neural Networks ...
• 825
1 vote
Accepted

### How do I know what a good mean absolute error value is?

Mean Absolute Error is nothing but the mean of absolute errors. If your model gave $n$ predictions $\{\hat{y}_i\}_{i = 1}^{n}$ against $n$ ground truths $\{y_i\}_{i = 1}^{n}$, then MAE is defines as ...
• 3,960
1 vote

### Is there any metric for calculating how natural a single image is given a dataset of the same class images?

Evaluating synthetically generated images is challenging and an active area of research. The problem is that the "how natural is an image"-task is not well-defined and subjective. To ...
• 967
1 vote

### Which metric should I use to assess the quality of the clusters?

One more popular metric for this is the Davies Bouldin Score. You can also take a look at the clustering metrics in scikit documentation.
1 vote

### Should I need to interpret the word "metric" in "performance metric" rigorously?

"Metric" should be understood as "a function of the trained model and of a dataset which returns a number". For example, in reinforcement learning, one can use as an evaluation ...
1 vote

### What inherent quality of a function makes it treated as either loss or evaluation metric?

Common loss functions, like the cross-entropy or mean squared error, are chosen because, if you minimize them, you are actually maximizing the likelihood of the parameters given the observed data. In ...
• 41.1k
1 vote

### Compare the efficiency of a trained ML model with a non-learning-based method for solving the same problem

The most generic answer to this question is: the same metrics you use to evaluate the quality of your model during training or in test phase. (Plus the timing of inference if you're referring to ...
• 5,378
1 vote
Accepted

### Does it make sense to use BLEU or ROUGE for any machine translation task?

Yes - and no. The important distinction is whether your data contains proper word boundaries and rigorous translation references. BLEU and ROGUE both work by comparing a candidate (ie, model output) ...
• 1,406
1 vote

### Aside from dice score, what other good metrics are used to evaluate segmentation models?

I agree fully with @a crazy Minon's answer. I will just slightly expand on it and provide a couple of additional references. While Dice is a popular metric for evaluating segmentation, it is ...
• 987
1 vote

### How should we interpret all the different metrics in reinforcement learning?

As you said, generally the most important one is reward per episode. If this isn't increasing overall, there's a problem (of course this metric can fluctuate, I mean to say that macroscopically it ...
• 1,136
1 vote

### What is meant by the expected BLEU cost when training with BLEU and SIMILE?

It looks like the method they use for training takes a set of candidate hypotheses $\mathcal{U}(x)$, along with associated probabilities, and then minimizes the expected value of the cost function ...
• 346
1 vote

### How can we compare, in terms of similarity, two pieces of text?

It depends what you mean by "comparison", but in general I would think not really. Neural networks operate on the sub-symbolic level, ie instead of handling discrete symbols (such as letters) they ...
• 5,397
1 vote

### Why is the perplexity a good evaluation metric for chatbots?

With perplexity you are trying to evaluate the similarity between the token (in your case probably sentences) distribution generated by the model and the one in the test data. For instance, assuming ...
• 146
1 vote

### What evaluation metric are used for sequence-to-sequence prediction problems?

This really depends on your data. MSE and its variant, the RMSE, are good for regression problems. In other words, when you want to produce a real number as an output, for example, in a time series ...
• 236
1 vote

### Using True Positive as a Cost Function

The vector functions for true positive, false positive etc all make use of the "magic" numbers $0$ and $1$ used to represent Boolean values. They are convenience methods that you can use in a ...
• 32.9k
1 vote

### Metrics for evaluating models that output probabilities

For a binary classifier, the cross-entropy loss is a natural measure of probability accuracy, if you care about relative probabilities. By that I mean if you care that the estimate $\hat{p}$ is within ...
• 32.9k

Only top scored, non community-wiki answers of a minimum length are eligible