Skip to main content
Share Your Experience: Take the 2024 Developer Survey
5 votes

How is the F1 score calculated in a question-answering system?

In QA, it's computed over the individual words in the prediction against those in the True Answer. The number of shared words between the prediction and the truth is the basis of the F1 score: ...
Chin Hui Chew's user avatar
4 votes
Accepted

Why is there more than one way of calculating the accuracy?

In machine learning, the accuracy is usually defined as the number of correct predictions divided by the total number of predictions. The correct predictions are the true positives ($\mathrm {TP}$) ...
nbro's user avatar
  • 40.8k
3 votes
Accepted

How can we compare, in terms of similarity, two pieces of text?

There are more than 1 way of doing this: You can compute the bleu score between them if you are looking at the quality of machine translation. Check this link. You can convert them into 2 vectors ...
varsh's user avatar
  • 562
2 votes

What loss function will be correlated with classification metrics?

Different metrics measure different quantities, so there is no reason to expect different metrics to move together unless one is a function of the other (such as MSE and RMSE). Further, metrics like ...
Dave's user avatar
  • 618
2 votes
Accepted

Do we need to know or verify properties of loss functions / metrics' implementations?

Why we would like a function to satisfy some properties? If we're talking about a loss function, you need to prove at the very least that the function has a minimum, otherwise you can't expect it to ...
Edoardo Guerriero's user avatar
2 votes
Accepted

How to calculate a meaningful distance between multidimensional tensors

You could try an earth mover distance in 2d or 3d over the image? For example you could follow this example, but call it sequentially. The idea would be something like the following (untested and ...
John St. John's user avatar
2 votes
Accepted

Aside from dice score, what other good metrics are used to evaluate segmentation models?

Typical metrics used with segmentation problems are Recall, Precision and the F1 Score (similar or the same as the Dice score depending on the definition used). These can be evaluated per class or for ...
a crazy Minion's user avatar
2 votes

Which metric should I use to assess the quality of the clusters?

You can compute Silhouette Coefficient for your aim. Its values mean: 1: Means clusters are well apart from each other and clearly distinguished. 0: Means clusters are indifferent, or we can say that ...
OmG's user avatar
  • 1,816
2 votes

How to choose the metric value when evaluating the performance of a deep learning model?

In short: you should provide the optimal (best) value, but this is not the whole story. As you said: I think sometimes the optimal value is a result of randomness. This is perfectly true. Therefore, ...
Luca Anzalone's user avatar
1 vote
Accepted

Why does the pass@k metric not "behave like" probability?

Very late to this question, but pass@k doesn't behave like you're describing because each single pass@k sample is itself a union of k independent events, so averaging them doesn't behave like a union ...
Brian Huang's user avatar
1 vote

How to determine alignment / correlation with comparison based judgments?

I ended up with converting the compares to an absolute scale using something like ELO scoring from chess (Wikipedia, a paper), than you can just use correlation as usual.
Osmosis D. Jones's user avatar
1 vote

What are the disadvantages to using a distance metric in character recognition prediction

As I see it, the question boils down to the comparison between distance (function/metric) based Optical Character Recognition (OCR) and (for example) OCR done by means of Convolutional Neural Networks ...
Daniel B.'s user avatar
  • 825
1 vote
Accepted

How do I know what a good mean absolute error value is?

Mean Absolute Error is nothing but the mean of absolute errors. If your model gave $n$ predictions $\{\hat{y}_i\}_{i = 1}^{n}$ against $n$ ground truths $\{y_i\}_{i = 1}^{n}$, then MAE is defines as ...
hanugm's user avatar
  • 3,890
1 vote

Is there any metric for calculating how natural a single image is given a dataset of the same class images?

Evaluating synthetically generated images is challenging and an active area of research. The problem is that the "how natural is an image"-task is not well-defined and subjective. To ...
Aray Karjauv's user avatar
1 vote

Which metric should I use to assess the quality of the clusters?

One more popular metric for this is the Davies Bouldin Score. You can also take a look at the clustering metrics in scikit documentation.
Abhishek Verma's user avatar
1 vote

Should I need to interpret the word "metric" in "performance metric" rigorously?

"Metric" should be understood as "a function of the trained model and of a dataset which returns a number". For example, in reinforcement learning, one can use as an evaluation ...
Raphael Lopez Kaufman's user avatar
1 vote

What inherent quality of a function makes it treated as either loss or evaluation metric?

Common loss functions, like the cross-entropy or mean squared error, are chosen because, if you minimize them, you are actually maximizing the likelihood of the parameters given the observed data. In ...
nbro's user avatar
  • 40.8k
1 vote

Compare the efficiency of a trained ML model with a non-learning-based method for solving the same problem

The most generic answer to this question is: the same metrics you use to evaluate the quality of your model during training or in test phase. (Plus the timing of inference if you're referring to ...
Edoardo Guerriero's user avatar
1 vote
Accepted

Does it make sense to use BLEU or ROUGE for any machine translation task?

Yes - and no. The important distinction is whether your data contains proper word boundaries and rigorous translation references. BLEU and ROGUE both work by comparing a candidate (ie, model output) ...
Recessive's user avatar
  • 1,396
1 vote

Aside from dice score, what other good metrics are used to evaluate segmentation models?

I agree fully with @a crazy Minon's answer. I will just slightly expand on it and provide a couple of additional references. While Dice is a popular metric for evaluating segmentation, it is ...
Snehal Patel's user avatar
1 vote

How should we interpret all the different metrics in reinforcement learning?

As you said, generally the most important one is reward per episode. If this isn't increasing overall, there's a problem (of course this metric can fluctuate, I mean to say that macroscopically it ...
harwiltz's user avatar
  • 1,136
1 vote

What is meant by the expected BLEU cost when training with BLEU and SIMILE?

It looks like the method they use for training takes a set of candidate hypotheses $\mathcal{U}(x)$, along with associated probabilities, and then minimizes the expected value of the cost function ...
Ray's user avatar
  • 346
1 vote

How can we compare, in terms of similarity, two pieces of text?

It depends what you mean by "comparison", but in general I would think not really. Neural networks operate on the sub-symbolic level, ie instead of handling discrete symbols (such as letters) they ...
Oliver Mason's user avatar
  • 5,387
1 vote

Why is the perplexity a good evaluation metric for chatbots?

With perplexity you are trying to evaluate the similarity between the token (in your case probably sentences) distribution generated by the model and the one in the test data. For instance, assuming ...
ginge's user avatar
  • 146
1 vote

What evaluation metric are used for sequence-to-sequence prediction problems?

This really depends on your data. MSE and its variant, the RMSE, are good for regression problems. In other words, when you want to produce a real number as an output, for example, in a time series ...
Marcus's user avatar
  • 236
1 vote

Using True Positive as a Cost Function

The vector functions for true positive, false positive etc all make use of the "magic" numbers $0$ and $1$ used to represent Boolean values. They are convenience methods that you can use in a ...
Neil Slater's user avatar
  • 32.5k
1 vote

Metrics for evaluating models that output probabilities

For a binary classifier, the cross-entropy loss is a natural measure of probability accuracy, if you care about relative probabilities. By that I mean if you care that the estimate $\hat{p}$ is within ...
Neil Slater's user avatar
  • 32.5k

Only top scored, non community-wiki answers of a minimum length are eligible