Hot answers tagged

4 votes

How is the F1 score calculated in a question-answering system?

In QA, it's computed over the individual words in the prediction against those in the True Answer. The number of shared words between the prediction and the truth is the basis of the F1 score: ...
user avatar
4 votes
Accepted

Why is there more than one way of calculating the accuracy?

In machine learning, the accuracy is usually defined as the number of correct predictions divided by the total number of predictions. The correct predictions are the true positives ($\mathrm {TP}$) ...
user avatar
  • 33.8k
2 votes
Accepted

How can we compare, in terms of similarity, two pieces of text?

There are more than 1 way of doing this: You can compute the bleu score between them if you are looking at the quality of machine translation. Check this link. You can convert them into 2 vectors ...
user avatar
  • 532
2 votes

Which metric should I use to assess the quality of the clusters?

You can compute Silhouette Coefficient for your aim. Its values mean: 1: Means clusters are well apart from each other and clearly distinguished. 0: Means clusters are indifferent, or we can say that ...
user avatar
  • 1,663
1 vote

Should I need to interpret the word "metric" in "performance metric" rigorously?

"Metric" should be understood as "a function of the trained model and of a dataset which returns a number". For example, in reinforcement learning, one can use as an evaluation ...
user avatar
1 vote

What inherent quality of a function makes it treated as either loss or evaluation metric?

Common loss functions, like the cross-entropy or mean squared error, are chosen because, if you minimize them, you are actually maximizing the likelihood of the parameters given the observed data. In ...
user avatar
  • 33.8k
1 vote

Compare the efficiency of a trained ML model with a non-learning-based method for solving the same problem

The most generic answer to this question is: the same metrics you use to evaluate the quality of your model during training or in test phase. (Plus the timing of inference if you're referring to ...
user avatar
1 vote

Does it make sense to use BLEU or ROUGE for any machine translation task?

Yes - and no. The important distinction is whether your data contains proper word boundaries and rigorous translation references. BLEU and ROGUE both work by comparing a candidate (ie, model output) ...
user avatar
  • 1,344
1 vote
Accepted

How do I know what a good mean absolute error value is?

Mean Absolute Error is nothing but the mean of absolute errors. If your model gave $n$ predictions $\{\hat{y}_i\}_{i = 1}^{n}$ against $n$ ground truths $\{y_i\}_{i = 1}^{n}$, then MAE is defines as ...
user avatar
  • 3,099
1 vote

Is there any metric for calculating how natural a single image is given a dataset of the same class images?

Evaluating synthetically generated images is challenging and an active area of research. The problem is that the "how natural is an image"-task is not well-defined and subjective. To ...
user avatar
1 vote

Which metric should I use to assess the quality of the clusters?

One more popular metric for this is the Davies Bouldin Score. You can also take a look at the clustering metrics in scikit documentation.
user avatar
1 vote

What are the disadvantages to using a distance metric in character recognition prediction

As I see it, the question boils down to the comparison between distance (function/metric) based Optical Character Recognition (OCR) and (for example) OCR done by means of Convolutional Neural Networks ...
user avatar
  • 705
1 vote

How should we interpret all the different metrics in reinforcement learning?

As you said, generally the most important one is reward per episode. If this isn't increasing overall, there's a problem (of course this metric can fluctuate, I mean to say that macroscopically it ...
user avatar
  • 961
1 vote

Using True Positive as a Cost Function

The vector functions for true positive, false positive etc all make use of the "magic" numbers $0$ and $1$ used to represent Boolean values. They are convenience methods that you can use in a ...
user avatar
  • 23.9k
1 vote

Metrics for evaluating models that output probabilities

For a binary classifier, the cross-entropy loss is a natural measure of probability accuracy, if you care about relative probabilities. By that I mean if you care that the estimate $\hat{p}$ is within ...
user avatar
  • 23.9k
1 vote

Why is the perplexity a good evaluation metric for chatbots?

With perplexity you are trying to evaluate the similarity between the token (in your case probably sentences) distribution generated by the model and the one in the test data. For instance, assuming ...
user avatar
  • 146

Only top scored, non community-wiki answers of a minimum length are eligible