Hot answers tagged

4

In machine learning, the accuracy is usually defined as the number of correct predictions divided by the total number of predictions. The correct predictions are the true positives ($\mathrm {TP}$) and true negatives ($\mathrm {TN}$), so the usual formula to calculate the accuracy is the following one (your first one). \begin{align} \text{Accuracy}=\frac {\...


2

There are more than 1 way of doing this: You can compute the bleu score between them if you are looking at the quality of machine translation. Check this link. You can convert them into 2 vectors using doc2vec and find the similarity between the vectors using cosine similarity. Siamese networks are something similar to what you are asking. They are neural ...


2

In QA, it's computed over the individual words in the prediction against those in the True Answer. The number of shared words between the prediction and the truth is the basis of the F1 score: precision is the ratio of the number of shared words to the total number of words in the prediction, and recall is the ratio of the number of shared words to the total ...


1

As you said, generally the most important one is reward per episode. If this isn't increasing overall, there's a problem (of course this metric can fluctuate, I mean to say that macroscopically it should increase). Policy loss (I assume you mean the "actor loss"?) is generally harder to interpret. You should think of this more as a source of ...


1

The most generic answer to this question is: the same metrics you use to evaluate the quality of your model during training or in test phase. (Plus the timing of inference if you're referring to computational efficiency) And I'm not referring to any specific metric yet cause that's really task dependent. But in general if you have a model that perform a task ...


1

Yes - and no. The important distinction is whether your data contains proper word boundaries and rigorous translation references. BLEU and ROGUE both work by comparing a candidate (ie, model output) to reference text (ie, training data). In a translation task (what these metrics are typically used for) this works quite well, as you can normally assume the ...


1

Mean Absolute Error is nothing but the mean of absolute errors. If your model gave $n$ predictions $\{\hat{y}_i\}_{i = 1}^{n}$ against $n$ ground truths $\{y_i\}_{i = 1}^{n}$, then MAE is defines as follows $$MAE_{model} = \dfrac{\sum\limits_{i = 1}^{n} |y_i - \hat{y}_i|}{n} $$. Thus, MAE gives the average amount of error. So, the machine learning model with ...


1

Evaluating synthetically generated images is challenging and an active area of research. The problem is that the "how natural is an image"-task is not well-defined and subjective. To evaluate generated images we can define two abstract properties: fidelity and diversity, as we want to generate not only a single high-quality image, but also ...


1

As I see it, the question boils down to the comparison between distance (function/metric) based Optical Character Recognition (OCR) and (for example) OCR done by means of Convolutional Neural Networks (CNNs). Particularly, it focuses on the cons of the former option. There are a few potential problems associated with using distance based OCR systems. First ...


1

The vector functions for true positive, false positive etc all make use of the "magic" numbers $0$ and $1$ used to represent Boolean values. They are convenience methods that you can use in a numerical library, but you do need to be aware of the fundamental Boolean nature of the data. The $0$ and $1$ values allow the maths for calculating TP et al, but are ...


1

For a binary classifier, the cross-entropy loss is a natural measure of probability accuracy, if you care about relative probabilities. By that I mean if you care that the estimate $\hat{p}$ is within some ratio of the true value. So an estimate of $\hat{p} = 0.1$ is a better estimate if the true value is $p = 0.2$ than if the true value is $p = 0.01$ (even ...


1

With perplexity you are trying to evaluate the similarity between the token (in your case probably sentences) distribution generated by the model and the one in the test data. For instance, assuming you have $M$ sentences $s_1, \dots, s_M$, each with probability $P(s_i)$, the perplexity is $$2^{-l},$$ where $l = \frac{1}{M} \sum P(s_i) \log P(s_i)$ for $i \...


Only top voted, non community-wiki answers of a minimum length are eligible