3
$\begingroup$

Some sources consider the true negatives (TN) when computing the accuracy, while some don't.

Source 1: https://medium.com/greyatom/performance-metrics-for-classification-problems-in-machine-learning-part-i-b085d432082b

Considers TN

Source 2:https://www.researchgate.net/profile/Mohammad_Sorower/publication/266888594_A_Literature_Survey_on_Algorithms_for_Multi-label_Learning/links/58d1864392851cf4f8f4b72a/A-Literature-Survey-on-Algorithms-for-Multi-label-Learning.pdf

enter image description here

which can be translated as

enter image description here

which one of these must be considered for my multi-label model.

$\endgroup$
  • 1
    $\begingroup$ There are many different metrics for all sorts of features; generally there is no right or wrong, and it depends on your application. If you have multiple labels, you probably don't have TN, so the second one looks to be a better fit. $\endgroup$ – Oliver Mason Feb 21 at 10:16
4
$\begingroup$

In machine learning, the accuracy is usually defined as the number of correct predictions divided by the total number of predictions. The correct predictions are the true positives ($\mathrm {TP}$) and true negatives ($\mathrm {TN}$), so the usual formula to calculate the accuracy is the following one (your first one).

\begin{align} \text{Accuracy}=\frac {\mathrm {TP} + \mathrm {TN}}{\mathrm {TP} + \mathrm {TN} + \mathrm {FP} + \mathrm {FN}} \label{0}\tag{0} \end{align}

The following formula corresponds to the threat score ($\mathrm{TS}$) or critical success index ($\mathrm {CSI}$).

\begin{align} {\displaystyle \mathrm {TS} ={\frac {\mathrm {TP} }{\mathrm {TP} +\mathrm {FN} +\mathrm {FP} }}} \label{1} \tag{1} \end{align}

In section 7 of the paper A Literature Survey on Algorithms for Multi-label Learning, it is written

In traditional classification such as multi-class problems, accuracy is the most common evaluation criteria. Additionally, there exists a set of standard evaluation metrics that includes precision, recall, F-measure, and ROC area defined for single-label multi-class classification problems. However, in multi-label classification, predictions for an instance is a set of labels and, therefore, the prediction can be fully correct, partially correct (with different levels of correctness) or fully incorrect. None of these existing evaluation metrics capture such notion in their original form.

Then they say

Depending on the target problem, evaluation measures for multi-label data can be grouped into at least three groups: evaluating partitions, evaluating ranking and using label hierarchy. The first one evaluates the quality of the classification into classes, the second one evaluates if the classes are ranked in order relevance and the third one evaluates how effectively the learning system is able to take into account an existing hierarchical structure of the labels.

In section 7.1, the authors state

To capture the notion of partially correct, one strategy is to evaluate the average difference between the predicted labels and the actual labels for each test example, and then average over all examples in the test set. This approach is called example-based evaluations.

To capture the notion of partially incorrect in multi-label classification problems, they then use the following definition of accuracy (proposed in section 5.2 of the paper Discriminative Methods for Multi-labeled Classification)

\begin{align} \frac{1}{n} \sum_{i=1}^{n} \frac{|Y_i \cap Z_i|}{|Y_i \cup Z_i|} \label{2} \tag{2} \end{align}

where $Y_i$ is the true set of labels and $Z_i$ the predicted set of labels for the single instance (or observation) $i$, so $\frac{|Y_i \cap Z_i|}{|Y_i \cup Z_i|}$ is the accuracy for the instance $i$. If $|Y_i|=1$ and $|Z_i|=1$, then we have a single-label classification problem.

The metric \ref{2} does not correspond to the metric \ref{1}. In \ref{2}, the accuracy is calculated for each example (or instance) and then we average all these accuracies, because, for each example, you may have that one predicted label is correct, two predicted labels are correct, and so on (also depending on the number of labels you need to predict for each of the examples). In \ref{1}, there is no notion of multiple labels.

| improve this answer | |
$\endgroup$
  • $\begingroup$ In equation.1, though there is no notion of multiple labels, I can consider confusion matrix for vector of predicted π‘Œπ‘– and vector of actual 𝑍𝑖 to calculate TS(Theta Score) and if I average over all the examples or instances isn't it same as equation.2? $\endgroup$ – Stephen Philip Feb 21 at 17:09
  • 1
    $\begingroup$ @StephenPhilip I don't think so. $Y_i$ and $Z_i$ should be sets (rather than vectors). $\endgroup$ – nbro Feb 21 at 18:12
  • $\begingroup$ Yes, π‘Œπ‘– and 𝑍𝑖 are sets, just to disambiguate I was using programming language terminology(C++ uses generally vectors are used to store as arrays, in python lists are used), but my doubt still persists, isn't eq.1 is representation of eq.2 with average over all the observations. $\endgroup$ – Stephen Philip Feb 21 at 18:32
  • 1
    $\begingroup$ @StephenPhilip I don't understand where you see the equivalence between equation 1 and 2, but maybe I am blind now. Does $|Y_i \cap Z_i|$ represent only the number of true positives (i.e. the numerator of equation 1)? $Y_i \cap Z_i$ is the intersection between two sets. It doesn't matter the order of the elements, but only whether the elements are or not in the sets. When you're counting the true positives, the order matters (i.e. you should compare the labels of instance $i$ with the corresponding predictions $i$). $\endgroup$ – nbro Feb 21 at 18:40
  • 1
    $\begingroup$ Btw, you also have sets in Python and C++, you don't have to use lists or vectors, which represent ordered sequences. $\endgroup$ – nbro Feb 21 at 18:42

Your Answer

By clicking β€œPost Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.