Why is there more than one way of calculating the accuracy?

Question

Some sources consider the true negatives (TN) when computing the accuracy, while some don't.

Source 1: https://medium.com/greyatom/performance-metrics-for-classification-problems-in-machine-learning-part-i-b085d432082b

Source 2:https://www.researchgate.net/profile/Mohammad_Sorower/publication/266888594_A_Literature_Survey_on_Algorithms_for_Multi-label_Learning/links/58d1864392851cf4f8f4b72a/A-Literature-Survey-on-Algorithms-for-Multi-label-Learning.pdf

which can be translated as

which one of these must be considered for my multi-label model.

Answer 1

In machine learning, the accuracy is usually defined as the number of correct predictions divided by the total number of predictions. The correct predictions are the true positives ($\mathrm {TP}$) and true negatives ($\mathrm {TN}$), so the usual formula to calculate the accuracy is the following one (your first one).

\begin{align} \text{Accuracy}=\frac {\mathrm {TP} + \mathrm {TN}}{\mathrm {TP} + \mathrm {TN} + \mathrm {FP} + \mathrm {FN}} \label{0}\tag{0} \end{align}

The following formula corresponds to the threat score ($\mathrm{TS}$) or critical success index ($\mathrm {CSI}$).

\begin{align} {\displaystyle \mathrm {TS} ={\frac {\mathrm {TP} }{\mathrm {TP} +\mathrm {FN} +\mathrm {FP} }}} \label{1} \tag{1} \end{align}

In section 7 of the paper A Literature Survey on Algorithms for Multi-label Learning, it is written

In traditional classification such as multi-class problems, accuracy is the most common evaluation criteria. Additionally, there exists a set of standard evaluation metrics that includes precision, recall, F-measure, and ROC area defined for single-label multi-class classification problems. However, in multi-label classification, predictions for an instance is a set of labels and, therefore, the prediction can be fully correct, partially correct (with different levels of correctness) or fully incorrect. None of these existing evaluation metrics capture such notion in their original form.

Then they say

Depending on the target problem, evaluation measures for multi-label data can be grouped into at least three groups: evaluating partitions, evaluating ranking and using label hierarchy. The first one evaluates the quality of the classification into classes, the second one evaluates if the classes are ranked in order relevance and the third one evaluates how effectively the learning system is able to take into account an existing hierarchical structure of the labels.

In section 7.1, the authors state

To capture the notion of partially correct, one strategy is to evaluate the average difference between the predicted labels and the actual labels for each test example, and then average over all examples in the test set. This approach is called example-based evaluations.

To capture the notion of partially incorrect in multi-label classification problems, they then use the following definition of accuracy (proposed in section 5.2 of the paper Discriminative Methods for Multi-labeled Classification)

\begin{align} \frac{1}{n} \sum_{i=1}^{n} \frac{|Y_i \cap Z_i|}{|Y_i \cup Z_i|} \label{2} \tag{2} \end{align}

where $Y_i$ is the true set of labels and $Z_i$ the predicted set of labels for the single instance (or observation) $i$, so $\frac{|Y_i \cap Z_i|}{|Y_i \cup Z_i|}$ is the accuracy for the instance $i$. If $|Y_i|=1$ and $|Z_i|=1$, then we have a single-label classification problem.

The metric \ref{2} does not correspond to the metric \ref{1}. In \ref{2}, the accuracy is calculated for each example (or instance) and then we average all these accuracies, because, for each example, you may have that one predicted label is correct, two predicted labels are correct, and so on (also depending on the number of labels you need to predict for each of the examples). In \ref{1}, there is no notion of multiple labels.