0
$\begingroup$

Over the last few days I have been seeing a lot of buzz and news articles about an AI death predictor that is 'highly accurate' based on a 'life story'. Due accuracy being such a poor indicator of predictive power, I tend to be highly suspicious of claims that focus on it. For example, I can write function return True which will be extremely accurate for many questions such as, "is person X on earth?" I realize that most mainstream media articles are not going to get into more meaningful statistics so that doesn't mean it's necessarily bogus so I decided to look into the specifics of the claim more closely. I was able to access the article here from my home computer (but not through my work VPN.)

I was a little underwhelmed by what I found. First off, I cannot find a confusion matrix or values that could be used to build one. Secondly, I cannot find any reference to the "78% correctness" mentioned in some of the articles I have seen. I also don't see anything like the claims that this will predict "when you'll die" given your life story. It seems clear that many of the news articles misrepresent the actual study, but the paper itself has some puzzling aspects.

The actual prediction was whether a person would survive for the 4 years after 2016 based on a number of factors. However, I can't find the values for: true positives (TP), true negatives (TN), false positives (FP), or false negatives (FN). All I can find is the mean-corrected Matthew's correlation coefficient (C-MCC) which is given as 0.41 which ranges from -1 to 1 where a value of 1 is completely correct, 0 is random chance, and values below 0 are worse than random.

Assuming I haven't just missed something (I searched visually and on key terms,) this seems a little strange. The MCC is calculated from these values but you cannot compute those inputs if all you have is the MCC value. The closest it seems to get is a (pretty confusing, IMO) plot which seems to show a lot of incorrect predictions and kind of looks to me like it mostly predicts that older people are more likely to die.

Question: Is it suspect to exclude the values that were used to calculate the MCC? This seems pretty basic to me. At the very least I would expect a paper of this type to provide precision and recall.

$\endgroup$
2
  • $\begingroup$ Too many questions in the same post. It seems that your main concern boils down to the definition of MCC and it relation to precision, recall, true positives, etc. If your questions are just subquestions of a main question, can you please put your main specific question in the title? Thanks. $\endgroup$
    – nbro
    Jan 2 at 13:11
  • $\begingroup$ "main concern boils down to the definition of MCC and it relation to precision, recall, true positives, etc" I understand how those relate. Anyone can look that up. I'm trying to understand why the core statistics seem to be obfuscated in the paper. $\endgroup$
    – JimmyJames
    Jan 2 at 16:05

1 Answer 1

1
$\begingroup$

I don't find it inherently suspect to report results in the form of summary statistics like the MCC, F1 score, or chi-squared statistic or p-value. Space and reader attention are limited in journal articles, it is simply not possible to include all underlying data or run all possible statistical tests. You mention that you'd have been happier seeing precision and recall, but those themselves are just other choices of summary statistics from a wide range of options. Different readers want different analyses or levels of detail. I'd say it's typically good research practice to release as much underlying data as possible when publishing, but it's quite common not to, so failure to include raw confusion matrices doesn't by itself strike me as a red flag.

Now, it's always possible that the authors have purposefully omitted particular representations of their results in order to make their method look more impressive than it is. There is of course no justification for bad-faith selective presentation of the statistics, but all authors must be selective in some way when communicating to their readers.

In this particular case, I agree it might have been interesting and informative to see if there is any bias toward Type 1 or Type 2 errors. The authors found the summary statistics sufficient, however - from their domain perspective, they're not particularly interested in whether they're incorrectly predicting that survivors will die or that the soon-to-die will survive. The paper seems to be written more from the perspective of general feasibility of the topic rather than actual application, which would be fraught with ethical concerns.

$\endgroup$
8
  • $\begingroup$ "Space and reader attention are limited in journal articles, it is simply not possible to include all underlying data" They can include huge diagrams but not 4 values because of space? Really? $\endgroup$
    – JimmyJames
    Jan 2 at 18:19
  • $\begingroup$ "The paper seems to be written more from the perspective of general feasibility of the topic rather than actual application,." That's sort of the problem I'm pointing to. The paper is basically saying, "this is doable because we did it." But did they? They claim their results performed 11% better than an existing benchmark, which is what, exactly? I'm also not completely sure whether they used 2020 data in their evaluation which would definitely throw the general applicability of this into question. $\endgroup$
    – JimmyJames
    Jan 2 at 18:37
  • $\begingroup$ @JimmyJames Sometimes more data detracts from the message - the authors don't seem to care at all about Type 1 vs. Type 2 errors, so they're not going to point the reader's focus in that direction. The paper reports performance statistics from several dozen models, reporting all the underlying confusion matrices would almost surely have been overkill. I don't really see how their choice of benchmark or practical use case relates to your original question, I'm just pointing out that omitting data may (or may not) be done for practical or well-justified reasons. $\endgroup$ Jan 2 at 18:43
  • $\begingroup$ "several dozen models" I guess I missed that. How do they only end up with one MCC value, then? $\endgroup$
    – JimmyJames
    Jan 2 at 19:07
  • $\begingroup$ @JimmyJames They get one final model, but do report a lot more performance stats in the model tuning section. You could argue all that data should be reported too, but you have to draw the line somewhere. $\endgroup$ Jan 2 at 19:10

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .