Is there any justification for excluding core statistics in 'AI Death Predictor' Paper?

Question

Over the last few days I have been seeing a lot of buzz and news articles about an AI death predictor that is 'highly accurate' based on a 'life story'. Due accuracy being such a poor indicator of predictive power, I tend to be highly suspicious of claims that focus on it. For example, I can write function return True which will be extremely accurate for many questions such as, "is person X on earth?" I realize that most mainstream media articles are not going to get into more meaningful statistics so that doesn't mean it's necessarily bogus so I decided to look into the specifics of the claim more closely. I was able to access the article here from my home computer (but not through my work VPN.)

I was a little underwhelmed by what I found. First off, I cannot find a confusion matrix or values that could be used to build one. Secondly, I cannot find any reference to the "78% correctness" mentioned in some of the articles I have seen. I also don't see anything like the claims that this will predict "when you'll die" given your life story. It seems clear that many of the news articles misrepresent the actual study, but the paper itself has some puzzling aspects.

The actual prediction was whether a person would survive for the 4 years after 2016 based on a number of factors. However, I can't find the values for: true positives (TP), true negatives (TN), false positives (FP), or false negatives (FN). All I can find is the mean-corrected Matthew's correlation coefficient (C-MCC) which is given as 0.41 which ranges from -1 to 1 where a value of 1 is completely correct, 0 is random chance, and values below 0 are worse than random.

Assuming I haven't just missed something (I searched visually and on key terms,) this seems a little strange. The MCC is calculated from these values but you cannot compute those inputs if all you have is the MCC value. The closest it seems to get is a (pretty confusing, IMO) plot which seems to show a lot of incorrect predictions and kind of looks to me like it mostly predicts that older people are more likely to die.

Question: Is it suspect to exclude the values that were used to calculate the MCC? This seems pretty basic to me. At the very least I would expect a paper of this type to provide precision and recall.

Answer 1

I don't find it inherently suspect to report results in the form of summary statistics like the MCC, F1 score, or chi-squared statistic or p-value. Space and reader attention are limited in journal articles, it is simply not possible to include all underlying data or run all possible statistical tests. You mention that you'd have been happier seeing precision and recall, but those themselves are just other choices of summary statistics from a wide range of options. Different readers want different analyses or levels of detail. I'd say it's typically good research practice to release as much underlying data as possible when publishing, but it's quite common not to, so failure to include raw confusion matrices doesn't by itself strike me as a red flag.

Now, it's always possible that the authors have purposefully omitted particular representations of their results in order to make their method look more impressive than it is. There is of course no justification for bad-faith selective presentation of the statistics, but all authors must be selective in some way when communicating to their readers.

In this particular case, I agree it might have been interesting and informative to see if there is any bias toward Type 1 or Type 2 errors. The authors found the summary statistics sufficient, however - from their domain perspective, they're not particularly interested in whether they're incorrectly predicting that survivors will die or that the soon-to-die will survive. The paper seems to be written more from the perspective of general feasibility of the topic rather than actual application, which would be fraught with ethical concerns.