How could facts be distinguished from opinions?

Question

As a software engineer, I am searching for an existing solution or, if none exists, willing to create one that will be able to process texts (e.g. news from online media) to extract/paraphrase dry facts from them, leaving all opinions, analysis, speculations, humor, etc., behind.

If no such solution exists, what would be a good way to start creating it (considering that I have zero experience in AI/machine learning)?

It would be no problem to manually create a set of examples (pairs of original news + dry facts extracted), but is that basically what it takes? I doubt so.

(This knowledge domain is already huge, so which parts of it need to be learned first and foremost to figure out how to achieve the goal?)

Answer 1

I will be starting my PhD in natural language processing in a few days and this is very similar to my proposed topic. It's an open problem that ties NLP and AI into philosophy of science and epistemology and is, I think, extremely interesting. I say all this to drive home the point that this is not a simple problem.

Two major theoretical concerns come to my mind:

1. What is a "fact"? Is it a universal truth, if there is such a thing? Or is it a generally accepted theory, and if so how do you measure acceptance? That is, accepted by whom, where, when?

2. Are there any linguistic markers of opinions vs. facts? Only in rare cases, such as when the speaker prefaces their statement with something like "I believe". In most cases, I think, opinions will be stated linguistically similarly to facts. For example, compare "Cats are felines." (a "fact") with "Cats are aliens." (an opinion some may hold). They have the exact same syntactic structure. The difference here is deeply semantic, and probably relates to the speaker's intention. I'd venture that often people state their opinions with the intention of communicating a "fact".

Some more practical concerns are:

1. Information extraction (also called relationship extraction, text mining, etc.), which for the most part assumes that the "facts" given in the labeled datasets are correct, is far from a solved problem. E.g. the state of the art model developed for a task released in 2010 has an F1 of only 76! What you propose adds significant uncertainty to these types of tasks.

2. I suspect that even if you were able to compile a dataset of facts and opinions with corresponding labels you would encounter a number of modeling problems. Given the linguistic similarity between the statements of facts and opinions, I'd guess that your model will simply memorize the dataset, making it generalize poorly to your test set. Either that or it would would pick up on random, hidden correlations in the data to solve the problem (neural nets are really good at this), perhaps generalizing to the test set, but failing to apply to any other data.

3. Fact vs. opinion is something that is embedded in a cultural milieu, so a model would, I think, need access to some proxy for what is culturally accepted in order to make this distinction, perhaps a via knowledge base. This may be feasible for limited, highly curated domains (e.g. biomedicine), but there is currently nothing suitable for a general-purpose fact finder.

tldr: No, it is not enough to simply create a dataset of facts vs. opinions. This problem poses major theoretical concerns related to epistemology, linguistics, and cognitive science. Additionally, there are more mundane (but non-trivial!) modeling issues to consider. @Sceptre is right that it will be impossible to start this without knowledge of AI/ML/NLP, especially a rather deep knowledge of what current AI systems are really capable of.