Are any AI systems available, or in development, for finding and analysing fallacious inference in natural language text?

Question

Poor reasoning, and ignorance in general, is the source of a lot of suffering and evil. Covertly erroneous logic is often used in manipulation. And much of this broken thought is being used directly in the training of AI.

There has been talk of, and development in, fact-checking, such as for language transformers. But what about reasoning?

The function in mind is specifically being able to process a potentially large text body, analysing all logic and implied relations for fallacy and other misleading reasoning. Perhaps shades of colour could indicate level of error. A bonus would be output listing and explaining the mistakes, maybe like compiler errors -- "fallacy x between premise y and conclusion z".

Are any AI systems available, or in development, for finding and analysing fallacious inference in natural language text?

Answer 1

There's some impressive work going on at IBM under the name of "Project Debater", which already produced some impressive results, that you can check in this video.

The project is about creating a system capable of debating with human on random topics by scraping the internet to get ground knowledge (mainly scientific papers, not random gibberish of course). The system is pretty massive, it include speech synthesis and lot of other components, what's interesting for you are the parts I highlighted in red, especially those on the left that contribute to generate the counter arguments to the human opponent (rebuttal construction).

I won't go into the details cause all those three topics constitute entire branches of Natural Language Processing. But you can easily see how combined together they try to accomplish a rough version of deductive reasoning:

• claim detection: identifying the main argument of a sentence/document
• evidence detection: identifying what facts are presented along with the argument
• stance detection: identifying the positive/neutral/negative stance of each evidence (and possibly opinion based facts presented) toward the claim has been made.

Assuming that we managed to extract all this information from a text (big time, unfortunately not easy at all) then we can build reasoning out of it. For example:

• Public schools are worse than private ones for children education -> detected negative claim
• Researches show that there's an equal amount of graduated students coming from public and private schools. -> detected evidence
• The previous evidence has a positive stance toward the initial claim -> stance detection

Putting everything together, since we have a negative claim followed by a positive evidence toward the same fact, we just found out that the document contains a contradiction.

Answer 2

Yes, current large language models since GPT-4 are able to find errors in reasoning, probably zero-shot, not much prompt engineering needed.

Giving a precise probability for something being wrong is much harder, even if the the model knows it, it is hard to query. Rough estimates should be easy.

---

Some examples using GPT-4:

Prompt:

Alice had blue shoes. Therefore, Bob had to eat the cake.

Analyze the reasoning.

Result:

The reasoning presented in the statement is illogical and does not follow a coherent thought process. There is no apparent connection or correlation between Alice having blue shoes and Bob having to eat the cake. The conclusion does not follow from the premise, and the argument lacks a proper cause-and-effect relationship.

Starting a new chat:

Prompt:

Alice had birthday cake. Therefore, Alice had to eat the cake.

Result:

It is not necessarily true that Alice had to eat the cake just because she had a birthday cake. While it is common for people to eat cake on their birthdays, Alice could have chosen not to eat it, shared it with others, or saved it for later.

Starting a new chat:

Prompt:

Alice had blue shoes. Therefore, Alice had to shoe the blues.

Result:

This statement is a play on words. "Alice had blue shoes" means that Alice owned a pair of blue-colored shoes. "Shoe the blues" is a pun on the phrase "sing the blues," which means to express one's troubles or sadness, often through music. In this case, the pun implies that because Alice had blue shoes, she had to express her sadness or troubles, possibly through wearing her blue shoes.

These examples are not cherry picked.