Selecting the right algorithm to predict disease from questions

Question

I'm trying to come up with the right algorithm for a system in which the user enter symptoms and we starts triggering questions related to that symptoms and his answers will result a disease which is related to the answer which is given by the user

Let's assume that the user entered the following input:

Symptom - Deafness

Q1. How long have you had a problem with deafness

A)From few days B)From few weeks to months C)More than month D)Since Birth

Q2. What was the onset of the deafness

A) Sudden B) Gradual

Now we have a knowledge base like if a user select option 1 from question 1 and option 2 form question 2 then we will give him some disease. But i need an algorithm which will give % of success in backend so that i can throw the results of disease for example if a user select option 2 from question1 and option 1 from question 2, then when we compare from our knowledge base there will be one set over there which has option 1 from question 1 and option 2 from question 2 then its a "SOME" disease.. now if we compare from our knowledgebase and we found even 50% of the choices is resulting this disease we will throw that disease name.

NOW i am confused what algo should be use for this approach for ai.

Answer 1

There is no defined rules for choosing a machine learning algorithm to learn some type of pattern. However, there are some guidelines to help you better select an algorithm which will yield a higher probability of success.

Some important considerations are:

• Number of features: This is the number of questions that each patient had to answer.
• Number of instances: This is the number of patients that took your survey.
• Number of output classes: This is how specific you want your diagnosis of the disease. Is this a yes/no, or a 5-stage progression.

The larger your feature space and the more output classes you have, the higher the complexity of your model. This is problematic because a more complex model will require more instances of data to learn the underlining patterns. You need to have a good balance between the number of examples you have in your dataset and the complexity of your model.

If you have limited data then you will want to stay very far away from deep learning. In such cases, I prefer to use shallow methods such as SVM, Naive Bayes or Random Forests. These techniques have been shown to be able to capture non-linear relationships. SVM is particularly powerful, you can use the kernel trick! This will transform your feature space into a space that makes the differentiation between the classes easier to distinguish. Do not underestimate the power of this algorithm.

If you have a very large dataset (i.e. 100,000 instances) then you will want to use deep learning. Recently, these techniques have been shown to outperform shallow machine learning algorithms in almost all categories where data is plentiful. You will want to start with a shallow neural network and then increase the complexity of your graph as you see fit. By adding additional nodes per layer, or by adding additional layers. Each node you add will increase the number of variables you need to tune, thus increasing the complexity of your model. If your data is expected to have some type of time dependency which often the case in medical data. Then you can use a long-short term memory neural network (LSTM). These are capable of capturing latent relationships. You can also try a stacked auto-encodder.

Answer 2

Medical diagnosis often employs abductive inference (also known as "inference to the best explanation"), and automated approaches to abductive reasoning have been applied to medical diagnosis. More concretely, a mechanism known as Parsimonious Covering Theory has been extensively researched as an approach to automated abductive reasoning in medical scenarios (among others). You might consider giving that a look.

FWIW, one of the main researchers in the field is Professor James Reggia from UMD. He and Yun Peng wrote a very accessible book on the subject - Abductive Inference Models for Diagnostic Problem Solving.

And to add to what JahKnows says - I'd say that where something like deep-learning would be most likely to come into play, would be building the initial knowledgebase that is used in the diagnostic system. That is, if you have lots of data relating symptoms to diseases, you could use deep learning to help define the weighting between specific diseases and specific symptoms. And note that while abductive inference is considered a branch of logic, and PCT dates back to the "GOFAI" era, it is not a strictly symbolic approach. The Reggia & Peng book explains how to incorporate Bayesian reasoning into the abductive system. Just FYI.

Disclosure: I've recently been researching this area and am working on a modern implementation of PCT using the Semantic Web stack.