How do I determine the most appropriate classifier for a certain problem?

Question

Consider a Bayesian classifier used in spam e-mail filtering. It converts an e-mail to a vector, most of the time using the bag-of-words method. Although it learns first before getting employed, it can be made to work as an online system, i.e. it can be used to filter and learn from examples even after deployment.

Now, on the other hand, now comes the perceptron. It calculates a mean vector of spam and not spam, and then classifies them into the appropriate categories. The model adjusts the mean vectors each time it makes mistakes.

Now, comes neural nets, they too are capable of taking a vector-like bag of words or image pixels of dogs and cats and classify them into yes or no.

So, while designing and implementing them into the system, how to determine which one of the methods (Bayesian classifier, perceptron or neural network) is the most appropriate for a given situation or task? One factor to consider is the time complexity (or speed), but what are other factors, and how to rank them?

Answer 1

This is one of the main skills that separates someone with a deep understanding of, and experience in, machine learning learning, from a neophyte. There are several approaches:

1. Try several methods, perhaps with automated hyperparameter optimization, and see if there's a big difference in typical model quality. This is pretty common if you don't have a lot of experience, but also something experts may try in a more targeted way.

2. Visualize the shape of your problem, perhaps by using a dimensionality reduction technique like PCA or tSNE, or maybe an auto-encoder. If you compress the data to 2d, are there clear linear patterns? Maybe try a linear model like logistic regression. Are there several distinct groups? How many lines would you need to draw to separate them? If it's a lot, maybe you're going to need a very non-linear model. If it's just a few, maybe a small multi-layer perceptron can help. If there are spiral bands or circular shapes, maybe and SVM with a non-linear kernel. Knowing how to translate the visualizations into intuition about the kinds of model that can help is an advanced skill. You need to understand what shapes of patterns each kind of model can learn to fit, and how these do or do not translate into a higher dimensional space.

3. Read the literature. If you're working in computer vision, you should try a CNN. Why? Well, everyone else is using them. They work great on most computer vision problems. They hold most competition records. It'd be silly not to try them. If you're working on spam classification though, CNNs are a bad choice. People use compression classifiers, Bayesian models, and sometimes multi-stage models to build complex features. If you look at the recent literature in your area, you can tell what to use. Reading and understanding this literature well enough to interpret it usually requires more advanced scientific training in ML and/or the application area.

As a final note, the No Free Lunch theorems for ML tell us that this is always going to be an art (at least, that's one interpretation experts argue for), and not a science, so the more practice you get with it, the better you'll become.