What are the domains where SVMs are still state-of-the-art?

Question

It seems that deep neural networks and other neural network based models are dominating many current areas like computer vision, object classification, reinforcement learning, etc.

Are there domains where SVMs (or other models) are still producing state-of-the-art results?

Answer 1

State-of-the-art is a tough bar, because it's not clear how it should be measured. An alternative criteria, which is akin to state-of-the-art, is to ask when you might prefer to try an SVM.

SVMs have several advantages:

1. Through the kernel trick, the runtime of an SVM does not increase significantly if you want to learn patterns over many non-linear combinations of features, rather than the original feature set. In contrast, a more modern approach like a deep neural network will need to get deeper or wider to model the same patterns, which will increase its training time.
2. SVMs have an inherent bias towards picking "conservative" hypotheses, that are less likely to overfit the data, because they try to find maximum margin hypotheses. In some sense, they "bake-in" Occam's razor.
3. SVMs have only two hyperparameters (the choice of kernel and the regularization constant), so they are very easy to tune to specific problems. It is usually sufficient to tune them by performing a simple grid-search through the parameter space, which can be done automatically.

SVMs also have some disadvantages:

1. SVMs have a runtime that scales cubically in the number of datapoints you want to train on (i.e. $O(n^3)$ runtime)¹. This does not compare well with, say, a typical training approach for a deep neural network which runs in $O(w*n*e)$ time, where $n$ is the number of data points, $e$ is the number of training epochs, and $w$ is the number of weights in the network. Generally $w, e << n$.
2. To make use of the Kernel trick, SVMs cache a value for the kernelized "distance" between any two pairs of points. This means they need $O(n^2)$ memory. This is far, far, more trouble than the cubic runtime on most real-world sets. More than a few thousand datapoints will leave most modern servers thrashing, which increases effective runtime by several orders of magnitude. Together with point 1, this means SVMs will tend to become unworkably slow for sets beyond maybe 5,000-10,000 datapoints, at the upper limit.

All of these factors point to SVMs being relevant for exactly one use case: small datasets where the target pattern is thought, apriori, to be some regular, but highly non-linear, function of a large number of features. This use case actually arises fairly often. A recent example application where I found SVMs to be a natural approach was building predictive models for a target function that was known to be the result of interactions between pairs of features (specifically, communications between pairs of agents). An SVM with a quadratic kernel could therefore efficiently learn conservative, reasonable, guesses.

---

¹ There are approximate algorithms that will solve the SVM faster than this, as noted in the other answers.

Answer 2

Deep Learning and Neural Networks are getting most of the focus because of recent advances in the field and most experts believe it to be the future of solving machine learning problems.

But make no mistake, classical models still produce exceptional results and in certain problems, they can produce better results than deep learning.

Linear Regression is still by far the most used machine learning algorithm in the world.

It’s difficult to identify a specific domain where classical models always perform better as the accuracy is very much determined on the shape and quality of the input data.

So algorithm and model selection is always a trade-off. It’s a somewhat accurate statement to make that classical models still perform better with smaller data sets. However, a lot of research is going into improving deep learning model performance on less data.

Most classical models require less computational resources so if your goal is speed then its much better.

Also, classical models are easier to implement and visualize which can be another indicator for performance, but it depends on your goals.

If you have unlimited resources, a massive observable data set that is properly labeled and you implement it correctly within the problem domain then deep learning is likely going to give you better results in most cases.

But in my experience, the real-world conditions are never this perfect

Answer 3

Totally agree with @John's answer. Will try and complement that with some more points.

Some advantages of SVMs:

a) SVM is defined by a convex optimisation problem for which there are efficient methods to solve, like SMO.

b) Effective in high dimensional spaces and also in cases where number of dimensions is greater than the number of samples.

c) Uses a subset of training points in the decision function (called support vectors), so it is also memory efficient.

d) Different Kernel functions can be specified for the decision function.. In its simplest form, the kernel trick means transforming data into another dimension that has a clear dividing margin between classes of data.

The disadvantages of support vector machines include:

a) If the number of features is much greater than the number of samples, avoiding over-fitting in choosing Kernel functions and regularization term is crucial. Kernel models can be quite sensitive to over-fitting the model selection criterion

b) SVMs do not directly provide probability estimates. In many classification problems you actually want the probability of class membership, so it would be better to use a method like Logistic Regression, rather than post-process the output of the SVM to get probabilities.

Answer 4

For datasets of low-dimensional tabular data. DNN are not efficient on low-dimensional input because of huge overparametrisation. So even if dataset is huge in size, but each sample is low-dimensional SVM would beat DNN.

More generally if data is tabular and the correlation between fields of the sample is weak and noisy, SVM may still beat DNN even for high dimensional data, but that depend on specific of data.

Unfortunately I can't recall any specific papers on subject, so it's mostly common sense reasoning, you don't have to trust it.