# What are Support Vector Machines?

What are Support Vector Machines? Is an SVM a kind of a neural network, meaning it has nodes and weights, etc.?

What is it best used for?

Where I can find information about these?

I find the chapter on machine learning from Russell & Norvig is a pretty good place to start with SVMs. I think this is Chapter 18?

One way to understand an SVM is as a kind of neural network, but this is not usually an intuitive approach for a beginner (unless your NN knowledge is already quite good).

A better way to understand SVMs is as consisting of three simple ideas rolled into one algorithm. Here's an attempt at a "For Dummies" answer though:

1. Maximum Margin Classification. SVMs are usually used to find a pattern in a set of data. Often, the data allow an infinite set of possible patterns that are all equally descriptive. For example, maybe The relationship is "Lives within 5 miles of a Coast -> Income High". It's easy to imagine that this pattern is just as good as "Lives within 5.0001 miles of a Coast -> Income High" or "Lives within 4.999 miles of a Coast -> Income High". There might actually be a lot more play than that in the data (e.g. 3 miles might work out too). If all these are equally good, then the maximum margin idea says you should pick the one that's "in the middle" of the data. So maybe all values between 5.5 and 4.8 are equally good. In that case, we might pick 5.15 (in the middle). This example is super simplified. Real world data would have a lot more variables, and the idea of "in the middle" ends up being a little more complex, but this is the intuition. It turns out that finding the maximum margin pattern is easy when the patterns are linear. That is, when they can be represented by drawing straight lines through a plot of the dataset.

2. Projection into higher dimensions. This one needs a bit of math to visualize. Consider a dataset consisting of a circular pattern (for instance, maybe the pattern is that higher incomes are found in the middle of the city). There is no linear relationship that captures this pattern. That is, you can't draw a straight line through the data, and say something meaningful about all the values on one side or the other. However, if you add a new feature to your data that is the square of the original coordinates, it's easy to find such a pattern. Basically, if you pre-compute "circular" functions of the original data, you can add them to the dataset, and then find a pattern that is a linear function of this new feature. This idea generalizes: if you compute a complex enough function of your original data, and then apply the maximum margin idea, you can learn any pattern you like. The problem is that it's slow: adding more features makes it take longer to find the patterns you want.

3. The Kernel Trick. The thing that made SVMs useful was the kernel trick: finding the maximum margin didn't depend on anything except the product of the coordinates of the various points. It turned out that this product could be computed first, and then run through certain functions to produce a problem that was identical to the one you'd get by first adding extra features and then doing the multiplication. However, computing the problem this way didn't require adding any new features! This made SVMs one of the first reliable, well understood, and fast methods for finding non-linear patterns in data.

Hope that provides a starting point. Consider reading Russell & Norvig as a next starting point, or Bishop if you want to go deeper.

• Excellent explanation! So, from my understanding of (2), SVMs do not only classify data that are separable by a single line but they classify also data that could be inside/outside of a circle (any kind of pattern)? Is their output a kind of a binary classification (meaning that the data belongs to a specific class or not) or it can classify the data in more than 2 (3, 4, 5, etc.) categories? eg. "Lives within 1mile of coast -> Income High, lives from 1 to 2 miles from coast -> Income medium, lives to more than 2 miles of coast -> Income Low). – ekalyvio Aug 3 '18 at 12:45
• Sort of. They're a combination of all three ideas. So the key is that they can find classification patterns that are non-linear (any kind of pattern), but they do this by turning that problem into the problem of finding a line in a different space. Regular SVMs do binary classification only, but there are lots of variations, some of which do multi-label or multi-class classification. You can also use something like 1vA classification to achieve the same thing (see Rifkin & Klautau: jmlr.org/papers/volume5/rifkin04a/rifkin04a.pdf). – John Doucette Aug 3 '18 at 13:33

In case this is still relevant to you I can share my tutorial on SVM with Python implementation in Jupyter notebook:

Primer to support vector machines

The tutorial assumes some mathematics and programming background knowledge. The SVM codes utilize no external machine learning packages and tries to teach the reader to build a SVM model him-/herself.

I hope it helps you!

Support vector machines are supervised learning models with associated learning algorithms that analyze data and are used for classification and regression analysis.

Here is a link where you can learn more about it from a introduction level: "Support Vector Machine — Introduction to Machine Learning Algorithms" (Medium)