What is the purpose of the "gamma" parameter in SVMs?

Question

I want to understand what the gamma parameter does in an SVM. According to this page.

Intuitively, the gamma parameter defines how far the influence of a single training example reaches, with low values meaning ‘far’ and high values meaning ‘close’. The gamma parameters can be seen as the inverse of the radius of influence of samples selected by the model as support vectors.

I don't understand this part "of a single training example reaches", does it refer to the training dataset?

Answer 1

I've summarized the key ideas of SVMs. So this is how $\gamma$ is used with a gaussian Kernel:

$$K_{\text{Gauss}}(\mathbf{x}_i, \mathbf{x}_j) = e^{\frac{-\gamma\|\mathbf{x}_i - \mathbf{x}_j\|^2}{2 \sigma^2}}$$

The bigger the $\gamma$, the more "linear" the decision boundary will be. The closer to 0, the more support vectors you have/the more non-linear the boundary is (see interactive example)

You can also find it on udacity: SVM Gamma Parameter

In practice, you can use a grid search or random search to get good values.

Answer 2

Rougly speaking, the higher the gamma, the more complex the model, the higher the risk of overfitting.

In fact, as you can read on the page you linked:

If gamma is too large, the radius of the area of influence of the support vectors only includes the support vector itself[...]

When gamma is very small, the model is too constrained and cannot capture the complexity or “shape” of the data. The region of influence of any selected support vector would include the whole training set.