If I have the Gaussian kernel

$$ k(x, x') = \operatorname{exp}\left( -\| x - x' \|^2 / 2\sigma^2 \right) $$

What is $x$ and $x'$ in the context of training an SVM?


$\mathbf{x} \in \mathbb{R}^p$ and $\mathbf{x}' \in \mathbb{R}^p$ are two inputs (or feature vectors).

In the context of classification with an SVM, you are given a dataset $D = \{(\mathbf{x}_i, y_i) \}_{i=1}^N$, where $\mathbf{x}_i \in \mathbb{R}^p$ is an input (or point) and $y_i$ the corresponding label. The goal is to find a hyperplane that classifies the points $\mathbf{x}_i$. The hyperplane actually corresponds to a binary classifier that splits the plane into two, so the assumption is that there are two labels. However, these points $ \mathbf{x}_i$ may not be linearly separable in $\mathbb{R}^p$, i.e. there may not be a hyperplane (in 2d, i.e. when $p=2$, a hyperplane is a line) that separates them. The kernel trick, i.e. the use of kernels (such as the Gaussian radial basis), allows an SVM to perform non-linear classification by transforming the inputs to a space where they are linearly separable.

  • $\begingroup$ Just to solidify my understanding, if I take the MNIST dataset for example and reduce it to just working with 2 classes (instead of the actual 10), with images of size 28x28 and say there are 500 examples in this example dataset. I would then have a 500x28x28 dataset with corresponding y labels of 500x2. If I then wanted to apply this gaussian kernel to this dataset, what are the actual values I would plug in? $\endgroup$ – FeedMeInformation Mar 1 '20 at 15:47
  • $\begingroup$ In your example, $\mathbf{x}$ (or $\mathbf{x}'$) would be a MNIST image, which can be represented as a vector in $\mathbb{R}^{784}$ (where $784 = 28*28$). In your example, you would have $500$ examples of the form $\mathbf{x}$. The labels don't have to be a vector, they can simply be a scalar, so your labels could simply be a vector of the form $\mathbb{R}^{500}$. Regarding how you apply the kernel, this is more a detail of how the SVM works, but, to give you the intuition, essentially, you will be replacing the dot products in the SVM computations with $k(\mathbf{x}, \mathbf{x}')$. $\endgroup$ – nbro Mar 1 '20 at 15:59
  • $\begingroup$ so is x and x' two of the datapoints(image vectors) in my dataset? sorry but its just not clicking. $\endgroup$ – FeedMeInformation Mar 1 '20 at 18:08
  • $\begingroup$ @FeedMeInformation Yes. (But the application of a kernel function $k$ is not just restricted to the data points (image vectors), but, given that it is a general function, it can be applied to any pair of vectors. You can ignore this, if you don't want to know the details of SVMs. But, if you read the Wikipedia entry on SVMs, you will get a better feeling of the whole picture (even though Wikipedia may not be the most reliable source ever)). $\endgroup$ – nbro Mar 1 '20 at 18:09
  • 1
    $\begingroup$ @FeedMeInformation In the SVM formulations, you will see dot products, which, in general, are often used to calculate a notion of "distance" between vectors. The kernel function (e.g. your Gaussian radial basis) has the same purpose (in this and other contexts), i.e. it's still a way of calculating distances between vectors, but you do it in a more "sophisticated way". So, you can think of a kernel function as a function that computes a notion of "distance". $\endgroup$ – nbro Mar 1 '20 at 18:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.