# What is the gradient of a non-linear SVM with respect to the input?

The objective function of an SVM is the following:

$$J(\mathbf{w}, b)=C \sum_{i=1}^{m} \max \left(0,1-y^{(i)}\left(\mathbf{w}^{t} \cdot \mathbf{x}^{(i)}+b\right)\right)+\frac{1}{2} \mathbf{w}^{t} \cdot \mathbf{w}$$ where

• $$\mathbf{w}$$ is the model's feature weights and $$b$$ is its bias parameter
• $$\mathbf{x}^{(i)}$$ is the $$i^\text{th}$$ training instance's feature vector
• $$y^{(i)}$$ is the target class ($$-1$$ or $$1$$) for the $$i^\text{th}$$ instance
• $$m$$ is the number of training instances
• $$C$$ is the regularisation hyper-parameter

And if I was to use a kernel, this would become:

$$J(\mathbf{w}, b)=C \sum_{i=1}^{m} \max \left(0,1-y^{(i)}\left(\mathbf{u}^{t} \cdot \mathbf{K}^{(i)}+b\right)\right)+\frac{1}{2} \mathbf{u}^{t} \cdot \mathbf{K} \cdot \mathbf{u}$$

where the kernel can be the Gaussian kernel:

$$K(\mathbf{u}, \mathbf{v})=e^{-\gamma\|\mathbf{u}-\mathbf{v}\|^{2}}$$

How would I go about finding its gradient with respect to the input?

I need to know this as to then apply this to a larger problem of a CNN with its last layer being this SVM, so I can then find the gradient of this output wrt the input of the CNN.

• if by max you mean the actual maximum between two values, you may be in trouble since the gradient of the maximum is not smooth and hence not differentiable. So you can only compute the gradient as long as you have 1-yy_pred !=0. – FirefoxMetzger Apr 11 '20 at 9:23
• I found some work done on this here: ecmlpkdd2013.org/wp-content/uploads/2013/07/527.pdf They talk about something similar in Section 3.1. Does it hold any weight with what I want to do? – FeedMeInformation Apr 11 '20 at 9:34
• @FeedMeInformation Have you found a solution to this problem? – nbro May 16 '20 at 16:48