The objective function of an SVM is the following:

$$J(\mathbf{w}, b)=C \sum_{i=1}^{m} \max \left(0,1-y^{(i)}\left(\mathbf{w}^{t} \cdot \mathbf{x}^{(i)}+b\right)\right)+\frac{1}{2} \mathbf{w}^{t} \cdot \mathbf{w}$$ where

  • $\mathbf{w}$ is the model's feature weights and $b$ is its bias parameter
  • $\mathbf{x}^{(i)}$ is the $i^\text{th}$ training instance's feature vector
  • $y^{(i)}$ is the target class ($-1$ or $1$) for the $i^\text{th}$ instance
  • $m$ is the number of training instances
  • $C$ is the regularisation hyper-parameter

And if I was to use a kernel, this would become:

$$J(\mathbf{w}, b)=C \sum_{i=1}^{m} \max \left(0,1-y^{(i)}\left(\mathbf{u}^{t} \cdot \mathbf{K}^{(i)}+b\right)\right)+\frac{1}{2} \mathbf{u}^{t} \cdot \mathbf{K} \cdot \mathbf{u}$$

where the kernel can be the Gaussian kernel:

$$K(\mathbf{u}, \mathbf{v})=e^{-\gamma\|\mathbf{u}-\mathbf{v}\|^{2}}$$

How would I go about finding its gradient with respect to the input?

I need to know this as to then apply this to a larger problem of a CNN with its last layer being this SVM, so I can then find the gradient of this output wrt the input of the CNN.

  • $\begingroup$ if by max you mean the actual maximum between two values, you may be in trouble since the gradient of the maximum is not smooth and hence not differentiable. So you can only compute the gradient as long as you have 1-yy_pred !=0. $\endgroup$ – FirefoxMetzger Apr 11 '20 at 9:23
  • 1
    $\begingroup$ I found some work done on this here: ecmlpkdd2013.org/wp-content/uploads/2013/07/527.pdf They talk about something similar in Section 3.1. Does it hold any weight with what I want to do? $\endgroup$ – FeedMeInformation Apr 11 '20 at 9:34
  • $\begingroup$ @FeedMeInformation Have you found a solution to this problem? $\endgroup$ – nbro May 16 '20 at 16:48
  • $\begingroup$ See also ai.stackexchange.com/q/8281/2444. $\endgroup$ – nbro May 18 '20 at 14:32
  • $\begingroup$ Have you found the solution/answer to this problem/question? If yes, feel free to write it below. If not, I will consider starting a bounty on this post, if you still need an answer. $\endgroup$ – nbro Jun 25 '20 at 12:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.