The objective function of an SVM is the following:
$$J(\mathbf{w}, b)=C \sum_{i=1}^{m} \max \left(0,1-y^{(i)}\left(\mathbf{w}^{t} \cdot \mathbf{x}^{(i)}+b\right)\right)+\frac{1}{2} \mathbf{w}^{t} \cdot \mathbf{w}$$ where
- $\mathbf{w}$ is the model's feature weights and $b$ is its bias parameter
- $\mathbf{x}^{(i)}$ is the $i^\text{th}$ training instance's feature vector
- $y^{(i)}$ is the target class ($-1$ or $1$) for the $i^\text{th}$ instance
- $m$ is the number of training instances
- $C$ is the regularisation hyper-parameter
And if I was to use a kernel, this would become:
$$J(\mathbf{w}, b)=C \sum_{i=1}^{m} \max \left(0,1-y^{(i)}\left(\mathbf{u}^{t} \cdot \mathbf{K}^{(i)}+b\right)\right)+\frac{1}{2} \mathbf{u}^{t} \cdot \mathbf{K} \cdot \mathbf{u}$$
where the kernel can be the Gaussian kernel:
$$K(\mathbf{u}, \mathbf{v})=e^{-\gamma\|\mathbf{u}-\mathbf{v}\|^{2}}$$
How would I go about finding its gradient with respect to the input?
I need to know this as to then apply this to a larger problem of a CNN with its last layer being this SVM, so I can then find the gradient of this output wrt the input of the CNN.
max
you mean the actual maximum between two values, you may be in trouble since the gradient of the maximum is not smooth and hence not differentiable. So you can only compute the gradient as long as you have1-yy_pred !=0
. $\endgroup$ – FirefoxMetzger Apr 11 '20 at 9:23