1
$\begingroup$

A neural network without a hidden layer is the same as just linear regression.

If I then use squared hinge loss and encoporate the l2 regularisation term, is it fair to then call this network the same as a linear SVM?

Going by this assumption, then if I need to implement a multiclass SVM, i can just have n output nodes (where n is the number of classes). Would this then be equivalent to having n number of SVMs, similar to a one-vs-rest method?

If I then wanted to encoporate a kernel into my SVM, could I then use an activation function or layer prior to the final output nodes (where I compute loss and add regularisation) which would then transfer this data into another feature plane the same as that of an SVM kernel?

This is my current hunch, but would like some confirmation or correction where my understanding is incorrect.

Improve this question
$\endgroup$
1
  • $\begingroup$ "A neural network without a hidden layer is the same as just linear regression.", this is not true. This is only true if the activation functions are identities. So you'd better review your post and question. $\endgroup$
    – nbro
    Commented Apr 27, 2020 at 20:26

1 Answer 1

Reset to default
2
$\begingroup$

First, what makes the neural network different than linear regression is the non-linearity (activation function), not the number of layers. So, a neural network with $n$ layers with no non-linearities is still the same as linear regression. Second, SVM finds the hyperplane of maximum margin. You are not guaranteed to find that hyperplane that has maximum margin with a neural network (using non-linearities).

Improve this answer
$\endgroup$
3
  • $\begingroup$ okay fair enough, does this (see link below) then have any valid points? or will it ultimately just be an aproximation to the functionality of an SVM? stackoverflow.com/questions/54414392/… $\endgroup$
    – FeedMeInformation
    Commented Apr 27, 2020 at 21:23
  • $\begingroup$ it will be an approximation. Unless you mathematically guarantee that is the maximum margin, it is just an approximation (That's why SVM is interesting, because it guarantees maximum margin, otherwise you stay with linear regression). Whenever you use a neural network, you can't guarantee you satisfy a constraint on the hyperplane found by the neural net. The math to do that doesn't exist yet. $\endgroup$
    – Schach21
    Commented Apr 27, 2020 at 21:30
  • $\begingroup$ okay that makes sense, cheers for the insight. $\endgroup$
    – FeedMeInformation
    Commented Apr 27, 2020 at 21:35

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .