My understanding is that neural networks are definitely not linear classifiers, as the point of functions like ReLU is to introduce non-linearity.

However, here's where my understanding starts to break down. A classifier, like Softmax or SVM is considered to be a "linear classifier". I'm using the definitions from CS 231N. There may be another definition of an SVM, but I'm not considering that.

In a linear classifier we have the following: $W \cdot x$ where

  • $W$ is the weights
  • $x$ is the input vector.

The output is a vector of scores. 1 score per label.

An SVM classifier uses hinge loss to update the weights:

enter image description here

A softmax classifier normalizes the output values using the softmax function and then uses cross-entropy loss to update the weights:

enter image description here

From the lecture CS231n Winter 2016: Lecture 2: Data-driven approach, kNN, Linear Classification 1, we have the following image to help visualize what a linear classifier does:

Image about linear classifiers

Essentially, each image can be considered a point in 3072 dimensional space, and we are drawing lines through this space. On one side of the "car" line, the car score for a given point will go up. On the other side of the "car" line, the car score for a given point will go down.

However, this doesn't seem to be that much different from ReLU (Taken from this post: https://stats.stackexchange.com/questions/158549/why-are-activation-functions-needed-in-neural-networks):

enter image description here

So what is the fundamental difference between a "linear" classifier like Softmax / SVM and a multi-layer neural network? Why can't a SVM classifier learn any function but a neural network can?

  • $\begingroup$ In my view neural nets are more universal than SVMs: they can approximate any functions from one multi-dimensional space to another. They are often used as classifiers by adding SoftMax or some other trick. SVMs, on the other hand, are purely classifiers, but they can do non-linear classification using so called "kernel trick" (en.wikipedia.org/wiki/Support-vector_machine). $\endgroup$
    – blamocur
    Commented Jun 23, 2022 at 13:52
  • $\begingroup$ @blamocur Ideally, our site is not meant to share "views" or "opinions" but facts and evidence. Just because there's a lot of hype around NNs now, it doesn't mean that they are more universal than SVMs, whatever that means. See also this. $\endgroup$
    – nbro
    Commented Jun 23, 2022 at 13:58

1 Answer 1


I was confused because the images look similar even though in reality the problems the 2 images are solving are completely different:

  • The first image shows a linear classifier assigning scores for each of $C$ classes to an input image
  • The second image shows a binary classifier (I think) scoring an item as ether white or purple (or something like that)

Consider a single class in the linear classifier. The linear classifier can only draw a single line through $N$ dimensional space. Items on one side of this line have negative car scores. Items on the other side have positive car scores.

For example: enter image description here

Here, the 2nd image in the row represents the SVM's attempt to extract all the features of a car into a single image. As images are "closer" to this image, the SVM will give them a higher car score. As images are farther away from this image, the SVM will give them a lower car score.

The SVM cannot extract individual features, it has to smush them all into a single line in $D$ dimensional space (where $D$ is the number of pixels). That line can be represented as:

enter image description here

On the other hand. A multi-layer neural network, for example: enter image description here

Can now partition the space into a 100 different ways. The example is a bit convoluted, since the neural net is classifying 1 thing into several buckets instead of doing binary classification, but the principle is the same. Maybe 10 out of a 100 neurons in the second layer are dedicated to partitioning the space of images with regard to cars.


You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .