My understanding is that neural networks are definitely not linear classifiers, as the point of functions like ReLU is to introduce non-linearity.
However, here's where my understanding starts to break down. A classifier, like Softmax or SVM is considered to be a "linear classifier". I'm using the definitions from CS 231N. There may be another definition of an SVM, but I'm not considering that.
In a linear classifier we have the following: $W \cdot x$ where
- $W$ is the weights
- $x$ is the input vector.
The output is a vector of scores. 1 score per label.
An SVM classifier uses hinge loss to update the weights:
A softmax classifier normalizes the output values using the softmax function and then uses cross-entropy loss to update the weights:
From the lecture CS231n Winter 2016: Lecture 2: Data-driven approach, kNN, Linear Classification 1, we have the following image to help visualize what a linear classifier does:
Essentially, each image can be considered a point in 3072 dimensional space, and we are drawing lines through this space. On one side of the "car" line, the car score for a given point will go up. On the other side of the "car" line, the car score for a given point will go down.
However, this doesn't seem to be that much different from ReLU (Taken from this post: https://stats.stackexchange.com/questions/158549/why-are-activation-functions-needed-in-neural-networks):
So what is the fundamental difference between a "linear" classifier like Softmax / SVM and a multi-layer neural network? Why can't a SVM classifier learn any function but a neural network can?