Is Deep Learning the repeated application of Linear Regression?

Question

Is Deep Learning the repeated application of Linear Regression?

Answer 1

There's no point to fit a linear regression model (such as OLS) with neural network because it's really designed for non-linear models. But if you want to do that, you'll just need to set linear activation units.

Answer 2

A neural network can be reduced to a linear regression model only if we use linear activation functions (i.e. $\sigma(x) = x$), and only if we do not use any neural network specific techniques such as convolution, residuals, etc., as shown below:

$\text{neural network}(x) = \sigma_n(W_{n} \sigma_{n-1}(W_{n-1}\dots\sigma_1(W_1 x + b_1) + \dots + b_{n-1}) + b_n) \\ = W_n (W_{n-1} \dots (W_1 x + b_1) + \dots + b_{n-1}) + b_n \\ = \left( W_n W_{n-1} \dots W_1 \right) x + \left( W_n W_{n-1} \dots W_2 \right) b_1 + \left( W_n W_{n-1} \dots W_3 \right) b_2 + \dots + W_n b_{n-1} + b_n \\ = W_z x + b_z$

where $W_z = \displaystyle \prod_i W_i$ is a weight matrix and $b_z$ is some vector constant.

This follows the linear regression model form $y = Wx + b$, where $W$ is the weight matrix and $b$ is the vector constant. As a result, you can analytically solve for $W_z$ and $b_z$ using linear regression techniques, and no longer need gradient descent.

Note that for this to work well as a linear regression, you need to check for the OLS data assumptions, such as making sure the regressors have no collinearity, the residuals have no heteroskedasticity, there's no auto-regression, and that the errors are roughly normally distributed (more info). Deep neural networks with non-linear activation functions do not require these assumptions since they are universal approximators, although checking for some conditions may help make the task easier to predict (more info*).

*Note - this link talks specifically about time series data with neural network, but the same concept applies to any task in general.

The neural network specific techniques, such as ReLu, convolutions, residuals, etc., is what allows the network to learn non-linear relationships, and therefore make neural networks something more than just repeated applications of linear regression.

Answer 3

Generally speaking, you can say this:

1. there is a relationship between neural network learning (I'm assuming a "vanilla" ANN here, no CNN's or RNN's or anything) and linear/logistic regression.

2. But they're not the same thing. Just related. You could maybe consider them "cousins" to use a real-life analogy.

The big obvious difference is this: standard linear regression is, well, linear, that is, it's based on a straight line. So it can only separate points on a plane which can be separated by a straight line drawn on that plane. An ANN however, is non-linear and can fit all sorts of crazy looking curves. The reason why this is true has to do with a combination of the "activation" functions that are used, as well as the layering effect of your hidden layers.

To be fair, if you extend linear regression to be polynomial regression, you can fit more complicated curves, but that has its own downsides. And while they are also related, linear regression and polynomial regression aren't - strictly speaking - the same thing (although they may both be special cases of the same general technique).

All of that may be over-simplifying a bit. If you really want a good explanation of both linear/logistic regression and ANN's and some explanation of how they relate and differ, I recommend Andrew Ng's ML courses on Coursera. Both the original one and the new DeepLearning.ai ones.