# Any explanation why multiple linear layers work better than a single linear layer in practice?

It is a well-known math fact that composition of linear/affine transformations is still linear/affine. For a naive example,

$$\textbf{A}_1\textbf{A}_2\textbf{x}$$ is simply $$\textbf{A}\textbf{x}$$ where $$\textbf{A}=\textbf{A}_1\textbf{A}_2\textbf{x}$$

Any one knows why in practice multiple linear layers tend to work better, even though it is mathematically equivalent to a single linear layer? Any reference is appreciated!

• Are you sure those "multiple linear layers" don't have nonlinear activations between them? – user76284 Oct 10 '19 at 1:54
• can you post a link to a source that explains the result you mention. There is some confusion as to whether or not you mean that a linear network performs better than its "mathematically reduced" network or if you misunderstand network activations. So a link to a source would be helpful. – respectful Oct 10 '19 at 3:41

The key is that the layers of neurons in neural networks are not affine transformations. All commonly used neurons have some kind of non-linearity. The simplest of these is the Rectified Linear Unit (ReLU), which takes the form $$y = x$$ when $$x > 0$$ and $$y = 0$$ for all other values, where $$x$$ is a weighted sum of the inputs to the neuron.