It is a well-known math fact that composition of linear/affine transformations is still linear/affine. For a naive example,

$\textbf{A}_1\textbf{A}_2\textbf{x}$ is simply $\textbf{A}\textbf{x}$ where $\textbf{A}=\textbf{A}_1\textbf{A}_2\textbf{x}$

Any one knows why in practice multiple linear layers tend to work better, even though it is mathematically equivalent to a single linear layer? Any reference is appreciated!

  • 1
    $\begingroup$ Are you sure those "multiple linear layers" don't have nonlinear activations between them? $\endgroup$ – user76284 Oct 10 '19 at 1:54
  • $\begingroup$ can you post a link to a source that explains the result you mention. There is some confusion as to whether or not you mean that a linear network performs better than its "mathematically reduced" network or if you misunderstand network activations. So a link to a source would be helpful. $\endgroup$ – respectful Oct 10 '19 at 3:41

The key is that the layers of neurons in neural networks are not affine transformations. All commonly used neurons have some kind of non-linearity. The simplest of these is the Rectified Linear Unit (ReLU), which takes the form $y = x$ when $x > 0$ and $y = 0$ for all other values, where $x$ is a weighted sum of the inputs to the neuron.

| improve this answer | |
  • $\begingroup$ I'm wondering if he read somewhere that linear networks perform better than the reduced architecture. Perhaps someone did an experiment with a linear network and its reduced equivalent and discovered that in practice the network performs better than the reduced form. I asked OP to clarify. $\endgroup$ – respectful Oct 10 '19 at 3:44
  • 2
    $\begingroup$ @respectful Maybe. I think a lot of modern presentations show a linear algebraic presentation of neutral networks, and that for a new reader, it can be easy to miss the activation functions and their roles. $\endgroup$ – John Doucette Oct 10 '19 at 13:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.