# When do two identical neural networks have uncorrelated errors?

In Chapter 9, section 9.1.6, Raul Rojas describes how committees of networks can reduce the prediction error by training N identical neural networks and averaging the results.

If $$f_i$$ are the functions approximated by the $$N$$ neural nets, then:

$$Q=\left|\frac{1}{N}(1,1, \ldots, 1) \mathbf{E}\right|^{2}=\frac{1}{N^{2}}(1,1, \ldots, 1) \mathbf{E} \mathbf{E}^{\mathrm{T}}(1,1, \ldots, 1)^{\mathrm{T}}\tag{9.4}\label{9.4}$$ is the quadratic error of the average of the networks, where $$\mathbf{E}=\left(\begin{array}{cccc} e_{1}^{1} & e_{2}^{1} & \cdots & e_{m}^{1} \\ \vdots & \vdots & \ddots & \vdots \\ e_{1}^{N} & e_{2}^{N} & \cdots & e_{m}^{N} \end{array}\right),$$ and $$\mathbf{E}$$'s rows are the errors of the approximations of the $$N$$ functions, i.e. $$\mathbf{e}^{i} = f_i(\mathbf{x}^{i}) - t_i$$, for each of the input-output pairs $$\left(\mathbf{x}^{1}, t_{1}\right), \ldots,\left(\mathbf{x}^{m}, t_{m}\right)$$ used in training.

Is there a way to assure that the errors for a neural network are uncorrelated to the errors of the others?

Raul Rojas says that the uncorrelation of residual errors is true for a not too large $$N$$ (i.e. $$N < 4$$). Why is that?

• Maybe these notes https://people.math.umass.edu/~johnpb/s706/notes_mixed.pdf could be useful. – nbro Jan 20 at 13:51
• I think that the tag that you actually want to use is ensemble-learning and not committees-of-networks, which I've never heard of, but it's possible it exists or has been used to refer to this approach. – nbro Jan 20 at 16:12
• @nbro section 9.1.6 is dedicated to it. Ensemble seems to be mixing different networks, or models. Committees is for identical networks, trained with the same data. – EmmanuelMess Jan 20 at 16:31
• I don't think that ensemble learning is restricted to different networks. Nowadays, it's typically used to indicate that you train multiple models and then combine them somehow, but I'm also not an expert in ensemble learning, to be honest. In any case, I've just looked at that chapter 9.1.6 again and he cites [399], which is a paper entitled "When Networks Disagree: Ensemble Methods for Hybrid Neural Networks". – nbro Jan 20 at 16:36
• Regarding the link above, I don't really know if it could help you. It talks about the correlation of errors in linear regression, so I thought it could be useful, at least, to understand what R. Rojas meant by "correlation". You could also interpret neural networks as performing non-linear regression, so that's another reason why I provided the link to that pdf, which, to be honest, I have only skimmed through. – nbro Jan 20 at 16:39