In section 13.2 Other Multilayer Machines (pp. 231-232) of the book Perceptrons: An Introduction to Computational Geometry (expanded edition, third printing, 1988) Minsky and Papert actually talk about their knowledge of or opinions about the capabilities of what they call the multilayered machines (i.e. perceptrons with many layers or MLPs).
Have you considered "perceptrons" with many layers?
Well, we have considered Gamba machines, which could be described as "two layers of perceptron". We have not found (by thinking or by studying the literature) any other really interesting class of multilayered machine, at least none whose principles seem to have a signiﬁcant relation to those of the perceptron. To see the force of this qualiﬁcation it is worth pondering the fact, trivial in itself, that a universal computer could be built entirely out of linear threshold modules. This does not in any sense reduce the theory of computation and programming to the theory of perceptrons. Some philosophers might like to express the relevant general principle by saying that the computer is so much more than the sum of its parts that the computer scientist can aﬁord to ignore the nature of the components and consider only their connectivity. More concretely, we would call the student's attention to the following considerations:
Multilayer machines with loops clearly open all the questions of the general theory of automata.
A system with no loops but with an order restriction at each layer can compute only predicates of ﬁnite order.
On the other hand, if there is no restriction except for the absence of loops, the monster of vacuous generality once more raises its head.
The problem of extension is not merely technical. It is also strategic. The perceptron has shown itself worthy of study despite (and even because of!) its severe limitations, It has many features to attract attention: its linearity; its intriguing learning theorem; its clear paradigmatic simplicity as a kind of parallel computation. There is no reason to suppose that any of these virtues carry over to the many-layered version. Nevertheless, we consider it to be an important research problem to elucidate (or reject) our intuitive judgment that the extension is sterile. Perhaps some powerful convergence theorem will be discovered, or some profound reason for the failure to produce an interesting
"learning theorem" for the multilayered machine will be found.