The current machine learning trend is interpreted by some new to the disciplines of AI as meaning that MLPs, CNNs, and RNNs can exhibit human intelligence. It is true that these orthogonal structures derived from the original perceptron design can categorize, extract features, adapt in real time, and learn to recognize objects in images or words in speech.
Combinations of these artificial networks can mimic design and control patterns. Even the approximation of more complex functions like cognition or dialog are considered theoretically possible with stateful networks such as RNNs because they are Turing complete.
This question centers around whether the impression created by the success of deep networks based on purely orthogonal extensions of the original perceptron design is limiting creativity.
How realistic is it to assume that tweaking the dimensions of arrays and matrices, which are convenient in most programming languages, will lead from artificial networks to artificial brains?
The network depth required to make a computer learn to choreograph a dance or develop a complex proof would not likely converge, even if a hundred racks of dedicated and advanced hardware ran for a year. Local minima in the error surface and gradient saturation would plague the runs, rendering convergence unrealistic.
The primary reason that orthogonality is found in MLP, CNN, and RNN design is because loops used for array iteration compile to simple tests and backward jumps in machine language. And that fact caries into all higher level languages from FORTRAN and C to Java and Python.
The most natural machine level data structure for trivial loops are arrays. Nesting loops provides the same direct trivial alignment with multidimensional arrays. These map to the mathematical structures of vectors, matrices, cubes, hyper-cubes, and their generalization: Tensors.
Although graph based libraries and object oriented databases have existed for decades and the use of recursion to traverse hierarchies is covered in most software engineering curricula, two facts deter the general trend away from less constricted topologies.
- Graph theory (vertices connected by edges) is not consistently included in computer science curricula.
- Many people that write programs have worked only with structures built into their favorite languages, such as arrays, ordered lists, sets, and maps.
The structure of the brain is not oriented to Cartesian topologies1 like vectors or matrices. The neural nets in biology are not orthogonal. Neither their physical orientation nor the graphical representations of their signal paths are boxy. Brain structure is not naturally represented in ninety degree angles.
Real neural circuits cannot be directly represented in Cartesian forms. Neither do they directly fit into recursive hierarchies. This is because of four distinctive characteristics.
- Parallelism in the mind is by trend not by iteration — The neurons in what appear as parallel structures are not identical and are wrought with exceptions to the apparent pattern.
- Cycles appear in the structure — Groups of neurons do not all point in a single direction. Cycles exist in the directed graph that represents many networks. There are many circuits where an ancestor in signal direction is also a descendant. This is like the stabilizing feedback in analog circuits.
- Neural structures that are not parallel are not always orthogonal either. If a ninety degree angle forms, it is by chance, not design.
- Neural structure is not static — Neuroplasticity is the phenomena that is observed where an axon or dendrite may grow in new directions that are not restricted to ninety degrees. Cell apoptosis may eliminate a neuron. A new neuron may form.
There is almost nothing about the brain that fits naturally into an orthogonal digital circuit structure like a vector, matrix, or cube of registers or contiguous memory addresses. Their representation in silicon and the feature demands they place on higher level programming languages are radically different than the multidimensional arrays and loops of basic algebra and analytic geometry.
The brain is constructed with unique topological1 structures that realize sophisticated signal propagation. They are unconstrained by Cartesian coordinate systems or grids. Feedback is nested and non-orthogonal. They have chemical and electrical equilibria that form balances of higher and lower thought, motivation, and attention.
Is that topological1 sophistication necessary or merely a bi-product of how DNA constructs a vector, matrix, cube, or hyper-cube?
As brain research progresses, it becomes increasingly unlikely that brain structures can be efficiently morphed into orthogonal signal pathways. It is unlikely that the needed signal structures are homogeneously typed arrays. It is even possible that stochastic or chaotic processing structures possess an advantage for AI development.
The brain's topologically1 sophisticated features may be a catalyst or even a necessity for the emergence of human forms of thought. When we set out to achieve convergence across hundreds of perceptron layers, we can only sometimes make it work. Are we in some way trapped by the conceptual limitations that began with Descartes?
Can we escape from those limitations by simply abandoning the programming convenience of orthogonal structures? Several researchers are working to discover new orientations in the design of VLSI chips. There may be a need to develop new kinds of programming languages or new features to existing ones to facilitate the description of mental function in code.
Some have suggested that new forms of mathematics are indicated, but significant theoretical framework has been created already by Leonhard Euler (graphs), Gustav Kirchhoff (networks), Bernhard Riemann (manifolds), Henri Poincaré (topology), Andrey Markov (graphs of action), Richard Hook Richens (computational linguistics), and others to support significant AI progress before mathematics need be extended further.
Is the next step in AI development to embrace topological sophistication?
 This question only uses the word topology to refer to the longstanding mathematical definition of the word. Although the term has been distorted by some emerging jargon, none of those distortions are meant in this question. Distortions include (a) calling an array of layer widths the network's topology and (b) calling the texture of a surface its topoLOGy when the correct term would be topoGRAPHy. Such distortions confound the communication of ideas like the ones described in this question, which is unrelated to (a) or (b).
Cliques of Neurons Bound into Cavities Provide a Missing Link between Structure and Function Frontiers in Computational Neuroscience, 12 June 2017, Michael W. Reimann et. al. https://www.frontiersin.org/articles/10.3389/fncom.2017.00048/full, https://doi.org/10.3389/fncom.2017.00048
An On-Line Self-Constructing Neural Fuzzy, Inference Network and Its Applications, Chia-Feng Juang and Chin-Teng Lin, IEEE Transactions on Fuzzy Systems, v6, n1, 1998, https://ir.nctu.edu.tw/bitstream/11536/32809/1/000072774800002.pdf
Gated Graph Sequence Neural Networks Yujia Li and Richard Zemel, ICLR conference paper, 2016, https://arxiv.org/pdf/1511.05493.pdf
Building Machines That Learn and Think Like People, Brenden M. Lake, Tomer D. Ullman, Joshua B. Tenenbaum, and Samuel J. Gershman, Behavioral and Brain Sciences, 2016, https://arxiv.org/pdf/1604.00289.pdf
Learning to Compose Neural Networks for Question Answering, Jacob Andreas, Marcus Rohrbach, Trevor Darrell, and Dan Klein, UC Berkeley, 2016, https://arxiv.org/pdf/1601.01705.pdf
Learning multiple layers of representation Geoffrey E. Hinton, Department of Computer Science, University of Toronto, 2007, http://www.csri.utoronto.ca/~hinton/absps/ticsdraft.pdf
Context-Dependent Pre-Trained Deep Neural Networks for Large-Vocabulary Speech Recognition, George E. Dahl, Dong Yu, Li Deng, and Alex Acero, IEEE Transactions on Audio, Speach, and Language Processing 2012, https://s3.amazonaws.com/academia.edu.documents/34691735/dbn4lvcsr-transaslp.pdf?AWSAccessKeyId=AKIAIWOWYYGZ2Y53UL3A&Expires=1534211789&Signature=33QcFP0JGFeA%2FTsqjQZpXYrIGm8%3D&response-content-disposition=inline%3B%20filename%3DContext-Dependent_Pre-Trained_Deep_Neura.pdf
Embedding Entities and Relations for Learning and Inference in Knowledge Bases, Bishan Yang1, Wen-tau Yih2, Xiaodong He2, Jianfeng Gao2, and Li Deng2, ICLR conference paper, 2015, https://arxiv.org/pdf/1412.6575.pdf
A Fast Learning Algorithm for Deep Belief Nets, Geoffrey E. Hinton, Simon Osindero, Yee-Whye Teh (communicated by Yann Le Cun), Neural Computation 18, 2006, http://axon.cs.byu.edu/Dan/778/papers/Deep%20Networks/hinton1*.pdf
FINN: A Framework for Fast, Scalable Binarized Neural Network Inference Yaman Umuroglu, et al, 2016, https://arxiv.org/pdf/1612.07119.pdf
From Machine Learning to Machine Reasoning, Léon Bottou, 2/8/2011, https://arxiv.org/pdf/1102.1808.pdf
Progress in Brain Research, Neuroscience: From the Molecular to the Cognitive, Chapter 15: Chemical transmission in the brain: homeostatic regulation and its functional implications, Floyd E. Bloom (editor), 1994, https://doi.org/10.1016/S0079-6123(08)60776-1
Neural Turing Machine (slideshow), Author: Alex Graves, Greg Wayne, Ivo Danihelka, Presented By: Tinghui Wang (Steve), https://eecs.wsu.edu/~cook/aiseminar/papers/steve.pdf
Neural Turing Machines (paper), Alex Graves, Greg Wayne, Ivo Danihelka, 2014, https://pdfs.semanticscholar.org/c112/6fbffd6b8547a44c58b192b36b08b18299de.pdf
Reinforcement Learning, Neural Turing Machines, Wojciech Zaremba, Ilya Sutskever, ICLR conference paper, 2016, https://arxiv.org/pdf/1505.00521.pdf?utm_content=buffer2aaa3&utm_medium=social&utm_source=twitter.com&utm_campaign=buffer
Dynamic Neural Turing Machine with Continuous and Discrete Addressing Schemes, Caglar Gulcehre1, Sarath Chandar1, Kyunghyun Cho2, Yoshua Bengio1, 2017, https://arxiv.org/pdf/1607.00036.pdf
Deep learning, Yann LeCun, Yoshua Bengio3 & Geoffrey Hinton, Nature, vol 521, 2015, https://www.evl.uic.edu/creativecoding/courses/cs523/slides/week3/DeepLearning_LeCun.pdf
Context-Dependent Pre-Trained Deep Neural Networks for Large-Vocabulary Speech Recognition, IEEE Transactions on Audio, Speach, and Language Processing, vol 20, no 1 George E. Dahl, Dong Yu, Li Deng, and Alex Acero, 2012, https://www.cs.toronto.edu/~gdahl/papers/DBN4LVCSR-TransASLP.pdf
Clique topology reveals intrinsic geometric structure in neural correlations, Chad Giusti, Eva Pastalkova, Carina Curto, Vladimir Itskov, William Bialek PNAS, 2015, https://doi.org/10.1073/pnas.1506407112, http://www.pnas.org/content/112/44/13455.full?utm_content=bufferb00a4&utm_medium=social&utm_source=twitter.com&utm_campaign=buffer
UCL, London Neurological Newsletter, July 2018 Barbara Kramarz (editor), http://www.ucl.ac.uk/functional-gene-annotation/neurological/newsletter/Issue17