Is topological sophistication necessary to the furtherance of AI?

Question

The current machine learning trend is interpreted by some new to the disciplines of AI as meaning that MLPs, CNNs, and RNNs can exhibit human intelligence. It is true that these orthogonal structures derived from the original perceptron design can categorize, extract features, adapt in real time, and learn to recognize objects in images or words in speech.

Combinations of these artificial networks can mimic design and control patterns. Even the approximation of more complex functions like cognition or dialog are considered theoretically possible with stateful networks such as RNNs because they are Turing complete.

This question centers around whether the impression created by the success of deep networks based on purely orthogonal extensions of the original perceptron design is limiting creativity.

How realistic is it to assume that tweaking the dimensions of arrays and matrices, which are convenient in most programming languages, will lead from artificial networks to artificial brains?

The network depth required to make a computer learn to choreograph a dance or develop a complex proof would not likely converge, even if a hundred racks of dedicated and advanced hardware ran for a year. Local minima in the error surface and gradient saturation would plague the runs, rendering convergence unrealistic.

The primary reason that orthogonality is found in MLP, CNN, and RNN design is because loops used for array iteration compile to simple tests and backward jumps in machine language. And that fact caries into all higher level languages from FORTRAN and C to Java and Python.

The most natural machine level data structure for trivial loops are arrays. Nesting loops provides the same direct trivial alignment with multidimensional arrays. These map to the mathematical structures of vectors, matrices, cubes, hyper-cubes, and their generalization: Tensors.

Although graph based libraries and object oriented databases have existed for decades and the use of recursion to traverse hierarchies is covered in most software engineering curricula, two facts deter the general trend away from less constricted topologies.

• Graph theory (vertices connected by edges) is not consistently included in computer science curricula.
• Many people that write programs have worked only with structures built into their favorite languages, such as arrays, ordered lists, sets, and maps.

The structure of the brain is not oriented to Cartesian topologies¹ like vectors or matrices. The neural nets in biology are not orthogonal. Neither their physical orientation nor the graphical representations of their signal paths are boxy. Brain structure is not naturally represented in ninety degree angles.

Real neural circuits cannot be directly represented in Cartesian forms. Neither do they directly fit into recursive hierarchies. This is because of four distinctive characteristics.

1. Parallelism in the mind is by trend not by iteration — The neurons in what appear as parallel structures are not identical and are wrought with exceptions to the apparent pattern.
2. Cycles appear in the structure — Groups of neurons do not all point in a single direction. Cycles exist in the directed graph that represents many networks. There are many circuits where an ancestor in signal direction is also a descendant. This is like the stabilizing feedback in analog circuits.
3. Neural structures that are not parallel are not always orthogonal either. If a ninety degree angle forms, it is by chance, not design.
4. Neural structure is not static — Neuroplasticity is the phenomena that is observed where an axon or dendrite may grow in new directions that are not restricted to ninety degrees. Cell apoptosis may eliminate a neuron. A new neuron may form.

There is almost nothing about the brain that fits naturally into an orthogonal digital circuit structure like a vector, matrix, or cube of registers or contiguous memory addresses. Their representation in silicon and the feature demands they place on higher level programming languages are radically different than the multidimensional arrays and loops of basic algebra and analytic geometry.

The brain is constructed with unique topological¹ structures that realize sophisticated signal propagation. They are unconstrained by Cartesian coordinate systems or grids. Feedback is nested and non-orthogonal. They have chemical and electrical equilibria that form balances of higher and lower thought, motivation, and attention.

Is that topological¹ sophistication necessary or merely a bi-product of how DNA constructs a vector, matrix, cube, or hyper-cube?

As brain research progresses, it becomes increasingly unlikely that brain structures can be efficiently morphed into orthogonal signal pathways. It is unlikely that the needed signal structures are homogeneously typed arrays. It is even possible that stochastic or chaotic processing structures possess an advantage for AI development.

The brain's topologically¹ sophisticated features may be a catalyst or even a necessity for the emergence of human forms of thought. When we set out to achieve convergence across hundreds of perceptron layers, we can only sometimes make it work. Are we in some way trapped by the conceptual limitations that began with Descartes?

Can we escape from those limitations by simply abandoning the programming convenience of orthogonal structures? Several researchers are working to discover new orientations in the design of VLSI chips. There may be a need to develop new kinds of programming languages or new features to existing ones to facilitate the description of mental function in code.

Some have suggested that new forms of mathematics are indicated, but significant theoretical framework has been created already by Leonhard Euler (graphs), Gustav Kirchhoff (networks), Bernhard Riemann (manifolds), Henri Poincaré (topology), Andrey Markov (graphs of action), Richard Hook Richens (computational linguistics), and others to support significant AI progress before mathematics need be extended further.

Is the next step in AI development to embrace topological sophistication?

---

Footnotes

[1] This question only uses the word topology to refer to the longstanding mathematical definition of the word. Although the term has been distorted by some emerging jargon, none of those distortions are meant in this question. Distortions include (a) calling an array of layer widths the network's topology and (b) calling the texture of a surface its topoLOGy when the correct term would be topoGRAPHy. Such distortions confound the communication of ideas like the ones described in this question, which is unrelated to (a) or (b).

---

References

Cliques of Neurons Bound into Cavities Provide a Missing Link between Structure and Function Frontiers in Computational Neuroscience, 12 June 2017, Michael W. Reimann et. al. https://www.frontiersin.org/articles/10.3389/fncom.2017.00048/full, https://doi.org/10.3389/fncom.2017.00048

An On-Line Self-Constructing Neural Fuzzy, Inference Network and Its Applications, Chia-Feng Juang and Chin-Teng Lin, IEEE Transactions on Fuzzy Systems, v6, n1, 1998, https://ir.nctu.edu.tw/bitstream/11536/32809/1/000072774800002.pdf

Gated Graph Sequence Neural Networks Yujia Li and Richard Zemel, ICLR conference paper, 2016, https://arxiv.org/pdf/1511.05493.pdf

Building Machines That Learn and Think Like People, Brenden M. Lake, Tomer D. Ullman, Joshua B. Tenenbaum, and Samuel J. Gershman, Behavioral and Brain Sciences, 2016, https://arxiv.org/pdf/1604.00289.pdf

Learning to Compose Neural Networks for Question Answering, Jacob Andreas, Marcus Rohrbach, Trevor Darrell, and Dan Klein, UC Berkeley, 2016, https://arxiv.org/pdf/1601.01705.pdf

Learning multiple layers of representation Geoffrey E. Hinton, Department of Computer Science, University of Toronto, 2007, http://www.csri.utoronto.ca/~hinton/absps/ticsdraft.pdf

Context-Dependent Pre-Trained Deep Neural Networks for Large-Vocabulary Speech Recognition, George E. Dahl, Dong Yu, Li Deng, and Alex Acero, IEEE Transactions on Audio, Speach, and Language Processing 2012, https://s3.amazonaws.com/academia.edu.documents/34691735/dbn4lvcsr-transaslp.pdf?AWSAccessKeyId=AKIAIWOWYYGZ2Y53UL3A&Expires=1534211789&Signature=33QcFP0JGFeA%2FTsqjQZpXYrIGm8%3D&response-content-disposition=inline%3B%20filename%3DContext-Dependent_Pre-Trained_Deep_Neura.pdf

Embedding Entities and Relations for Learning and Inference in Knowledge Bases, Bishan Yang1, Wen-tau Yih2, Xiaodong He2, Jianfeng Gao2, and Li Deng2, ICLR conference paper, 2015, https://arxiv.org/pdf/1412.6575.pdf

A Fast Learning Algorithm for Deep Belief Nets, Geoffrey E. Hinton, Simon Osindero, Yee-Whye Teh (communicated by Yann Le Cun), Neural Computation 18, 2006, http://axon.cs.byu.edu/Dan/778/papers/Deep%20Networks/hinton1*.pdf

FINN: A Framework for Fast, Scalable Binarized Neural Network Inference Yaman Umuroglu, et al, 2016, https://arxiv.org/pdf/1612.07119.pdf

From Machine Learning to Machine Reasoning, Léon Bottou, 2/8/2011, https://arxiv.org/pdf/1102.1808.pdf

Progress in Brain Research, Neuroscience: From the Molecular to the Cognitive, Chapter 15: Chemical transmission in the brain: homeostatic regulation and its functional implications, Floyd E. Bloom (editor), 1994, https://doi.org/10.1016/S0079-6123(08)60776-1

Neural Turing Machine (slideshow), Author: Alex Graves, Greg Wayne, Ivo Danihelka, Presented By: Tinghui Wang (Steve), https://eecs.wsu.edu/~cook/aiseminar/papers/steve.pdf

Neural Turing Machines (paper), Alex Graves, Greg Wayne, Ivo Danihelka, 2014, https://pdfs.semanticscholar.org/c112/6fbffd6b8547a44c58b192b36b08b18299de.pdf

Reinforcement Learning, Neural Turing Machines, Wojciech Zaremba, Ilya Sutskever, ICLR conference paper, 2016, https://arxiv.org/pdf/1505.00521.pdf?utm_content=buffer2aaa3&utm_medium=social&utm_source=twitter.com&utm_campaign=buffer

Dynamic Neural Turing Machine with Continuous and Discrete Addressing Schemes, Caglar Gulcehre1, Sarath Chandar1, Kyunghyun Cho2, Yoshua Bengio1, 2017, https://arxiv.org/pdf/1607.00036.pdf

Deep learning, Yann LeCun, Yoshua Bengio3 & Geoffrey Hinton, Nature, vol 521, 2015, https://www.evl.uic.edu/creativecoding/courses/cs523/slides/week3/DeepLearning_LeCun.pdf

Context-Dependent Pre-Trained Deep Neural Networks for Large-Vocabulary Speech Recognition, IEEE Transactions on Audio, Speach, and Language Processing, vol 20, no 1 George E. Dahl, Dong Yu, Li Deng, and Alex Acero, 2012, https://www.cs.toronto.edu/~gdahl/papers/DBN4LVCSR-TransASLP.pdf

Clique topology reveals intrinsic geometric structure in neural correlations, Chad Giusti, Eva Pastalkova, Carina Curto, Vladimir Itskov, William Bialek PNAS, 2015, https://doi.org/10.1073/pnas.1506407112, http://www.pnas.org/content/112/44/13455.full?utm_content=bufferb00a4&utm_medium=social&utm_source=twitter.com&utm_campaign=buffer

UCL, London Neurological Newsletter, July 2018 Barbara Kramarz (editor), http://www.ucl.ac.uk/functional-gene-annotation/neurological/newsletter/Issue17

Answer 1

Yes ... definitely.

AI software should not be overly bound to tensors. The natural representation of a network is a network, not an array.

Math Well Fitted for AI

Vertices connected by directed edges is the closest mathematical equivalent to working nervous systems. Even the control circuit of a CPU, FPU, or GPU is a complex network of gates, not a grid like RAM.

The end points of connections are not the only significant properties of an edge either. The length of the edges in neural circuits are definitely part of the design that works. That means that the layout of an artificially intelligent circuit in a quantum or near quantum computer may determine whether it works well. The NIH article Effect of the Topology and Delayed Interactions in Neuronal Networks Synchronization explains why. And this image and its source article confirms.

Growth Connectomics – The organization and reorganization of brain networks during normal and abnormal development

Traumatic Brain Injury

Those who have traumatic brain injury relearn tasks that were disrupted by the injury. Neuroplasticity causes a physical change in network structure and circuitry around the damaged area that helps in the re-acquisition of the lost abilities due to the injury.

Neurologists believe that changes to the axons, dendrites, and synapses occur first, followed by changes in the internals to the neurons that have connected and the chemistry at the synopses that all work together to restore function, but the function is never completely restored. There is always a signature of the injury exhibited in behavior.

That means that where in the brain system the gray matter that handles specific functions matter.

Topology in AI

I concur that people are using the term topology without any idea what it is.

The Seven Bridges of Königsberg may be the first well known problem in topology and Poincaré did extend the idea of topology into N-space, but A.F. Möbius and Felix Klein created the constructions that made the idea of 2D and 3D objects that have only one side or surface.

Mobius Strip

Klein Bottle

Problems of Illustration vs. Problems in Real Life

Below is the concept illustration of Euler's famous Seven Bridges of Königsberg. The goal is to cross all seven exactly once. It is a problem that shows that topologies have properties that can be determined by trying every possibility or by applying a mathematical proof.

The topology of that and other bridge problems can be represented in a tensor (specifically a matrix) and algorithms can be used to brute force attack the problem. Production rule systems can infer properties from the tensor, and deep networks fed with randomly generated sequences of turns can learn whether the destination or objective can be reached and a navigation sequence if it can, using the information in the tensor and leaving the result in another tensor.

Here's one of the actual turn points in Königsberg.

Deciding where to turn and which way currently requires a human brain, even if a map is available to the computer. The image could theoretically be processed in a tensor by a computer, and it may be theoretically possible to derive the meaning of the objects in view and place them in another tensor, but that's not the way the visual system of animals work and that's not how cognition works.

It's simply not the naturally matched architecture for data storage of this kind of information. Tensor arithmetic is not the naturally matched computational model either.

The networks and computational algorithms that work best have structure that is very different from those used to store and process tensors. The networks of vision and decision are embodied as networks physically, syntactically, semantically, and cognitively.

Critique of the Traditions of Computer Science Education

The mathematics does not seem to be the bottleneck in the development in AI. It is an educational product. It almost seems like a good idea to not allow beginner students to cheat by using arrays or for loops during the first few years of computer science education. Even discussing arrays, tensors, and loops could plague the ability of the student to think in a more transcendent AI way.

Answer 2

Recent advancement in neural networks around the topic of multi-modal learning and ontology parsing has shown, that the topology of a neural network is not very important. Instead the input data are the most dominant factor to determine the success rate. Let us make an example. We are taking a plaintext file as input for a neural network. As a topology we are using the most complicated LSTM network available which is simulated on a high-end computer-cluster. Is our neural network able to speak like a human after the training? No. The bottleneck is the not the number of layers or the learning algorithm, the bottleneck is, that the input data makes no sense. Now we add annotations to the input data, that means we are not only give plain text, but define what the subject, verb and object is . If we are using the same layout and the same hardware, the success rate is much greater.

What I want to explain is, that very simple neural networks with only 3 layers and an outdated back propagation algorithm are sufficient if the input data was pre-processed and makes sense. A modern neural network works like a database to get access to the data storage. It is some kind of error tolerant memory.

This explanation has a philosophical dimension. The question is: where is the location in which thinking take place? According to psychology, thinking is done in the brain. According to sociology, thinking is done by agents in a game situation. If a neural network is seen as a temporal memory the data which are stored in the network become a greater importance. If thinking is located inside the brain, that the data are not important because the brain is intelligent but not it's environment. Focusing on the neural network topology is equal to ignoring the environment.