The simplistic neural networks that have been given away for free after they prove insufficient by themselves in field use consist solely of two orthogonal dimensions.
- Layer width — the number of ordinal numbers or floating point numbers that represent the signal path through any given layer comprise of an array of layer elements
- Network depth — the number of layers in the primary signal path, which is the number of activation function arrays or convolution kernels or whatever other elements Usually neural networks consist from layers
However, in large corporations that have AI pipelines, this is not the case. We are beginning to see more interesting topologies in open source. We see this in generative systems for images, text, and speech. We see this in robotic control of robots. The truth is that these more sophisticated topologies have been in play for years, but were just not appearing in the open source community because they were company confidential. Enough academic work, releasing of portions of corporate IP, and the accumulation of independent OSS work has occurred to start to see these topologies in GIT repos.
Cyclic Not Acyclic
Artificial network topologies are generally cyclic, not acyclic in terms of their causality or their signal pathways, depending on how you depict them theoretically. These are three basic examples from among dozens in the literature and in the open source repositories.
Back-propagation represents the introduction of a deliberate cycle in signal paths in a basic multilayer perceptron, making that topology a sequence of layers represented by vertices, connected sequentially by a set of directed edges representing forward propagation, and a set of directed edges in the reverse direction to distribute the corrective error determined at the network output according to the principle of gradient descent. For efficiency, the corrective signal is distributed recursively backward through the layers to the $N - 1$ matrices of parameters attenuating the signals between $N$ layers. Back propagation requires the formation of these $N - 1$ cycles for convergence to occur.
In a generative adversarial network (GAN), we have the signal path of each of the two networks feeding the training criteria of the other. Such a topological arrangement is like negative feedback in a stable control system in that an equilibrium is formed between the generative network and discriminative network. The two directed edges, (a) the one that causally affects G with Ds result, and (b) the one that causally affects D with Gs result, create a cycle on top of the cycles in each of G and D.
In attention based networks being touted as theoretically advantageous over LSMT (which has been dominating over CNNs) has a much more complex topology and more cycles above those in supervisory layer than those in GANs.
Analysis of Answer One of Three
It is true that every directed graph can be realized in an arbitrarily large RNN because they are Turing complete, but that doesn't mean they are a great topology for all finite algorithms.
Turing was aware that his punched tape model was not the best general purpose, high speed computing architecture. He was not intending to prove anything about computing speed but rather what could be computed. His Turing machine had a trivial topology deliberately. He wanted to illustrate his completeness theorem to others and resurrect the forward movement of rationalism after Gödel disturbed it with his two incompleteness theorems.
Similarly, John von Neumann proposed his computing architecture, with a central processing unit (CPU) and unified data and instruction bus, to reduce the number of relays or vacuum tubes, not to maximize parallel algorithm execution. That topology as a directed graph has the instruction controller and the arithmetic unit in the center and everything else branching out from the data and address bus leading from them.
That a topology can accomplish a task is no longer a justification for persisting in the use of that topology, which is why Intel acquired Nirvana, which deviates from traditional von Neumann architecture, DSP architecture, and the current CUDA core architecture that NVidia GPUs use and offer for artificial network realization through C libraries that can be called via integrated Java and Python adapters.
There is definitely sense to seek for more general topologies, if they are fit for the purpose, just as with Turing's or von Neuman's.
Analysis of Answer Two of Three
General topologies exist, the most economically viable of which is the CUDA cores begun by NVidia, which can be configured for MLPs, CNNs, RNNs, and general 2D and 3D video processing. They can be configured with or without cycles depending on the characteristics of the parallelism desired.
The realization of topologies unlike the Cartesian arrangements of activation functions in artificial networks or the kernel cells in convolution engines do have barriers to use, but they are not fundamental restrictions. The primary barrier is not one of hardware or software. It is one of linguistics. We don't think topologically because we don't talk topologically. That's what is great about this question's challenge.
FORTRAN began to dominate over LISP during the time when general purpose programming began to emerge in many corporations. That is not surprising because humans communicate in orthogonal ways. It is cultural. When a child scribbles, teachers are indoctrinated to say nice things but respond by drawing a shape. If the child draws a square, the teacher smiles. The child is given blocks. The books are rectangular. Text is justified into rectangles.
We can see this in building architecture dating back to Stonehenge. Ninety degree angles are clearly dominant in artificial things, whereas nature doesn't seem to have that bias.
Although directed graphs were easy to implement and traverse in recursive structure and were commonplace in the LISP community. FORTRAN with its realization of vectors and matrices in one and two dimensional arrays respectively were easier to grasp by those with less theoretical background in data structures.
The result is that even if learning EMMASCRIPT (JavaScript) which has its seed from the LISP community and is not biased toward orthogonal data structures, people tend to proceed from HelloWorld.js to something with a basic loop in it, with an underlying array through which the loop iterates.
There are three wonderfully inquisitive and insightful phrases in answer two of three.
- Maybe learning is not converging in them — Interestingly an algorithm cannot learn without a cycle. Directly applying a formula or converging using a known convergent series of terms does not qualify as learning. Gradient descent relies entirely on the cyclical nature of a corrective action at the end of each sample processing or batch of them.
- Maybe they generate chaotic [oscillations] — This gets into chaos theory and control theory's concept of stability. They can do so, but so can a basic multilayer perceptron if the learning rate is set to high.
- Maybe they generate bifurcations — Now we have fully entered the realm of chaos, which is arguably closely related to creativity. Mendelbrot proposed the relationship between new forms of order and the apparent chaotic behavior arising from the appropriate level of feedback in a system with signal path components that cannot be modelled with a first degree equation. Since then, we find that most phenomena in nature are actually strange attractors. The plot of the training of a network from a continuous feed of consistently distributed data in phase space will reveal ... you guessed it ... a strange attractor. When purtibations are deliberately injected into a training epoch from a pseudo-random number generator, the specific purpose is a bifurcation, so that the global optimization can be found when the training gets stuck in a local optimization.
Analysis of Answer Three of Three
General topologies exist and are promising and researchers are ready to work with them. It is enthusiasts that can have a dismissive attitude. They don't yet understand the demos they've downloaded and painstakenly tweaked to run on their computer, they're about to launch their AI carrier amidst the growing demand from all the media hype, and now someone is introducing something interesting and not yet implemented in code. The motivational direction is generally to either dismiss or resist the creative proposals.
In this case, Google, CalTech, IBM, MIT, U Toronto, Intel, Tesla, Japan, and a thousand other governments, institutions, corporations, and open source contributors will solve that problem, provided people keep talking about topology and the restrictions inherent in purely Cartesian thinking.
Misunderstanding Topology to Mean Dimensionality or Topography
There has been some confusion in terms. The SO reference in the question is an example of thinking that changing an array dimension is changing the topology. If such were so, then there would be no change one could make to the geometry of an AI system that would not be topological. Topology can only have meaning if there are features that are not topological. When one draws a layer, they don't need to increase the height of the rectangle representing it if the number of activitations, the width of the layer, is changed from 100 to 120.
I've also seen academic papers that called the texture or roughness of an error surface its topology. That completely undermines the concept of topology. They meant to use the term topography. Unfortunately neither the publisher nor the editor noticed the error.
Software or Tools
Most programming languages support directed graphs in recursive hashmaps. LISP and its derivatives supported them at a more machine instruction efficient level, and that's still the case. Object oriented databases and graph libraries exist and are in use. Google uses them extensively in web indexing and lookup. FaceBook's API is called the Graph API, because it is a query and insert API into the graph that is FaceBook's user data store.
The explosion is here in global software giants. There is open source for it. The revolution that is missing is among those who are not yet educated as to the meaning of topology, the difference between a hierarchy and a network or the role of feedback in any learning system.
Regarding Java and Python there are many barriers to the revolution in thinking, primarily these.
- There are now key words in either Java or Python to directly deal with directed graphs other than the idea of a class with references to instances of other classes, which is quite limited. None of these languages can add edge types with a single simple language construct.
- There is no mapping to hardware yet, although Nirvana allegedly developed one, and Intel acquired Nirvana, so that barrier may evaporate soon.
- The bias still exists in preschool, kindergarten, and first grade
- Hilbert spaces are not generally taught in calculus
Graphviz and other graphing software that auto-generates diagrams from unconstrained directed or bidirectional graph representations have done much to bust through the barriers because the generated images are visible across the web. It may be through visual representations of graphs that linguistic representations, thought, hardware, and software begin to emerge representing the paradigm shift the question investigates.
It is not that constraints are not useful. Only some patterns and paradigms produce results, but since the results from the human brain demand attention, and the human brain is
- Not at all orthogonal,
- Not implemented using Cartesian neural patterns, and
- Not topologically a box,
One can all but conclude that those are not particularly well chosen constraints. Neither is the acyclic criteria. Nature is cyclic, and intelligence probably requires it in many ways and at many levels.
