Geometry and AI
Matrices, cubes, layers, stacks, and hierarchies are what we could accurately call topologies. Consider topology in this context the higher level geometrical design of a learning system.
As complexity rises, it is often useful to represent these topologies as directed graph structures. State diagrams and Markov's work on game theory are two places where directed graphs are commonly used. Directed graphs have vertices (often visualized as closed shapes) and edges often visualized as arrows connecting the shapes.
We can also represent GANs as a directed graph, where the output of each net drives the training of the other in adversarial fashion. GANs resemble a Möbius strip topologically.
We cannot discover new designs and architectures without understanding not only the mathematics of converging on an optimal solution or tracking one but also topologies of network connections that can support such convergence. It is like first developing a processor while imagining what an operating system would need before writing the operating system.
To glimpse what topologies we have NOT YET considered, let's first look at which ones have been.
Step One — Extrusion in a Second Dimension
In the 1980s, success was achieved with the extension of the original perceptron design. Researchers added a second dimension to create a multi-layered neural network. Reasonable convergence was achieved through back-propagation of an error function's gradient through the gradients of the activation functions attenuated by learning rates and dampened with other meta-parameters.
Step Two — Adding Dimensions to the Discrete Input Signal
We see the emergence of convolutional networks based on existing manually tuned image convolution techniques introduced dimensions to the network input: Vertical position, color components, and frame. This last dimension is critical to CGI, face replacement, and other morphological techniques in contemporary movie making. Without it, we have image generation, categorization, and noise removal.
Step Three — Stacks of Networks
We see stacks of neural nets emerge in the late 1990s, where the training of one network is supervised by another. This is the introduction of conceptual layers, neither in the sense of sequential layers of neurons nor in the sense of layers of color in an image. This type of layering is not recursion either. It is more like the natural world where one structure is an organ within another completely different kind of structure.
Step Four — Hierarchies of Networks
We see hierarchies of neural nets appearing frequently in the research arising out of the 2000s and early 2010s (Laplacian and others), which continues the interaction between neural nets and continuing the mammalian brain analogy. We now see meta-structure, where entire networks become vertices in a directed graph representing a topology.
Layers have ordinally valued activation functions for vertices and attenuation matrices mapped to an exhaustive set of directed edges between adjacent layers . Image convolution layers are often in two dimensional vertex arrangements with attenuation cubes mapped to an abridged set of directed edges between adjacent layers . Stacks have entire layered nets as vertices in a meta-directed-graph, and those meta-vertices are connected in a sequence with each edge being either a training meta-parameter, a reinforcement (real time feedback) signal, or some other learning control. Hierarchies of nets reflect the notion that multiple controls can be aggregated and direct lower level learning, or the flip case where multiple learning elements can be controlled by one higher level supervisor network.
Analysis of the Trend in Learning Topologies
We can analyze trends in machine learning architecture. We have three topological trends.
Depth in the causality dimension — Layers to the signal processing where the output of one layer of activations is fed through a matrix of attenuating parameters (weights) to the input of the next layer. As greater controls are established, only beginning with basic gradient descent in back propatagion, greater depth can be achieved.
Input signal dimensionality — From scalar input to hypercubes (video has horizontal, vertical, color depth including transparency, and frame — Note that this is not the same as the number of inputs in the perceptron sense.
Topological development — The above two are Cartesian in nature. Dimensions are added at right angles to the existing dimensional. As networks are wired in hierarchies (as in Laplacian Hierarchies) and Möbius strip like circles (as in GANs), the trends are topographical and are best represented by directed graphs where the vertices are not neurons but smaller networks of them.
What Topologies are Missing?
This section expands on the meaning of the title question.
- Is there any reason why multiple meta-vertices, each representing a neural net, can be arranged such that multiple supervisor meta-vertices can, in conjunction, supervise multiple employee meta-vertices?
- Why is the back-propagation of an error signal the only non-linear equivalent of negative feedback?
- Can't collaboration between meta-vertices rather than supervision be employed, where there are two reciprocal edges representing controls?
- Since neural nets are employed mainly for learning of nonlinear phenomena, why prohibits other types of closed paths in the design of the nets or their interconnection?
- Is there any reason why sound cannot be added to picture so that video clips can be categorized automatically? If that is the case, is a screenplay a possible feature extraction of a movie and can an adversarial architecture be used to generate screenplays and produce the movies without the movie studio system? What would that topology look like as a directed graph?
Artificial cells in MLPs use of floating or fixed point arithmetic transfer functions rather than electro-chemical pulse transmissions based on amplitude and proximity based threshold. They are not realistic simulations of neurons, so calling the vertices neurons would be a misnomer for this kind of analysis.
Correlation of image features and relative changes between pixels in close proximity is much higher than that of distant pixels.