9
$\begingroup$

Geometry and AI

Matrices, cubes, layers, stacks, and hierarchies are what we could accurately call topologies. Consider topology in this context the higher level geometrical design of a learning system.

As complexity rises, it is often useful to represent these topologies as directed graph structures. State diagrams and Markov's work on game theory are two places where directed graphs are commonly used. Directed graphs have vertices (often visualized as closed shapes) and edges often visualized as arrows connecting the shapes.

We can also represent GANs as a directed graph, where the output of each net drives the training of the other in adversarial fashion. GANs resemble a Möbius strip topologically.

We cannot discover new designs and architectures without understanding not only the mathematics of converging on an optimal solution or tracking one but also topologies of network connections that can support such convergence. It is like first developing a processor while imagining what an operating system would need before writing the operating system.

To glimpse what topologies we have NOT YET considered, let's first look at which ones have been.

Step One — Extrusion in a Second Dimension

In the 1980s, success was achieved with the extension of the original perceptron design. Researchers added a second dimension to create a multi-layered neural network. Reasonable convergence was achieved through back-propagation of an error function's gradient through the gradients of the activation functions attenuated by learning rates and dampened with other meta-parameters.

Step Two — Adding Dimensions to the Discrete Input Signal

We see the emergence of convolutional networks based on existing manually tuned image convolution techniques introduced dimensions to the network input: Vertical position, color components, and frame. This last dimension is critical to CGI, face replacement, and other morphological techniques in contemporary movie making. Without it, we have image generation, categorization, and noise removal.

Step Three — Stacks of Networks

We see stacks of neural nets emerge in the late 1990s, where the training of one network is supervised by another. This is the introduction of conceptual layers, neither in the sense of sequential layers of neurons nor in the sense of layers of color in an image. This type of layering is not recursion either. It is more like the natural world where one structure is an organ within another completely different kind of structure.

Step Four — Hierarchies of Networks

We see hierarchies of neural nets appearing frequently in the research arising out of the 2000s and early 2010s (Laplacian and others), which continues the interaction between neural nets and continuing the mammalian brain analogy. We now see meta-structure, where entire networks become vertices in a directed graph representing a topology.

Summarizing

Layers have ordinally valued activation functions for vertices and attenuation matrices mapped to an exhaustive set of directed edges between adjacent layers [1]. Image convolution layers are often in two dimensional vertex arrangements with attenuation cubes mapped to an abridged set of directed edges between adjacent layers [2]. Stacks have entire layered nets as vertices in a meta-directed-graph, and those meta-vertices are connected in a sequence with each edge being either a training meta-parameter, a reinforcement (real time feedback) signal, or some other learning control. Hierarchies of nets reflect the notion that multiple controls can be aggregated and direct lower level learning, or the flip case where multiple learning elements can be controlled by one higher level supervisor network.

Analysis of the Trend in Learning Topologies

We can analyze trends in machine learning architecture. We have three topological trends.

  • Depth in the causality dimension — Layers to the signal processing where the output of one layer of activations is fed through a matrix of attenuating parameters (weights) to the input of the next layer. As greater controls are established, only beginning with basic gradient descent in back propatagion, greater depth can be achieved.

  • Input signal dimensionality — From scalar input to hypercubes (video has horizontal, vertical, color depth including transparency, and frame — Note that this is not the same as the number of inputs in the perceptron sense.

  • Topological development — The above two are Cartesian in nature. Dimensions are added at right angles to the existing dimensional. As networks are wired in hierarchies (as in Laplacian Hierarchies) and Möbius strip like circles (as in GANs), the trends are topographical and are best represented by directed graphs where the vertices are not neurons but smaller networks of them.

What Topologies are Missing?

This section expands on the meaning of the title question.

  • Is there any reason why multiple meta-vertices, each representing a neural net, can be arranged such that multiple supervisor meta-vertices can, in conjunction, supervise multiple employee meta-vertices?
  • Why is the back-propagation of an error signal the only non-linear equivalent of negative feedback?
  • Can't collaboration between meta-vertices rather than supervision be employed, where there are two reciprocal edges representing controls?
  • Since neural nets are employed mainly for learning of nonlinear phenomena, why prohibits other types of closed paths in the design of the nets or their interconnection?
  • Is there any reason why sound cannot be added to picture so that video clips can be categorized automatically? If that is the case, is a screenplay a possible feature extraction of a movie and can an adversarial architecture be used to generate screenplays and produce the movies without the movie studio system? What would that topology look like as a directed graph?

Notes

  1. Artificial cells in MLPs use of floating or fixed point arithmetic transfer functions rather than electro-chemical pulse transmissions based on amplitude and proximity based threshold. They are not realistic simulations of neurons, so calling the vertices neurons would be a misnomer for this kind of analysis.

  2. Correlation of image features and relative changes between pixels in close proximity is much higher than that of distant pixels.

$\endgroup$
0
$\begingroup$

Topology is the study of geometric forms differentiated by intersection and bifurcation. The term is used for the graphic aspects network architectures. It is apropos to use it to consider the extension of the neural network analogy, with the understanding that ANNs are not much like biological neurons in the way they activate. Because of that, it is difficult to limit discussion to topological concerns when considering what is largely unexplored.

The supervisor employee paradigm is what stacks and Laplacian hierarchies use, whereas the collaborator paradigm is what adversarial networks use. Although the feedback is negative, the generative model (G) and the discriminative model (D) are actually in collaboration to achieve a goal, as a devils advocate is used in discourse to converge on truths. Certainly other designs where the vertices are not artificial neurons but entire ANNs or CNN elements are forthcoming.

The teacher-student and supervisor-employee paradigms are probably only two of many. To simulate neural plasticity, the gardener-plant, appliance-repairman, and engineer-product paradigms need investigation.

Back-propagation of an error signal isn't the only non-linear equivalent of negative feedback. The circular topology of GANs is negative feedback too, as you indicated in your use of the Möbius strip analogy. There should be more thought along those lines though.

Collaboration between meta-vertices is interesting. Must collaboration be of the pretended adversary type? Can positive feedback be useful in artificial intelligence topologies? Farm owners and food distribution truck drivers buy foods at supermarkets that are at the end of a chain of processes of which their role is only a part. Larger cycles in directed graph representations of topologies and designs can probably employ positive or negative feedback usefully.

The artificial production of motion pictures may come out of research like the Cornell U work on Video Generation From Text — Li, Min, Shen, Carlson, and Carin.

$\endgroup$
-1
$\begingroup$

may be offtop. so, delete it.

in electronic circuitry we have logical blocks - generators, triggers, memory cells, selectors, alus, fpus, buses and many others chips. and from this we have computers, and from next level we have computer networks...

for machine learning we must have a similar organization of things, but if we have 64-bits computers, our neural networks may have more complex inputs/outputs AND more logical functions than defined in any programming language.

so, for X input bits we have X^(2^2) states for one output bit, and 2^X bits for choice a needed logical function.

So, we must consistently study these functions, highlighting the necessary, as first opencv-filters as for examples.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.