11
$\begingroup$

Geometry and AI

Matrices, cubes, layers, stacks, and hierarchies are what we could accurately call topologies. Consider topology in this context the higher level geometrical design of a learning system.

As complexity rises, it is often useful to represent these topologies as directed graph structures. State diagrams and Markov's work on game theory are two places where directed graphs are commonly used. Directed graphs have vertices (often visualized as closed shapes) and edges often visualized as arrows connecting the shapes.

We can also represent GANs as a directed graph, where the output of each net drives the training of the other in adversarial fashion. GANs resemble a Möbius strip topologically.

We cannot discover new designs and architectures without understanding not only the mathematics of converging on an optimal solution or tracking one but also topologies of network connections that can support such convergence. It is like first developing a processor while imagining what an operating system would need before writing the operating system.

To glimpse what topologies we have NOT YET considered, let's first look at which ones have been.

Step One — Extrusion in a Second Dimension

In the 1980s, success was achieved with the extension of the original perceptron design. Researchers added a second dimension to create a multi-layered neural network. Reasonable convergence was achieved through back-propagation of an error function's gradient through the gradients of the activation functions attenuated by learning rates and dampened with other meta-parameters.

Step Two — Adding Dimensions to the Discrete Input Signal

We see the emergence of convolutional networks based on existing manually tuned image convolution techniques introduced dimensions to the network input: Vertical position, color components, and frame. This last dimension is critical to CGI, face replacement, and other morphological techniques in contemporary movie making. Without it, we have image generation, categorization, and noise removal.

Step Three — Stacks of Networks

We see stacks of neural nets emerge in the late 1990s, where the training of one network is supervised by another. This is the introduction of conceptual layers, neither in the sense of sequential layers of neurons nor in the sense of layers of color in an image. This type of layering is not recursion either. It is more like the natural world where one structure is an organ within another completely different kind of structure.

Step Four — Hierarchies of Networks

We see hierarchies of neural nets appearing frequently in the research arising out of the 2000s and early 2010s (Laplacian and others), which continues the interaction between neural nets and continuing the mammalian brain analogy. We now see meta-structure, where entire networks become vertices in a directed graph representing a topology.

Step Five %mdash; Departures from Cartesian Orientation

Non-Cartesian systematically repeating arrangements of cells and connections between them have begun to emerge in the literature. For example, Gauge Equivariant Convolutional Networks and the Icosahedral CNN (Taco S. Cohen, Maurice Weiler, Berkay Kicanaoglu, Max Welling, 2019) examines the use an arrangement based on a convex regular icosahedron.

Summarizing

Layers have ordinally valued activation functions for vertices and attenuation matrices mapped to an exhaustive set of directed edges between adjacent layers [1]. Image convolution layers are often in two dimensional vertex arrangements with attenuation cubes mapped to an abridged set of directed edges between adjacent layers [2]. Stacks have entire layered nets as vertices in a meta-directed-graph, and those meta-vertices are connected in a sequence with each edge being either a training meta-parameter, a reinforcement (real time feedback) signal, or some other learning control. Hierarchies of nets reflect the notion that multiple controls can be aggregated and direct lower level learning, or the flip case where multiple learning elements can be controlled by one higher level supervisor network.

Analysis of the Trend in Learning Topologies

We can analyze trends in machine learning architecture. We have three topological trends.

  • Depth in the causality dimension — Layers to the signal processing where the output of one layer of activations is fed through a matrix of attenuating parameters (weights) to the input of the next layer. As greater controls are established, only beginning with basic gradient descent in back propatagion, greater depth can be achieved.

  • Input signal dimensionality — From scalar input to hypercubes (video has horizontal, vertical, color depth including transparency, and frame — Note that this is not the same as the number of inputs in the perceptron sense.

  • Topological development — The above two are Cartesian in nature. Dimensions are added at right angles to the existing dimensional. As networks are wired in hierarchies (as in Laplacian Hierarchies) and Möbius strip like circles (as in GANs), the trends are topographical and are best represented by directed graphs where the vertices are not neurons but smaller networks of them.

What Topologies are Missing?

This section expands on the meaning of the title question.

  • Is there any reason why multiple meta-vertices, each representing a neural net, can be arranged such that multiple supervisor meta-vertices can, in conjunction, supervise multiple employee meta-vertices?
  • Why is the back-propagation of an error signal the only non-linear equivalent of negative feedback?
  • Can't collaboration between meta-vertices rather than supervision be employed, where there are two reciprocal edges representing controls?
  • Since neural nets are employed mainly for learning of nonlinear phenomena, why prohibits other types of closed paths in the design of the nets or their interconnection?
  • Is there any reason why sound cannot be added to picture so that video clips can be categorized automatically? If that is the case, is a screenplay a possible feature extraction of a movie and can an adversarial architecture be used to generate screenplays and produce the movies without the movie studio system? What would that topology look like as a directed graph?
  • Although orthogonally arranged cells can simulate an arbitrary regular packing arrangement of non-orthogonal vertices and edges, is it efficient to do so in computer vision where the tilt of the camera other than plus or minus 90 degrees is common?
  • Is it efficient to arrange individual cells in networks or networks of cells in AI systems orthogonally in learning systems that are aimed at natural language comprehension and assembly or artificial cognition?

Notes

  1. Artificial cells in MLPs use of floating or fixed point arithmetic transfer functions rather than electro-chemical pulse transmissions based on amplitude and proximity based threshold. They are not realistic simulations of neurons, so calling the vertices neurons would be a misnomer for this kind of analysis.

  2. Correlation of image features and relative changes between pixels in close proximity is much higher than that of distant pixels.

$\endgroup$
  • $\begingroup$ i've read this question over once or twice and i must admit i have no idea what is being asked. in particular "topology" doesn't refer to any of the concepts you mention. perhaps you mean "architecture"? but that doesn't seem to make sense either...... i think this question falls squarely within the "does not even" category. $\endgroup$ – k.c. sayz 'k.c sayz' Oct 22 at 22:07
0
$\begingroup$

Topology is the study of geometric forms differentiated by intersection and bifurcation. The term is used for the graphic aspects network architectures. It is apropos to use it to consider the extension of the neural network analogy, with the understanding that ANNs are not much like biological neurons in the way they activate. Because of that, it is difficult to limit discussion to topological concerns when considering what is largely unexplored.

The supervisor employee paradigm is what stacks and Laplacian hierarchies use, whereas the collaborator paradigm is what adversarial networks use. Although the feedback is negative, the generative model (G) and the discriminative model (D) are actually in collaboration to achieve a goal, as a devils advocate is used in discourse to converge on truths. Certainly other designs where the vertices are not artificial neurons but entire ANNs or CNN elements are forthcoming.

The teacher-student and supervisor-employee paradigms are probably only two of many. To simulate neural plasticity, the gardener-plant, appliance-repairman, and engineer-product paradigms need investigation.

Back-propagation of an error signal isn't the only non-linear equivalent of negative feedback. The circular topology of GANs is negative feedback too, as you indicated in your use of the Möbius strip analogy. There should be more thought along those lines though.

Collaboration between meta-vertices is interesting. Must collaboration be of the pretended adversary type? Can positive feedback be useful in artificial intelligence topologies? Farm owners and food distribution truck drivers buy foods at supermarkets that are at the end of a chain of processes of which their role is only a part. Larger cycles in directed graph representations of topologies and designs can probably employ positive or negative feedback usefully.

The artificial production of motion pictures may come out of research like the Cornell U work on Video Generation From Text — Li, Min, Shen, Carlson, and Carin.

$\endgroup$
0
$\begingroup$

Edge of Chaos and Machine Learning; and Benefits in Decision Making


Direct Answer to Your Question:--

Edge of Chaos


Layperson Explanation:--

(https://www.lucd.ai/post/the-edge-of-chaos#!)


What this Answer is About:--

The Edge of Chaos in chaos theory might be an important topic of research in artificial intelligence.

What is the edge of chaos? This field is hypothesised to exist within a wide variety of systems. It has many applications in such fields. This field is a transition zone between interplay between order and disorder.

I am interested in the intersection between A.I. and chaos theory. The edge of chaos serves as a potential topology that is largely unexplored in machine learning.

This is a rich field that offers much potential. It is, both, largely unknown and under-estimated.

I will explore the benefits of analysing such a field in this answer. The benefits show up in decision making, such as the optimal way to invest and manage the manpower in an organisation.


Technical Explantion:--

"Matrices, cubes, layers, stacks, and hierarchies are what we could accurately call topologies. Consider topology in this context the higher level geometrical design of a learning system." ~ Douglas Daseeco, Opening Poster

Compare that to this extract from the paper's abstract below:--

" ... Through dynamical stability analysis on various computer vision models, we find direct evidence that optimal deep neural network performance occur near the transition point separating stable and chaotic attractors. ... " Feng, Ling, and Choy Heng Lai. — "Optimal Machine Intelligence Near the Edge of Chaos." arXiv preprint arXiv:1909.05176 (2019).

-

"The edge of chaos is a transition space between order and disorder that is hypothesized to exist within a wide variety of systems. This transition zone is a region of bounded instability that engenders a constant dynamic interplay between order and disorder.

Even though the idea of the edge of chaos is abstract and unintuitive, it has many applications in such fields as ecology, business management, psychology, political science, and other domains of the social science. Physicists have shown that adaptation to the edge of chaos occurs in almost all systems with feedback." Wikipedia contributors." — "Edge of chaos." Wikipedia, The Free Encyclopedia. Wikipedia, The Free Encyclopedia, 10 Sep. 2019. Web. 22 Sep. 2019.


The Benefits From Studying Such a Field:--

" [...] Strategy, protocol, teams, departments, hierarchies. All meticulously organized for optimal performance.

Or at least, that’s how it’s supposed to be. But when we apply a complexity theorist’s lens to the business we do, we see matters are rather more complex. We no longer view organizations as organizations, or departments as departments, but as complex adaptive systems, most helpfully understood in the three parts:

EMPLOYMENT

Using mental models to make better decisions at work Professional life is littered with difficult choices. Am I ready for this promotion? Which of my managers should I pick as a mentor? What should I eat at lunch? There is no foolproof method for consistently taking the best course of action —even the best of us make mistakes— but, with the right tools, it is possible to maximise the chances of success.

Firstly, employees (in complexity speak: heterogeneous agents). Each employee has different and evolving decision rules that both reflect the environment and attempt to anticipate change in it. Secondly, employees interacting with one another, and the structures that these interactions create – scientists call this emergence. Lastly, the overarching structure that emerges, behaving like a higher-level system with properties and characteristics distinct from those of its underlying agents. This last part is the reason we often say ‘the whole is greater than the sum of its parts’.

Given managers’ desire for control, complexity is far from a convenient reality. Rather than face the brutal reality of the system they are working to sustain, managers often work in silos, creating models and mechanisms that impose a veneer of certainty. In so doing, they help themselves and their colleagues to make decisions with fewer variables. Meeting the goals set out by these models generates evidence of success – but it is a simplified success that may not be in the best interests of the system as a whole.

For instance, placing a rigid priority on maximizing shareholder returns makes things clear for workers: in the case of a difficult tradeoff, the option that lends itself to immediate profitability is the preferable option. But, of course, we are all aware that cutting down on expenses and investments to boost short-term margins can be detrimental to the long-term health of a company. Only by embracing complexity can we effectively balance competing values and priorities (and the effects of decisions on all of them). [...] " — Fresno, Blanca González del. “Order from Chaos: How to Apply Complexity Theory at Work: BBVA.” NEWS BBVA, BBVA, 4 Dec. 2017,< www.bbva.com/en/order-from-chaos-how-to-apply-complexity-theory-at-work/ >.


Further Reading:--


Sources and References:--

$\endgroup$
-1
$\begingroup$

This may be off-topic. If so, delete it.

In electronic circuitry we have logical blocks - generators, triggers, memory cells, selectors, alus, fpus, buses and many others chips. And from this we have computers, and from next level we have computer networks...

For machine learning we must have a similar organization of things, but if we have 64-bits computers, our neural networks may have more complex inputs/outputs AND more logical functions than defined in any programming language.

So, for X input bits we have X^(2^2) states for one output bit, and 2^X bits for choice a needed logical function.

So, we must consistently study these functions, highlighting the necessary, as first opencv-filters as for examples.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.