# What is an artificial neural network?

What is an artificial neural network in artificial intelligence?

It is apparently used to find patterns in data and it is loosely inspired by human neural networks.

• en.wikipedia.org/wiki/Artificial_neural_network – Oliver Mason Apr 23 '19 at 16:52
• Very surprisingly, this doesn't appear to be a duplicate of any existing question on this site (at least, I cannot find one). I don't think this is too broad to answer, so probably it should not be closed. – John Doucette Apr 23 '19 at 16:56
• @JohnDoucette It is difficult to describe NNs in one answer, so that a person that knows nothing about them understands them after having read such answer. However, we can try to give simplistic and, hopefully, intuitive answers. – nbro Apr 23 '19 at 18:16

You can find a lot of content regarding the mathematical definitions. Let's take it simpler: It is a universal function approximator. Think about this: You can approximate this locus with a line, two parameters: slope and offset. That's the simplest you can do, with a straight line. You approximated it with planes and curvatures, didn't you? That's a little gist of what Neural Networks do.

• They approximate functions with given I/O.
• They find patterns.
• They find behaviours in data (image, text, speech, numbers).

We think it is artificial intelligence, and it has worked out pretty well so far.

• A neural network is a universal function approximator provided certain conditions are satisfied, so not all NNs are univeral function approximators. Furthermore, not all function approximators are NNs. – nbro Apr 28 '19 at 13:01
• You're right. They approximate functions, but for certain I/O. Finding patterns using numbers and matrices can be strongly linked to the behaviours of Neural Networks. – Karan Shah Apr 28 '19 at 13:04

An (artificial) neural network (NN) can be described at different levels of detail, but, essentially, an NN is a data structure (or model), which behaves like a function: that is, it can receive one or more inputs and it produces one or more outputs.

Most of the NNs can be thought of as graph-like data structures that contain nodes which are connected by weighted edges, where some of the nodes are associated with the inputs and some of the nodes are associated with the outputs. The remaining nodes are often called "hidden nodes". However, there are NNs which might not be easily understood as a graph-like structure: for example, convolutional neural networks.

The number of nodes and the connections between them defines the architecture of the NN and, intuitively, its "expressivity", that is, the number or type of functions it can approximate. Each node in an NN often computes a "function" of its inputs: for example, it could sum the inputs.

The weights of the edges connecting the nodes can change, during a "learning or training phase", so that the NN produces the desired outputs given certain inputs. For example, we can train an NN to "approximate" the function $$f(x) = x$$. Thus, an NN is often described as a "function approximator".

There are several algorithms that can be used to "train" an NN to approximate a desired (and often unknown) function. The most common one is based on an optimisation algorithm (like gradient descent) and the back-propagation algorithm. In order to approximate a function, an NN requires data, often in the form of pairs of inputs and outputs, $$(x, y)$$, where $$x$$ is the input and $$y$$ is the output of the function $$f$$ (that we want to approximate). Intuitively, after the learning phase, an NN that receives an input similar to $$x$$, denoted by $$x'$$, will then produce an output similar to $$y$$, $$y'$$.

There are a lot of tasks that can be thought of as functions. For example, the task of machine translation can be thought of as a function $$f(x) = y$$, where the input, $$x$$, is a sentence in one language (e.g. English) and the output, $$y$$, is a sentence in another language (e.g. Chinese). In this case, $$f$$ is unknown (to a lot of people), but an NN can still learn (from the data) an approximation of it. Therefore, NNs have wide applicability and are thus quite useful.

• I tried to simplify as much as I could and not to describe or mention certain details (e.g. the ones associated with the training phase). I will add a picture later. – nbro Apr 23 '19 at 18:06
• To be more precise, a neural network needs a datamodel as input vector. This input vector is stored on the harddrive, but never the weights of the neural network. The neural network can be understand as an index for the content. – Manuel Rodriguez Apr 24 '19 at 11:49
• @ManuelRodriguez What the heck are you talking about?! I don't know which sources you use to study and acquire knowledge, but it seems like you're reading "alternative" books. – nbro Apr 24 '19 at 12:22

Defining Contemporary Artificial Networks

The definition of an artificial neural network is a challenge. The most traveled sites on the Internet do not provide any adequate formal definitions and include much misconception. Wikipedia's articles contain some truth but read much like a blog in places, with too much ambiguity and misconception to be considered authoritative.

The question in the body of this question is phrased in the wider field of AI rather than in the narrower sub-field of machine learning, which is excellent.

What is an artificial neural network in artificial intelligence?


That's the question to be answered in this particular post.

The answer given in the question is a synopsis of common understanding in a single sentence.

It is apparently used to find patterns in data
and it is loosely inspired by human neural networks.


Common Misconception

This synopsis sentence is an accurate representation of common understanding, but it that common conception is inaccurate in a few ways.

• There is very little in current artificial networks inspired specifically from human brains. Insect brains could just as easily be construed as the inspiration. Current artificial networks do not yet simulate the cognitive abilities of the cerebral cortex of people.
• Pattern recognition can be accomplished with an artificial network, but that is not their defining characteristic. When machine learning is directed at the task of recognizing a pattern, it is not the same as targeting the AlphaZero at winning a Chess or Go tournament. It is not the same as a child recognizing her or his mother's face when walking among many adults at the mall. The concept class of a specific pattern is trained into the network by providing a loss function that compares the network output with an expectation from labels, some external signal, or a correlative model.
• Current artificial networks don't actually recognize anything. They can be trained to do many things that are tiny subsets of recognition, but they don't see and understand that what they are seeing is a car, pedestrian, missile, submarine, tapped hole into which a screw can be screwed, or crumb to be vacuumed up. They don't think, "Oh no, and ICBM. I'd better send out some countermeasures and notify the President." They don't know what a missile is, but they can detect the features of one, having been trained to do so, and perform some action they were trained to perform.

What Then is an Artificial Network in the Context of AI?

Whatever definition we pick, it must apply to unadorned feed forward networks, CNNs, attention based networks, GRU cell networks, and other designs inspired by the original Perceptrons and CNNs, developed prior to the discovery of usable gradient descent and back-propagation mathematics and corresponding software technique.

Our fitting of artificial networks into the context of AI is limited by the lack of cross-cultural or academic consensus on what intelligence is. Because of that, we cannot yet produce a formal definition on what qualifies as artificial intelligence.

Allan Turing's Imitation Game was intended by him to be a thought experiment to consider what we think of as intelligence when in conversation. He never intended it to be a definition or a test of intelligence, but rather a way to test natural language and perhaps cognitive ability in machines against a human. If the machine could not be differentiated from the human in a double blind experiment, then the machine was, at at least a superficial level, acting human.

Definition By Universal Features of Artificial Networks

Let's consider what is common to all contemporary artificial networks.

• The idea of interconnected cells that together carry a signal, often in sequences of layers but not necessarily so, producing a structure that is conducive to learning an output behavior as a function of input stimuli: $$\mathcal{Y} = f(\mathcal{X})$$, where $$\mathcal{X}$$ is a representation of input stimuli and \$\mathcal{Y} is a representation of output behavior. — If the cells are, as is the typical convention, arranged in layers, signals propagate from the input layer to the output layer to produce a behavior. Each layer contain cells identical to one another except for the values of signal attenuation parameters (usually called weights or simply parameters), which can be initialized in some way and then change in subsequent learning iterations based improvement strategies that have been shown to work both theoretically and empirically.

• The mathematical concept of convergence on an optimum set of output behaviors in response to input stimuli, which we call training, and which relies on a numerical computation that indicates the degree of optimization — This can be called the error function, loss function, deviation function, pain function, or disparity function. The inverse function can be called a gain function, value function, pleasure function, accuracy function, or some other similar name. In systems theory terms, these are names of a feedback device, a corrective signal that causes a well designed learning system to adapt its behavior to hopefully achieve the optimum or sufficiently close to it to satisfy the system requirements. It is a hope because artificial networks are not guaranteed to converge on an optimum in every situation. There may be a deficiency in the training data, its presentation to the learning system, computing resources, overall approach, or hyper-parameter.

• Cell non-linearity — not representable as a first degree polynomial of the form $$y = A \, \vec{x} + B$$, where $$\vec{x}$$ is the cell input vector from immediately upstream cells in the network and $$y$$ is the cell output — This is a critical requirement to permit the learning of nontrivial behavioral complexity. Although this first degree polynomial representation is the representation of a typical attenuation at the front end of the cell, at which time $$A$$ is a parameter vector or matrix and $$B$$ is an offset, the right hand side of the expression is enclosed in what is normally called an activation function. This name arose from the obsolete conception of neurons, when it was thought that some learned synaptic attenuation of signals (the weights in $$A$$) and some learned threshold (-$$B$$) determined when a neuron fired. This is now known to be a gross oversimplification, for reasons about two thirds of the way down in this answer.

Although there are many other characteristics that are common across many artificial network types, such as gradient descent, back-propagation, learning rate strategies, the set of commonly used activation functions, hyper-parameters, batching, data set analytics, approaches to training and use, and libraries used, none of these are absolute requirements on a design that makes it an artificial network. These three sum up the universal requirements.

Fitting Artificial Networks in AI

We can see that components with the above three characteristics have been used effectively in many systems that bear marks of intelligence. Some of the most notable cases can be enumerated.

• The ability to prevail in the games of chess and go
• The ability to identify and locate objects of certain types in an image
• The ability to map trajectories or expression changes in a movie
• The ability to generate images of certain types

The question of whether these abilities are examples of intelligence is debated. Artificial networks cannot get honest and explain to some humans why it made a particular game move, how it determined what pixels were what object, or what it had in mind when producing a new living room interior design. Artificial networks are not yet cognitive in the way a cognitive scientist uses that term, although no one has proved that they cannot one day develop cognitive abilities.

References

1. Convergence of ion channel genome content in early animal evolution, Benjamin J. Liebeskind, David M. Hillis, and Harold H. Zakon, 2014

2. A measure for brain complexity: relating functional segregation and integration in the nervous system, G Tononi, O Sporns, and G M Edelman, PNAS, 1994

3. How artificial networks and biological neural systems are dissimilar