# What are neural networks?

I have recently got into AI and I am eager to learn about its concepts. In some of the information I saw about AI, there was a lot about neural networks. Neural networks seem to be (something along the lines of) a type of algorithm that creates a graph which works based on a theory about how neurons interact, in order to create self-learning programs.

## The graph:

Firstly, the diagram I keep seeing (I'm assuming it's a graph). It shows a set of nodes (which is apparently the input), each directed to each of another set of nodes, each of which is then directed to each of another set of nodes, etc. until it reaches what is apparently the output.

How can something like an image, or a complex piece of data be represented by those input nodes? What goes on at each set of nodes? Does every node from each set have to always be connected to every node from the next set? Do they always have to be directed to the next set, or can they go back and fourth as well? Can I have some code given in relation to one of these diagrams?

(A node is a variable shown in a graph.)

## AI progress:

Secondly, I read that these sort of algorithms could eventually create a conscious program if "enough neurons work together". I do look forward to when people start to manage to do this, but it seems like no one is trying to rethink/expand-on that theory as much as they should. I'd expect people to try to look for new human behaviours to represent with AI. For example: a new-born would take their first breath despite the pain, while usually the mind tries to avoid pain. This causes crying.

Has anyone tried to mimic this sort of behaviour scenario (or any behaviour scenario) in a program? If not, why not? How close are we to creating a conscious mind in a program? Are people challenging the current theory that created our neural network model? Is it lightly for this community to propose a better theory and model?

Neural networks seem to be (something along the lines of) a type of algorithm that creates a graph which works based on a theory about how neurons interact, in order to create self-learning programs.

Technically, a neural network is a combination of

• group of high dimensional arrays/vectors storing 'weights' (more on this soon)
• a list of instructions or algorithms to be used to manipulate these weights

Somewhat similar to what you said, it is easy to visualize a neural network as a graph. Precisely because, in a mathematical sense, a network is a graph. If you consider an abstract entity, a node of the neural network, and assign our weights from the high dimensional arrays/vectors to each edge between such nodes, then you have a way to construct a mathematical graph.

How can something like an image, or a complex piece of data be represented by a few nodes?

It cannot. Which is, ironically, why neural networks are so useful. They condense data and represent it in such a way to get accurate results. This is similar (as an analogy) to what happens in our brain. We don't necessarily store high-quality mental representations. But we do store ways to reconstruct them based on experience, or training, in the neural networks lingo.

Does every node from each set have to always be connected to every node from the next set?

No. Layers of a neural network may be dense (connected) or sparse (partially connected). Additionally, a particular method of training may choose to leave out certain nodes in each cycle, to prevent something called overfitting. This method is called dropout, and is pretty recent.

[...] but it seems like no one is trying to rethink/expand-on that theory as much as they should [...]

Neural networks are not too new, but advances in this field are quite new and recent. For instance, distributed semantic representation of language has seen major activity in the period 2011-2018 (ongoing). Additionally, research in this area requires huge computing power, time, and distributed processing hardware. We are yet discovering ways to optimize almost all of these, and although processor companies may advertise chipsets that compute faster than your thoughts, they are still inadequate to train large-scale neural networks. One reason why our brain is efficient is that each neuron is an individual computation unit. On the other hand, not each node of an artificial neural network is a separate computer. Each node still has to be evaluated and manually processed by one of a computer's many CPU or GPU cores. So work is in progress but certainly, it will take some time to reach even a little bit closer to being able to create actual intelligence.

Can I have some code given in relation to one of these diagrams.

Here you will find a (relatively) intuitive example of a Recurrent Neural Network being used to perform addition. It is coded in python using a library called keras, built on top of tensorflow. It would help to read the keras documentation, to get an idea of what's going on, as well as to find out what else is possible and how you could construct your own neural network.

• The code you have provided does not seem to give any reference to one of those diagrams. I can't run it because I don't seem to have the keras module. Also, sorry if I was unclear, but by "How can something like an image, or a complex piece of data be represented by a few nodes?", I was sort of asking how an image, or complex piece of data gets input to this model. – Super S Jun 21 '18 at 8:26
• @SuperS naturally, to run code written with a particular module/library as a back-end, you would need that module/library – Aalok Jun 25 '18 at 17:16

AI progress:

So you are asking great questions that show your interest in AI. Regarding the progress, there has been amazing progress yearly in neural networks. Neural Networks (NNs) were originally created about 60 years ago but we didn't have even close to the compute power, quantity of data necessary, and memory resources to make real deep learning possible. Deep Learning, as opposed to a shallow NN, simple has many more layers. But with each added layer, the complexity significantly increases. Additionally, activation functions within each layer were updated over the years which has greatly increased NNs capability. Brief history here.

So the last 5 years have been quite remarkable. Groups from MIT, Google, and others have created better and better NNs within each 'family'(RNNs, CNNs, DNNs). But everyone is working on 'narrow' AI at the moment-- I really believe narrow AI the best path forward. Narrow AI is trained on very specific use cases and only truly works in that use case. For example, you couldn't use an autonomous driving AI to help predict stock prices-- they're just separate use cases. However, what you're asking about is called "general AI" which is when a machine has the ability to think across use cases and across domains. To me, we will get there but it starts with narrow AI and we will gradually increase the use cases where we will have a master NN possibly working with each "slave" narrow NN.

Also, if you follow things like the singularity which is the point that AI crosses the intellectual threshold of humans, that would be the epitome of general AI. Experts from Google, Microsoft, and other say the singularity will probably come around the year 2045. As compute power and resources continue to evolve, I believe the pace that we move forward with AI will only accelerate.

You seem to be on the right path. With AI, there are some basic mathematical and computer science fundamentals that you need to be familiar with.

The graph
This is a pictorial representation of how the overall architecture of your neural network(algorithm) is outlined. This is backed by mathematical notation, which is one of the fundamentals you need to be familiar with.
In certain instances, common with research papers, the graph is dropped altogether and statistical methods are used to prove the network (normally Bayesian theorems).

AI progress
Currently we are at the stage of making multiple "strong narrow AI". The reason why we haven't made a complete replica of the human mind is mainly due to these facts:

• The human brain is too complex and we don't fully comprehend it yet.
• The amount of data required in AI is a lot and hard for one person or group to accumulate and process.
• The compute power required to run the algorithms is beyond the reach of many.
• AI talent is just not that abundant. The little that is there is spread across many fields.
• I am aware that we haven't got all that close to creating an actual mind within a computer, but part of my question asked how close we are. We don't fully understand the brain yet, but how many known attributes of the mind have we been able to represent? – Super S Jun 21 '18 at 8:03
• Well no one can pin point the actual time when a singularity might occur but most experts put the peg on 20 years from now. – Simbarashe Timothy Motsi Jun 21 '18 at 8:20

An artificial neural network is simply a glorified template matching algorithm that is trainable. Neurons correspond to templates, and they fire when something that matches their prototype closely enough is presented. The graph concept comes into play because neural networks are usually hierarchical, consisting of many layers, hence the "deep" in "deep learning". This merely makes our templates more flexible and abstract, and it enables feature reuse of lower-level components to construct higher-level features.

Neural networks in particular and machine learning algorithms generally constitute a tool-set of universal function approximators. What this means practically is that you can learn to approximate any function f:

y = f(x)

over a finite domain, given enough paired samples (x,y).

When will artificial neural nets attain consciousness? Never. A function or a function approximator isn't the same thing as consciousness; conscious beings are capable of infinite arbitration and can't be reduced to any finite or deterministic algorithm. Furthermore, data-driven function approximators will always be subject to the usual weaknesses of any finite algorithm (and all algorithms are finite) competing with humans via a Turing test.