11
$\begingroup$

Everything related to Deep Learning (DL) and deep(er) networks seems "successful", at least progressing very fast, and cultivating the belief that AGI is at reach. This is popular imagination. DL is a tremendous tool to tackle so many problems, including the creation of AGIs. It is not enough, though. A tool is a necessary ingredient, but often insufficient.

Leading figures in the domain are looking elsewhere to make progress. This report/claim gathers links to statements by Yoshua Bengio, Yann LeCun and Geoff Hinton. The report also explains:

The main weaknesses of DL (as I see them) are: reliance on the simplest possible model neurons (“cartoonish” as LeCun calls them); use of ideas from 19th century Statistical Mechanics and Statistics, which are the basis of energy functions and log-likelihood methods; and the combination of these in techniques like backprop and stochastic gradient descent, leading to a very limited regime of application (offline, mostly batched, supervised learning), requiring highly-talented practitioners (aka “Stochastic Graduate Descent”), large amounts of expensive labelled training data and computational power. While great for huge companies who can lure or buy the talent and deploy unlimited resources to gather data and crunch it, DL is simply neither accessible nor useful to the majority of us.

Although interesting and relevant, such kind of explanation does not really address the gist of the problem: What is lacking?

The question seems broad, but it may be by lack of a simple answer. Is there a way to pin-point what DL is lacking for an AGI ?

$\endgroup$
7
$\begingroup$

Everyone dealing with neural networks misses an important point when comparing systems with human like intelligence. A human takes many months to do anything intelligible, let alone being able to solve problems where adult humans can barely manage. That and the size of human brain is enormous compared to our neural networks. Direction might be right, but the scale is way off. Number of neurons in human brain can be matched memory-wise but the amount of parallelism to simulate it real-time cannot yet be achieved (at least for a random researcher). While a little old this might give you an idea of how much we lack the processing power.

$\endgroup$
  • $\begingroup$ Thank you for this concise answer. Are you saying that only size matters to achieve an AGI and above, only with DL technologies? Size does matter, but something is probably missing yet. (Any pun in this paragraph is entirely intended). $\endgroup$ – Eric Platon Feb 16 '17 at 22:56
  • 1
    $\begingroup$ For example, let's consider cellular automata a la Wolfram. Very simple, yet leading to surprising complexity. $\endgroup$ – Eric Platon Feb 16 '17 at 23:06
  • $\begingroup$ Is quantum computing, whatever form that may ultimately take, one of the proposed solutions to this processing issue? $\endgroup$ – DukeZhou Feb 17 '17 at 1:25
  • $\begingroup$ Quantum processing can be used to reach to a decision at a point but it cannot be used to simulate continuous flow like in human brain. Once the system is observed quantum waveform collapses, reducing it to essentially a slow sequential system. $\endgroup$ – Cem Kalyoncu Feb 17 '17 at 6:27
  • 1
    $\begingroup$ @CemKalyoncu Indeed. But an elephant has almost 3 times as many as humans. The size argument matters, sure, but size alone does not seem to suffice. $\endgroup$ – Eric Platon Feb 21 '17 at 9:31
6
$\begingroup$

Deep Learning is mostly successful in supervised learning, whereas the brain builds categories mostly in an unsupervised way. We don't yet know how to do that. (Take a look at google brain: 16,000 cores and all this thing can do is recognise cats and human faces with pretty abysmal accuracy.)

Deep Learning uses highly unstructured activations, i.e. the high level representations of "dog" and "cat" in a neural network classifier don't have to be similar at all. The brain on the other hand uses inhibitory neurons to create sparse distributed representations which are decomposable into their semantic aspects. That's probably important for abstraction and reasoning by analogy.

The brain has many different parts which work together. Deep Learning researchers are only just beginning to integrate memory or attention mechanisms into their architecture.

The brain integrates information from many different senses. Most Deep Learning applications use just one type of input, like text or pictures.

The brain is capable of modelling sequences as categories. (Basically every verb names a sequential (i.e. temporal) category.) It can then arrange these categories into long-term hierarchical plans. So far I haven't seen anything in that direction in Deep Learning.

Also neural networks can't yet operate on the same scale as the human brain. If you look at the answers to this question, the human brain will be ahead in neuron count for another couple of decades. A neural network might not need the same number of neurons as the brain to reach a similar performance (because of higher accuracy), but right now for example video processing is still pretty limited in terms of input and throughput.

$\endgroup$
  • $\begingroup$ Interesting points here too, thank you. My concern here is that it is contrasting process ([deep] learning) and structure (of the network or brain). If this trend is correct, AGI is just a question of time based on what we have. You do mention semantic issues in deep networks, probably best viewed in adversarial models. This indicates that something is missing, and is one of the best argument in this answer. I do understand that current structures are insufficient (i.e. early memory models). This is indirectly addressing the "why" issue, though. Do you see ways to refine your answer? $\endgroup$ – Eric Platon Feb 17 '17 at 0:42
  • $\begingroup$ Are the recent "optical only" approaches to ML game play an attempt to break free from supervised learning? $\endgroup$ – DukeZhou Feb 17 '17 at 1:28
  • $\begingroup$ @DukeZhou: I do think that RL-techniques might play a role in unsupervised learning, but right now it seems to me that RL isn't learning high level concepts, yet. $\endgroup$ – BlindKungFuMaster Feb 17 '17 at 7:30
  • 1
    $\begingroup$ @EricPlaton: I'm not sure I understand your comment. What I think is missing is 1. structure and 2. scale. And of course algorithms, but those are intertwined with the structure. $\endgroup$ – BlindKungFuMaster Feb 17 '17 at 7:42
3
$\begingroup$

IMHO the first hurdle is scale: even Google's largest DNN doesn't come close to the scale of the brain, and by a factor of several orders of magnitude...

$\endgroup$
2
$\begingroup$

I think it's missing still the aspects what makes a human brain; having a lot of different networks working with each other.

Just like meditation improves cognitive abilities by having the brain work more synergistically, we could apply that to machines too.

For example google is learning a computer to dream, just like we do, to reinforce what we already learned. https://medium.com/@tannistho/why-is-google-teaching-its-ai-to-dream-e9ae9ecd0e3a#.gljal6pww

And here is pathnet, a network of neural network. https://medium.com/@thoszymkowiak/deepmind-just-published-a-mind-blowing-paper-pathnet-f72b1ed38d46#.ed0f6pdq7

Creating all these mechanics and putting them all together, with enough power and we will get pretty close!

$\endgroup$
  • $\begingroup$ Could you clarify what "it" is? It could be the process of deep learning, or equally deep networks. These are different. $\endgroup$ – Eric Platon Feb 16 '17 at 22:51
  • 1
    $\begingroup$ Both, but mainly having deep neural networks work with each other, I speculate that DNN should also have good neural plasticity features. But this is something we could only touch the basics on, we don't even know even know how exactly a human brain works $\endgroup$ – Alexander Feb 17 '17 at 9:58
1
$\begingroup$

Artificial intelligence proponents today are focused on the problem of computability - the ability to solve complex problems fast. It is my belief that any amount of success in this direction will not lead to human (general) intelligence although it certainly will outperform humans in certain domains. Instead, efforts should be toward a study of what neurological events cause sensation (the experience of qualia). Of course, this is the hard problem of philosophy but I believe it is the unique key to general intelligence and its capabilities. Reverse engineering and also testable theories should be advanced toward this end.

$\endgroup$
  • 1
    $\begingroup$ Quaila might be an interesting trait for machines to have (especially if we want people to give rights to these machines), but aside from quaila itself being a very hard problem in philosophy, there's two main issues at play. (A) intelligence itself may not require quaila, you can be smart without being able to have subjective first-person experiences..i.e., the philosophical zombie. $\endgroup$ – Left SE On 10_6_19 Feb 20 '17 at 6:55
  • $\begingroup$ (B) Industry only cares about solving complex problems fast and doesn't really worry about whether said fast machine can think or feel. AGI is only desired to the extent that it could solve a complex problems fast...intelligence is only a means to an end. (In fact, industry may not want a thinking and feeling machine, as such machines may deserve rights...and rights are kinda like regulations, limiting what a business can do with their tool.) $\endgroup$ – Left SE On 10_6_19 Feb 20 '17 at 7:00

Not the answer you're looking for? Browse other questions tagged or ask your own question.