4
$\begingroup$

So recently I have been learning about new NN's which are used for specialised purposes like speech recognition, image recognition, etc. The more I discover the more I get amazed by the cleverness behind models such as RNN's and CNN's. Questions about the working, intuition, mathematics have been asked a lot in this community, all with vague answers and apparent understandings.

So my question is that, did the researchers come up with these specialised models accidentally or did they follow particular steps to get to the model (like in a mathematical framework)? And how did they look at a particular class of problem and think "Yeah, a better solution might exist"? For me since understanding of NN's are so vague, these are 'high risk, high reward' scenarios, since you might be chasing only the mirage (illusion) of a solution.

$\endgroup$
2
$\begingroup$

Although there is a strong element of "try and see" that has driven successful architectures, the drivers for what to try are often inspired by underlying theory or knowledge from other disciplines.

Specifically for basic CNN, which led to AlexNet and many of the best image processing, the concept of using local receptive fields in layers was inspired by study of neurons in the cat visual system.

Modern RNNs also did not appear out of nowhere, there has long been an appreciation of the difference between a feed-forward network and a recurrently-connected one, and the different applications possible. The step change to LSTM was deliberate response to analysis of problems training simplest forms of RNN.

Like much of science, these things are also driven by success in the real world following the research. Many promising ideas have been tried and rejected. Some have been used for a while then superseded, e.g. using RBMs or stacked auto-encoders to pre-train deep networks before ReLUs and Xavier initialisation were discovered - although both RBMs and auto-encoders still have their niches.

Tweaks to architectures, such as variants of LSTM/GRU, may even be deliberately searched and assessed as part of research. That is done with the explicit knowledge that this part of finding a good design is best done as a search across possibilities.

Despite the evolution-like progress, presenting all such advances as completely random or pure GA-like search is ignoring the conscious effort and research that leads to the designs. If you search literature on any major successful design (such as the existence of RNNs or CNNs in the first place), and read the papers, you will often find that modern neural network architectures have deep roots in older research, plus have mathematical and/or scientific justifications for the choices made.

$\endgroup$
2
$\begingroup$

Researchers may follow specific mathematical frameworks, techniques to come-up with amazing works just like in any field, but I believe in Darwinian natural selection as a base theory for human's discoveries as well as for the Evolutionary Neural Net Architectures.

"Principle by which each slight variation [of a trait], if useful, is preserved".

$\endgroup$
1
$\begingroup$

They are found using the infinite monkeys approach:

The infinite monkey theorem states that a monkey hitting keys at random on a typewriter keyboard for an infinite amount of time will almost surely choose a neural network architecture that appears to work well on the given data set and parameters.

You assign thousand grad students around the world with deep learning tasks. Someone will be lucky at picking the right architecture and parameters and data to get a "good" result. This one gets published. The x999 other grad students are unhappy and wait for their lucky draw.

Yes, I am being cynical. But truth isn't far from this, unfortunately.

$\endgroup$
  • 1
    $\begingroup$ But I don't think the ones that win are collg students $\endgroup$ – DuttaA Jun 22 '18 at 18:32
  • $\begingroup$ If you consider "writing a thesis" to be a win, then they win if their network passes the test. $\endgroup$ – Anony-Mousse Jun 22 '18 at 19:39
  • 1
    $\begingroup$ I am not talking about small improvement networks..I am talking of networks which change the way we thought about the application of NN to a particular problem $\endgroup$ – DuttaA Jun 23 '18 at 3:05
  • 1
    $\begingroup$ That is why you need millions of monkeys. One of them writes Shakespeare. $\endgroup$ – Anony-Mousse Jun 23 '18 at 7:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.