10
$\begingroup$

I'm interested in knowing whether there exist any neural network, that solves (with >=80% accuracy) any nontrivial problem, that uses very few nodes (where 20 nodes is not a hard limit). I want to develop an intuition on sizes of neural networks.

$\endgroup$
  • $\begingroup$ Universal approximation theorem: a neural network with one hidden layer can approximate any "reasonable" function given a sufficient number of nodes in the hidden layer. $\endgroup$ – nbro Mar 1 at 15:06
  • 2
    $\begingroup$ I think it will not be easy to answer your question. What do you mean by "non-trivial problem"? $\endgroup$ – nbro Mar 1 at 15:08
  • $\begingroup$ I want few nodes even in hidden layers. $\endgroup$ – Guillermo Mosse Mar 1 at 20:11
  • $\begingroup$ @nbro my sense of "non-trivial" in this context is intractable or unsolved. $\endgroup$ – DukeZhou Mar 1 at 22:54
13
$\begingroup$

Even if it’s impossible to answer this question properly, as non trivial is not well defined (maybe the author will edit this questions later, to specify it better), I take the opportunity to point out this paper which looks interesting to me

Smallest Neural Network to Learn the Ising Criticality

Assuming you have a general idea of the Ising Model I think the problem of identifying the critical temperature from a data driven perspective can be considered as non trivial and the paper shows how the authors have improved the performance related to solve this task with NN passing from 100 Hidden Neurons, as performed in this paper Machine learning phases of matter from 2017, to only 2 Hidden Neurons

Just my cents:

  • reducing the neurons, while keeping good performance, should help in terms of neural processing interpretability which is notoriously obscure and its complexity grows (exponentially) with the number of neurons
$\endgroup$
  • 3
    $\begingroup$ I just wanted examples that the community itself found interesting. I think it is actually non trivial to define non triviality. Right? $\endgroup$ – Guillermo Mosse Mar 1 at 20:12
  • 1
    $\begingroup$ @GuillermoMosse No, it's quite trivial to define non-trivial. The definition is more or less arbitrary depending on what best fits the situation. I'm sure a very simple NN could learn to tell whether or not a given number is a non-trivial square root... $\endgroup$ – forest Mar 2 at 3:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.