Episode #125 of the Stack Overflow podcast is here. We talk Tilde Club and mechanical keyboards. Listen now

Questions tagged [neural-networks]

For questions about a artificial networks, such as MLPs, CNNs, RNNs, LSTM, and GRU networks, their variants or any other AI system components that qualify as a neural networks in that they are, in part, inspired by biological neural networks.

Filter by
Sorted by
Tagged with
69
votes
8answers
12k views

Do scientists know what is happening inside artificial neural networks?

Do scientists or research experts know from the kitchen what is happening inside complex "deep" neural network with at least millions of connections firing at an instant? Do they understand the ...
29
votes
3answers
17k views

Why is Lisp such a good language for AI?

I've heard before from computer scientists and from researchers in the area of AI that that Lisp is a good language for research and development in artificial intelligence. Does this still apply, with ...
14
votes
2answers
8k views

What is the time complexity for training a neural network using back-propagation?

Suppose that a NN contains $n$ hidden layers, $m$ training examples, $x$ features, and $n_i$ nodes in each layer. What is the time complexity to train this NN using back-propagation? I have a basic ...
15
votes
3answers
1k views

How are Artificial Neural Networks and the Biological Neural Networks similar and different?

I've heard multiple times that "Neural Networks are the best approximation we have to model the human brain", and I think it is commonly known that Neural Networks are modelled after our brain. I ...
13
votes
6answers
6k views

What is the purpose of an activation function in Neural Networks?

It is said that activation functions in neural networks help introduce non-linearity. What does this mean? What does non-linearity mean in this context? How does introduction of this non-linearity ...
7
votes
3answers
668 views

What makes animal brain so special?

So this is an introductory question. Whenever I read any book about Neural Nets or Machine Learning, their introductory chapter says that we haven't been able to replicate the brain's power due to its ...
3
votes
1answer
237 views

Is it possible to build human-brain-level artificial intelligence based on neuromorphic chips and neural networks?

I read a lot about the structure of the human brain and artificial neural networks. I wonder if it is possible to build an artificial intelligence with neural networks that would be divided into ...
41
votes
3answers
27k views

How can neural networks deal with varying input sizes?

As far as I can tell, neural networks have a fixed number of neurons in the input layer. If neural networks are used in a context like NLP, sentences or blocks of text of varying sizes are fed to a ...
34
votes
5answers
21k views

How does Hinton's “capsules theory” work?

Geoffrey Hinton has been researching something he calls "capsules theory" in neural networks. What is this and how does it work?
37
votes
4answers
9k views

Are neural networks prone to catastrophic forgetting?

Imagine you show a neural network a picture of a lion 100 times and label with "dangerous", so it learns that lions are dangerous. Now imagine that previously you have shown it millions of images of ...
27
votes
9answers
5k views

Is artificial intelligence vulnerable to hacking?

The paper The Limitations of Deep Learning in Adversarial Settings explores how neural networks might be corrupted by an attacker who can manipulate the data set that the neural network trains with. ...
24
votes
2answers
811 views

How is a deep neural network different from other neural networks?

How is a neural network having the "deep" adjective actually distinguished from other similar networks?
5
votes
3answers
737 views

Are neural networks statistical models?

By reading the abstract of Neural Networks and Statistical Models paper it would seem that ANNs are statistical models. In contrast Machine Learning is not just glorified Statistics. I am looking ...
21
votes
4answers
3k views

Can deep networks be trained to prove theorems?

Assume we have a large number of proofs in first order predicate calculus. Assume we also have the axioms, corollaries, and theorems in that area of mathematics in that form too. Consider the each ...
10
votes
2answers
2k views

Which layer consumes more time in CNN training ? Convolution layers vs FC layers

In Convolutional Neural Network, which layer consumes maximum time in training? Convolution layers or Fully Connected layers? We can take AlexNet architecture to understand this. I want to see time ...
8
votes
1answer
749 views

Mathematical intuition for the use of Re-Lu's in Machine Learning

So, currently the most commonly used activation functions are Re-Lu's. So I answered this question What is the purpose of an activation function in Neural Networks? and while writing the answer it ...
4
votes
1answer
118 views

Where can I find the proof of the universal approximation theorem?

The Wikipedia article for the universal approximation theorem cites a version of the universal approximation theorem for Lebesgue-measurable functions from this conference paper. However, the paper ...
3
votes
1answer
91 views

Use cross-validation to train after model selection

I have been recently reading about model selection algorithms (for example to decide which value of the regularisation parameter or what size of a neural network to use, broadly hyper-parameters). ...
28
votes
4answers
38k views

How to select number of hidden layers and number of memory cells in an LSTM?

I am trying to find some existing research on how to select the number of hidden layers and the size of these of an LSTM-based RNN. Is there an article where this problem is being investigated, i.e., ...
16
votes
3answers
28k views

What is the difference between a Convolutional Neural Network and a regular Neural Network?

I've seen these terms thrown around this site a lot, specifically in the tags convolutional-neural-networks and neural-networks. I know that a Neural Network is a system based loosely on the human ...
14
votes
3answers
21k views

How to handle images of large sizes in CNN?

Suppose there are 10K images of sizes 2400 x 2400 are required to use in CNN.Acc to my view conventional computers the people use will be of use. Now the question is how to handle such large image ...
23
votes
5answers
14k views

Is it possible to train the neural network to solve math equations?

I'm aware that neural networks are probably not designed to do that, however asking hypothetically, is it possible to train the deep neural network (or similar) to solve math equations? So given the ...
23
votes
2answers
8k views

Is it possible to train a neural network incrementally?

I would like to train a neural network where the output classes are not (all) defined from the start. More and more classes will be introduced later based on incoming data. This means that, every time ...
18
votes
4answers
9k views

What is a Dynamic Computational Graph?

Frameworks like PyTorch and TensorFlow through TensorFlow Fold support Dynamic Computational Graphs and are receiving attention from data scientists. However, there seems to be a lack of resource to ...
14
votes
2answers
1k views

What's the main concept behind Capsule Networks? [duplicate]

As you might know, Capsule Networks have been recently introduced by Hinton. There also have been several heads up within his talks. As expected, the paper elaborates on the idea way theoretically! ...
16
votes
3answers
21k views

Understanding GAN loss function

I'm struggling to understand the GAN loss function as provided in Understanding Generative Adversarial Networks (a blog post written by Daniel Seita). In the standard cross-entropy loss, we have an ...
7
votes
4answers
4k views

What is the fundamental difference between CNN and RNN?

What is the fundamental difference between convolutional neural networks and recurrent neural networks? Where are they applied?
21
votes
4answers
7k views

Could a neural network detect primes?

I am not looking for an efficient way to find primes (which of course is a solved problem). This is more of a "what if" question. So, in theory: Could you train a neural network to predict whether ...
9
votes
4answers
7k views

How to classify data which is spiral in shape?

I have been messing around in tensorflow playground. One of the input data sets is a spiral. No matter what input parameters I choose, no matter how wide and deep the neural network I make, I cannot ...
10
votes
1answer
778 views

Why is the merged neural network of AlphaGo Zero more efficient than two separate neural networks?

AlphaGo Zero contains several improvements compared to its predecessors. Architectural details of Alpha Go Zero can be seen in this cheat sheet. One of those improvements is using a single neural ...
9
votes
3answers
844 views

What size of neural networks can be trained on current consumer grade GPUs? (1060,1070,1080)

is it possible to give a rule of thumb estimate about the size of neural networks that are trainable on common consumer grade GPUs? For example: The Emergence of Locomotion (Reinforcement) paper ...
13
votes
2answers
361 views

How should I encode the structure of a neural network into a genome?

For a deterministic problem space, I need to find a neural network with the optimal node and link structure. I want to use a genetic algorithm to simulate many neural networks to find the best network ...
9
votes
2answers
693 views

Power efficiency of human brains vs. neural networks

How big artificial neural networks can we run now (either with full train-backprop cycle or just evaluating network outputs) if our total energy budget for computation is equivalent to human brain ...
7
votes
5answers
2k views

Why do activation functions need to be differentiable in the context of neural networks?

Why should an activation function of a neural network be differentiable? Is it strictly necessary or is it just advantageous?
4
votes
1answer
1k views

Neural Network Cell (Node) Types

I found this nice-ish-looking diagram, but it has a wholly inadequate descriptions for each of the cell types, aside from including names. What is the definition/description of each of these cell ...
3
votes
3answers
434 views

What is the most effective way to learn natural language processing online?

There are many books, courses, etc. out there, but not sure which path to take. So what would be the most effective way (shortest) to learn natural language processing online? p.s. I mean learning ...
7
votes
3answers
1k views

How to represent Tic-Tac-Toe vs Checkers or Chess for a Neural Network

I've been reading a lot about TD-Gammon recently as I'm exploring options for AI in a video-game I'm making. The video game is a turn-based positional sort of game, i.e. a "units", or game piece's, ...
6
votes
2answers
5k views

How is gradient calculated for middle layer weights?

I am trying to understand backpropagation. I used a simple neural network with one input x, one hidden layer h and one output layer y, with weight w1 connecting x to h, and w2 connecting h to y. x--...
5
votes
1answer
3k views

How to avoid falling into the “local minima” trap?

How do I avoid my gradient descent algorithm into falling into the "local minima" trap while backpropogating on my neural network? Are there any methods which help me avoid it?
4
votes
2answers
465 views

Why exactly do neural networks require i.i.d. data?

In reinforcement learning, in general, successive states (actions and rewards) are highly correlated. An "experience replay" buffer was used, in the DQN architecture, to avoid training the neural ...
4
votes
3answers
255 views

Can someone direct me to a sites and/or videos that can bring an absolute beginner up to speed with AI?

To start, I'm not a programmer/computer scientist/et al... - I work in Finance and have, through my job, self-thought myself VBA for excel and outlook and would consider myself as being in the upper ...
3
votes
1answer
56 views

Combining different trained neural networks

I'm relatively new to this whole AI thing and have a question.. Let's say I have two different fully trained neural networks. The first one is trained for mathematical addition and the second one on ...
1
vote
4answers
895 views

How to evaluate a NEAT neural network?

I'm trying to write my own implementation of NEAT and I'm stuck on the network evaluate function, which calculates the output of the network. NEAT as you may know contains a group of neural networks ...
1
vote
1answer
231 views

How do I restrict the neural network structure to be acyclic in NEAT?

I want my neural network structure to not have a circular/looping structure something similar like a directed acyclic graph (DAG). How do I do that?
7
votes
1answer
205 views

Why does a one-layer hidden network get more robust to poor initialization with growing number of hidden neurons?

In a nutshell: I want to understand why a one hidden layer neural network converges to a good minimum more reliably when a larger number of hidden neurons is used. Below a more detailed explanation of ...
6
votes
2answers
551 views

Are ReLUs incapable of solving certain problems?

Background: I've been interested in, and reading about, Neural Networks for several years, but I haven't gotten around to testing them out until recently. Both for fun and to increase my understanding,...
5
votes
3answers
554 views

Use of machine learning for analyzing companies enlisted in stock market

Can current trends and tools, in the field of machine learning, replicate the complexity of financial market? If yes, then what are the tools available in this domain. Q. I am trying to build a model ...
5
votes
2answers
120 views

Neural network to detect “spam”?

I've inherited a neural network project at the company I work for. The person who developed gave me some very basic training to get up and running. I've maintained it for a while. The current neural ...
5
votes
2answers
591 views

Does NEAT require only connection genes to be marked with a global innovation number?

Does NEAT require only connection genes to be marked with a global innovation number? From the NEAT paper Whenever a new gene appears (through structural mutation), a global innovation number is ...
4
votes
2answers
90 views

Can we optimize an optimization algorithm?

In this answer to the question Is an optimization algorithm equivalent to a neural network?, the author stated that, in theory, there is some recurrent neural network that implements a given ...