Questions tagged [neural-networks]

For questions about a artificial networks, such as MLPs, CNNs, RNNs, LSTM, and GRU networks, their variants or any other AI system components that qualify as a neural networks in that they are, in part, inspired by biological neural networks.

Filter by
Sorted by
Tagged with
81
votes
8answers
14k views

Do scientists know what is happening inside artificial neural networks?

Do scientists or research experts know from the kitchen what is happening inside complex "deep" neural network with at least millions of connections firing at an instant? Do they understand the ...
32
votes
3answers
20k views

Why is Lisp such a good language for AI?

I've heard before from computer scientists and from researchers in the area of AI that that Lisp is a good language for research and development in artificial intelligence. Does this still apply, with ...
11
votes
2answers
2k views

Where can I find the proof of the universal approximation theorem?

The Wikipedia article for the universal approximation theorem cites a version of the universal approximation theorem for Lebesgue-measurable functions from this conference paper. However, the paper ...
2
votes
1answer
517 views

Is back propagation applied for each data point or for a batch of data points?

I am new to deep learning and trying to understand the concept of back propagation. I have a doubt on when the back propagation is applied. Assume that I have a training data set of 1000 images for ...
22
votes
6answers
11k views

What is the purpose of an activation function in neural networks?

It is said that activation functions in neural networks help introduce non-linearity. What does this mean? What does non-linearity mean in this context? How does the introduction of this non-...
22
votes
5answers
17k views

What is the time complexity for training a neural network using back-propagation?

Suppose that a NN contains $n$ hidden layers, $m$ training examples, $x$ features, and $n_i$ nodes in each layer. What is the time complexity to train this NN using back-propagation? I have a basic ...
47
votes
4answers
11k views

Are neural networks prone to catastrophic forgetting?

Imagine you show a neural network a picture of a lion 100 times and label with "dangerous", so it learns that lions are dangerous. Now imagine that previously you have shown it millions of images of ...
12
votes
2answers
2k views

What is a recurrent neural network?

Surprisingly, this wasn't asked before - at least I didn't find anything besides some vaguely related questions. So, what is a recurrent neural network, and what are their advantages over regular (or ...
24
votes
3answers
4k views

Can neural networks be used to prove conjectures?

Imagine I have a list (in a computer-readable form) of all problems (or statements) and proofs that math relies on. Could I train a neural network in such a way that, for example, I enter a problem ...
17
votes
3answers
1k views

How are Artificial Neural Networks and the Biological Neural Networks similar and different?

I've heard multiple times that "Neural Networks are the best approximation we have to model the human brain", and I think it is commonly known that Neural Networks are modelled after our brain. I ...
4
votes
2answers
241 views

How to estimate the capacity of a neural network?

Is it possible to estimate the capacity of a neural network model? If so, what are the techniques involved?
30
votes
9answers
5k views

Is artificial intelligence vulnerable to hacking?

The paper The Limitations of Deep Learning in Adversarial Settings explores how neural networks might be corrupted by an attacker who can manipulate the data set that the neural network trains with. ...
15
votes
3answers
6k views

How to choose an activation function?

I choose the activation function for the output layer depending on the output that I need and the properties of the activation function that I know. For example, I choose the sigmoid function when I'm ...
7
votes
3answers
741 views

What makes animal brain so special?

So this is an introductory question. Whenever I read any book about Neural Nets or Machine Learning, their introductory chapter says that we haven't been able to replicate the brain's power due to its ...
3
votes
1answer
254 views

Is it possible to build human-brain-level artificial intelligence based on neuromorphic chips and neural networks?

I read a lot about the structure of the human brain and artificial neural networks. I wonder if it is possible to build an artificial intelligence with neural networks that would be divided into ...
36
votes
6answers
21k views

How do capsule neural networks work?

Geoffrey Hinton has been researching something he calls "capsules theory" in neural networks. What is it? How do capsule neural networks work?
59
votes
3answers
39k views

How can neural networks deal with varying input sizes?

As far as I can tell, neural networks have a fixed number of neurons in the input layer. If neural networks are used in a context like NLP, sentences or blocks of text of varying sizes are fed to a ...
40
votes
4answers
58k views

How to select number of hidden layers and number of memory cells in an LSTM?

I am trying to find some existing research on how to select the number of hidden layers and the size of these of an LSTM-based RNN. Is there an article where this problem is being investigated, i.e., ...
20
votes
5answers
20k views

Can a neural network be used to predict the next pseudo random number?

Is it possible to feed a neural network the output from a random number generator and expect it learn the hashing (or generator) function, so that it can predict what will be the next generated pseudo-...
28
votes
2answers
1k views

How is a deep neural network different from other neural networks?

How is a neural network having the "deep" adjective actually distinguished from other similar networks?
10
votes
5answers
6k views

What is the fundamental difference between CNN and RNN?

What is the fundamental difference between convolutional neural networks and recurrent neural networks? Where are they applied?
5
votes
2answers
2k views

Are neural networks statistical models?

By reading the abstract of Neural Networks and Statistical Models paper it would seem that ANNs are statistical models. In contrast Machine Learning is not just glorified Statistics. I am looking ...
7
votes
2answers
1k views

Why exactly do neural networks require i.i.d. data?

In reinforcement learning, in general, successive states (actions and rewards) are highly correlated. An "experience replay" buffer was used, in the DQN architecture, to avoid training the neural ...
7
votes
2answers
2k views

When should I use 3D convolution?

I am new to convolutional neural networks, and I am learning 3D convolution. What I could understand is that 2D convolution gives us relationships between low-level features in the X-Y dimension, ...
7
votes
1answer
856 views

What is the time complexity of the forward pass algorithm of a neural network?

How do I determine the time complexity of the forward pass algorithm of a feedforward neural network? How many multiplications are done to generate the output?
5
votes
3answers
201 views

How is neural architecture search performed?

I have come across something that IBM offers called neural Architecture search. You feed it a data set and it outputs an initial neural Architecture that you can train. How is neural architecture ...
10
votes
1answer
1k views

Mathematical intuition for the use of Re-Lu's in Machine Learning

So, currently the most commonly used activation functions are Re-Lu's. So I answered this question What is the purpose of an activation function in Neural Networks? and while writing the answer it ...
2
votes
1answer
690 views

What are some examples of LSTM architectures?

I've been doing some class assignments recently on building various neural networks. For convolutional networks, there are several well-known architectures such as LeNet, VGG etc. Such "classic" ...
5
votes
1answer
110 views

What is the formula used to calculate the loss in the FaceNet model?

The FaceNet model returns the loss of the predictions and ground-truth classes. How is this loss calculated?
3
votes
1answer
148 views

Use cross-validation to train after model selection

I have been recently reading about model selection algorithms (for example to decide which value of the regularisation parameter or what size of a neural network to use, broadly hyper-parameters). ...
52
votes
11answers
8k views

What are some well-known problems where neural networks don't do very well?

What are some well-known cases, problems or real-world applications where neural networks don't do very well? Specification: I'm looking for specific regression tasks (with accessible data-sets) ...
27
votes
6answers
47k views

What is the difference between a convolutional neural network and a regular neural network?

I've seen these terms thrown around this site a lot, specifically in the tags convolutional-neural-networks and neural-networks. I know that a neural network is a system based loosely on the human ...
16
votes
3answers
28k views

How do I handle large images when training a CNN?

Suppose that I have 10K images of sizes $2400 \times 2400$ to train a CNN. How do I handle such large image sizes without downsampling? Here are a few more specific questions. Are there any ...
27
votes
5answers
18k views

Is it possible to train the neural network to solve math equations?

I'm aware that neural networks are probably not designed to do that, however asking hypothetically, is it possible to train the deep neural network (or similar) to solve math equations? So given the ...
24
votes
2answers
10k views

Is it possible to train a neural network incrementally?

I would like to train a neural network where the output classes are not (all) defined from the start. More and more classes will be introduced later based on incoming data. This means that, every time ...
19
votes
4answers
11k views

What is a Dynamic Computational Graph?

Frameworks like PyTorch and TensorFlow through TensorFlow Fold support Dynamic Computational Graphs and are receiving attention from data scientists. However, there seems to be a lack of resource to ...
27
votes
4answers
11k views

Could a neural network detect primes?

I am not looking for an efficient way to find primes (which of course is a solved problem). This is more of a "what if" question. So, in theory: Could you train a neural network to predict whether ...
9
votes
6answers
10k views

How to classify data which is spiral in shape?

I have been messing around in tensorflow playground. One of the input data sets is a spiral. No matter what input parameters I choose, no matter how wide and deep the neural network I make, I cannot ...
19
votes
3answers
24k views

Understanding GAN loss function

I'm struggling to understand the GAN loss function as provided in Understanding Generative Adversarial Networks (a blog post written by Daniel Seita). In the standard cross-entropy loss, we have an ...
10
votes
1answer
854 views

Why is the merged neural network of AlphaGo Zero more efficient than two separate neural networks?

AlphaGo Zero contains several improvements compared to its predecessors. Architectural details of Alpha Go Zero can be seen in this cheat sheet. One of those improvements is using a single neural ...
11
votes
2answers
4k views

Which layer in a CNN consumes more training time: convolution layers or fully connected layers?

In a convolutional neural network, which layer consumes more training time: convolution layers or fully connected layers? We can take AlexNet architecture to understand this. I want to see the time ...
9
votes
3answers
1k views

What size of neural networks can be trained on current consumer grade GPUs? (1060,1070,1080)

is it possible to give a rule of thumb estimate about the size of neural networks that are trainable on common consumer grade GPUs? For example: The Emergence of Locomotion (Reinforcement) paper ...
14
votes
2answers
418 views

How should I encode the structure of a neural network into a genome?

For a deterministic problem space, I need to find a neural network with the optimal node and link structure. I want to use a genetic algorithm to simulate many neural networks to find the best network ...
9
votes
2answers
785 views

Power efficiency of human brains vs. neural networks

How big artificial neural networks can we run now (either with full train-backprop cycle or just evaluating network outputs) if our total energy budget for computation is equivalent to human brain ...
9
votes
5answers
3k views

Why do activation functions need to be differentiable in the context of neural networks?

Why should an activation function of a neural network be differentiable? Is it strictly necessary or is it just advantageous?
8
votes
2answers
5k views

How do we choose the kernel size depending on the problem?

Obviously, finding suitable hyper-parameters for a neural network is a complex task and problem or domain-specific. However, there should be at least some "rules" that hold most times for the size of ...
6
votes
1answer
6k views

How is the gradient calculated for the middle layer's weights?

I am trying to understand backpropagation. I used a simple neural network with one input $x$, one hidden layer $h$ and one output layer $y$, with weight $w_1$ connecting $x$ to $h$, and $w_2$ ...
5
votes
1answer
3k views

What is the definition of each of these neural network cell types?

I found this nice-ish-looking diagram, but it has a wholly inadequate descriptions for each of the cell types, aside from including names. What is the definition/description of each of these cell ...
4
votes
5answers
2k views

How to evaluate a NEAT neural network?

I'm trying to write my own implementation of NEAT and I'm stuck on the network evaluate function, which calculates the output of the network. NEAT as you may know contains a group of neural networks ...
3
votes
3answers
481 views

What is the most effective way to learn natural language processing online? [closed]

There are many books, courses, etc. out there, but not sure which path to take. So what would be the most effective way (shortest) to learn natural language processing online? p.s. I mean learning ...