Questions tagged [deep-learning]

For questions related to deep learning, which refers to a subset of machine learning methods based on artificial neural networks (ANNs) with multiple hidden layers. The adjective deep thus refers to the number of layers of the ANNs. The expression deep learning was apparently introduced (although not in the context of machine learning or ANNs) in 1986 by Rina Dechter in the paper "Learning while searching in constraint-satisfaction-problems".

Filter by
Sorted by
Tagged with
85
votes
8answers
15k views

Do scientists know what is happening inside artificial neural networks?

Do scientists or research experts know from the kitchen what is happening inside complex "deep" neural network with at least millions of connections firing at an instant? Do they understand the ...
4
votes
2answers
337 views

How to estimate the capacity of a neural network?

Is it possible to estimate the capacity of a neural network model? If so, what are the techniques involved?
32
votes
3answers
22k views

Why is Lisp such a good language for AI?

I've heard before from computer scientists and from researchers in the area of AI that that Lisp is a good language for research and development in artificial intelligence. Does this still apply, with ...
29
votes
5answers
14k views

What is the purpose of an activation function in neural networks?

It is said that activation functions in neural networks help introduce non-linearity. What does this mean? What does non-linearity mean in this context? How does the introduction of this non-...
3
votes
1answer
489 views

How can the convolution operation be implemented as a matrix multiplication?

How can the convolution operation used by CNNs be implemented as a matrix-vector multiplication? We often think of the convolution operation in CNNs as a kernel that slides across the input. However, ...
46
votes
8answers
37k views

In a CNN, does each new filter have different weights for each input channel, or are the same weights of each filter used across input channels?

My understanding is that the convolutional layer of a convolutional neural network has four dimensions: input_channels, filter_height, filter_width, number_of_filters. Furthermore, it is my ...
18
votes
3answers
8k views

How to choose an activation function?

I choose the activation function for the output layer depending on the output that I need and the properties of the activation function that I know. For example, I choose the sigmoid function when I'm ...
5
votes
1answer
413 views

What is a graph neural network?

What is a graph neural network (GNN)? Here are some sub-questions How is a GNN different from a NN? How exactly is a GNN related to graphs? What are the components of a GNN? What are the inputs and ...
2
votes
1answer
185 views

Why do we need convolutional neural networks instead of feed-forward neural networks?

Why do we need convolutional neural networks instead of feed-forward neural networks? What is the significance of a CNN? Even a feed-forward neural network will able to solve the image classification ...
2
votes
1answer
118 views

How is the depth of filters of hidden layers determined?

I am a bit confused about the depth of the convolutional filters in a CNN. At layer 1, there are usually about 40 3x3x3 filters. Each of these filters outputs a 2d array, so the total output of the ...
5
votes
2answers
2k views

Are neural networks statistical models?

By reading the abstract of Neural Networks and Statistical Models paper it would seem that ANNs are statistical models. In contrast Machine Learning is not just glorified Statistics. I am looking ...
11
votes
1answer
282 views

What are the state-of-the-art results on the generalization ability of deep learning methods?

I've read a few classic papers on different architectures of deep CNNs used to solve varied image-related problems. I'm aware there's some paradox in how deep networks generalize well despite ...
65
votes
3answers
44k views

How can neural networks deal with varying input sizes?

As far as I can tell, neural networks have a fixed number of neurons in the input layer. If neural networks are used in a context like NLP, sentences or blocks of text of varying sizes are fed to a ...
53
votes
11answers
9k views

What are some well-known problems where neural networks don't do very well?

Background: It's well-known that neural networks offer great performance across a large number of tasks, and this is largely a consequence of their universal approximation capabilities. However, in ...
26
votes
4answers
11k views

Is it possible to train a neural network as new classes are given?

I would like to train a neural network (NN) where the output classes are not (all) defined from the start. More and more classes will be introduced later based on incoming data. This means that, every ...
21
votes
5answers
24k views

Can a neural network be used to predict the next pseudo random number?

Is it possible to feed a neural network the output from a random number generator and expect it learn the hashing (or generator) function, so that it can predict what will be the next generated pseudo-...
13
votes
1answer
6k views

How can policy gradients be applied in the case of multiple continuous actions?

Trusted Region Policy Optimization (TRPO) and Proximal Policy Optimization (PPO) are two cutting edge policy gradients algorithms. When using a single continuous action, normally, you would use some ...
6
votes
2answers
434 views

Is reinforcement learning using shallow neural networks still deep reinforcement learning?

Often times I see the term deep reinforcement learning to refer to RL algorithms that use neural networks, regardless of whether or not the networks are deep. For example, PPO is often considered a ...
9
votes
2answers
467 views

What is experience replay in laymen's terms?

I've been reading Google's DeepMind Atari paper and I'm trying to understand the concept of "experience replay". Experience replay comes up in a lot of other reinforcement learning papers (...
6
votes
1answer
2k views

Why do ResNets avoid the vanishing gradient problem?

I read that, if we use the sigmoid or hyperbolic tangent activation functions in deep neural networks, we can have some problems with the vanishing of the gradient, and this is visible by the shapes ...
12
votes
2answers
6k views

How to implement an “unknown” class in multi-class classification with neural networks?

For example, I need to detect classes for MNIST data. But I want to have not 10 classes for digits, but also I want to have 11th class "not a digit", so that any letter, any other type of ...
11
votes
3answers
246 views

What is a deep neural network? [duplicate]

What is the definition of a deep neural network? Why are they so popular or important?
4
votes
1answer
424 views

How to detect vanishing gradients?

Can vanishing gradients be detected by the change in distribution (or lack thereof) of my convolution's kernel weights throughout the training epochs? And if so how? For example, if only 25% of my ...
3
votes
1answer
66 views

What's the right way of building a deep Q-network?

I'm new to RL and to deep q-learning and I have a simple question about the architecture of the neural network to use in an environment with a continous state space a discrete action space. I tought ...
1
vote
1answer
264 views

How to generate labels for self-supervised training?

I've been reading a lot lately about self-supervised learning and I didn't understand very well how to generate the desired label for a given image. Let's say that I have an image classification task, ...
7
votes
2answers
2k views

How to write a C decompiler using AI?

I would like to learn more about whether it is possible and how to write a program that decompiles executable binary (an object file) to the C source. I'm not asking exactly 'how', but rather how this ...
3
votes
1answer
58 views

Is batch learning with gradient descent equivalent to “rehearsal” in incremental learning?

I am learning about incremental learning and read that rehearsal learning is retraining with old data. In essence, isn't this the exact same thing as batch learning (with stochastic gradient descent)? ...
3
votes
1answer
126 views

Is continuous learning possible with a deep convolutional neural network, without changing its topology?

In general, is continuous learning possible with a deep convolutional neural network, without changing its topology? In my case, I want to use a convolutional neural network as a classifier of ...
2
votes
1answer
149 views

What is the scope of real-world deep learning applications in 2020?

2015 was a milestone year for AI--"deep learning" was validated in a very public way with AlphaGo. However, at the time, the question was raised: "What else is deep learning good for?&...
1
vote
1answer
67 views

How to determine the number of hidden layers and units of a deep auto-encoder?

I am using a deep autoencoder for my problem. However, the way I choose the number of hidden layers and hidden units in a hidden layer is still based on my feeling. The size of the model that ...
1
vote
1answer
916 views

What is the computational complexity of the forward pass of a convolutional neural network?

How do I determine the computational complexity (big-O notation) of the forward pass of a convolutional neural network? Let's assume for simplicity that we use zero-padding such that the input size ...
19
votes
3answers
32k views

How do I handle large images when training a CNN?

Suppose that I have 10K images of sizes $2400 \times 2400$ to train a CNN. How do I handle such large image sizes without downsampling? Here are a few more specific questions. Are there any ...
17
votes
3answers
17k views

What is the concept of Tensorflow Bottlenecks?

What is the concept and how does one calculate Bottleneck values? How do these values help image classification? Please explain in simple words.
15
votes
2answers
1k views

When is deep learning overkill?

For example, for classifying emails as spam, is it worthwhile - from a time/accuracy perspective - to apply deep learning (if possible) instead of another machine learning algorithm? Will deep ...
9
votes
3answers
6k views

What is non-Euclidean data?

What is non-Euclidean data? Here are some sub-questions Where does this type of data arise? I have come across this term in the context of geometric deep learning and graph neural networks. ...
12
votes
1answer
968 views

What are all the different kinds of neural networks used for? [closed]

I found the following neural network cheat sheet (Cheat Sheets for AI, Neural Networks, Machine Learning, Deep Learning & Big Data). What are all these different kinds of neural networks used ...
11
votes
5answers
2k views

Why are deep neural networks and deep learning insufficient to achieve general intelligence?

Everything related to Deep Learning (DL) and deep(er) networks seems "successful", at least progressing very fast, and cultivating the belief that AGI is at reach. This is popular imagination. DL is a ...
19
votes
3answers
25k views

How can we process the data from both the true distribution and the generator?

I'm struggling to understand the GAN loss function as provided in Understanding Generative Adversarial Networks (a blog post written by Daniel Seita). In the standard cross-entropy loss, we have an ...
17
votes
6answers
2k views

What is the difference between machine learning and deep learning?

Can someone explain to me the difference between machine learning and deep learning? Is it possible to learn deep learning without knowing machine learning?
13
votes
2answers
2k views

How are generative adversarial networks trained?

I am reading about generative adversarial networks (GANs) and I have some doubts regarding it. So far, I understand that in a GAN there are two different types of neural networks: one is generative ($...
13
votes
3answers
16k views

Measuring Object size using Deep Neural Network

I have a large dataset of vehicles with the ground truth of their lengths (Over 100k samples). Is it possible to train a deep network to measure/estimate vehicle length ? I haven't seen any papers ...
4
votes
3answers
2k views

What are the differences between transfer learning and meta learning?

What are the differences between meta-learning and transfer learning? I have read 2 articles on Quora and TowardDataScience. Meta learning is a part of machine learning theory in which some ...
11
votes
2answers
5k views

Which layer in a CNN consumes more training time: convolution layers or fully connected layers?

In a convolutional neural network, which layer consumes more training time: convolution layers or fully connected layers? We can take AlexNet architecture to understand this. I want to see the time ...
4
votes
2answers
1k views

Each training run for DDQN agent takes 2 days, and still ends up with -13 avg score, but OpenAi baseline DQN needs only an hour to converge to +18?

Status: For a few weeks now, I have been working on a Double DQN agent for the PongDeterministic-v4 environment, which you can find here. A single training run ...
8
votes
2answers
2k views

Why should the number of neurons in a hidden layer be a power of 2?

I have read somewhere on the web (I lost the reference) that the number of units (or neurons) in a hidden layer should be a power of 2 because it helps the learning algorithm to converge faster. Is ...
6
votes
4answers
3k views

What is the purpose of the hidden layers?

Why would anybody want to use "hidden layers"? How do they enhance the learning ability of the network in comparison to the network which doesn't have them (linear models)?
6
votes
3answers
4k views

CPU preferences and specifications for a multi GPU deep-learning setup [closed]

For a multi (4xTitan Xp) GPU deep learning setup what kind of CPU is preferable? Specifically I am comparing: Intel Xeon E5-2620 with 8x2.1GHz 20MB L3 Cache Intel Xeon E5-1620K with 4x3.5Ghz 10MB L3 ...
5
votes
1answer
86 views

Is it possible to combine two neural networks trained on different tasks into one that knows both tasks?

I'm relatively new to artificial intelligence and neural networks. Let's say I have two different fully trained neural networks. The first one is trained for mathematical addition and the second one ...
5
votes
1answer
1k views

Why is the target $r + \gamma \max_{a'} Q(s', a'; \theta_i^-)$ in the loss function of the DQN architecture?

In the paper Human-level control through deep reinforcement learning, the DQN architecture is presented, where the loss function is as follows $$ L_i(\theta_i) = \mathbb{E}_{(s, a, r, s') \sim U(D)} \...
3
votes
1answer
147 views

In novelty search, are the novel structures or behaviour of the neural network rewarded?

I have been reading a lot lately about some very promising work coming out of Uber's AI Labs using mutation algorithms enhanced with novelty search to evolve deep neural nets. See the paper Safe ...