10
votes
Accepted
What is the benefit of using identity mapping layers in deep neural networks like ResNet?
TL;DR: Deep networks have some issues that skip connections fix.
To address this statement:
As I understand Resnet has some identity mapping layers that their task is to create the output as the ...
- 1,106
7
votes
Accepted
Why do ResNets avoid the vanishing gradient problem?
Before proceeding, it's important to note that ResNets, as pointed out here, were not introduced to specifically solve the VGP, but to improve learning in general. In fact, the authors of ResNet, in ...
- 41.4k
6
votes
Accepted
If vanishing gradients are NOT the problem that ResNets solve, then what is the explanation behind ResNet success?
They explained in the paper why they introduce residual blocks. They argue that it's easier to learn residual functions $F(x) = H(x) - x$ and then add them to the original representation $x$ to get ...
- 2,406
5
votes
Why aren't there neural networks that connect the output of each layer to all next layers?
Actually, this already exists!
I happened to make a presentation of a paper that talks about this topic. These networks are called DenseNets, which stands for densely connected convolutional networks....
- 51
3
votes
Accepted
How can I use larger input images when using a pre-trained CNN without resizing?
TL;DR:
It's definitely worth trying to benefit from the learned features from the ResNet. As it's made of mainly pretrained convolutional layers with pooling, adding new resizing layers upfront is ...
- 321
3
votes
Does it make sense to apply batch normalization to a batch size of 1?
It's not possible to use batch normalization with a batch size of 1. Batch normalization requires you to calculate the variance of activation values in the current batch, and variance is undefined for ...
- 31
3
votes
Residual Blocks - why do they work?
First a little intro, skip to the end for the straight answers: residual networks were proposed after observing that deeper models tend to perform worse than their shallow counterpart if we just keep ...
- 5,428
3
votes
If the point of the ResNet skip connection is to let the main path learn the residual relative to identity, why are there convolutional skips?
Well, I found an answer that satisfies me.
The zero-padded identity is not ideal. Suppose we're mapping from 64 channels to 128 channels. Then the zero-padded identity will map to an output where ...
- 1,369
3
votes
What is the benefit of using identity mapping layers in deep neural networks like ResNet?
As explained in this paper , the major benefit of identity mapping is that it enables backpropagation signal to reach from output (last) layers to input (first) layers.
You can see on the paper at ...
- 688
3
votes
Accepted
Can residual connections be beneficial when we have a small training dataset?
Can residual connections be beneficial when we have a small training dataset?
The usual rule of data science investigations applies here: Try it, measure the results, then you will know.
It is very ...
- 33.3k
3
votes
Are neurons instantly feed forward when input arrives?
It is unclear what kind of network your are referring to, there is not a single neural-network model so conceivable both cases could exist and serve some purpose, yet if you are looking for one that ...
- 575
3
votes
Accepted
Wouldn't residual connections in RNNs solve the vanishing/exploding gradient problem?
In my opinion your idea indeed holds merit. Something worth noting though is that it is cruder than the LSTM/GRU that have trainable weights that guide what features are remembered and forgotten. ...
2
votes
Is a VGG-based CNN model sometimes better for image classfication than a modern architecture?
Below is a listing of Keras application models that can be used easily in transfer learning. Note VGG has on the order of 140 million parameters which is why it is slow.
...
- 724
2
votes
Accepted
Can residual neural networks use other activation functions different from ReLU?
The problem with certain activation functions, such as the sigmoid, is that they squash the input to a finite interval (i.e. they are sometimes classified as saturating activation functions). For ...
- 41.4k
2
votes
What is the name of this neural network architecture with layers that are also connected to non-neighbouring layers?
Such a network could be either a Residual Network or a Highway Network depending upon the underlying architecture of the skip layers. They are primarily used to to tackle the problem of vanishing ...
- 370
2
votes
Accepted
What is the name of this neural network architecture with layers that are also connected to non-neighbouring layers?
This could be called a residual neural network (ResNet), which is a neural network with skip connections, that is, connections that skip layers.
Here's a screenshot of a figure from the paper Deep ...
- 41.4k
2
votes
Accepted
Why do skip layer connections require the same layer sizes?
One can concatenate with the previous layer outputs as well, and this approach in pursued in DenseNets. A nice illustration, that compares difference between ResNets and DenseNets is presented below:
...
- 2,754
2
votes
Why are weights not initialized with mean=1?
Interesting question,
I can come with 2 explanations why we don't initialize weights with 1 mean value :
It may be easier for the network to learn identity function, but we may have a similar issue ...
- 211
2
votes
Accepted
How can I use a ResNet as a function approximator for pixel based reinforcement learning?
Residual Network are usually deeper and hence take more time to train. EfficientNet are trying to tackle this. However, the latest advice show that the architecture tend to play a crucial role in the ...
2
votes
Accepted
Darknet as a part of Yolo v3
Lets start by listing what is what.
RCNN : Is a type of CNN Model
Resnet50, DarkNet53 and VGG : Are implementations of a CNN Model
Now moving to your questions.
Yes Darknet53 is the backbone of ...
- 76
2
votes
Accepted
Whats wrong with my resnet50 training on CIFAR10 pytorch?
The articles you mention likely meant pretrained resnet50. It can get to 85%+ accuracy in 5 epochs with Adam and 1e-3 learning rate indeed. You'd need to replace the last layer or use the timm wrapper ...
- 136
2
votes
Exact definition of WRN-d-k (Wide ResNet)
I also found the WRN-$n$-$k$ notation confusing, but I think I can explain it:
$n$ is the total number of convolutions in the model. So to understand the architecture associated with each $n$, we ...
- 121
2
votes
How do I use ResNet for text processing?
ResNet as a name is defined as a CNN with a specific architecture, but the more general concept of Residual Networks are not necessarily CNNs, but networks that use skip connections.
You could make a ...
- 1,355
2
votes
Accepted
How to handle the size difference of highway network or residual network in cnn?
You can just use padding='same'. As noted from the documentation:
When padding="same" and ...
- 1,160
2
votes
If a residual block learns to approximate the identity function, should we reinitialize the weights?
I think what is done in practice is that you just train multiple networks with different random seeds. This greatly reduces the possibility for getting bad models due to bad initialization.
At the end ...
- 915
2
votes
Wouldn't residual connections in RNNs solve the vanishing/exploding gradient problem?
Your idea is exactly the idea behind state-space models. They have a linear "residual" connection from previous hidden states, skipping activations. In fact, it works very well! I'd ...
- 166
1
vote
Keep weights of output layer in transfer learning?
Transfer learning is a complete field of research, and there are multiple possibilities for what might work best in each situation.
There are various ways in which you can employ a pretrained model ...
- 2,660
1
vote
Retraining ResNet-50 for iris flower classification
It looks like you're defining an optimizer in the line sgd = optimizers.SGD(lr = 0.01, decay = 1e-6, momentum = 0.9, nesterov = True) and then not using it, since ...
- 259
1
vote
How many unique angles of an object do you need in your image training set in order to correctly classify it?
[I wanted it to be a comment but it's too long :)]
I don't think it's a good approach to split point of views into a group of 12 angles. The main purpose of using neural net is to have model that is ...
- 242
1
vote
Residual Blocks - why do they work?
The residual layer was not invented to solve the vanishing gradient problem. Citing from the official ResNet paper:
An obstacle to answering this question was the notorious problem of vanishing/...
- 915
Only top scored, non community-wiki answers of a minimum length are eligible