Questions tagged [residual-networks]

For questions related to residual networks (ResNets), introduced in "Deep Residual Learning for Image Recognition" (2015) by Kaiming He et al. and that won the first place at "Large Scale Visual Recognition Challenge 2015" (ILSVRC2015).

Filter by
Sorted by
Tagged with
2
votes
0answers
37 views

How to implement a (3 + 2)-dimensional convolutional layer where the 2d space is “internal”?

I am trying to train a CNN to learn 5D (kind of) data. The data is structured as follows. It has three spatial dimensions [x, y, z], but it also has two "...
1
vote
0answers
18 views

Does it make sense to apply batch normalization to a batch size of 1?

I am interested in your opinion on the topic if you think that it makes sense to use batch normalization layer in a network that is trained with a batch size of 1. This is a special case as part of an ...
1
vote
1answer
27 views

Why are weights not initialized with mean=1?

I wonder why weights are initialized with zero-mean. It is one of the reasons, why deep architectures cannot be trained without skip connections. Without the skip connections, the zero initialization ...
1
vote
1answer
30 views

Can residual connections be beneficial when we have a small training dataset?

I have a classification problem, for which an inadequate amount of training data is available. Also, there is no known practical data augmentation approach for this problem (as no unlabelled data is ...
1
vote
0answers
79 views

What are the benefits of Cross Stage Partial Connections over Residual Connections?

Cross Stage Partial Connections (CSPC) try to solve the next problems: Reduce the computations of the model in order to make it more suitable for edge devices. Reduce memory usage. Better ...
1
vote
1answer
70 views

Regression For Elliptical Curve Public Key Generation Possible?

As part of a learning more about deep learning, I have been experimenting with writing ResNets with Dense layers to do different types of regression. I was interested in trying a harder problem and ...
1
vote
0answers
129 views

How to use residual learning applied to fully connected networks?

Is there any reason why skip connections would not provide the same benefits to fully connected layers as it does for convolutional? I've read the ResNet paper and it says that the applications should ...
3
votes
1answer
95 views

Can residual neural networks use other activation functions different from ReLU?

In many diagrams, as seen below, residual neural networks are only depicted with ReLU activation functions, but can residual NNs also use other activation functions, such as the sigmoid, hyperbolic ...
2
votes
1answer
89 views

If the point of the ResNet skip connection is to let the main path learn the residual relative to identity, why are there convolutional skips?

In the original ResNet paper they talk about using plain identity skip connections when the input and output of a block have the same dimensions. When the input and output have different dimensions ...
2
votes
1answer
202 views

If vanishing gradients are NOT the problem that ResNets solve, then what is the explanation behind ResNet success?

I often see blog posts or questions on here starting with the premise that ResNets solve the vanishing gradient problem. The original 2015 paper contains the following passage in section 4.1: We ...
3
votes
1answer
55 views

Do deeper residual networks perform better or worse?

If you have an $18$ layer residual network versus and a $32$ layer residual network, why would the former do better at object detection than the latter, if you have both models are training using the ...
2
votes
2answers
82 views

What is the name of this neural network architecture with layers that are also connected to non-neighbouring layers?

Consider a feedforward neural network. Suppose you have a layer of inputs, which is feedforward to a hidden layer, and feedforward both the input and hidden layers to an output layer. Is there a name ...
6
votes
1answer
3k views

Why do ResNets avoid the vanishing gradient problem?

I read that, if we use the sigmoid or hyperbolic tangent activation functions in deep neural networks, we can have some problems with the vanishing of the gradient, and this is visible by the shapes ...
1
vote
3answers
487 views

Is a VGG-based CNN model sometimes better for image classfication than a modern architecture?

I have an image classification task to solve, but based on quite simple/good terms: There are only two classes (either good or not good) The images always show the same kind of piece (either with or ...
7
votes
4answers
3k views

What is the benefit of using identity mapping layers in deep neural networks like ResNet?

As I understand, ResNet has some identity mapping layers, whose task is to create the output as the same as the input of the layer. The ResNet solved the problem of accuracy degrading. But what is the ...
1
vote
1answer
1k views

Why are there transition layers in DenseNet?

The DenseNet architecture can be summarizde with this figure: Why there are transition layers between each block? In the papers, they justify the use of transition layers as follow : The ...
7
votes
1answer
628 views

Why aren't there neural networks that connect the output of each layer to all next layers?

Why aren't there neural networks that connect the output of each layer to all next layers? For example, the output of layer 1 would be fed to the input of layers 2, 3, 4, etc. Beyond computational ...
5
votes
1answer
107 views

Are neurons instantly feed forward when input arrives?

Let's say I have a neural network with 5 layers, including the input and output layer. Each layer has 5 nodes. Assume the layers are fully connected, but the 3rd node in the 2nd layer is connected to ...
14
votes
2answers
710 views

Should deep residual networks be viewed as an ensemble of networks?

The question is about the architecture of Deep Residual Networks (ResNets). The model that won the 1-st places at "Large Scale Visual Recognition Challenge 2015" (ILSVRC2015) in all five main tracks: ...