Skip to main content

Questions tagged [residual-networks]

For questions related to residual networks (ResNets), introduced in "Deep Residual Learning for Image Recognition" (2015) by Kaiming He et al. and that won the first place at "Large Scale Visual Recognition Challenge 2015" (ILSVRC2015).

Filter by
Sorted by
Tagged with
0 votes
0 answers
24 views

Why the architeture of Resnet18 is suitable to images classification?

I am studying convolutional networks and in particular I have focused on the ResNet18 network. I've been studying ResNet18 and understand the purpose of skip connections and residual network. However,...
Domme's user avatar
  • 1
1 vote
2 answers
72 views

Wouldn't residual connections in RNNs solve the vanishing/exploding gradient problem?

I was recently brushing up on my deep-learning basics and came back to RNNs. LSTMs/GRUs and the Transformer architecture were invented to solve RNN's vanishing/exploding gradient problem. I was at ...
Vladislav Korecký's user avatar
0 votes
1 answer
14 views

Information cannot directly flow from past layer to future layer in resnet?

Resnet block: ...
JobHunter69's user avatar
0 votes
2 answers
289 views

What is the difference between densenet and resnet?

Is the only difference between the two how the skip connection is combined? Resnet combines skip connections through addition and Densenet through concatenating. The Densenet paper appears to be ...
JobHunter69's user avatar
0 votes
1 answer
56 views

Does ResNext split data or copy it?

I have been learning how to create ResNext neural networks, and am confused how input works with cardinality. In this answer, it seems that it's saying that the data is added together, which I assumed ...
eop3's user avatar
  • 11
0 votes
0 answers
31 views

Attention module (CBAM) in CNN tend to saturate values to 1

In the context of image classification, I am using a feature extractor based on a resnet-like architecture (ResNet12): four residual blocks, each of which is made of two consecutive conv3x3, batch ...
Lorenzo's user avatar
0 votes
0 answers
12 views

Training ImageNet on Resnet - Dropping LR has little improvement on accuracy

I'm trying to train Resnet50 on Imagenet following this paper [1] as well as this one[2]. They say that at approximately every 30 epochs, I should drop the learning rate by 10. Since I'm training on 8 ...
Liam F-A's user avatar
0 votes
0 answers
23 views

Discussion about Improving Visual Search Model Accuracy

My Visual Search Model is only achieving an accuracy of about 42% If anyone can give me advice to drastically improve this number I would greatly appreciate it. Below is my current flow of image ...
rileylivingston's user avatar
0 votes
1 answer
52 views

If a residual block learns to approximate the identity function, should we reinitialize the weights?

Now, the main idea behind ResNet is to enable the network to learn the residual value needed to modify the input for achieving the best result. In other words, given an input x, the output y is ...
Legendary's user avatar
0 votes
2 answers
1k views

Layer Norm in a ResNet MLP

Where do you insert layer norm in a residual block? After the addition or before the activation function (RELU in this case)?
postnubilaphoebus's user avatar
0 votes
2 answers
466 views

How to handle the size difference of highway network or residual network in cnn?

For highway network, it looks like this: For residual network, it looks like this: Pictures are from What is the name of this neural network architecture with layers that are also connected to non-...
liaoming999's user avatar
0 votes
1 answer
176 views

How do I use ResNet for text processing?

I need to implement a deep neural network [residual neural network (ResNet)] that takes some text as an input [length M x N] and then processes it. Now as far as my understanding goes, ResNet is used ...
Python's user avatar
  • 1
2 votes
0 answers
46 views

Can Inception-ResNet be inverted layer-by-layer?

It has already been shown that by using a normalization layer during training, it is possible to invert a residual network layer-by-layer. I wonder how similar Inception-ResNet is and whether a ...
Richie Bendall's user avatar
0 votes
1 answer
83 views

What underlying network is typically meant with ResNET?

When people talk about a ResNet architecture, they are talking about a neural network architecture with skip connections. But what basis network are they typically referring to? Feedforward-networks ...
postnubilaphoebus's user avatar
0 votes
1 answer
45 views

Keep weights of output layer in transfer learning?

I'm seeing conflicting info on what to do with the fully-connected output layer of a pre-trained network when it's used in transfer learning. A previous answer seems to imply that the network is kept ...
Fijoy Vadakkumpadan's user avatar
0 votes
1 answer
1k views

How can I use larger input images when using a pre-trained CNN without resizing?

I have a ResNet18 model trained on the Places365 image dataset, and I'd like to use this pre-trained model to expedite the training needed to identify distressed houses. My dataset is images of size ...
Cole Tritch's user avatar
0 votes
1 answer
475 views

Whats wrong with my resnet50 training on CIFAR10 pytorch?

I've been trying to construct resnet50 architecture from scratch using pytorch for classification. After construction I've run training job on CIFAR10 torchvision dataset, in 20 epochs with lr of 0.01 ...
qvuer7's user avatar
  • 3
2 votes
1 answer
235 views

Exact definition of WRN-d-k (Wide ResNet)

I am a little confused about the WRN-d-k notation from Wide Residual Networks. To quote the paper, In the rest of the paper we use the following notation: WRN-n-k denotes a residual network that has ...
nalzok's user avatar
  • 321
0 votes
1 answer
316 views

Retraining ResNet-50 for iris flower classification

I am trying to retrain ResNet-50 for iris flower classification in tensorflow (TensorFlow version: 2.3.0) using the following code ...
root's user avatar
  • 11
0 votes
1 answer
115 views

Darknet as a part of Yolo v3

I am pretty new to ML and my question may look strange. Especially the last part of it. 1)As far as I understand Darknet53 is an integral part of Yolo just as Resnet50 is a part of R-CNN Am I right? 2)...
Igor's user avatar
  • 183
2 votes
2 answers
540 views

How many unique angles of an object do you need in your image training set in order to correctly classify it?

I'm interested in using ResNet-50 to classify images of objects for around 1000 unique classes. I'm wondering if there is any way to estimate how many unique angles I need in my training set to ...
Tyler Hilbert's user avatar
1 vote
1 answer
494 views

How can I use a ResNet as a function approximator for pixel based reinforcement learning?

I'd like to use a residual network to improve learning in image-based reinforcement learning, specifically on Atari Games. My main question is divided into 3 parts. Would it be wise to integrate a ...
desert_ranger's user avatar
3 votes
0 answers
36 views

Why actual mapping is called as unreferenced mapping in this context of residual framework?

Consider the following statements from the research paper titled Deep Residual Learning for Image Recognition by Kaiming He et al. #1: We explicitly reformulate the layers as learning residual ...
hanugm's user avatar
  • 3,890
1 vote
3 answers
2k views

Residual Blocks - why do they work?

I've learnt that idea that the residual block was invented to solve the vanishing gradient problem due to the deep layer to layer multiplication. I understand that for example if I have 10 layers, and ...
user49443's user avatar
1 vote
3 answers
655 views

Why do skip layer connections require the same layer sizes?

I know how skip connections work: you add the activations of the previous layer to the activations of a successive layer to stabilize information/gradient flow. My question is, why doesn't it just get ...
profPlum's user avatar
  • 434
2 votes
0 answers
43 views

How to implement a (3 + 2)-dimensional convolutional layer where the 2d space is "internal"?

I am trying to train a CNN to learn 5D (kind of) data. The data is structured as follows. It has three spatial dimensions [x, y, z], but it also has two "...
play's user avatar
  • 133
1 vote
1 answer
89 views

Does it make sense to apply batch normalization to a batch size of 1?

I am interested in your opinion on the topic if you think that it makes sense to use batch normalization layer in a network that is trained with a batch size of 1. This is a special case as part of an ...
user3352632's user avatar
1 vote
1 answer
65 views

Why are weights not initialized with mean=1?

I wonder why weights are initialized with zero-mean. It is one of the reasons, why deep architectures cannot be trained without skip connections. Without the skip connections, the zero initialization ...
spadel's user avatar
  • 31
1 vote
1 answer
602 views

Can residual connections be beneficial when we have a small training dataset?

I have a classification problem, for which an inadequate amount of training data is available. Also, there is no known practical data augmentation approach for this problem (as no unlabelled data is ...
Reza_va's user avatar
  • 89
1 vote
0 answers
433 views

What are the benefits of Cross Stage Partial Connections over Residual Connections?

Cross Stage Partial Connections (CSPC) try to solve the next problems: Reduce the computations of the model in order to make it more suitable for edge devices. Reduce memory usage. Better ...
IgnacioGaBo's user avatar
1 vote
1 answer
85 views

Regression For Elliptical Curve Public Key Generation Possible?

As part of a learning more about deep learning, I have been experimenting with writing ResNets with Dense layers to do different types of regression. I was interested in trying a harder problem and ...
superuser's user avatar
3 votes
0 answers
635 views

How to use residual learning applied to fully connected networks?

Is there any reason why skip connections would not provide the same benefits to fully connected layers as it does for convolutional? I've read the ResNet paper and it says that the applications should ...
rocksNwaves's user avatar
3 votes
1 answer
258 views

Can residual neural networks use other activation functions different from ReLU?

In many diagrams, as seen below, residual neural networks are only depicted with ReLU activation functions, but can residual NNs also use other activation functions, such as the sigmoid, hyperbolic ...
jr123456jr987654321's user avatar
4 votes
1 answer
332 views

If the point of the ResNet skip connection is to let the main path learn the residual relative to identity, why are there convolutional skips?

In the original ResNet paper they talk about using plain identity skip connections when the input and output of a block have the same dimensions. When the input and output have different dimensions ...
Alexander Soare's user avatar
6 votes
1 answer
1k views

If vanishing gradients are NOT the problem that ResNets solve, then what is the explanation behind ResNet success?

I often see blog posts or questions on here starting with the premise that ResNets solve the vanishing gradient problem. The original 2015 paper contains the following passage in section 4.1: We ...
Alexander Soare's user avatar
3 votes
1 answer
116 views

Do deeper residual networks perform better or worse?

If you have an $18$ layer residual network versus and a $32$ layer residual network, why would the former do better at object detection than the latter, if you have both models are training using the ...
Stockguu's user avatar
2 votes
2 answers
164 views

What is the name of this neural network architecture with layers that are also connected to non-neighbouring layers?

Consider a feedforward neural network. Suppose you have a layer of inputs, which is feedforward to a hidden layer, and feedforward both the input and hidden layers to an output layer. Is there a name ...
scorpdaddy's user avatar
7 votes
1 answer
12k views

Why do ResNets avoid the vanishing gradient problem?

I read that, if we use the sigmoid or hyperbolic tangent activation functions in deep neural networks, we can have some problems with the vanishing of the gradient, and this is visible by the shapes ...
FraMan's user avatar
  • 199
2 votes
3 answers
2k views

Is a VGG-based CNN model sometimes better for image classfication than a modern architecture?

I have an image classification task to solve, but based on quite simple/good terms: There are only two classes (either good or not good) The images always show the same kind of piece (either with or ...
Matthias's user avatar
  • 165
10 votes
4 answers
8k views

What is the benefit of using identity mapping layers in deep neural networks like ResNet?

As I understand, ResNet has some identity mapping layers, whose task is to create the output as the same as the input of the layer. The ResNet solved the problem of accuracy degrading. But what is the ...
Ali Abdari's user avatar
1 vote
1 answer
3k views

Why are there transition layers in DenseNet?

The DenseNet architecture can be summarizde with this figure: Why there are transition layers between each block? In the papers, they justify the use of transition layers as follow : The ...
Astariul's user avatar
  • 371
9 votes
1 answer
944 views

Why aren't there neural networks that connect the output of each layer to all next layers?

Why aren't there neural networks that connect the output of each layer to all next layers? For example, the output of layer 1 would be fed to the input of layers 2, 3, 4, etc. Beyond computational ...
Christopher Jernigan's user avatar
5 votes
1 answer
127 views

Are neurons instantly feed forward when input arrives?

Let's say I have a neural network with 5 layers, including the input and output layer. Each layer has 5 nodes. Assume the layers are fully connected, but the 3rd node in the 2nd layer is connected to ...
Miemels's user avatar
  • 389
14 votes
2 answers
828 views

Should deep residual networks be viewed as an ensemble of networks?

The question is about the architecture of Deep Residual Networks (ResNets). The model that won the 1-st places at "Large Scale Visual Recognition Challenge 2015" (ILSVRC2015) in all five main tracks: ...
Erba Aitbayev's user avatar