0
$\begingroup$

Resnet block:

    identity = x

    out = self.conv1(x)
    out = self.bn1(out)
    out = self.relu(out)

    out = self.conv2(out)
    out = self.bn2(out)

    if self.downsample is not None:
        identity = self.downsample(x)

    out += identity
    out = self.relu(out)

we can see that the skip connection comes out of the activation. In order for the output of layer n to be an input of an arbitrary layer m where m > n, either the identity addition should come after self.relu, or the incoming input should come before the relu in the previous block. Why do they do it like this? The information cannot flow now

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

I found that this was the old Resnet design and they shortly came up with a newer design that does what I'm talking about: Identity Mappings in Deep Residual Networks

Improve this answer
$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .