0
$\begingroup$

For highway network, it looks like this:

enter image description here

For residual network, it looks like this:

enter image description here

Pictures are from What is the name of this neural network architecture with layers that are also connected to non-neighbouring layers?

My question is, how to handle the size difference between different layers in CNN to make highway network or residual network? For example, I am working on a text classification problem. By using the embedding, I have the input size as follows:

input.shape =[batch_size, embedding_dim, max_length]

I also has a CNN layer as follows:

Conv1d(in_channels= embedding_dim, out_channels=hidden_dim, kernel_size=n)

So that the size of the output of Conv1d is [batch_size, hidden_dim, max_length-n+1].

Here is the question, the input size of the CNN layer is different from the output size. How do handle the size difference so that highway network or residual network can be built?

Thank you.

$\endgroup$

2 Answers 2

2
$\begingroup$

You can just use padding='same'. As noted from the documentation:

When padding="same" and strides=1, the output has the same size as the input.

Note that strides is default to 1, and if kernel_size=1, the output also has the same shape as the input.

I look at two different implementations and can confirm this:

class Residual(tf.keras.Model):  #@save
    """The Residual block of ResNet."""
    def __init__(self, num_channels, use_1x1conv=False, strides=1):
        super().__init__()
        self.conv1 = tf.keras.layers.Conv2D(num_channels, padding='same',
                                            kernel_size=3, strides=strides)
        self.conv2 = tf.keras.layers.Conv2D(num_channels, kernel_size=3,
                                            padding='same')
        self.conv3 = None
        if use_1x1conv:
            self.conv3 = tf.keras.layers.Conv2D(num_channels, kernel_size=1,
                                                strides=strides)
        self.bn1 = tf.keras.layers.BatchNormalization()
        self.bn2 = tf.keras.layers.BatchNormalization()

    def call(self, X):
        Y = tf.keras.activations.relu(self.bn1(self.conv1(X)))
        Y = self.bn2(self.conv2(Y))
        if self.conv3 is not None:
            X = self.conv3(X)
        Y += X
        return tf.keras.activations.relu(Y)

which we see conv1 and conv2 has padding='same', strides=1 everywhere.

Here's the visualization of how different padding works. In short, 'same' automatically calculates the padding dimension based on the kernel size so that the output has the same shape as the input for you.

$\endgroup$
0
$\begingroup$

When using residual connections you want to have your input dimensions match the output dimension so that you can perform the addition operation. In a standard ResNet-style architecture you mostly have layers that keep the dimensions (i.e. they use padding="same").

However, there are a few places where you change the dimensions. Usually, your convolutional layer will reduce the spatial dimensions in half and will double the number of channels, i.e. $C \times H \times W \rightarrow 2C \times H//2 \times W//2$. Such a layer can be initialized like this for example:

nn.Conv2d(in_chan, 2*in_chan, kernel_size=3, stride=2, padding=1)

In this case your skip connection will not be a simple identity function, but you will actually apply a 1x1 convolution instead. Thus, instead of having $F(x) = f(x) + x$, you will have $F(x) = f(x) + g(x)$, where g(x) is the 1x1 convolution. This additional convolutional layer will apply the needed modification of the dimensions of $x$ so that you can apply the addition operation. For images and more explanations please see: https://pi-tau.github.io/posts/res-nets/#the-architecture-of-the-resnet

Now, before using this architecture for nlp tasks, I think it is a good idea to know WHY we do this dimension reduction. You usually start with a tensor of some size $C \times H \times W$. Then you apply some ResBlocks that do not modify the dimensions and extract features from this fixed dimensional space. Then you apply one special block that modifies the dimensions and again you stack ResBlocks that extract features from this reduced dimensional space. The idea is to squash the spatial dimensions and to expand the channel dimensions so that information that is contained across multiple nearby pixels in the original image is, at the end, concentrated in a single "pixel" which is now embedded in a much larger space. (see the link above for even more info)

Now with NLP sequences we are talking about 1D convolutions, so replicating this behavior would mean gradually reducing your sequence length in order to increase the embedding size. But I think that this design might fail to capture long term relationships which are quite common in text-related tasks. Also if you don't reduce the sequence length and only increase the embedding size, then your computational cost is quite higher in the deeper layers, and you really want near-constant cost across the layers. If you take a look at the transformer, for example, there they use self-attention to capture long-distance relations, and they also keep the embedding size constant throughout making every encoder block equally expensive.

Best of luck with your problem :)

$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .