Concatenate convolution layers with different strides in tensorflow.

Question

I am trying to do an inception layer, but it only works if the convolution strides, pool strides and pool size are the same, otherwise I get an error in

tf.concat

that Dimesion 1 is not the same. So If I change something in the last three tuples, I get the error.

conv1 = conv2d_maxpool(x, 64, (5, 5), (1, 1), (2, 2), (2, 2)) 
conv2 = conv2d_maxpool(x, 64, (4, 4), (1, 1), (2, 2), (2, 2)) 
conv3 = conv2d_maxpool(x, 32, (2, 2), (1, 1), (2, 2), (2, 2)) 
conv4 = conv2d_maxpool(x, 32, (1, 1), (1, 1), (2, 2), (2, 2)) 
conv = tf.concat([conv1, conv2, conv3, conv4], 3)

For example, this is the error I get if I change the 5x5 filter to have strides 3:

conv1 = conv2d_maxpool(x, 64, (5, 5), (3, 3), (2, 2), (2, 2))

Dimension 1 in both shapes must be equal, but are 6 and 16 for 'concat' (op: 'ConcatV2') with input shapes: [?,6,6,64], [?,16,16,64], [?,16,16,32], [?,16,16,32], [].

This is the conv2d_maxpool function:

def conv2d_maxpool(x_tensor, conv_num_outputs, conv_ksize, conv_strides, pool_ksize, pool_strides):
    """
    Apply convolution then max pooling to x_tensor
    :param x_tensor: TensorFlow Tensor
    :param conv_num_outputs: Number of outputs for the convolutional layer
    :param conv_strides: Stride 2-D Tuple for convolution
    :param pool_ksize: kernal size 2-D Tuple for pool
    :param pool_strides: Stride 2-D Tuple for pool
    : return: A tensor that represents convolution and max pooling of x_tensor
    """
    # TODO: Implement Function
    weights = tf.Variable(tf.truncated_normal(
        shape = [*conv_ksize, int(x_tensor.get_shape().dims[3]), conv_num_outputs], 
        mean = 0.0, 
        stddev=0.1, 
        dtype=tf.float32))
    bias = tf.Variable(tf.zeros(conv_num_outputs)) 

    conv_layer = tf.nn.conv2d(x_tensor, weights, strides=[1, *conv_strides, 1], padding='SAME')
    conv_layer = tf.nn.bias_add(conv_layer, bias)
    conv_layer = tf.nn.relu(conv_layer)

    conv_layer_max_pool = tf.nn.max_pool(conv_layer, ksize=[1, *pool_ksize, 1], strides=[1, *pool_strides, 1], padding='SAME')

    return conv_layer_max_pool

How can I combine convolution filters with different strides and/or different pooling to create an inception layer?

Answer 1

Dimension 1 in both shapes must be equal, but are 6 and 16 for 'concat' (op: 'ConcatV2') with input shapes: [?,6,6,64], [?,16,16,64], [?,16,16,32], [?,16,16,32], [].

Pablo's answer is correct. Your problem is that the convolved images (output of conv-layers) must match in spatial dimensionality in order to concatenate them. This makes perfectly sense, because how would you combine images of shape 6x6 with images of shape 16x16? You can not.

Either you have to ensure that the convolutions produce output of equal spatial dimenions, i.e. using the same padding and strides strategy or you have to use tf.image.resize_images to down-/upscale the different output to the same spatial dimensionality (or some other down-/upscaling strategy).

Answer 2

Not 100% sure, but the problem is that when you work with different strides, the size of the convolved image change, so you should ensure, that all the convolved images have the same shape before concatenating the output. You can fill with 0s, or considering that the image is periodical in time, so filling with reflections of the image