Is there a best practice for creating multiple convolutional layers from small image inputs?

Question

With all the work being done on larger and larger images, I'd like to ask if a best practice(s) has arisen for allowing multiple convolutional layers on small image inputs?

For instance, in my case I have a 7x7x4 input that I convolve (coming from a small visual piece of a RL environment). Obviously if I want to allow a few layers with increasing depth/channels to allow the network to learn abstractions, many kernel/padding combinations will swiftly reduce the first two dimensions to a point where arguably convolutional layers don't make sense.

For my own case, I've managed to get 3 seemingly reasonable layers by playing with kernel size, stride, and padding, and while it "works" for my use-case, it feels outright arbitrary:

# Values for the first conv layer
self.ksize_1 = 3
self.stride_1 = 1
self.padding_1 = 1
# retains the input_dimz x input_dimz shape

# Values for the second conv layer
self.ksize_2 = 4
self.stride_2 = 1
self.padding_2 = 1
# roughly makes dimension even (looks like input_dimz - 1)

# Values for the third conv layer
self.ksize_3 = 3
self.stride_3 = 1
self.padding_3 = 0
# outputs dimensions roughly 2 less in width by height from above
# should be (input_dimz - 3) x (input_dimz - 3)

# Now our actual conv layers
self.conv1 = nn.Conv2d(self.input_chan, self.conv_dim, self.ksize_1,
                       self.stride_1, self.padding_1, padding_mode=self.padding_mode)
self.conv2 = nn.Conv2d(self.conv_dim, self.conv_dim * 2, self.ksize_2,
                       self.stride_2, self.padding_2, padding_mode=self.padding_mode)
self.conv3 = nn.Conv2d(self.conv_dim * 2, self.conv_dim * 3, self.ksize_3,
                       self.stride_3, self.padding_3, padding_mode=self.padding_mode)

(where conv_dim is essentially a hyperparam to tweak the channel depth)

Are small dimensional cases like this still pretty arbitrary and the above reasonable, or has a methodological approach arisen that can be followed?

Answer 1

A bit late to the party but the only thing I could come up with is VGG-oriented architectures. What I mean by that is use blocks of convolutional layers that leave the input the same by for instance using stride one and same padding. Finish the previous block with a convolutional layer that has stride 2 or valid padding and then create a new block that utilizes a zero padding layer.

For tensorflow you would do something like this:

   inputs = keras.Input(shape=input_shape)
   x = layers.Conv2D(filters=num_filters, kernel_size=kern_size, padding="same", strides=1,activation='relu')(inputs)
   x = layers.Conv2D(filters=num_filters, kernel_size=kern_size, padding="same", strides=1,activation='relu')(x)
   ...
   x = layers.Conv2D(filters=num_filters, kernel_size=kern_size, padding="same", strides=2,activation='relu')(x)
   # new block
   x = layers.ZeroPadding2D(padding=((padding_num)(x)
   ...

This way you can have several layers that leave the input size untouched until you want to go further. Ofcourse instead of changing the stride, you could add a MaxPooling layer etc. My answer is based on subjective empirical evidence with small images and with trying to expand networks without reducing the size of the initial input to too small dimensions.