I have a convolutional encoder (a CNN) consisting of DenseBlocks and a total of 50 layers (cf. FC-DenseNet103). The receptive field of the encoder (after last layer) is 660 according to Tensorflow function compute_receptive_field_from_graph_def(..)) whereas the input image is 64x64 pixels. Obviously the receptive field is way too big.
How can the receptive field be reduced to say 46 but the capacity of the encoder be more or less kept at the same level? By capacity I simply mean the number of parameters of the model. The capacity requirement is justified due to the complex dataset to be processed.
Using less layers or smaller kernels reduces the receptive field size but also the capacity. Should I then just increase the number of filters in the remaining layers in order to keep the capacity?