I train a fully convoluted network for semantic segmentation. To each convolution blocks, I associate a module pruning feature maps to reduce the quantity of information generated by the network. From these modules, I extract a sparsity ratio, i.e. the number of feature maps containing information over the total number of feature maps in the given layer. Averaging this ratio at the network scale (given all modules), I obtain a total_sparsity_ratio which is added to the loss function.
total_sparsity_ratio is differentiable, but since I use a step function to obtain calculate it, the gradient is 0.
Is it acceptable to use a "non-trainable" regularizer in the loss function? Training results show slightly better results.