I'm building a CNN/3DCNN model that classifies hand gestures. The problem is that the actual gesture occupies only like 1% of the whole image. That means that an enormous amount of convolutional operations is done on the "empty" parts of the image, which is useless.
Is there a way to solve this problem? I was thinking about a MaxPooling layer with a giant pool size, but near features that are extracted from the gesture will be probably "compressed" in only 1 feature.