From what I understand, (Faster/Mask-)RCNN is fully convolutional. The backbone is fully convolutional, and the region proposal network (RPN) creates anchors on the feature map with a fixed stride.
This means that a forward pass on a crop of a larger image should give the same output (barring any edge issues) as a forward pass on the full image for that cropped area, since feature sizes are the same. Alternatively, padding an image with zeros shouldn't affect the output (again, barring edge issues).
However, that doesn't seem to be the case when I try it with PyTorch's Mask-RCNN implementation. What am I missing here?
I've already turned off the resizing that the PyTorch Mask-RCNN implementation does internally. Is there another resolution-dependent process at play here? Is this specific to this implementation, or am I misunderstanding something more general about how the RCNN architecture works?