I've been using matterport's Mask R-CNN to train on a custom dataset. However, there seem to be some parameters that i failed to correctly define because on practically all of the images, the bottom or top of the object's mask is cut off:

As you can see, the bounding box is fine since it covers the whole blade, but the mask seems to suddenly stop in a horizontal line on the bottom.

On another hand, there is a stair-like effect on masks of larger and curvier objects such as this one (in addition to the bottom and top cut-offs):

• The original images are downscaled to IMAGE_MIN_DIM = IMAGE_MAX_DIM = 1024 using the "square" mode.
• USE_MINI_MASK is set to true with MINI_MASK_SHAPE = (512, 512) (somehow if i set it off, RAM gets filled and training chrashes).
• RPN_ANCHOR_SCALES = (64, 128, 256, 512, 1024) since the objects occupy a large space of the image.

It doesn't feel like the problem comes from the amount of training. These two predictions come from 6 epochs of 7000 steps per epoch (took around 17 hours). And the problem appears from early stage and persists along all the epochs.

I posted the same question on stack overflow, and an answer pointed out that this issue is common when using mask r-cnn. It also suggested to look at PointRend, an implementation to mask r-cnn that addresses this issue.

Nevertheless I feel like I could still optimize my model and use the full potential of mask r-cnn before looking for an alternative.

Any idea on what changes to make ?