I trained Masked Autoencoder-based models in order to use the encoder as a backbone for another task. Pretraining has been done in a Self-Supervised manner on image data. Now that it comes to my downstream task, I was wondering what is the best way of creating feature maps on the whole image for the 2 models:
- ConvNextV2 (fully convolutional masked autoencoder) FCMAE with a mask ratio of 60% in pretraining: Since we deal with sparse convolutions, it might make sense to just set the mask ratio to 0, such that we do not mask out anything and create features for the whole image area, right? I tried to compare it with creating 2 deterministic masks (50% each) and do 2 forward passes such that I can simply add up the feature maps, but the results are not equal. Can someone explain the reason for that?
- ViT-MAE (MAE with vision transformer backbone): From my knowledge, Transformers are constrained to their pre-trained sequence length and thus do not perform well when changing the number of input patches (e.g. 100%) for inference to create feature maps for the whole image. Any remarks on that approach?
