How can I use larger input images when using a pre-trained CNN without resizing?

Question

I have a ResNet18 model trained on the Places365 image dataset, and I'd like to use this pre-trained model to expedite the training needed to identify distressed houses. My dataset is images of size 800x800x3, but the inputs are of size 224x224x3. I do not want to resize the image to 224x224 since I'm worried it will lose house distress indicators (chipped paint and loose shingles) during conversion.

My idea was to add extra layers that can handle the larger images before feeding them into ResNet. I have the following pytorch model:

import torch
from torch import nn
from torchvision import models

class NewModel(nn.Model):
    def __init__(self, pretrain_model_path) -> None:
        # Not sure here
        self.pre_layers = None
        # import the trained model
        model = models.resnet18(num_classes=365)
        checkpoint = torch.load(pretrain_model_path, map_location=lambda storage, loc: storage)
        state_dict = {str.replace(k,'module.',''): v for k,v in checkpoint['state_dict'].items()}
        model.load_state_dict(state_dict)
        # change prediction class count
        model.fc = nn.Linear(model.fc.in_features, 4)
        self.backbone = model

    def forward(self, x):
        x = self.pre_layers(x)
        x = self.backbone(x)
        return x

Is this a common practice or is it better to make one from scratch that is built for this size image and problem specifically? How would I go about implementing it if so?

Answer 1

TL;DR:

It's definitely worth trying to benefit from the learned features from the ResNet. As it's made of mainly pretrained convolutional layers with pooling, adding new resizing layers upfront is unnecessary and would very likely hurt the performance. On the plus side, this means you should be able to approach the problem with your original resolution and fine-tune the exisiting ResNet with you own custom output layers.

More Detail:

Off the bat, Convolutional layers tend to learn to work with different scales of features through the model, with early layers working with pixel patterns and later layers finding more abstract patterns as discussed in this wonderful Distill piece, which contains the image below:

Adding convolutional layers before the input of a trained backbone model will add a layer of abstraction from the original pixels which the early layers of the backbone won't be trained to work on. As such, the additional layers you suggest in the question would likely interfere quite heavily with the backbone's performance.

---

However, as the majority of the ResNet18 architecture is convolutional layers it's possible (as you may already be aware) to feed any size image into the model and get a prediction on the output. This is due to the convolutional and global pooling layers reshaping the input as it goes through the model, discussed briefly here, which means you won't lose pixel-resolution at the input stage.

This means you could follow the traditional fine-tuning route using the ResNet18 archicture as-is, replacing and retraining the final fully-connected layer, freezing the rest of the parameters to speed up the training.

That said, due to the difference between the original and your image resolutions it's possible that the performance of the backbones's convolutions will be degraded regardless. To mitigate this, you could not freeze any of the ResNet18 parameters, effectively using it as a pre-trained warm-start for you problem and leveraging transfer learning.

Happy experimenting!