A1. There are different alternatives to fine-tune on a dataset with a different shape than the one used for training (assuming training dataset's shape is
s1 and target shape is
- If the target shape is similar (i.e. slightly lower or greater): you can crop (if
s2 is smaller than
s1) or pad (if
s2 is larger than
s1). Padding is a simple method that works for tabular, sequence, and image data. The usual value is zero, but if that has a precise meaning from training dataset
d1 you may want to pick a value that is meaningless: anyway, the network should learn that the padded value is always constant having no correlation with the targets.
- If the target shape is quite different:
- In the strict case of image data (note: restrictions about the learning framework can apply), a fully-convolutional network (FCN) - made of only convolutional layers, replacing even the Dense (or fully-connected) layers with $1\times 1$ convolutions to handle varying images sizes - may do the job but you still need to handle varying output size.
- Perhaps, a more practical alternative would be to train an intermediate model that converts (i.e. tries to learn a compatible representation of) a sample with target shape
s2 to source shape
s1 such that you can reuse your trained model on
d1. This, in principle, should work on any kind of data.
More details on the "intermediate model approach"
Say you have a trained model $f(x)$ on dataset
d1 with shape
s1, you want to transfer its knowledge to a model $f_2(\bar x)$, and then fine-tune on dataset
d2 with shape
s2. To do you learn a dataset-specific model $g:X\to \bar X$ that learns to represent samples from
d2 as if they were from
During training (but also inference) you would use the model $g$ as follows (in pseudo-code):
x_bar, y = sample from d2
x = g(x_bar)
y_hat = f_2(x)
# gradient step
loss = criterion(y, y_hat)
grads = loss.backward()
optimizer.apply_gradients(grads, weights=g + f_2) # <--
Basically, you learn $g$ by letting the gradient of the error flow to it from $f_2$.
Such an approach should work well if the features in
d1 are somehow correlated (even by some hidden relation) with the ones in
d2: but, in general, if there is no correlation you can't transfer the learning at all!
If you want to try such approach I would suggest to follow a two-phase training strategy (assuming $f_2$ is a clone of $f$):
- Warmup: You freeze (i.e. make them fixed, or not learnable) the weights of $f_2$. Pick a small learning rate (the common
3e-4 may work well) and train $g$ on the target dataset until the error (yield by $f_2$) stabilizes (is not important that is low.) This is to ensure that when you later fine-tune the gradient magnitude is sufficiently low.
- Fine-tuning: according to the problem (i.e. how much different the features are in
d2) you may want to train only the last output layer, the penultimate layers, or even all layers (if the features are quite different). Anyway, is important to use a much smaller learning rate (especially if re-training all layers): say 10-20x smaller than before, or compared to the lr used to learn
A2. If you change the shape of the input layer, the weight matrix of the next layer (i.e. first hidden layer) is usually dependent on that shape, and so it's very likely that that would change to. For convolutional layers, and so VGG, in principle, you should have that the conv layers adapt to the input size but what can be problematic are still the last dense layers.