A1. There are different alternatives to fine-tune on a dataset with a different shape than the one used for training (assuming training dataset's shape is s1
and target shape is s2
):
- If the target shape is similar (i.e. slightly lower or greater): you can crop (if
s2
is smaller than s1
) or pad (if s2
is larger than s1
). Padding is a simple method that works for tabular, sequence, and image data. The usual value is zero, but if that has a precise meaning from training dataset d1
you may want to pick a value that is meaningless: anyway, the network should learn that the padded value is always constant having no correlation with the targets.
- If the target shape is quite different:
- In the strict case of image data (note: restrictions about the learning framework can apply), a fully-convolutional network (FCN) - made of only convolutional layers, replacing even the Dense (or fully-connected) layers with $1\times 1$ convolutions to handle varying images sizes - may do the job but you still need to handle varying output size.
- Perhaps, a more practical alternative would be to train an intermediate model that converts (i.e. tries to learn a compatible representation of) a sample with target shape
s2
to source shape s1
such that you can reuse your trained model on d1
. This, in principle, should work on any kind of data.
More details on the "intermediate model approach"
Say you have a trained model $f(x)$ on dataset d1
with shape s1
, you want to transfer its knowledge to a model $f_2(\bar x)$, and then fine-tune on dataset d2
with shape s2
. To do you learn a dataset-specific model $g:X\to \bar X$ that learns to represent samples from d2
as if they were from d1
.
During training (but also inference) you would use the model $g$ as follows (in pseudo-code):
x_bar, y = sample from d2
x = g(x_bar)
y_hat = f_2(x)
# gradient step
loss = criterion(y, y_hat)
grads = loss.backward()
optimizer.apply_gradients(grads, weights=g + f_2) # <--
Basically, you learn $g$ by letting the gradient of the error flow to it from $f_2$.
Such an approach should work well if the features in d1
are somehow correlated (even by some hidden relation) with the ones in d2
: but, in general, if there is no correlation you can't transfer the learning at all!
If you want to try such approach I would suggest to follow a two-phase training strategy (assuming $f_2$ is a clone of $f$):
- Warmup: You freeze (i.e. make them fixed, or not learnable) the weights of $f_2$. Pick a small learning rate (the common
3e-4
may work well) and train $g$ on the target dataset until the error (yield by $f_2$) stabilizes (is not important that is low.) This is to ensure that when you later fine-tune the gradient magnitude is sufficiently low.
- Fine-tuning: according to the problem (i.e. how much different the features are in
d2
) you may want to train only the last output layer, the penultimate layers, or even all layers (if the features are quite different). Anyway, is important to use a much smaller learning rate (especially if re-training all layers): say 10-20x smaller than before, or compared to the lr used to learn f
.
A2. If you change the shape of the input layer, the weight matrix of the next layer (i.e. first hidden layer) is usually dependent on that shape, and so it's very likely that that would change to. For convolutional layers, and so VGG, in principle, you should have that the conv layers adapt to the input size but what can be problematic are still the last dense layers.
dataset1
which is (28, 9, 1) in shape, you then train your model, next you takedataset2
that is (20, 5, 1) and pad it (e.g. with zeros) to get a shape of (28, 9, 1). Finally you fine-tune on the padded data. Does it solve you doubt? $\endgroup$