# Extract Features at Multiple Image-Scales

I try to replicate the results of this paper. They state, that they used VGG16- and VGG19-models pretrained on imagenet and used the output of the last convolutional layer (without relu and max-pooling) as feature vectors.

To configure the model accordingly i do:

from tensorflow.keras.models import Model
from tensorflow.keras.applications import VGG16

base_model = VGG16(include_top=False)  # Cut off the fully-connected-layers
model = Model(inputs=base_model.input, outputs=base_model.get_layer('block5_conv3').output)  # Discard the last max-pooling-layer
model.layers[-1].activation = None  # Change activation-function of last layer from Relu to Linear


and i get a feature-tensor of shape (1, 14, 14, 512) for the default input-shape of (224, 224, 3). So far, this looks the way i would want it to be. However, the authors state:

The VGG-M [equivalent to VGG16 in the context of the paper] convolutional features are extracted as the output of the last convolutional layer, directly from the linear filters excluding ReLU and max pooling, which yields a field of 512-dimensional descriptor vectors

Now, i have stated above, that the number 512 is part of my output-shape. However i thought, that it means that i get back 512 individual image-patches of size 14x14! The only way i could think of, that would get me 512-dimensional descriptor vectors would be something like this:

features = model.predict(img)

feature_vectors = []
for i in range(features.shape[1]):
for j in range(features.shape[2]):
feature_vectors.append(features[0, i, j, :])
feature_vectors = np.array(feature_vectors)


But then i would slice through all of the existing image-patches!

Question 1:

Did the authors mean to do just that? Is this a common practice anyone her has used before? All the tutorials or blogposts i found online just flatten() the output-tensor and add it to the database of existing feature-vectors.

Question 2:

The authors also state, that:

...local descriptors are extracted at multiple scales, obtained by rescaling the image by factors $$2^s, s=−3,−2.5,\dots,1.5$$ (but, for efficiency, discarding scales that would make the image larger than $$1024^2$$ pixels).

I can totally extract images at different scales, by specifying the input_shape, when instantiating the VGG16-model. My method of getting 512-dimensional feature-vectors stated above would also work in that case, even though the output-tensor could be much larger (i.e. a shape of (1, 64, 64, 512)) in case of a bigger image.

Is this the way to do feature-extraction at multiple scales?

Not sure I have understood well your second question but I am gonna try to see if I can help you.

Question 1

Yes, the authors meant just that. I see were the confusion might come. So:

• The authors say: "[...] yields a field of 512-dimensional descriptor vectors"
• You say: "[...] i get back 512 individual image-patches of size 14x14"
• You get: "i get a feature-tensor of shape (1, 14, 14, 512)"

So when the authors say a field of K-dimensional vector, with the word field they implicitly say that is a 2D vector (so $$[H,W]$$) of K dimensions, so $$[H,W,K]$$ which is exactly what your are getting (only with the batch dimension first, that Keras ignores in its input, that's why I prefer pytorch, it is more explicit).

So being explicit, from an input image $$I:[B,H,W,C]=[1, 224, 224, 3]$$ you get a field (or patch, or feature) vector, a 2D vector, of $$K=512$$ dimensions (or features, or descriptors) so $$F:[B,H_f, W_f, K]=[1,14, 14, 512]$$

What you are doing in your snippet is taking 1D vectors of $$K=512$$ dimensions, so you are extracting K-dimensional vectors from a K-dimensional feature map (or patch or field).

The flatten() goes in the second question.

Question 2

There are a lot of ways of doing feature-extraction at multiple scales. Say for example you extract features at 3 levels so you get:

• $$P_1:[H_1, W_1, C_1]=[56, 56, 100]$$
• $$P_2:[H_2, W_2, C_2]=[26, 26, 200]$$
• $$P_3:[H_3, W_3, C_3]=[13, 13, 300]$$

This vectors are not concatenable. You indeed could flatten() them over the $$[H,W]$$ dimensions and concat over the channel dimension. But you would loose the 2D spatial relationship of images. What is normally done (from FPN, Retinanet EfficientNet, EfficientDet...) is the following:

• You extract the previous feature maps at different scales
• You process them over a FCN head of your network (so last dimension is bounding box coordinates or class probabilities)
• You concatenate over last dimension

After extracting $$P_1, P_2, P_3$$ and passing by a FCN classification head on your network you would get (given you have 10 classes):

• $$P_1':[H_1, W_1, C_1]=[56, 56, 10]$$
• $$P_2':[H_2, W_2, C_2]=[26, 26, 10]$$
• $$P_3':[H_3, W_3, C_3]=[13, 13, 10]$$

Now you have a classification vector per each "pixel" on each feature dimension. So now it makes more sense to flatten them to get 1D vectors of 10 class probs:

• $$P_{concat}=[N, 10]$$ where $$N$$ is the flattened number of dimensions from $$H,W$$ of each scale feature map

If you want the implementation details go to RetinaNet in papers with code. I would link one very nice implementation but it is in pytorch, and I see you use Keras.

• Thanks for your explanation. Regarding Question 2: The "different scales" part of the question regards the resizing of the original image. The default input-shape of VGG16 is 224 x 224 x 3. I have images that are 2048x2048, so in order to get to the input shape i would have to shrink them. "Different scales" would mean i rescale not only to 224 x 224 but also to other sizes, for example 1024x1024. Because of the bigger input size, the output at the last conv-layer would also yield bigger output-patches (64x64 instead of 14x14 in the case of 224 x 224 input-images) – Tim Hilt Nov 12 '20 at 9:08
• Ah! OK, I understood that you extracted multiple feature maps at different scales in your VGG net. Thing is, from some years till now the multiple scales are gotten from different parts of the nets not by doing multiple pass on the same image with different scales. But OK, it is a paper from 4 years ago. Anyways the flatten strategy still applies – JVGD Nov 12 '20 at 10:09