Extract Features at Multiple Image-Scales

Question

I try to replicate the results of this paper. They state, that they used VGG16- and VGG19-models pretrained on imagenet and used the output of the last convolutional layer (without relu and max-pooling) as feature vectors.

To configure the model accordingly i do:

from tensorflow.keras.models import Model
from tensorflow.keras.applications import VGG16
    
base_model = VGG16(include_top=False)  # Cut off the fully-connected-layers
model = Model(inputs=base_model.input, outputs=base_model.get_layer('block5_conv3').output)  # Discard the last max-pooling-layer
model.layers[-1].activation = None  # Change activation-function of last layer from Relu to Linear

and i get a feature-tensor of shape (1, 14, 14, 512) for the default input-shape of (224, 224, 3). So far, this looks the way i would want it to be. However, the authors state:

The VGG-M [equivalent to VGG16 in the context of the paper] convolutional features are extracted as the output of the last convolutional layer, directly from the linear filters excluding ReLU and max pooling, which yields a field of 512-dimensional descriptor vectors

Now, i have stated above, that the number 512 is part of my output-shape. However i thought, that it means that i get back 512 individual image-patches of size 14x14! The only way i could think of, that would get me 512-dimensional descriptor vectors would be something like this:

features = model.predict(img)
    
feature_vectors = []
    for i in range(features.shape[1]):
        for j in range(features.shape[2]):
            feature_vectors.append(features[0, i, j, :])
feature_vectors = np.array(feature_vectors)

But then i would slice through all of the existing image-patches!

Question 1:

Did the authors mean to do just that? Is this a common practice anyone her has used before? All the tutorials or blogposts i found online just flatten() the output-tensor and add it to the database of existing feature-vectors.

Question 2:

The authors also state, that:

...local descriptors are extracted at multiple scales, obtained by rescaling the image by factors $2^s, s=−3,−2.5,\dots,1.5$ (but, for efficiency, discarding scales that would make the image larger than $1024^2$ pixels).

I can totally extract images at different scales, by specifying the input_shape, when instantiating the VGG16-model. My method of getting 512-dimensional feature-vectors stated above would also work in that case, even though the output-tensor could be much larger (i.e. a shape of (1, 64, 64, 512)) in case of a bigger image.

Is this the way to do feature-extraction at multiple scales?

Answer 1

Not sure I have understood well your second question but I am gonna try to see if I can help you.

Question 1

Yes, the authors meant just that. I see were the confusion might come. So:

• The authors say: "[...] yields a field of 512-dimensional descriptor vectors"
• You say: "[...] i get back 512 individual image-patches of size 14x14"
• You get: "i get a feature-tensor of shape (1, 14, 14, 512)"

So when the authors say a field of K-dimensional vector, with the word field they implicitly say that is a 2D vector (so $[H,W]$) of K dimensions, so $[H,W,K]$ which is exactly what your are getting (only with the batch dimension first, that Keras ignores in its input, that's why I prefer pytorch, it is more explicit).

So being explicit, from an input image $I:[B,H,W,C]=[1, 224, 224, 3]$ you get a field (or patch, or feature) vector, a 2D vector, of $K=512$ dimensions (or features, or descriptors) so $F:[B,H_f, W_f, K]=[1,14, 14, 512]$

What you are doing in your snippet is taking 1D vectors of $K=512$ dimensions, so you are extracting K-dimensional vectors from a K-dimensional feature map (or patch or field).

The flatten() goes in the second question.

Question 2

There are a lot of ways of doing feature-extraction at multiple scales. Say for example you extract features at 3 levels so you get:

• $P_1:[H_1, W_1, C_1]=[56, 56, 100]$
• $P_2:[H_2, W_2, C_2]=[26, 26, 200]$
• $P_3:[H_3, W_3, C_3]=[13, 13, 300]$

This vectors are not concatenable. You indeed could flatten() them over the $[H,W]$ dimensions and concat over the channel dimension. But you would loose the 2D spatial relationship of images. What is normally done (from FPN, Retinanet EfficientNet, EfficientDet...) is the following:

• You extract the previous feature maps at different scales
• You process them over a FCN head of your network (so last dimension is bounding box coordinates or class probabilities)
• You concatenate over last dimension

After extracting $P_1, P_2, P_3$ and passing by a FCN classification head on your network you would get (given you have 10 classes):

• $P_1':[H_1, W_1, C_1]=[56, 56, 10]$
• $P_2':[H_2, W_2, C_2]=[26, 26, 10]$
• $P_3':[H_3, W_3, C_3]=[13, 13, 10]$

Now you have a classification vector per each "pixel" on each feature dimension. So now it makes more sense to flatten them to get 1D vectors of 10 class probs:

• $P_{concat}=[N, 10]$ where $N$ is the flattened number of dimensions from $H,W$ of each scale feature map

If you want the implementation details go to RetinaNet in papers with code. I would link one very nice implementation but it is in pytorch, and I see you use Keras.