0
$\begingroup$

A CNN model is based on a series of filters applied to an image. However, these filters can only "see" a small portion of the image and they have no information of the relative position of the pixels that they are analyzing with respect to the image coordinates. Likewise, deeper in the network, they have no information of the the position of the field of view that they are analyzing with respect to the coordinate-system of the image. In other words, if trained correctly, these modes are translation-invariant and they do not "know" the region of the image to which some features belong (ie, they have no information if some extracted features belong to the lower, upper, right or left part of the image).

I was wondering how a model like, for example, YOLO can infer the relative coordinates of the objects that it is detecting, without having this information. I understand that the image is divided in in a grid of cells and that gives some notion of region (ie, part of the image that is being analyzed). However, we still have to guess the width, height and relative x,y position with respect to the cell. Why don't we overlap the input image (or extracted features) with some kind of positional encoding (the x,y coordinates for example), so that the model has information of the region of the image to which some features belong?

This question also applies to other CNN models like, for example, U-Net architectures. I think that it may be useful to have some information of the position of the pixels/feature map with respect to the image's coordinates in some problems. For example, to segment the road in self-driving problems, it will be very useful to "know" if the region of the image that is being analyzed is in the bottom part of the image as, most-likely, the road will be found there.

How does these models infer this information? And, why is not common to add this "positional encoding" in deep learning projects applied to images?

Improve this question
$\endgroup$
5
  • $\begingroup$ Can you explain what you mean when you say it infers the coordinates? do you give it a single pixel and it tells you x=100 y=200? $\endgroup$
    – Criticizing Israel not allowed
    Commented Mar 15, 2023 at 21:06
  • $\begingroup$ With this I mean that it is some useful information. It depends on the model, but in some problems, knowing that some pixels belong to a particular region of the image may be useful (for example, when segmenting the road it is usefull to know if the region is the lower part of the image or the upper part). As well, some models like YOLO directly outputs the coordinates (x, y, w, h) of the objects it is detecting. $\endgroup$
    – IgnacioGaBo
    Commented Mar 16, 2023 at 8:52
  • 1
    $\begingroup$ After the convolutional layers there are usually dense layers. $\endgroup$
    – Criticizing Israel not allowed
    Commented Mar 16, 2023 at 12:08
  • $\begingroup$ You are right. I guess that the dense layers may give a sense of spatial location as they are analyzing the image as a whole. However, I see it as a very indirect way to get this sense of spatial location. Why don't we usually add a positional encoding to the input directly? In addition, you don't have this in purely convolutional models such as CNNs for segmentation. $\endgroup$
    – IgnacioGaBo
    Commented Mar 16, 2023 at 15:47
  • $\begingroup$ I don't know. You could try it and see. For segmentation the network is probably not aware of which part of the picture it's looking at although I suppose some positional encoding could help it decide (e.g. sky is at the top) $\endgroup$
    – Criticizing Israel not allowed
    Commented Mar 16, 2023 at 15:54

2 Answers 2

Reset to default
0
$\begingroup$

A filter value in a smaller deeper layer is in a fixed relationship to the geometry of the input.

Its position in the filter defines its relation to an image.

But positional encoding is actually needed and common in transformers etc, where that is not the case.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Your answer could be improved with additional supporting information. Please edit to add further details, such as citations or documentation, so that others can confirm that your answer is correct. You can find more information on how to write good answers in the help center. $\endgroup$
    – Community Bot
    Commented Mar 15, 2023 at 20:53
  • $\begingroup$ Well, I see what you say that the deeper you go, the bigger is the field of view and the features extracted will be related to that field of view and nothing else. However, you still have the "problem" of not knowing to what exact region of the image (coordinates x,y) does those features belong. $\endgroup$
    – IgnacioGaBo
    Commented Mar 16, 2023 at 7:59
0
$\begingroup$

In a purely convolutional network, shift equivariance is inherent (ignoring boundary effects), meaning the network lacks an awareness of specific x/y coordinates. Introducing positional information by overlaying the pixel grid with coordinates would enable the model to gain a sense of distance within the image.

However, incorporating absolute positional encoding would break this shift equivariance. Consequently, the model would lose this advantageous property and would function more similarly to applying an MLP directly to the entire image.

It is important to note that shift equivariance is typically considered a beneficial feature of CNNs, as it allows for the detection of objects regardless of their location within the image.

Nonetheless, if you require a sense of absolute position while still retaining the shared parameter architecture of CNNs, you can indeed opt for absolute positional encoding.

Improve this answer
$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .