2
$\begingroup$

I'm currently studying CNNs and I had the idea of building a model without a fully connected layer at the end. I think this could be beneficial, if one can somehow model the desired outputs as a matrix. A basic problem solvable in this way is the following: let's say we want to place a dot on the nose of a person in an image. The basic approach is

img -> [conv layer] -> [pooling] -> [...] -> [fully connected layer] -> pred_x, pred_y

where pred_x and pred_y are the two nodes in the final layer, after the fully connected one, and represents the coordinates of the dot.

Every coordinate x, y can be represented with a matrix of size image_width x image_height with all zeros except a 1 in the position int(x), int(y). This means that it might be possible to have a CNN like the following:

img -> [conv layer] -> [pooling] -> [conv layer] -> [...] -> act_map

where act_map is the activation map resulting from the last convolutional layer. Given act_map we can extract pred_x, pred_y by getting the two indexes for which act_map is maximum.

Maybe this doesn't make sense with a single output, but imagine a problem where we want to predict the position of n dots on an image. Then we could simply have n activation maps at the end.

I have a few questions:

  1. where can I find more resources on this topic? I'm sure I'm not the first one to have thought about this
  2. could this really work? I don't really know how to create a loss function

Thanks

$\endgroup$

2 Answers 2

3
+100
$\begingroup$

It looks like you are thinking about something like UNET architecture.

The final layer to your toy problem with a "dot" on the nose can be modelled as a convolution layer from last convolution layer of your network of a shape [BatchSize, Channels, Width, Hight] to the "output" layer of size [BatchSize, 1, Width, Hight].

You may have a dimensionality problem, since the width and hight of the output image can differ from the input. This can happened if you decide to, for example, perform pooling in your architecture, luckily there are a lot of upsampling methods. You also can use architecture of CNN, that does not change image's size, like UNET, YOLO. I recommend you to look at the problem of image segmentation, since it looks very similar to the example given by you.

As of your second point. If you would want to predict a mask of onces and zeros you can use sigmoid function for each activation ( pixel ) of last convolution layer to get the probability as an output and then train the network by minimising cross entropy between predicted probabilities and true "mask" of an image.

$\endgroup$
2
$\begingroup$

I agree with the answer by @vl_knd and want to add my two cents:

This problem does sound a lot like a special case of image segmentation. You want an image where each pixel gets assigned a class. In your case, most pixels will be class 0 and some individual pixels will be class 1, that's where the points are. A very similar field of research is 2D pose estimation. Given an image, you want to map a skeleton onto that image that indicates the pose of a depicted person. This is mostly mapping 2D points onto the joints of a person. There exists work for both the ideas you mentioned (regression of $x,y$ and prediction a heat-map representation of the $x,y$-points on an image) Here is a reading list of such papers. And this paper predicts the $x, y$ pixels in the form of heat maps.

You might also find some ideas in papers that tackle high-precision segmentation problems. The medical domain might be interesting (i.e. blood vessel segmentation).

When it comes to the loss function, it should probably be a bit forgiving. When you want a pixel $(x,y)$ to be classified as 1, neighboring pixels likely have some ambiguity as to whether they should be 1 as well. What can come in handy is smooth labeling. You could for example center a Gaussian at $(x,y)$ with some variance to include neighboring pixels and provide a smoother training signal.

$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .