Detecting object position given the relative position of another object

Question

I know that the title might be redundant but I'm trying to understand if there is way to predict where a specific object will be if I provide a certain object as a reference.

See as an example the image . The red object (which can be provided as a set of coordinates) in always present as a feature. The blue object is the one to be predicted. In general the red polygon can vary (more elongated, larger or smaller) and the blue object change its position wrt the red one.

I am thinking, if possible, about a system where in a first training phase I provide both the objects with the blue object been the target of my training.

In a test phase I would provide only the red object and let the system try to guess where the blue object will be.

The underlying relation between the two objects relative position is not well understood in general so my idea was trying to understand if out there is something that could generalize and approach this problem.

It might be a conceptual question for the moment, in case more details are needed I would be happy to discuss them.

Thanks!

Answer 1

I always like to think that Theoretically, if there exists some function $f:R \rightarrow B$ that maps the set of points $R$ which represent your reference object to the set of points $B$ which represent your target object, then there should be a network $N_\theta$ that approximates $f$ with arbitrarily small precision. Sadly for us, it may be very difficult to find $N_\theta$, but hey let's try anyway!

In terms of the algorithmic approach, your suggestion seems likely. Another possibility would be to try and formulate a basic CNN that takes in normalized images of the reference object and outputs a $(x,y)$ that will represent the center of your target object. For that, of course, you will have to create data that consists only of images of rotated + scaled version of your initial reference object and labels that are the center points $\{(x_i,y_i)\}$ of the target object.

It might also be reasonable to consider transfer-learning using some pre-trained ImageNet-based network, as those were shown to improve results even for very different datasets (compared to ImageNet)