1
$\begingroup$

I have been researching if there are any existing machine learning models that would help mark objects (for example: cars) on the map having only image, camera location, and camera orientation.

For example:

We know the camera's location, FOV

From the camera's image we detect objects: 2 cars

Our model having all the info provided before marks those two cars on the map

Are there any existing models that could help solve this problem? With today's tools is it even possible to build such a model?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
2
$\begingroup$

It should be possible with a combination of object detection, monocular depth estimation (for example Monodepth, Godard et al.) and some math. Here is an example of how this could look like:

  1. Apply object detection/segmentation model to your image to find bounding boxes
  2. Determine the position of the center of the bounding box in the camera FOV, you could calculate a relative position here via: $\frac{\text{bbox}_x}{\text{img-width}} = p_x$
  3. Apply monocular depth estimation to generate a depth map from your camera image
  4. Take the bounding boxes to cut out regions in the depth maps and compute the average distance of the object to the camera lens.

Now you have: relative x-position in FOV $p_x$ and distance $d$, which gives you a vector in radial coordinates from your camera lens $\mathbf{\hat{p}} = (p_x, d)$ that directly translates to a position in the first image you showed.

Improve this answer
$\endgroup$
5
  • $\begingroup$ That's what I thought at first, this approach looks very straightforward and doable, but is very complex if we want to use different cameras. I failed to highlight in my question that I am looking for a little bit more exotic approach - is possible to estimate object location by surrounding objects in an image on a 2D map? For example: We can see a car near the corner of the building. Can we recognize that building on the map, locate it, and estimate that the car is near the building corner? $\endgroup$
    – user3500960
    Commented Feb 21 at 8:02
  • $\begingroup$ Ah, that will be more difficult. Here is something to think about first though: If your biggest problem is the changing cameras, I'd go for something like a CNN that estimates the FOV for each camera to make the approach in my answer applicable. This way would be simpler and would most likely also generalize better. But sounds like that's also not exactly what you are after. What do you think of the FOV estimation to account for changing cams? Or is it more that you'd like an "end-to-end AI solution" for this problem? $\endgroup$
    – Chillston
    Commented Feb 21 at 9:30
  • $\begingroup$ I think FOV estimation is a smart solution and I will try to build something using this approach. However having "end-to-end" solution would be amazing. I am still not very advanced in this field, so building something from scratch is still too complex for me. $\endgroup$
    – user3500960
    Commented Feb 21 at 10:48
  • $\begingroup$ Okay, IMO an end-to-end solution would have to 'understand' the concept of a top-down view and also the concept of stationary vs mobile objects. Then, the camera image can be matched to all stationary objects that are also visible from top-down on the map as the reference points. However, this will not be easy to implement reliably and with the ability to generalize to unseen scenery. $\endgroup$
    – Chillston
    Commented Feb 21 at 14:30
  • $\begingroup$ I'd try first with some modular AI system which should be easier to do and presents more structure on how to approach this. On the other hand you basically have a mountain of available AI models at your disposal but with no real 'recipe' how to apply it to your case, it can be quite frustrating when you are new to the field. Maybe some related area that deals with somewhat similar problems is SLAM, i.e. simultaneous locating and mapping - it sounds like its very related to your project, maybe you find some inspiration there. $\endgroup$
    – Chillston
    Commented Feb 21 at 14:31

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .