How can I detect thin objects (like pens and pencils) without a bounding box but only 2 endpoints and the orientation?

Question

I am looking to detect thin objects, like pens, pencils, and surgical instruments. The bounding box is not important, but I am looking to see if I can train a model to detect both the object as well as its orientation.

Typical object detection networks, like R-CNN, YOLO, and SSD encode the class name and bounding boxes. Instead of bounding boxes, I'm looking to encode only 2 points, one starting $x,y$ point and one ending $x,y$ point. The start point for objects is where one would grip the object. For instance:

• The pencil eraser(start point) is pointed 50 degrees to the top right.
• The surgical instrument is 10 degrees from the x-axis and the handle is pointed to the bottom right.
• Pen tip (endpoint) is pointing vertically upwards.
• Fork, the start point would be the grip handle part, and the endpoint would be in the middle where the 4 prongs are.

As long as I can encode the start and endpoints, then I can determine the orientation. I would need to define these points during training.

The question is whether there is an existing model (mobile net/inception/RCNN) that I can encode this information in? One potential way I was thinking was to use YOLO and for the bounding box, the top left $x,y$ would be the starting point $x,y$ (handle), whereas the bounding box's width and height would be replaced with the endpoint $x,y$ (pencil writing tip, fork prongs.

Answer 1

You could just go with any fully convolutional network that has the same output resolution as the image input resolution (e.g. from semantic segmentation) as your backbone.

Then

Solution 1: You could adapt the last layer to have 4 feature maps where the first 2 maps represent the proability of existance or non-existance of point 1. And the last 2 maps represent the proability of existance or non-existance of point 2. This then gives you a heatmap of the existance of such points. Here you should also do non-maximum supression to only get single likely points instead of areas. Furthermore you need to apply the softmax function correctly only on each 2 points separately. Or use a sigmoid layer where you only solve a binary problem and use 2 feature maps instead of 4.

Solution 2: If there is no clear physical/unique explaination of wich point should be regarded as point 1 and which point as point 2 then solution 1 might be problematic since you force the model to decide on that (by having dedicated point 1 and point 2 feature maps) which is then actually not possible.

In that case you could do e.g. this approach:

Use 4 feature maps where the first two maps represent the probability of existance or non-existance of a point in general. Then use the last 2 maps to represent the normalized x/y vector that points to the corresponding other point. Then afterwards you can filter out corresponding points by looking if their regression vector points towards each other. (This is somewhat similar and adapted to what center point net does). Here only the first 2 layers should be sigmoided and the last 2 are regression layers, but normalized. Non-max suppression should also be done here to filter out only single points.

These both approaches have the nice property of being fully convolutional and therefore translation invariant. It means for example you can easily adapt it to other image resolutions.

Answer 2

Recent work achieves a similar task: Object recognition together with the bounding box (e.g. YOLO---there are quite a few on Github too). The bounding box is not enough in your case, but it is an interesting pattern: Recognition plus some form of measurement. Such architectures could be good candidate to start with, and repurpose for stick orientation.

The problem could also leverage the current results in gait recognition. In fact, this looks closer to the problem at hand than object recognition. An example is this model based on multiview (many pictures input) recognition, with a demonstration on Github. Gait recognition is also popular these days, and many inspiring papers and OSS implementations are available.

The above presents two approaches your problem could benefit from, as a "combination". My gut feeling is that tilt and orientations may be easier than direction (i.e. where is the tip?).

---

The question calls for training a model. An alternative approach, perhaps to start with and get more insight, could be to go with "standard" computer vision algorithms, such as the Hough Transform. This transforms allows to find lines in an image. The mathematics are at reach, and it may work well enough for a quick demo. Also, your handle name suggests "embedded mobile" engineer, and a simple Hough Transform could be cheap on mobile.