2
$\begingroup$

I am looking to detect think objects like pens, pencils and surgical instruments. The bounding box is not important, but I am looking to see if I can train a model to detect both the object as well as its orientation. Typical object detection networks like R-CNN,YOLO,SSD encode class name and bound boxes. Instead of bounding boxes, I'm looking to encode only 2 points, one starting x,y point and one ending x,y point. The start point for objects is where one would grip the object. For instance:

  • The pencil eraser(start point) is pointed 50 degrees to the top right.
  • The surgical instrument is 10 degrees from the x-axis and handle is pointed to the bottom right.
  • Pen tip(end point) is pointing vertically upwards.
  • Fork, the start point would be the grip handle part, and the end point would be in the middle where the 4 prongs are.

As long as I can encode the start and end points, then I can determine the orientation. I would need to define these points during training and the question is whether there is an existing model (mobile net/inception/RCNN) that I can encode this information in? One potential way I was thinking was to use YOLO and for the bounding box, the top left x,y would be the starting point x,y(handle) whereas the bound box width, height would be replaced with the end point x,y(pencil writing tip, fork prongs.

Thank you.

|improve this question
$\endgroup$
1
$\begingroup$

Recent work achieves a similar task: Object recognition together with the bounding box (e.g. YOLO---there are quite a few on Github too). The bounding box is not enough in your case, but it is an interesting pattern: Recognition plus some form of measurement. Such architectures could be good candidate to start with, and repurpose for stick orientation.

The problem could also leverage the current results in gait recognition. In fact, this looks closer to the problem at hand than object recognition. An example is this model based on multiview (many pictures input) recognition, with a demonstration on Github. Gait recognition is also popular these days, and many inspiring papers and OSS implementations are available.

The above presents two approaches your problem could benefit from, as a "combination". My gut feeling is that tilt and orientations may be easier than direction (i.e. where is the tip?).

The question calls for training a model. An alternative approach, perhaps to start with and get more insight, could be to go with "standard" computer vision algorithms, such as the Hough Transform. This transforms allows to find lines in an image. The mathematics are at reach, and it may work well enough for a quick demo. Also, your handle name suggests "embedded mobile" engineer, and a simple Hough Transform could be cheap on mobile.

|improve this answer
$\endgroup$
  • $\begingroup$ Appreciate the feedback. I am considering a modified version of any bounding box model. Instead of the coordinates defining the box, it would just be the coordinates for the tip and end of the instrument. So for instance, for a pencil, the tip would be the sharp lead writing part while the end would be the eraser. For forceps, the tip would be the clamp of the forceps while the end would be the handle, where the forceps are attached. I just do not know if this approach can work. I wanted to get some confidence before labeling all the training data before going ahead with this approach. $\endgroup$ – iOScoder Jun 1 '18 at 3:16
  • 1
    $\begingroup$ Hough transform would be great for lines (like these), but cannot determine orientation. You're right though, it is indeed good for mobile. :) $\endgroup$ – iOScoder Jun 1 '18 at 3:19
  • $\begingroup$ Is there a way to update the question to get more specific answers, and perhaps refine mine? As for the Hough transform, sorry if the explanation was vague (possible edit too). The transform detects lines only. The orientation takes a bit more work, but knowing two points on each line, and the orientation of the photo (e.g. EXIF metadata) are enough. $\endgroup$ – Eric Platon Jun 1 '18 at 3:34
  • $\begingroup$ I've updated the question to be more specific. Thanks. $\endgroup$ – iOScoder Jun 5 '18 at 6:38
  • 1
    $\begingroup$ I would say pose recognition is also very similar to OP's requirement. E.g. static.googleusercontent.com/media/research.google.com/en//pubs/… - although achieving results like the paper requires a lot of labelled data, it might be achieved for simpler objects via using a game engine renderer. $\endgroup$ – Neil Slater Jun 5 '18 at 7:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.