# Is it difficult to learn the rotated bounding box for a (rotated) object?

I have checked out many methods and papers, like YOLO, SSD, etc., with good results in detecting a rectangular box around an object, However, I could not find any paper that shows a method that learns a rotated bounding box.

Is it difficult to learn the rotated bounding box for a (rotated) object?

Here's a diagram that illustrates the problem.

For example, for this object (see this), its bounding box should be of the same shape (the rotated rectangle is shown in the 2nd right image), but the prediction result for the YOLO will be Ist right.

Is there any research paper that tackles this problem?

## 3 Answers

Cartesian Bias and Pipeline Efficiency

You are experiencing a techno-cultural artifact of Cartesian-centric imaging running all the way back to the dawn of coordinate systems. It is the momentum of practice as a consequence of applying Cartesian 2D coordinates to rasterize images appearing at the focal planes of lenses from the dawn of television and the earliest standards of raster based capture and display.

Although some work was done toward adding tilt to bounding rectangles in the late 1990s and since, from a time and computing resource conservation perspective, it is computationally and programmatically less costly to include the four useless triangles of pixels and keep the bounding box orthogonal with the pixel grid.

Adding a tilt angle to the bounding boxes is marginally competitive when detecting ships from a satellite only because two conditions offset the inefficiencies in that narrow domain. The ship appears as an oblong rectangle with rounded corners from a satellite positioned in geosynchronous orbit. In the general case, adding a tilt angle can slow recognition significantly.

Biology Less Biased

An interesting side note is that the neural networks of animal and human vision systems do not have that Cartesian-centricity, but that doesn't help this question's solution, since non-orthogonal hardware and software is virtually nonexistent.

Early Non-Cartesian Research and Today's Rasterization

Gerber Scientific Techonology research and development in the 1980s (South Windsor, Connecticut, U.S.) investigated vector capture, storage, and display, but the R&D was not financially sustainable for a mid-side technology corporation for the reasons above.

What remains, because it is economically viable and necessary from an animation point of view, is rasterization on the end of the system that converts vector models into frames of pixels. We see this in on the rendering SVG, VRML, and the original intent of CUDA cores and other hardware rendering acceleration strategies and architectures.

On the object and action recognition side, the support of vector models directly from imaging is much less developed. This has not been a major stumbling block for computer vision because the wasted pixels at one tilt angle may be of central importance at another tilt angle, so there are no actual wasted input pixels if the centering of key scene elements is widely distributed in translation and tilt, which is often the case in real life (although not so much in hygienically pre-processed datasets).

Conventions Around Object Minus Camera Tilt and Skew from Parallax

Once edge detection, interior-versus-exterior, and 3D solid recognition come into play, the design of CNN pipelines and the way kernels can do radial transformation without actually requiring $$\; \sin, \, \cos, \, \text{and} \, \arctan \;$$ functions evaporate the computational burden of the Cartesian nature of pixel tensors. The end result is that the bounding box being orthogonal to the image frame is not as problematic as it initially appears. Efforts to conserve the four triangles of pixels and pre-process orientation is often a wasted effort by a gross margin.

Summary

The bottom line is that efforts to produce vector recognition from roster inputs have been significantly inferior in terms of resource and wait time burden, with the exception of insignificant gains in the narrow domain of naval reconnaissance satellite images. Trigonometry is expensive, but convolution kernels, especially now that they are moving from software into hardware accelerated computing paths in VLSI, is computable at lower costs.

Past and Current Work

Below is some work that deals with tilting with regard to objects and the effects of parallax in relation to the Cartesian coordinate system of the raster representation. Most of the work has to do with recognizing 3D objects in a 3D coordinate system to project trajectories and pilot or drive vehicles rationally on the basis of Newtonian mechanics.

Efficient Collision Detection Using Bounding Volume Hierarchies of k-DOPs, James T. Klosowski, Martin Held, Joseph S.B. Mitchell, Henry Sowizral, and Karel Zikan, 1998

Sliding Shapes for 3D Object Detection in Depth Images, Shuran Song and Jianxiong Xiao, 2014

Amodal Completion and Size Constancy in Natural Scenes, Abhishek Kar, Shubham Tulsiani, Joao Carreira and Jitendra Malik, 2015

HMD Vision-based Teleoperating UGV and UAV for Hostile Environment using Deep Learning, Abhishek Sawarkar1, Vishal Chaudhari, Rahul Chavan, Varun Zope, Akshay Budale and Faruk Kazi, 2016

Ship rotated bounding box space for ship extraction from high-resolution optical satellite images with complex backgrounds, Z Liu, H Wang, L Weng, Y Yang, 2016

3D Pose Regression using Convolutional Neural Networks, Siddharth Mahendran, 2017

Aerial Target Tracking Algorithm Based on Faster R-CNN Combined with Frame Differencing, Yurong Yang, Huajun Gong, Xinhua Wang and Peng Sun, 2017

A Semi-Automatic 2D solution for Vehicle Speed Estimation from Monocular Videos, Amit Kumar, Pirazh Khorramshahi, Wei-An Lin, Prithviraj Dhar, Jun-Cheng Chen, Rama Chellappa, 2018

Here's a recent paper that does what you're looking for. It looks like they achieve this simply by adding a couple rotated prior boxes and regressing the angles in between. This is similar to what standard object detectors do in terms of creating a bunch of prior box shapes and regressing the actual sizes.

It should not be much more difficult to predict a rotated rectangle compared to a bounding box.

A bounding box can be parameterized with 4 floats: $$x_c$$, $$y_c$$, width, height.

A rotated rectangle can be parameterized with 5 floats: $$x_c$$, $$y_c$$, width, height, angle.

However, to avoid the wrap-around issue with predicting the angle with one value (0° is same as 360°), it should be better to predict sine and cosine instead.

It is actually useful to predict rotated rectangles for text detection (each text field is a rotated rectangle). Indeed, in the wild, text can be in any orientation, and it is important to predict precisely rotated rectangles for OCR to work well. It is especially true for long text boxes near 45° (an axis-aligned bounding box around this would be useless because too big).

Here are 2 links I found about this topic: