Cartesian Bias and Pipeline Efficiency
You are experiencing a techno-cultural artifact of Cartesian-centric imaging running all the way back to the dawn of coordinate systems. It is the momentum of practice as a consequence of applying Cartesian 2D coordinates to rasterize images appearing at the focal planes of lenses from the dawn of television and the earliest standards of raster based capture and display.
Although some work was done toward adding tilt to bounding rectangles in the late 1990s and since, from a time and computing resource conservation perspective, it is computationally and programmatically less costly to include the four useless triangles of pixels and keep the bounding box orthogonal with the pixel grid.
Adding a tilt angle to the bounding boxes is marginally competitive when detecting ships from a satellite only because two conditions offset the inefficiencies in that narrow domain. The ship appears as an oblong rectangle with rounded corners from a satellite positioned in geosynchronous orbit. In the general case, adding a tilt angle can slow recognition significantly.
Biology Less Biased
An interesting side note is that the neural networks of animal and human vision systems do not have that Cartesian-centricity, but that doesn't help this question's solution, since non-orthogonal hardware and software is virtually nonexistent.
Early Non-Cartesian Research and Today's Rasterization
Gerber Scientific Techonology research and development in the 1980s (South Windsor, Connecticut, U.S.) investigated vector capture, storage, and display, but the R&D was not financially sustainable for a mid-side technology corporation for the reasons above.
What remains, because it is economically viable and necessary from an animation point of view, is rasterization on the end of the system that converts vector models into frames of pixels. We see this in on the rendering SVG, VRML, and the original intent of CUDA cores and other hardware rendering acceleration strategies and architectures.
On the object and action recognition side, the support of vector models directly from imaging is much less developed. This has not been a major stumbling block for computer vision because the wasted pixels at one tilt angle may be of central importance at another tilt angle, so there are no actual wasted input pixels if the centering of key scene elements is widely distributed in translation and tilt, which is often the case in real life (although not so much in hygienically pre-processed datasets).
Conventions Around Object Minus Camera Tilt and Skew from Parallax
Once edge detection, interior-versus-exterior, and 3D solid recognition come into play, the design of CNN pipelines and the way kernels can do radial transformation without actually requiring $\; \sin, \, \cos, \, \text{and} \, \arctan \;$ functions evaporate the computational burden of the Cartesian nature of pixel tensors. The end result is that the bounding box being orthogonal to the image frame is not as problematic as it initially appears. Efforts to conserve the four triangles of pixels and pre-process orientation is often a wasted effort by a gross margin.
Summary
The bottom line is that efforts to produce vector recognition from roster inputs have been significantly inferior in terms of resource and wait time burden, with the exception of insignificant gains in the narrow domain of naval reconnaissance satellite images. Trigonometry is expensive, but convolution kernels, especially now that they are moving from software into hardware accelerated computing paths in VLSI, is computable at lower costs.
Past and Current Work
Below is some work that deals with tilting with regard to objects and the effects of parallax in relation to the Cartesian coordinate system of the raster representation. Most of the work has to do with recognizing 3D objects in a 3D coordinate system to project trajectories and pilot or drive vehicles rationally on the basis of Newtonian mechanics.
Efficient Collision Detection Using Bounding Volume Hierarchies of k-DOPs, James T. Klosowski, Martin Held, Joseph S.B. Mitchell, Henry Sowizral, and Karel Zikan, 1998
Sliding Shapes for 3D Object Detection in Depth Images, Shuran Song and Jianxiong Xiao, 2014
Amodal Completion and Size Constancy in Natural Scenes, Abhishek Kar, Shubham Tulsiani, Joao Carreira and Jitendra Malik, 2015
HMD Vision-based Teleoperating UGV and UAV for Hostile
Environment using Deep Learning, Abhishek Sawarkar1, Vishal Chaudhari, Rahul Chavan, Varun Zope, Akshay Budale and Faruk Kazi, 2016
Ship rotated bounding box space for ship extraction from high-resolution optical satellite images with complex backgrounds, Z Liu, H Wang, L Weng, Y Yang, 2016
Amodal Detection of 3D Objects:
Inferring 3D Bounding Boxes from 2D Ones in RGB-Depth Images, Zhuo Deng, 2017
3D Pose Regression using Convolutional Neural Networks, Siddharth Mahendran, 2017
Aerial Target Tracking Algorithm Based on Faster R-CNN Combined with Frame Differencing, Yurong Yang, Huajun Gong, Xinhua Wang and Peng Sun, 2017
A Semi-Automatic 2D solution for Vehicle Speed Estimation from Monocular Videos, Amit Kumar, Pirazh Khorramshahi, Wei-An Lin, Prithviraj Dhar, Jun-Cheng Chen, Rama Chellappa, 2018
