# Algorithms for scene rotation

My goal is to take an image and return another image that looks as if the scene was viewed from another angle. The difference in angle can be small — let's say as if the hand holding the camera moved slightly sideways.

— Stereoscopic Synthesis —

The generation of an image that would likely appear in the right eye of a head from which you already have an image from the left eye (or vice versa) is too complex to expect simple convolution (linear matrix transformation) to achieve a reasonable result.

You are correct that rotation is not the correct description, simply because it is ambiguous. What is rotating? The best description is the synthesis of a stereoscopic image from a single eyed/camera one.

Although deep learning is an approach well suggested, it is a very general term into which a number of concepts, books, research projects, and software components fit. I agree with other answers that indicate one could easily find the target objective missed upon repeated attempts at finding a working solution. Shots in the dark are likely to waste time and effort.

For example, an auto-encoder may not work well because the modelling of depth may not be a feature extracted without having a host of stereoscopic image pairs of similar scenes to which automated feature extraction could be accomplished.

Should feature extraction be possible, it is not noise that needs to be removed, but pixelization and optical distortion that needs to be characterized, so that surfaces revealed by the shift in position could later be imbued with the same contour, focal blur, reflective properties, and edge continuation as the adjacent surfaces of the same objects when pixels corresponding to newly revealed surfaces are generated.

For greatest image authenticity, another noise profile to profile and imbue to generated pixels is the capture device noise profile.

— Formal Problem Restatement —

To narrow machine learning approaches so that we can take a shot in the light, let's consider the model of a three dimensional scene exposed to light sources with two cameras adjacent to one another. Let's consider the input, the output, and the internal architecture required to produce a fairly reliable and accurate second image more formally.

We have image pixel matrix I1 that represents light arriving at camera c1 containing a rectangular image capture surface in an x-z plane upon which a lens of effective focal length l1 and aperture a1 is focused over scene S over a time window starting at s1 and ending at e1. Some point is at the origin of a Cartesian coordinate axis, both camera c1 and another camera c2 point such that the origin is centered in the image capture and the point of lens focus. The three dimensional coordinates of c1 and c2 are known.

You wish to predict the second image I2 arriving at camera c2.

Let's assume, for simplicity, that l1 = l2, that a1 = a2, and that the scene is motionless so that time is not critical in the model. Let's also assume that the y coordinates and the image capture duration (e minus s) are the same for both cameras c1 and c2.

— Solution Architecture —

For this simplified case and assuming the object space is not an abyss containing only one object, the process architecture of the solution is the following. Each --> symbol is a sub-process. The horizontal and vertical positional difference between c1 and c2 is { x, z }.

{ I1, l, a, y, x, z }
--> { I1, l, a, y, { S1 ... Sn }}
--> { I1, l, a, y, { S1 ... Sn }, { E1 ... En }}
--> { I2 }


The first sub-process is a feature extraction, where the features are the three dimensional surfaces visible in the two dimensional image I1. This is a questionable extraction because no y information in the scene is available and there is no mention of y-labeled training data is in the problem statement.

The second sub-process is the extension of features extracted to provide needed surface representation for I2.

The last process is rendering I2, potentially using morphing pixels in I1 and filling transparent sections remaining using E1 through En and knowledge of the contour, reflective properties, edge continuation, and capture device noise profile from feature extraction.

— Practicality of Learning About Scenes —

The effectiveness of any deep learning architecture could benefit from the above understanding of vision and the comprehension of scenes in DNA based life. The problem of automated feature extraction is complicated because the data is unlabeled with y information as stated before.

Learning visual comprehension of arbitrary scenes by DNA based life is assisted by the fact that motion occurs and interaction with physical objects and liquid viscosity provides a vastly greater number of dimensions to the input data.

If deep learning is what you are trying to use here, you should keep in mind that the real intent behind deep learning is to learn a probability distribution, which means that if you were to use a deep learning model to "rotate" images, you can only do it on a specific class of images (e.g. faces, cats, etc...).

If that's your goal, generative models are the way to go:

# Autoencoders

You can train an Autoencoders to slightly change the angle. Autoencoders are a special type of neural network that is trained to output the same input you feed into them with a few imposed restrictions to prevent it from learning a trivial identity function. In your case, you could use a variation of a de-noising autoencoder. A de-noising autoencoder, as the name suggests, generates the same input image minus an artificial stochastic noise. The way this is achieved is by feeding a corrupted version of the image and then evaluating the loss on the non-corrupted version.

In your case, you could feed the original images into your autoencoder and evaluate them based on the rotated images. This will result in your autoencoder effectively learning the inner distribution that generates the images in order to generate a slightly "rotated" version of it. For more info on de-noising autoencoders, see the original paper.

For a more sophisticated approach, you can use generative adversarial networks. GANs are relatively harder to manipulate, but usually perform better than other generative models when it comes to images.