# Algorithms for scene rotation

My goal is to take an image and return another image that looks as if the scene was viewed from another angle. The difference in angle can be small — let's say as if the hand holding the camera moved slightly sideways.

If deep learning is what you are trying to use here, you should keep in mind that the real intent behind deep learning is to learn a probability distribution, which means that if you were to use a deep learning model to "rotate" images, you can only do it on a specific class of images (e.g. faces, cats, etc...).

If that's your goal, generative models are the way to go:

# Autoencoders

You can train an Autoencoders to slightly change the angle. Autoencoders are a special type of neural network that are trained to output the same input you feed into them with a few imposed restriction to prevent it from learning a trivial identity function. In your case, you could use a variation of a de-noising autoencoder. A de-noising autoencoder, as the name suggests, generates the same input image minus an artificial stochastic noise. The way this is achieved is by feeding a corrupted version of the image and then evaluating the loss on the non corrupted version.

### How can this be adapted in your case?

In your case, you could feed the original images into your autoencoder and evaluate it based on the rotated images. This will result in your autoencoder effectively learning the inner distribution that generates the images in order to generate a slightly "rotated" version of it. Foir more info on de-noising autoencoders, see the original paper.

# Generative adversarial networks

For a more sophisticated approach, you can use generative adversarial networks. GANs are relatively harder to manipulate, but usually perform better than other generative models when it comes to images.

### How can this be adapted in your case?

In general, GANs generate images from noise. However, in your case, you can use the original (non rotated) images as input for the generator. The generator can be a convolutional autoencoder for example. And the "real images" dataset will be your rotated images. This way, your model will learn to generate slightly rotated images by being fed an equivalent of noise in traditional GANs which in your case will be the original images. For more info on GANs, I suggest this and the original paper.
I should point out that one of the flaws of GANs is distorted perspectives so it's probably a shot in the dark; however, I think it won't be be a problem here because you would be using real images as input instead of complete noise.

# Related work

Now, as far as the literature goes, I don't think this has been done before except for this one (kinda) for faces representation learning. The only way this is similar to yours is that you can modify the implementation to only generate the faces it has been fed from different perspectives instead of an average of everything it has learned.

• This answer is great for me, thank you. I labeled deep learning only bc other labels were not also available. Other approaches are relevant just as well for me at this stage. – proton Mar 21 '18 at 15:47
• Convolution does not contract and extend surfaces, so that can't work, and pixel morphing is not mentioned. The noise that needs to be removed is not identified. Discussion of training data possibilities is not mentioned. – FauChristian Mar 21 '18 at 18:35
• @FauChristian I'm in a preliminary stage at my work. I want to know what has been done, or at least what to look for – proton Mar 21 '18 at 21:15
• @proton, Exactly. The beginning of an investigation is the time where a project is most sensitive to taking wrong turns, and others may read any given post and be steered down less than optimal paths. SO is set up so that one can down vote misleading responses and provide an explanation for both authors (to modify or improve the response) and readers who are seeking the right direction for work. ... Doing a web search or a scholarly article search for "three-dimensional feature extraction" will give you some sense of work done and options for initial direction. – FauChristian Mar 22 '18 at 15:18
• @FauChristian The reason why I suggested autoencoders is because we are using them for image enhancement and although we weren't expecting it, a few of the generated results had severely distorted perspective. Perhaps, the reason for this is the hybrid architecture and the semi supervised nature of training during which, several shots of the same scene has been provided. I can't provide further technical details as this is still a work in progress, but I can definitely tweak this architecture to produce something similar to what OP asked. – Achraf Oussidi Mar 22 '18 at 17:55

— Stereoscopic Synthesis —

The generation of an image that would likely appear in the right eye of a head from which you already have an image from the left eye (or vice versa) is too complex to expect simple convolution (linear matrix transformation) to achieve a reasonable result.

You are correct that rotation is not the correct description, simply because it is ambiguous. What is rotating? The best description is the synthesis of a stereoscopic image from a single eyed/camera one.

Although deep learning is an approach well suggested, it is a very general term into which a number of concepts, books, research projects, and software components fit. I agree with other answers that indicate one could easily find the target objective missed upon repeated attempts at finding a working solution. Shots in the dark are likely to waste time and effort.

For example, an auto-encoder may not work well because the modelling of depth may not be a feature extracted without having a host of stereoscopic image pairs of similar scenes to which automated feature extraction could be accomplished.

Should feature extraction be possible, it is not noise that needs to be removed, but pixelization and optical distortion that needs to be characterized, so that surfaces revealed by the shift in position could later be imbued with the same contour, focal blur, reflective properties, and edge continuation as the adjacent surfaces of the same objects when pixels corresponding to newly revealed surfaces are generated.

For greatest image authenticity, another noise profile to profile and imbue to generated pixels is the capture device noise profile.

— Formal Problem Restatement —

To narrow machine learning approaches so that we can take a shot in the light, let's consider the model of a three dimensional scene exposed to light sources with two cameras adjacent to one another. Let's consider the input, the output, and the internal architecture required to produce a fairly reliable and accurate second image more formally.

We have image pixel matrix I1 that represents light arriving at camera c1 containing a rectangular image capture surface in an x-z plane upon which a lens of effective focal length l1 and aperture a1 is focused over scene S over a time window starting at s1 and ending at e1. Some point is at the origin of a Cartesian coordinate axis, both camera c1 and another camera c2 point such that the origin is centered in the image capture and the point of lens focus. The three dimensional coordinates of c1 and c2 are known.

You wish to predict the second image I2 arriving at camera c2.

Let's assume, for simplicity, that l1 = l2, that a1 = a2, and that the scene is motionless so that time is not critical in the model. Let's also assume that the y coordinates and the image capture duration (e minus s) are the same for both cameras c1 and c2.

— Solution Architecture —

For this simplified case and assuming the object space is not an abyss containing only one object, the process architecture of the solution is the following. Each --> symbol is a sub-process. The horizontal and vertical positional difference between c1 and c2 is { x, z }.

{ I1, l, a, y, x, z }
--> { I1, l, a, y, { S1 ... Sn }}
--> { I1, l, a, y, { S1 ... Sn }, { E1 ... En }}
--> { I2 }


The first sub-process is a feature extraction, where the features are the three dimensional surfaces visible in the two dimensional image I1. This is a questionable extraction because no y information in the scene is available and there is no mention of y-labeled training data is in the problem statement.

The second sub-process is the extension of features extracted to provide needed surface representation for I2.

The last process is rendering I2, potentially using morphing pixels in I1 and filling transparent sections remaining using E1 through En and knowledge of the contour, reflective properties, edge continuation, and capture device noise profile from feature extraction.

— Practicality of Learning About Scenes —

The effectiveness of any deep learning architecture could benefit from the above understanding of vision and the comprehension of scenes in DNA based life. The problem of automated feature extraction is complicated because the data is unlabeled with y information as stated before.

Learning visual comprehension of arbitrary scenes by DNA based life is assisted by the fact that motion occurs and interaction with physical objects and liquid viscosity provides a vastly greater number of dimensions to the input data.