Feasibility of generating large images with a convnet

Question

I've spent the past couple of months learning about neural networks, and am thinking of projects that would be fun to work on to cement my understanding of this tech.

One thing that came to mind last night is a system that takes an image of a movie poster and predicts the genre of the movie. I think I have a good understanding of what'd be required to do this (put together a dataset, augment it, download a convnet trained on imagenet, finetune it on my dataset, and go from there).

I also thought that it would be pretty cool to run the system backwards at the end, so that I could put in e.g. a genre like 'horror' and have the system generate a horror movie poster. I expect that it will be very bad at this because I'm not a team of expert researchers, but I think I could have some fun hacking on it even if it only ever generated incomprehensible results.

Here's what I'm having trouble understanding: on the one hand, all the convnets whose architecture I've seen described seem to rely on being given very small, square input images (on the order of 220px by 220px iirc), and movie posters are rectangular, and a generated poster would have to be of a larger size in order for a human to make any sense of it. I've seen several examples of papers where researchers use convnets to generate images, e.g. the adversarial system that generates pictures of birds and flowers, and a system that generates the next few frames of video when given a feed of a camera sweeping across the interior of a room, but all of those generated images seemed to be of the small square size I've been describing.

On the other hand, I've seen lots of "deep dream" images over the past year or so that have been generated by convnets and are of a much larger size than ~220px by ~220px.

Here's my question: is it possible for me to build the system I describe, which takes a movie genre and outputs a movie poster of a size like e.g. 400px by 600px? [I'm not asking about whether or not the resulting poster would be any good - I'm curious about whether or not it's possible to use a convnet to generate an image of that size.]

If it is possible, how is it possible, given that these systems seem to expect small, square input images?

Answer 1

The way you would recognize images (image -> genre) would be very different from the other way around (genre -> image). For the former you are correct on what would be involved.

For the latter, if you want large images then GANs are indeed the way to go. Currently the largest images we can generate are on the order of 220 x 220 pixels, mostly due to memory constraints on the GPU. There is no fundamental problem with having rectangular images, it just so happens that we use squares. You would be able to use identical architectures to train on rectangular data as well.

The reason that some generated images you see (e.g. from DeepDream or NeuralStyle) are larger is that it's not a GAN. Both of these are not generative models in a standard sense, even if they do technically "generate" images. Instead, they merely modify an existing image by running backprop on a specifically designed loss function.

TLDR: your movie genre recognition idea is sound. For generation, have a look at things like this (https://github.com/Newmu/dcgan_code), where Alec generated album covers instead of posters. If you have enough data it will do something half sensible.

Answer 2

There is a fundamental difference between Generative Models and Discriminative Models, to simplify it we could say that

• given $S$ Semantic Space (low dimensionality) and

• given $A$ Appearance Space (high dimensionality, e.g. space of $W \times H$ images)

the Discriminator learns the function $f(A) \rightarrow S$ and the Generator learns the function $g(S) \rightarrow A$

According to this formalism we can say that $g \neq f^{-1}$ and a simple reason for this is $f$ is not invertible because it is not injective, in fact 2 different horror movie posters will be assigned the same label “horror movie” (assuming the network works well :) )

So you have to find another way to define a mapping from $S$ to $A$ and this kind of job is related to the Generative Models

Certainly GAN are state of the art in image generation but technically they are not the only choice, e.g. you could use a VAE as the Decoder Subnetwork is a generative model

However from a practical point of view do not expected VAE results would that good: you’ll probably get blurred stuff like this one

so go for GAN but beware that training them properly is definitely not easy