How important is it that the generator of a generative adversarial network doesn't take in information about input classes?

Question

I'm building a generative adversarial network that generates images based on an input image. From the literature I've read on GANs, it seems that the generator takes in a random variable and uses it to generate an image.

If I were to have the generator receive an input image, would it no longer be a GAN? Would the discriminator be extraneous?

Answer 1

Short Answer

Generative networks in generative network arrangements do not learn about input images directly. Their input during training is feedback from the discriminative network.

The Theory in Summary

The seminal paper, Generative Adversarial Networks, Goodfellow, Pouget-Abadie, Mirza, Xu, Warde-Farley, Ozair, Courville, and Bengio, June 2014, states, "We propose a new framework for estimating generative models via an adversarial process, in which we simultaneously train two models ..." The two models are defined as MLPs (multilayer perceptrons) in the paper.

• Generative model, G
• Discriminative model, D

These two models are interconnected such that they form a negative feedback loop.

• G is trained to capture the feature relational distribution of a set of examples and generate new examples based on that relational distribution well enough to fool D.
• D is trained to differentiate G's mocks from the set of externally procured examples.

Applying Concepts

If G were to receive input images, their presence would merely frustrate network training, in that the goal of the training would likely be inadequately defined. The objective of the convergence of G, stated above, is not the learning of how to process the images to produce some other form of output. Its objective in the generative approach is to learn how to generate well, an entirely incompatible objective with either image evaluation or image processing.

Additional Information

Additionally, one image is not nearly enough. There must be a sufficiently large set of example images for training to converge at all and then many more to expect the convergence to be both accurate and reliable. The PAC (probably approximately correct) learning analysis framework may be helpful to determine how many examples are needed for a specific case.

Essential Discriminator

The discriminator is essential to the generative approach because the feedback loop referenced above is essential to the convergence mechanism. The bidirectional interdependence between G and D is what allows a balanced approach toward accuracy in the feature relational distribution. That accuracy facilitates the human perception that the generated images fit adequately within a cognitive class.

.

---

Response to Comments

The attempt to use a generative approach, "To paint in gaps in an image," is reasonable. In such a case, using Goodfellow's nomenclature, G would be generating the missing pixels and D would be trying to discriminate between G's gap filling and the pixels that were in the scenes in the regions of gaps prior to their introduction.

There are two additional requirements in the scenario of filling in pixels.

• G must be strongly incentivized against allowing a large gradient between the generated pixels and the adjacent non-gap pixels, unless that gradient is appropriate to the scene, as in the case of an object edge, or a change in reflectivity, such as a surface abrasion or the edge of a spray painted shape.
• D must train using the entire image, which means the examples should be images without gaps, the gaps must be introduced in a way that matches the expected distribution of features of gaps that may be encountered later, and the result of G must be superimposed over the full image to produce the input arising from G and discriminated from the original by D.

It is recommended to begin with a standard GAN design, create a test for it (in TDD fashion), implement it, experiment with it, and otherwise become familiar with it and the mathematics involved. Most important to understand is how the balance between G's convergence and D's convergence is obtained in the loss (a.k.a. error or disparity) functions for each, and what concepts of feedback are employed using those functions.

• Does your point about input images frustrating network training apply to this kind of problem, or just to GANNs that generate from scratch?

It applies to both.

• Would I have to have the generator compare the original image with the generated image and pick which one it thinks is better in order to deal with the "adequately defined" issue?

D compares, not G. That is the delegated arrangement. It is not that other arrangements cannot work. They may. But Goodfellow and the others understood what worked in artificial networks long before they discovered a new approach, and they likely worked out the math of that approach and diagrammed it, perhaps on a white board, long before they typed a single line of code.

Answer 2

If you're building a straight "vanilla" generative adversarial network, it's best to understand the network as a statistical engine: You are training the generator on samples of a statistical distribution. (And you're training the discriminator to distinguish between "ground truth" images, and images from that generator.)

Once you replace the input noise with another image... well. Strictly speaking, it is probably still a generative adversarial network, if you're still doing everything else the same. It is still a generator and a discriminator, acting in an adversarial fashion.

But you've radically altered the input distribution, so there is a good chance that you're no longer accomplishing what you want to accomplish unless you're being very careful and clever.

That said, there are GAN variants which do take images rather than noise as inputs. See the wonderful paper on CycleGANs by Zhu, et al, along with a substantial body of followup literature. And note that CycleGANs use not one, but two discriminators, so even here the discriminator is necessary.