Generative networks in generative network arrangements do not learn about input images directly. Their input during training is feedback from the discriminative network.
The Theory in Summary
The seminal paper, Generative Adversarial Networks, Goodfellow, Pouget-Abadie, Mirza, Xu, Warde-Farley, Ozair, Courville, and Bengio, June 2014, states, "We propose a new framework for estimating generative models via an adversarial process, in which we simultaneously train two models ..." The two models are defined as MLPs (multilayer perceptrons) in the paper.
- Generative model, G
- Discriminative model, D
These two models are interconnected such that they form a negative feedback loop.
- G is trained to capture the feature relational distribution of a set of examples and generate new examples based on that relational distribution well enough to fool D.
- D is trained to differentiate G's mocks from the set of externally procured examples.
If G were to receive input images, their presence would merely frustrate network training, in that the goal of the training would likely be inadequately defined. The objective of the convergence of G, stated above, is not the learning of how to process the images to produce some other form of output. Its objective in the generative approach is to learn how to generate well, an entirely incompatible objective with either image evaluation or image processing.
Additionally, one image is not nearly enough. There must be a sufficiently large set of example images for training to converge at all and then many more to expect the convergence to be both accurate and reliable. The PAC (probably approximately correct) learning analysis framework may be helpful to determine how many examples are needed for a specific case.
The discriminator is essential to the generative approach because the feedback loop referenced above is essential to the convergence mechanism. The bidirectional interdependence between G and D is what allows a balanced approach toward accuracy in the feature relational distribution. That accuracy facilitates the human perception that the generated images fit adequately within a cognitive class.
Response to Comments
The attempt to use a generative approach, "To paint in gaps in an image," is reasonable. In such a case, using Goodfellow's nomenclature, G would be generating the missing pixels and D would be trying to discriminate between G's gap filling and the pixels that were in the scenes in the regions of gaps prior to their introduction.
There are two additional requirements in the scenario of filling in pixels.
- G must be strongly incentivized against allowing a large gradient between the generated pixels and the adjacent non-gap pixels, unless that gradient is appropriate to the scene, as in the case of an object edge, or a change in reflectivity, such as a surface abrasion or the edge of a spray painted shape.
- D must train using the entire image, which means the examples should be images without gaps, the gaps must be introduced in a way that matches the expected distribution of features of gaps that may be encountered later, and the result of G must be superimposed over the full image to produce the input arising from G and discriminated from the original by D.
It is recommended to begin with a standard GAN design, create a test for it (in TDD fashion), implement it, experiment with it, and otherwise become familiar with it and the mathematics involved. Most important to understand is how the balance between G's convergence and D's convergence is obtained in the loss (a.k.a. error or disparity) functions for each, and what concepts of feedback are employed using those functions.
- Does your point about input images frustrating network training apply to this kind of problem, or just to GANNs that generate from scratch?
It applies to both.
- Would I have to have the generator compare the original image with the generated image and pick which one it thinks is better in order to deal with the "adequately defined" issue?
D compares, not G. That is the delegated arrangement. It is not that other arrangements cannot work. They may. But Goodfellow and the others understood what worked in artificial networks long before they discovered a new approach, and they likely worked out the math of that approach and diagrammed it, perhaps on a white board, long before they typed a single line of code.