GAN model predictions before training is predictable

Question

I have a dataset of 3000 8x8 images, and I would like to train a GAN for an image generation purpose.

I am planning to start with a simple GAN model and see if it overfits. Before training, I try to do a comparison of the discriminator model prediction using real image input against the whole GAN model prediction using random seed input. My thought process is that since this model is not trained yet, the output for real images and fake images by the discriminator should not be predictable.

However, the discriminator model prediction using real image input always returns a value very close to 1.0, and the whole GAN model prediction using random seed input always returns a value near 0.5 with a small deviation. I suspect that during training, the model would simply pull the 0.5 value near 0.0 and would never actually learn from the dataset.

I try to increase the training parameters and different initializers, but the output is still the same.

By ruling out the possibility a bad dataset, what could be the reason for this situation?

This is some sneak peek of the generator and discriminator model building: https://pastebin.com/ehMDP7k6

Answer 1

I took a look at your model. It seems you have incorrect architecture. The Conv2D layers in your D should have following params: (n_filters, kernel=3, padding='same'), where n_filters is the number of filters and it usually should be doubled as per DCGAN architecture. You can also use strides but since your images are small it won't make any sense.

The D network also should not include UpSampling2D layer. This layer can be used in G network instead of Conv2DTranspose.

Your images should be normalized to the range of [-1, 1] and the activation function of G should be tanh.

You also included m.add(keras.layers.Reshape((8, 8, 3))) to your G network which is the final size of your data. Since you have the final size in the first layers you don't have to include upsampling Conv2DTranspose layers.

Your final model should look like

def make_discriminator_model():
    model = tf.keras.Sequential()
    model.add(layers.Conv2D(8, kernel=3, strides=1, padding='same',
                                     input_shape=[8, 8, 1]))
    model.add(layers.LeakyReLU())
    model.add(layers.Dropout(0.3))

    model.add(layers.Conv2D(16, kernel=3, strides=1, padding='same'))
    model.add(layers.LeakyReLU())
    model.add(layers.Dropout(0.3))

    model.add(layers.Flatten())
    model.add(layers.Dense(1))

    return model

def make_generator_model():
    model = tf.keras.Sequential()
    model.add(layers.Dense(8*8*16, use_bias=False, input_shape=(100,)))
    model.add(layers.BatchNormalization())
    model.add(layers.LeakyReLU())

    model.add(layers.Reshape((8, 8, 16)))

    model.add(layers.Conv2D(8, kernel=3, strides=1, padding='same'))
    model.add(layers.BatchNormalization())
    model.add(layers.LeakyReLU())

    model.add(layers.Conv2D(1, kernel=3, strides=1, padding='same', activation='tanh'))

    return model

Note that I did not test the model, so you should adapt it to the shape of your data.

I encourage you to follow this tutorial.

Update regarding overfitting

GANs do not have overfitting problem in the classical sense. Instead, vanilla GAN have other problems like vanishing gradient. It happens when the D becomes overconfident regarding fake samples. In that case, D stops providing useful information to the G, since its error becomes 0. To find out if training fell into this problem, you should plot D and G loss. It will look as follows:

Another problem is the mode collapse. In that case, the G tricks the D by producing only one type of samples which looks realistic but G won't represent the real data distribution. Therefore, the generated samples will be homogeneous.

For more information, see Improved Techniques for Training GANs and Wasserstein GAN.