0
$\begingroup$

I'm currently coding a GAN on the dataset MNIST. I'm using the following code to transform my data:

# MNIST Dataset
transform = transforms.Compose([
    transforms.ToTensor(),
    transforms.Normalize((0.1307,), (0.3081,))])
# the output of torchvision datasets are PILImage images of range [0, 1] and we 
# want data that is centered around 0 with a std of 1 (0.1307 and 0.3081 are the estimated values of the MNIST mean & std)

I will have data centered around 0 with a standard deviation of 1 ((0.1307,), (0.3081,) are the estimated mean & standard deviation on the training dataset)). So that means that there will occasionnaly be values above 1 and below -1 in the real data.

Now, my generator ends up with a tanh activation function:

return torch.tanh(self.fc4(x))  # outputs in[-1; 1]

That means there will never be values above 1 and below -1 in the faked data.

Is it possible that the discriminator picks on this phenomenon? This seems to be the case as its loss goes to 0 really quickly. However this also could also be the case that the discriminator is just "too strong" as I've seen numerous times on stackexchange posts. I however never seen nobody talking about the fact that it could pick on the fact that there are outliers in the "real data" and only pixels between -1 and 1 in the "fake data".

EDIT: the entirety of my code can be found here: https://github.com/JQuentinMendoza2008/PyTorch_GAN_for_MNIST_Dataset

Any suggestion is welcomed.

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

Is it possible that the discriminator picks on this phenomenon?

It's not just possible, it's a certainty. The generator should learn to translate an input distribution A to an output distribution B, if distribution B has range $(-\infty, \infty)$ and your generator can output only values in range $(-1, 1)$ the translation between the two simply can't happen.

Move to another normalization, like min max, and replace tanh with sigmoid as final activation function. If your GAN will still have trouble converging, only then you might start investigating other components like the discriminator depth.

Improve this answer
$\endgroup$
9
  • 1
    $\begingroup$ you could use no activation at all if you want to output any real value. But I would suggest against it, convergence is still possible but for sure it will become really slow and unstable. $\endgroup$
    – Edoardo Guerriero
    Mar 29, 2022 at 8:02
  • 1
    $\begingroup$ the min max normalization is actually probably the simplest one among normalizations. You just subtract the minimum value of the whole image to all pixels, and divide them by (max value of whole image - min value of whole image). The first step push down the min value to zero, the second step push down the max value to 1, so your guarantee to have values within the range $(0,1)$ (hence the necessity of using sigmoid activation with this normalization) $\endgroup$
    – Edoardo Guerriero
    Mar 29, 2022 at 8:05
  • 1
    $\begingroup$ @EdoardoGuerriero do you have anything where you base this on (that you advice against linear activations due to stability issues)? I'm being curious, because I've had great results with using linear output activations where my output distribution was not constrained. $\endgroup$
    – Chillston
    Mar 29, 2022 at 9:22
  • 1
    $\begingroup$ @EdoardoGuerriero Yeh of course, I just wanted to know why people were doing it in general. I'm gonna try to not use any activation at all and to use a sigmoid with a min max normalization (but I think that's already been done in the Dataset I'm using, I'd have to check) $\endgroup$
    – FluidMechanics Potential Flows
    Mar 29, 2022 at 10:05
  • 1
    $\begingroup$ @Chillston I got that experience while working on satellite data super resolution. Admittedly the data where codified in uint16, so the range to learn is quite huge compared to uint8. We tried to not normalized cause we didn't wanted to affect the raw sensor data but we gave up pretty quickly on this potential risk and after normalizing the training was much more easy. Still, the mnist data is quite a simple one so you might be right that this risk is probably not noticeable. $\endgroup$
    – Edoardo Guerriero
    Mar 29, 2022 at 10:08

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .