GAN generates exactly the same Images cross a batch only because of seeds distribution, Why?

问题

I have trained a GAN to reproduce CIFAR10 like images. Initially I notice all images cross one batch produced by the generator look always the same, like the picture below:

After hours of debugging and comparison to the tutorial which is a great learning source for beginners (https://machinelearningmastery.com/how-to-develop-a-generative-adversarial-network-for-a-cifar-10-small-object-photographs-from-scratch/), I just add only one letter on my original code and the generated images start looking normal (everyone starts looking different from each other cross one batch), like the picture below:

The magical one character change on the code is to make the following change:

Changing from:

def generate_latent_points(self, n_samples):
        return np.random.rand(n_samples, self.latent_dim)

to:

def generate_latent_points(self, n_samples):
        return np.random.randn(n_samples, self.latent_dim)

Hope this very subtle detail could help those who spend hours scratching off their head for their GAN training process.

np.random.rand gives uniform distribution over [0, 1)

np.random.randn gives a univariate “normal” (Gaussian) distribution of mean 0 and variance 1

So why the difference in the distribution of the latent seeds for generator would behave so differently?

回答1:

There might be few possible reasons that I can think of:

1. The distribution differences of these two are two folded: 1. rand gives only positive value while randn gives both negative and positive values around 0; 2. rand gives larger value in magnitude than randn for obvious reason. The much larger magnitude might lead to unstable learning because dL/dw (which is in proportional to x) will be larger than that in the case of randn.

2. The unsuccessful learning of generator puts no challenge on discriminator to distinguish real and fake images, so the discriminator is not learning anything new. (This is my observation on the loss and acc of the discriminator during training process)

来源：https://stackoverflow.com/questions/65122089/gan-generates-exactly-the-same-images-cross-a-batch-only-because-of-seeds-distri