0
$\begingroup$

I have built the following function which takes as input some data and runs a VAE on them:

def VAE(data, original_dim, latent_dim, test_size, epochs):
    
    x_train, x_test = train_test_split(data, test_size=test_size, random_state=42)
    
    # Define the VAE architecture
    #Encoder
    encoder_inputs = tf.keras.Input(shape=(original_dim,))
    x = layers.Dense(64, activation='relu')(encoder_inputs)
    x = layers.Dense(32, activation='relu')(x)
    x = layers.Dense(8, activation='relu')(x)

    #--- Custom Latent Space Layer
    z_mean = layers.Dense(units=latent_dim, name='Z-Mean', activation='linear')(x)
    z_log_sigma = layers.Dense(units=latent_dim, name='Z-Log-Sigma', activation='linear')(x)
    z = layers.Lambda(sampling, name='Z-Sampling-Layer')([z_mean, z_log_sigma, latent_dim]) # Z sampling layer

    # Instantiate the encoder
    encoder = tf.keras.Model(encoder_inputs, [z_mean, z_log_sigma, z], name='encoder')

    #Decoder
    latent_inputs = tf.keras.Input(shape=(latent_dim,))
    x = layers.Dense(8, activation='relu')(latent_inputs)
    x = layers.Dense(32, activation='relu')(x)
    x = layers.Dense(64, activation='relu')(x)
    decoder_outputs = layers.Dense(1, activation='relu')(x)

    # Instantiate the decoder
    decoder = tf.keras.Model(latent_inputs, decoder_outputs, name='decoder')

    # Define outputs from a VAE model by specifying how the encoder-decoder models are linked
    # Instantiate a VAE model
    vae = tf.keras.Model(inputs=encoder_inputs, outputs=decoder(encoder(encoder_inputs)[2]), name='vae')
    
    # Reconstruction loss compares inputs and outputs and tries to minimise the difference
    r_loss = original_dim * tf.keras.losses.mse(encoder_inputs, decoder(encoder(encoder_inputs)[2]))  # use MSE

    # KL divergence loss compares the encoded latent distribution Z with standard Normal distribution and penalizes if it's too different
    kl_loss = -0.5 * K.mean(1 + z_log_sigma - K.square(z_mean) - K.exp(z_log_sigma), axis=-1)

    #VAE total loss
    vae_loss = K.mean(r_loss + kl_loss)

    # Add loss to the model and compile it
    vae.add_loss(vae_loss)
    vae.compile(optimizer='adam')
    
    # train the model
    vae.fit(x_train, x_train, epochs=epochs, validation_data=(x_test, x_test))

where

def sampling(args):
    
    z_mean, z_log_sigma, latent_dim = args
    epsilon = K.random_normal(shape=(K.shape(z_mean)[0], latent_dim), mean=0., stddev=1., seed=42)
    return z_mean + K.exp(z_log_sigma) * epsilon

My question is, if I want to generate new data, by using the above VAE how can I achieve that ?

If I want to sample 100 new data, should I use this

   latent_mean = tf.math.reduce_mean(encoder(x_train)[2], axis=0) 
   latent_std = tf.math.reduce_std(encoder(x_train)[2], axis=0)
   latent_sample = tf.random.normal(shape=(100, latent_dim), mean=latent_mean, 
                                    stddev=latent_std)
   generated_data = decoder(latent_sample)

or

   latent_mean = tf.math.reduce_mean(encoder(x_train)[0], axis=0) 
   latent_std = tf.math.reduce_mean(tf.math.exp(encoder(x_train))[1], axis=0)
   latent_sample = tf.random.normal(shape=(100, latent_dim), mean=latent_mean, 
                                    stddev=latent_std)
   generated_data = decoder(latent_sample)

?

Basically should I infer z_mean and z_log_sigma from the z or should I use z_mean and z_log_sigma directly ? What is the difference ?

Moreover, I have seen that everytime tf.random.normal is used to generate new data from the latent space. Why not use lognormal for instance ? Is it because of the KL divergence ?

The end-goal is the distribution of the generated_data to be as close as possible to the distribution of the original data.

$\endgroup$
5
  • $\begingroup$ Why would you use log-normal? that's not the distribution you've learnt the parameters for, as far as I am aware. $\endgroup$
    – David
    Commented Jan 23, 2023 at 10:42
  • $\begingroup$ @DavidIreland As far as I understand, you can generate anything, since whatever you generate it will later pass from the decoder, and passing from the decoder does not necessarily mean that the distribution from which you sampled from will be preserved, right ? Or am I missing something ? $\endgroup$
    – quant
    Commented Jan 23, 2023 at 10:46
  • 1
    $\begingroup$ Once you have trained the VAE, you can discard the encoder and feed data directly in. You seem to know this however in your case it means that the z_mean and z_log_variance can be discorded since you are only generating a sample latent vector. This should be centered on zero, so you can simply generate vectors based on a normal curve. $\endgroup$ Commented Jan 23, 2023 at 11:41
  • $\begingroup$ @DavidHoelzer what is the purpose then of the z_mean and z_log_variance ? $\endgroup$
    – quant
    Commented Jan 23, 2023 at 14:06
  • 1
    $\begingroup$ The expectation is that those two layers model the distribution, which you are sampling with the reparameterization trick using your z layer. $\endgroup$ Commented Jan 25, 2023 at 2:29

1 Answer 1

2
$\begingroup$

Normally in synthetic data generation using VAEs, you do not use the encoder anymore upon inference. Instead, generating samples after training is usually done by simply sampling random noise from 'a distribution' (i.e. a normal distribution) in the dimensions of the latent space and decoding it using the decoder. Hence, generating samples would be usually done as follows:

noise = tf.random.normal(shape=(100, latent_dim))
generated_samples = decoder(noise)

if you use training samples in generating your new data, it is not representative of how good your generative model is as it then bases the output on the training data.

Answer However, if you do want to sample using the training data to get the means and standard deviations, then your second method is a more appropriate method than your first method. You need the standard deviation, and you only get the log(std) from the model. Taking the exponent gives you the standard deviation.

$\endgroup$
7
  • $\begingroup$ Why is the second method more appropriate ? Could you please elaborate $\endgroup$
    – quant
    Commented Jan 23, 2023 at 11:00
  • $\begingroup$ Because you need to convert your log_sigma which is equal in function to log(std), to sigma (equal to std). You can do so by taking the exponent. $\endgroup$ Commented Jan 23, 2023 at 11:15
  • $\begingroup$ In the first method there is no need to take the exponent because it calculates directly the std from $z$, so in terms of "rescaling" these methods are equivalent. The question refers to "should I use the z_log_sigma directly (after rescaling it using the exp ofc) or should I infer the std from $z$ ? $\endgroup$
    – quant
    Commented Jan 23, 2023 at 11:26
  • 1
    $\begingroup$ My bad, I misread your code. The more i look at it the more confused i get too be honest. So first a question: are you trying to generate only one class? or multiple classes? Having the same mean and standard deviation for sampling noise for all classes is not logical. For sampling one class, you would have to use a different mean and standard deviation than for sampling another class $\endgroup$ Commented Jan 23, 2023 at 12:57
  • $\begingroup$ I have one class $\endgroup$
    – quant
    Commented Jan 23, 2023 at 14:11

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .