0
$\begingroup$

I am trying to addapt the Keras variational autoencoder (VAE) here from a 2-D input/output (matrix of a picture) to a 1-D input/output (just a vector). I thought this would be a fearly easy task, but somehting verry weard is going on with the learning which leads me to believe that I have missed something.

This is my current modified code.

###Imports 
import numpy as np  
import tensorflow as tf  
from tensorflow import keras  
from tensorflow.keras import layers   
from data_prep import * #local python file for data prep get_data 
print("Imports sucsessfully.")


### Create a sampling layer 
class Sampling(layers.Layer):
    """Uses (z_mean, z_log_var) to sample z, the vector encoding the data."""

    def call(self, inputs):
        z_mean, z_log_var = inputs
        batch = tf.shape(z_mean)[0]
        dim = tf.shape(z_mean)[1]
        epsilon = tf.keras.backend.random_normal(shape=(batch, dim))
        return z_mean + tf.exp(0.5 * z_log_var) * epsilon


### Define the VAE as a Model with a custom Train_step 
class VAE(keras.Model):
    def __init__(self, encoder, decoder, **kwargs):
        super(VAE, self).__init__(**kwargs)
        self.encoder = encoder
        self.decoder = decoder
        self.total_loss_tracker = keras.metrics.Mean(name="total_loss")
        self.reconstruction_loss_tracker = keras.metrics.Mean(
            name="reconstruction_loss"
        )
        self.kl_loss_tracker = keras.metrics.Mean(name="kl_loss")

    @property
    def metrics(self):
        return [
            self.total_loss_tracker,
            self.reconstruction_loss_tracker,
            self.kl_loss_tracker,
        ]

    def train_step(self, data):
        with tf.GradientTape() as tape:
            z_mean, z_log_var, z = self.encoder(data)
            reconstruction = self.decoder(z)
            reconstruction_loss = tf.reduce_sum(keras.losses.binary_crossentropy(data, reconstruction))
            #reconstruction_loss = tf.reduce_mean(tf.reduce_sum(keras.losses.binary_crossentropy(data, reconstruction)))
            #reconstruction_loss = keras.losses.binary_crossentropy(data, reconstruction)#, from_logit=True))
            kl_loss = -0.5 * (1 + z_log_var - tf.square(z_mean) - tf.exp(z_log_var))
            kl_loss = keras.backend.sum(kl_loss, axis=1)
            total_loss = reconstruction_loss + kl_loss
        grads = tape.gradient(total_loss, self.trainable_weights)
        self.optimizer.apply_gradients(zip(grads, self.trainable_weights))
        self.total_loss_tracker.update_state(total_loss)
        self.reconstruction_loss_tracker.update_state(reconstruction_loss)
        self.kl_loss_tracker.update_state(kl_loss)
        return {
            "loss": self.total_loss_tracker.result(),
            "reconstruction_loss": self.reconstruction_loss_tracker.result(),
            "kl_loss": self.kl_loss_tracker.result(),
        }


### Build the Model 
input_dim = 73 
latent_dim = 10

### building the Encoder 
input_shape = keras.Input(shape=(input_dim,)) 
x = layers.Dense(64, activation='relu')(input_shape) 
x = layers.Dense(32, activation='relu')(x) 
x = layers.Dense(16, activation="relu")(x) 
z_mean = layers.Dense(latent_dim, name="z_mean")(x) 
z_log_var = layers.Dense(latent_dim, name="z_log_var")(x) 
z = Sampling()([z_mean, z_log_var]) 
encoder = keras.Model(input_shape, [z_mean, z_log_var, z], name="encoder") 
encoder.summary()


### building the decoder 
latent_inputs = keras.Input(shape=(latent_dim,)) 
x = layers.Dense(16, activation="relu")(latent_inputs) 
x = layers.Dense(32,activation='relu')(x) 
x = layers.Dense(64,activation='relu')(x) 
decoder_outputs = layers.Dense(input_dim, activation="sigmoid")(x) 
decoder = keras.Model(latent_inputs, decoder_outputs, name="decoder") 
decoder.summary()


### Training the VAE 
use_percentage = 0.1 
folder_path = r"<local_link>"

(x_train, _), (x_test, _) = data_prep.get_data(folder_path, use_percentage) 
all_data = np.concatenate([x_train, x_test], axis=0) 
print(all_data.shape)


vae = VAE(encoder, decoder) 
vae.compile(optimizer=keras.optimizers.Adam(
    learning_rate=1e-05,
    beta_1=0.9,
    beta_2=0.999,
    epsilon=1e-07,
    amsgrad=False,
    name="Adam"
    )) 
vae.fit(all_data, epochs=20, batch_size=128)

The only things I changed are the definition of the encoder and decoder, the loading of my own data and the reconstruction_loss of the custom training_step in the VAE class. The old reconstruction_loss for the 2-D Keras example is commented out underneath my new definition. Everything else in the VAE and the Sampling class is the same as in the keras example.

This model is doing something, but it looks like the loss function is not regulated. This is the output from the code:

2021-06-01 13:05:32.514186: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cudart64_110.dll
Flag
Imports sucsessfully.
2021-06-01 13:05:39.238654: I tensorflow/compiler/jit/xla_cpu_device.cc:41] Not creating XLA devices, tf_xla_enable_xla_devices not set
2021-06-01 13:05:39.254880: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library nvcuda.dll
2021-06-01 13:05:39.312607: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: 
pciBusID: 0000:01:00.0 name: NVIDIA GeForce RTX 2070 SUPER computeCapability: 7.5
coreClock: 1.77GHz coreCount: 40 deviceMemorySize: 8.00GiB deviceMemoryBandwidth: 417.29GiB/s
2021-06-01 13:05:39.321695: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cudart64_110.dll
2021-06-01 13:05:39.397924: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cublas64_11.dll  
2021-06-01 13:05:39.402967: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cublasLt64_11.dll
2021-06-01 13:05:39.447141: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cufft64_10.dll
2021-06-01 13:05:39.465101: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library curand64_10.dll
2021-06-01 13:05:39.583036: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cusolver64_10.dll
2021-06-01 13:05:39.611093: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cusparse64_11.dll
2021-06-01 13:05:39.619346: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cudnn64_8.dll
2021-06-01 13:05:39.624324: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2021-06-01 13:05:39.632668: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2021-06-01 13:05:39.645881: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: 
pciBusID: 0000:01:00.0 name: NVIDIA GeForce RTX 2070 SUPER computeCapability: 7.5
coreClock: 1.77GHz coreCount: 40 deviceMemorySize: 8.00GiB deviceMemoryBandwidth: 417.29GiB/s
2021-06-01 13:05:39.658036: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cudart64_110.dll
2021-06-01 13:05:39.662552: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cublas64_11.dll
2021-06-01 13:05:39.666954: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cublasLt64_11.dll
2021-06-01 13:05:39.671151: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cufft64_10.dll
2021-06-01 13:05:39.675108: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library curand64_10.dll
2021-06-01 13:05:39.679541: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cusolver64_10.dll
2021-06-01 13:05:39.684305: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cusparse64_11.dll
2021-06-01 13:05:39.688460: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cudnn64_8.dll
2021-06-01 13:05:39.692604: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2021-06-01 13:05:41.125810: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1261] Device interconnect StreamExecutor with strength 1 edge matrix:
2021-06-01 13:05:41.131058: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1267]      0
2021-06-01 13:05:41.134586: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1280] 0:   N
2021-06-01 13:05:41.139374: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1406] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 6611 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2070 SUPER, pci bus id: 0000:01:00.0, compute capability: 7.5)
2021-06-01 13:05:41.153092: I tensorflow/compiler/jit/xla_gpu_device.cc:99] Not creating XLA devices, tf_xla_enable_xla_devices not set
Model: "encoder"
__________________________________________________________________________________________________
Layer (type)                    Output Shape         Param #     Connected to
==================================================================================================
input_1 (InputLayer)            [(None, 73)]         0
__________________________________________________________________________________________________
dense (Dense)                   (None, 64)           4736        input_1[0][0]
__________________________________________________________________________________________________
dense_1 (Dense)                 (None, 32)           2080        dense[0][0]
__________________________________________________________________________________________________
dense_2 (Dense)                 (None, 16)           528         dense_1[0][0]
__________________________________________________________________________________________________
z_mean (Dense)                  (None, 10)           170         dense_2[0][0]
__________________________________________________________________________________________________
z_log_var (Dense)               (None, 10)           170         dense_2[0][0]
__________________________________________________________________________________________________
sampling (Sampling)             (None, 10)           0           z_mean[0][0]
                                                                 z_log_var[0][0]
==================================================================================================
Total params: 7,684
Trainable params: 7,684
Non-trainable params: 0
__________________________________________________________________________________________________
Model: "decoder"
_________________________________________________________________
Layer (type)                 Output Shape              Param #
=================================================================
input_2 (InputLayer)         [(None, 10)]              0
_________________________________________________________________
dense_3 (Dense)              (None, 16)                176
_________________________________________________________________
dense_4 (Dense)              (None, 32)                544
_________________________________________________________________
dense_5 (Dense)              (None, 64)                2112
_________________________________________________________________
dense_6 (Dense)              (None, 73)                4745
=================================================================
Total params: 7,577
Trainable params: 7,577
Non-trainable params: 0
_________________________________________________________________
0.1
Importing, preparing and splitting the data for training...
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 10/10 [00:12<00:00,  1.22s/it]
min and max of all data is: [-0.9911290322580646, 0.9999568268315888]
Training shape: (44691, 73)
Testing shape: (11173, 73)

(55864, 73)
2021-06-01 13:05:55.232180: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:116] None of the MLIR optimization passes are enabled (registered 2)
Epoch 1/20
2021-06-01 13:05:56.907519: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cublas64_11.dll
2021-06-01 13:05:58.087456: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cublasLt64_11.dll
437/437 [==============================] - 6s 8ms/step - loss: 88.4235 - reconstruction_loss: 87.1899 - kl_loss: 0.4491
Epoch 2/20
437/437 [==============================] - 3s 8ms/step - loss: 85.7526 - reconstruction_loss: 84.5157 - kl_loss: 0.3339
Epoch 3/20
437/437 [==============================] - 3s 8ms/step - loss: 81.8148 - reconstruction_loss: 78.8124 - kl_loss: 1.2687
Epoch 4/20
437/437 [==============================] - 3s 8ms/step - loss: 73.4287 - reconstruction_loss: 65.9878 - kl_loss: 4.3876
Epoch 5/20
437/437 [==============================] - 3s 8ms/step - loss: 61.0203 - reconstruction_loss: 50.1326 - kl_loss: 7.8043
Epoch 6/20
437/437 [==============================] - 3s 8ms/step - loss: 49.5439 - reconstruction_loss: 37.1917 - kl_loss: 10.0104
Epoch 7/20
437/437 [==============================] - 3s 8ms/step - loss: 40.2992 - reconstruction_loss: 26.5229 - kl_loss: 11.6247
Epoch 8/20
437/437 [==============================] - 3s 8ms/step - loss: 31.3508 - reconstruction_loss: 14.6711 - kl_loss: 14.1937
Epoch 9/20
437/437 [==============================] - 3s 8ms/step - loss: 20.3250 - reconstruction_loss: -3.3209 - kl_loss: 20.1333
Epoch 10/20
437/437 [==============================] - 3s 8ms/step - loss: 3.5804 - reconstruction_loss: -36.8813 - kl_loss: 34.3109
Epoch 11/20
437/437 [==============================] - 3s 8ms/step - loss: -26.6376 - reconstruction_loss: -102.3258 - kl_loss: 64.7776
Epoch 12/20
437/437 [==============================] - 3s 8ms/step - loss: -78.8574 - reconstruction_loss: -216.2937 - kl_loss: 119.0723
Epoch 13/20
437/437 [==============================] - 3s 8ms/step - loss: -165.1668 - reconstruction_loss: -401.3890 - kl_loss: 207.7007
Epoch 14/20
437/437 [==============================] - 3s 8ms/step - loss: -298.7348 - reconstruction_loss: -686.9670 - kl_loss: 345.0612
Epoch 15/20
437/437 [==============================] - 3s 8ms/step - loss: -499.4923 - reconstruction_loss: -1114.8948 - kl_loss: 552.3550
Epoch 16/20
437/437 [==============================] - 3s 8ms/step - loss: -788.9226 - reconstruction_loss: -1730.5520 - kl_loss: 851.7061
Epoch 17/20
437/437 [==============================] - 3s 8ms/step - loss: -1199.3429 - reconstruction_loss: -2595.0793 - kl_loss: 1273.4973
Epoch 18/20
437/437 [==============================] - 3s 8ms/step - loss: -1757.6337 - reconstruction_loss: -3780.2153 - kl_loss: 1852.7542
Epoch 19/20
437/437 [==============================] - 3s 8ms/step - loss: -2520.4012 - reconstruction_loss: -5371.1265 - kl_loss: 2629.8840
Epoch 20/20
437/437 [==============================] - 3s 8ms/step - loss: -3525.9451 - reconstruction_loss: -7478.2344 - kl_loss: 3660.9993
End

As you can see the reconstruction_loss is decreasing an then grows larger as a negative number. The kl_loss has some sort of exponential growth. I don't really know what to do from here. A kollege of mine thinks it is a hyperparameter problem, but i have never seen a loss function to go negative for exammple. FYI: The input data is a Vektro with normalized data from -1 to 1.

I would be verry thankfull if you have some ideas about what might be the problem, or what I might have missed and how I could fix the model to train properly.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.