1
$\begingroup$

I am creating a VAE for time series data using CNNs. The data has 4800 timesteps and 4 features. It is standardized and normalized. The network I am using is implemented in Keras as follows. I have used a MSE reconstruction error:

# network parameters
(_, seq_len, feat_init) = X_train.shape
input_shape = (seq_len, feat_init)
intermediate_dim = 512
batch_size = 128
latent_dim = 10
epochs = 10
img_chns = 3
filters = 32
num_conv = (2, 2)
epsilon_std = 1

inputs = Input(shape=input_shape)
conv1 = Conv1D(16, 3, 2, padding='same', activation = 'relu', data_format = 'channels_last')(inputs)
conv2 = Conv1D(32, 2, 2, padding='same', activation = 'relu', data_format = 'channels_last')(conv1)
conv3 = Conv1D(64, 2, 2, padding='same', activation = 'relu', data_format = 'channels_last')(conv2)
flat = Flatten()(conv3)
hidden = Dense(intermediate_dim, activation='relu')(flat)
z_mean = Dense(latent_dim, name = 'z_mean')(hidden)
z_log_var = Dense(latent_dim, name = 'z_log_var')(hidden)

def sampling(args):
    z_mean, z_log_var = args
    epsilon = K.random_normal(shape=(K.shape(z_mean)[0], latent_dim),
                              mean=0., stddev=epsilon_std)
    return z_mean + K.exp(z_log_var) * epsilon

z = Lambda(sampling, output_shape=(latent_dim,))([z_mean, z_log_var])

decoder_hid = Dense(intermediate_dim, activation='relu')(z)
decoder_upsample = Dense(38400, activation='relu')(decoder_hid)
decoder_reshape = Reshape((600,64))(decoder_upsample)

deconv1 = Conv1D(filters=32, kernel_size=2, strides=1,
             activation="relu", padding='same', name='conv-decode1')(decoder_reshape)
upsample1 = UpSampling1D(size=2, name='upsampling1')(deconv1)
deconv2 = Conv1D(filters=16, kernel_size=2, strides=1,
             activation="relu", padding='same', name='conv-decode2')(upsample1)
upsample2 = UpSampling1D(size=2, name='upsampling2')(deconv2)
deconv3 = Conv1D(filters=8
                 , kernel_size=2, strides=1,
             activation="relu", padding='same', name='conv-decode3')(upsample2)
upsample3 = UpSampling1D(size=2, name='upsampling3')(deconv3)
x_decoded_mean_squash = Conv1D(filters=4
                 , kernel_size=4, strides=1,
             activation="relu", padding='same', name='conv-decode4')(upsample3)

class CustomVariationalLayer(Layer):
    def __init__(self, **kwargs):
        self.is_placeholder = True
        super(CustomVariationalLayer, self).__init__(**kwargs)

    def vae_loss(self, x, x_decoded_mean_squash):
        x = K.flatten(x)
        x_decoded_mean_squash = K.flatten(x_decoded_mean_squash)
        xent_loss = mse(x, x_decoded_mean_squash)
        kl_loss = - 0.5 * K.mean(1 + z_log_var - K.square(z_mean) - K.exp(z_log_var), axis=-1)
        return K.mean(xent_loss + kl_loss)

    def call(self, inputs):
        x = inputs[0]
        x_decoded_mean_squash = inputs[1]
        loss = self.vae_loss(x, x_decoded_mean_squash)
        self.add_loss(loss, inputs=inputs)
        return x

outputs = CustomVariationalLayer()([inputs, x_decoded_mean_squash])

# entire model
vae = Model(inputs, outputs)
vae.compile(optimizer='adadelta', loss=None)
vae.summary()

I wanted to ask whether it is possible for the network to nearly perfectly reconstruct the test timeseries when passed through the entire VAE network, but still output junk when using a random Normal input. For further details, here is one of the inputs and outputs when passing a test signal through the network.

inputs and outputs

Here is a reconstruction generated purely from a random sample.

enter image description here

How can this be? Even if there was a posterior collapse, the VAE should still be able to generate a good output sample with a random input. To further test this I decided to split the network into two parts (encoder and decoder), and then pass the test image through it. The encoder and decoder networks were made by simply splitting the trained VAE network as follows:

idx = 9 
input_shape = vae.layers[idx].get_input_shape_at(0)

layer_input = Input(shape=(input_shape[1],)) 

x = layer_input
for layer in vae.layers[idx:-1]:
    x = layer(x)

decoder = Model(layer_input, x)
decoder.summary()
idx = 0
input_shape = vae.layers[idx].get_input_shape_at(0)

layer_input = Input(shape=input_shape)

x = layer_input
for layer in vae.layers[idx + 1:7]:
     x = layer(x)

encoder = Model(layer_input, x)
encoder.summary()

Interestingly, I also got junk output here. I'm not sure how it is possible. If the model itself is getting a near perfect reconstruction, surely just passing an image through the encoder, extracting the latent mean, and then passing that latent mean through the decoder should also create a near perfect image?

Is there something I am missing here?

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.