I'm using tf.Keras to build a deep-fully connected autoencoder. My input dataset is a dataframe with shape (19947,), and the purpose of the autoencoder is to predict normalized gene expression values. They are continuous values that range from [0,~620000].

I tried different architectures and I'm using relu activation for all layers. To optimize I'm using adam with mae loss.

The problem I have is the network trains successfully (although the train loss is still terrible) but when I'm predicting I notice that although the predictions do make sense for some nodes, there are always a certain number of nodes that only output 0. I've tried changing the number of nodes of my bottleneck layer (output) and it always happens even when I decrease the output number.

Any ideas on what I'm doing wrong?

tf.Keras code:

input_layer = keras.Input(shape=(19947,))
simple_encoder = keras.models.Sequential([
    keras.layers.Dense(512, activation='relu'),
    keras.layers.Dense(128, activation='relu'),
    keras.layers.Dense(16, activation='relu')
simple_decoder = keras.models.Sequential([
    keras.layers.Dense(128, activation='relu'),
    keras.layers.Dense(512, activation='relu'),
    keras.layers.Dense(19947, activation='relu')
simple_ae = keras.models.Sequential([simple_encoder, simple_decoder])
simple_ae.compile(optimizer='adam', loss='mae')
simple_ae.fit(X_train, X_train,
              validation_data=(X_valid, X_valid),

Output of encoder.predict with 16 nodes on the bottleneck layer. 7 nodes predict only 0's and 8 nodes predict "correctly" Output from encoder.predict with 16 nodes on bottleneck layer

  • $\begingroup$ Have you done any normalization of the data? The weights and biases will have a hard time influencing a number with a magnitude like 620000. $\endgroup$ – David Hoelzer Nov 29 '20 at 22:48
  • $\begingroup$ The data is already normalized using a specific algorithm to normalize genomic expression data, I could normalize on top of that using min-max or standard scaler but I'm not sure that would be accurate $\endgroup$ – beerzy Nov 29 '20 at 23:42

My closest guess is because you are using the activation function ReLU, which pushes data to be greater than zero. However, because of your data's nature, the autoencoder is highly dependent on negative calculations to reconstruct your data, but the best it can achieve is zero.

In the context of artificial neural networks, the rectifier is an activation function defined as the positive part of its argument: $$f(x) = x^+ = max(0, x)$$ https://en.wikipedia.org/wiki/Rectifier_(neural_networks)

  • $\begingroup$ I'm not sure I fully understand your answer. I do understand that relu pushes any negative numbers to zero, however my data contains no negative number so what's the logic behind the bottleneck layer predicting negative numbers? Anyway, I will try to use LeakyReLu as activation wich should in theory fix the problem right? Do you suggest I try this change only on the bottleneck layer or on all layers? $\endgroup$ – beerzy Nov 30 '20 at 15:30
  • $\begingroup$ I've tried the following: 1) Convert my dataset to ]0,~620000]; 2) Apply log2 transformation converting it to [-23.25, 19.24]. My hope is that changing to activations that accept negative numbers will fix it, but I still have the same issue. $\endgroup$ – beerzy Nov 30 '20 at 16:08
  • $\begingroup$ I mean, the auto encoder probably wanted to calculate negative outputs in hidden layers but the relu won't let it to be. Have you tried to use linear activation? $\endgroup$ – kiarash_kiani Nov 30 '20 at 18:05
  • $\begingroup$ Try to change all activation function in all layers $\endgroup$ – kiarash_kiani Nov 30 '20 at 18:14
  • $\begingroup$ Does decreasing bottleneck size affects reconstruction error? $\endgroup$ – kiarash_kiani Nov 30 '20 at 18:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.