Why does the denoising autoencoder always returns the same output?

Question

I am trying to implement a denoising autoencoder (DAE) to remove noise from 1024-point FFT spectra. I am using two types of spectra: (1) that contain a distinctive high amplitude spectral peak and (2) that contain only noise peaks.

If I understood correctly, I can train the DAE using the corruputed spectra (spectra+noise) and afterwards I can use it the remove noise from new datasets. The problem is that when testing the DAE, it returns the type (1) spectrum mentioned above, regardless of the input. The same case when I apply predict on the training data. This is the code I am using (Python/Tensorflow):

def BuildModel(nInput):
    input_dim = Input(shape = (nInput, ))

    # Encoder Layers
    encoded1 = Dense(896, activation = 'relu')(input_dim)
    encoded2 = Dense(768, activation = 'relu')(encoded1)
    encoded3 = Dense(640, activation = 'relu')(encoded2)
    encoded4 = Dense(512, activation = 'relu')(encoded3)
    encoded5 = Dense(384, activation = 'relu')(encoded4)
    encoded6 = Dense(256, activation = 'relu')(encoded5)
    encoded7 = Dense(encoding_dim, activation = 'relu')(encoded6)

    # Decoder Layers
    decoded1 = Dense(256, activation = 'relu')(encoded7)
    decoded2 = Dense(384, activation = 'relu')(decoded1)
    decoded3 = Dense(512, activation = 'relu')(decoded2)
    decoded4 = Dense(640, activation = 'relu')(decoded3)
    decoded5 = Dense(768, activation = 'relu')(decoded4)
    decoded6 = Dense(896, activation = 'relu')(decoded5)
    decoded7 = Dense(nInput, activation = 'sigmoid')(decoded6)

    # Combine Encoder and Deocoder layers
    autoencoder = Model(inputs = input_dim, outputs = decoded7)

    autoencoder.summary()
    # Compile the Model
    autoencoder.compile(optimizer=OPTIMIZER, loss='binary_crossentropy')
    #autoencoder.compile(loss='mean_squared_error', optimizer = RMSprop())

    return autoencoder

X_train, X_test, y_train, y_test = train_test_split(spectra.iloc[:,0:spectra.shape[1]-1], spectra['Class'], test_size=testDatasetSize, stratify=spectra.Class, random_state=seedValue)

X_train, y_train = shuffle(X_train, y_train, random_state=seedValue)
X_test, y_test = shuffle(X_test, y_test, random_state=seedValue)

X_unseen = X_train.to_numpy()[0:1000,:] # Data not used for training, only for testing
y_unseen = y_train.to_numpy()[0:1000]
X_train = X_train.iloc[1000:]
y_train = y_train.iloc[1000:]

# Scaling
maxVal = max(X_train)
X_train = (X_train/maxVal).to_numpy()
X_test = (X_test/maxVal).to_numpy()
X_unseen = (X_unseen/maxVal)#.to_numpy()

# Corrupted data
noise_factor = 0.01
X_train_noisy = X_train + noise_factor * np.random.normal(loc=0.0, scale=1.0, size=X_train.shape)
X_test_noisy = X_test + noise_factor * np.random.normal(loc=0.0, scale=1.0, size=X_test.shape)
X_unseen_noisy = X_unseen + noise_factor * np.random.normal(loc=0.0, scale=1.0, size=X_unseen.shape)

ae = BuildModel(X_train.shape[1])
PrintConsoleLine('Creating model finished')
print('')

history = ae.fit(X_train_noisy, X_train, epochs=NB_EPOCH, batch_size=BATCH_SIZE, validation_data=[X_test_noisy, X_test])
save_model(ae, modelFile, overwrite=True)

# Test
X = X_unseen
X_noisy = X_unseen_noisy
X_denoised = ae.predict(X_noisy) # X_train gives the same result (spectra type (1)) !?!
N = len(X_denoised[0,:])
index = 6
PlotDataSimple(3, np.linspace(0,N-1,N), X[index,:], 'Frequency domain', 'Index', 'Amplitude', None)
PlotDataSimple(4, np.linspace(0,N-1,N), X_noisy[index,:], 'Frequency domain', 'Index', 'Amplitude', None)
PlotDataSimple(5, np.linspace(0,N-1,N), X_denoised[index, :], 'Frequency domain', 'Index', 'Amplitude', None)

Dataset shape: (17000, 65, 65, 1) (files, samples X axis, samples Y axis, class)
Train on 12600 samples, validate on 3400 samples
Epoch 1/3
12600/12600 [==============================] - 26s 2ms/sample - loss: 0.6813 - val_loss: 0.4913
Epoch 2/3
12600/12600 [==============================] - 14s 1ms/sample - loss: 0.1621 - val_loss: 0.0578
Epoch 3/3
12600/12600 [==============================] - 16s 1ms/sample - loss: 0.0230 - val_loss: 0.0169

The results I am getting (Column 1 - Initial signal, Column 2 - Corrupted signal, Column 3 - Denoised signal):

So why does the DAE output the same spectra regardless of the inputs? Am I misunderstanding the DAE principle or is there a problem in my implementation?

Answer 1

You are using Dense layers, try 1d convolution instead. Have you tried a different activation function such as softmax and instead of Binary cross entropy try MSE loss? Are all your inputs between 0 and 1? Also, I think your noise amplitude is too much in 2nd and 3rd case as compared to the actual signal. Can you try training on different types of spectra separately and check the result, if the DAE is learning anything. Also, try trining for more epochs.

I am not sure and would have commented this but I can't due to low rep, but I am interested in the solution.

Answer 2

If the auto-encoder is converging to the same encoding for different instances, there may be a problem in the loss function. Check the size and shape of the output of the loss function, as it may be getting confused and evaluating the wrong tensors (i.e. you may need to transpose something somewhere).

Basically, assuming you are using an auto-encoder to encode $M$ features of $N$ training instances, your loss function should return $N$ values i.e., the size of your loss tensor should be the amount of instances in your training set.