3
$\begingroup$

I generate some non-Gaussian data, and use two kinds of DNN models, one with BN and the other without BN.

I find that the model DNN with BN can't predict well.

The codes is shown as follow:

import numpy as np
import scipy.stats
import matplotlib.pyplot as plt
from keras.models import Sequential
from keras.layers import Dense,Dropout,Activation, BatchNormalization

np.random.seed(1)

# generate non-gaussian data
def generate_data():
    distribution = scipy.stats.gengamma(1, 70, loc=10, scale=100)
    x = distribution.rvs(size=10000)
    # plt.hist(x)
    # plt.show()
    print ('[mean, var, skew, kurtosis]', distribution.stats('mvsk'))

    y = np.sin(x) + np.cos(x) + np.sqrt(x)
    plt.hist(y)
    # plt.show()
    # print(y)
    return x ,y 

x, y = generate_data()

x_train = x[:int(len(x)*0.8)]
y_train = y[:int(len(y)*0.8)]
x_test = x[int(len(x)*0.8):]
y_test = y[int(len(y)*0.8):]


def DNN(input_dim, output_dim, useBN = True):
    '''
    定义一个DNN model
    '''
    model=Sequential()

    model.add(Dense(128,input_dim= input_dim))
    if useBN:
        model.add(BatchNormalization())
    model.add(Activation('tanh'))
    model.add(Dropout(0.5))

    model.add(Dense(50))
    if useBN:
        model.add(BatchNormalization())
    model.add(Activation('tanh'))
    model.add(Dropout(0.5))

    model.add(Dense(output_dim))
    if useBN:
        model.add(BatchNormalization())
    model.add(Activation('relu'))

    model.compile(loss= 'mse', optimizer= 'adam')
    return model

clf = DNN(1, 1, useBN = True)
clf.fit(x_train, y_train, epochs= 30, batch_size = 100, verbose=2, validation_data = (x_test, y_test))

y_pred = clf.predict(x_test)
def mse(y_pred, y_test):
    return np.mean(np.square(y_pred - y_test))
print('final result', mse(y_pred, y_test))

The input x is like this shape:

enter image description here

If I add BN layers, the result is shown as follows:

Epoch 27/30
 - 0s - loss: 56.2231 - val_loss: 47.5757
Epoch 28/30
 - 0s - loss: 55.1271 - val_loss: 60.4838
Epoch 29/30
 - 0s - loss: 53.9937 - val_loss: 87.3845
Epoch 30/30
 - 0s - loss: 52.8232 - val_loss: 47.4544
final result 48.204881459013244

If I don't add BN layers, the predicted result is better:

Epoch 27/30
 - 0s - loss: 2.6863 - val_loss: 0.8924
Epoch 28/30
 - 0s - loss: 2.6562 - val_loss: 0.9120
Epoch 29/30
 - 0s - loss: 2.6440 - val_loss: 0.9027
Epoch 30/30
 - 0s - loss: 2.6225 - val_loss: 0.9022
final result 0.9021717561981543

Anyone knows the theory about why BN is not suitable for non-gaussian data ?

$\endgroup$
  • $\begingroup$ Interesting question, if you do the same thing with gaussian data, does BN work properly? $\endgroup$ – Djib2011 Nov 3 at 9:46
0
$\begingroup$

So batch-normalization helps descent based learning have an easier time traversing the loss manifold, but in your case you use it along with a relu as a final activation is problematic, it means the output is relatively associated with the other samples in the batch.

Remove that last BN and you get better results, but also understand BN is inherently problematic for this task. Think of DNNs as featurizers, and BN in this case takes out the 2 batch-wide statistics which if they don't align to the initial distribution will cause error, which will lead to error in the output. In theory if BN gets the perfect statistics of the gaussian it should not matter too much, so one thing I tried with your code was remove last BN and increase N to 100,000 while increasing the batch size to 10000 and you see a huge boost in performance.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.