0
$\begingroup$

I wish to get MSE < 0.5 on test data (https://easyupload.io/zr7xf3) which is 20% of given data chosen randomly. But I am reaching 0.73 using both plain Ridge Regression as well as a neural network with about 6 layers with some elementary regularization, dropout and choice of other parameters. Overfitting also occurs.

Suggest. I believe a Bayesian optimization or a genetic algorithm for parameters is required.

I did no feature selection (as top 4 features showed no improvement) and non-linear methods exploration.

My solutions - Ridge - Alpha = 0.002 (Grid searched)

Neural Network efforts =
reduce_lr = ReduceLROnPlateau(monitor='val_loss', factor=0.2,
                              patience=10, min_lr=0.001)


es = EarlyStopping(monitor='val_loss', mode='min', verbose=1, patience=10, restore_best_weights=True)


model_b = Sequential()


model_b.add(Dense(2048, kernel_initializer='he_uniform',input_dim = X.shape[1], activation='relu', kernel_regularizer=regularizers.l2(l2=1e-6)))
model_b.add(BatchNormalization(beta_regularizer = regularizers.l2(0.00001)))


# The Hidden Layers :
model_b.add(Dense(1024, kernel_initializer='lecun_normal',activation='selu',kernel_regularizer=regularizers.l2(l2=1e-6)))


model_b.add(BatchNormalization(beta_regularizer = regularizers.l2(0.00001)))


model_b.add(Dense(1024, kernel_initializer='lecun_normal',activation='selu',kernel_regularizer=regularizers.l2(l2=1e-6)))


model_b.add(BatchNormalization(beta_regularizer = regularizers.l2(0.00001)))


model_b.add(Dropout(0.5))


model_b.add(Dense(512, kernel_initializer='normal',activation='relu'))


model_b.add(Dense(512, kernel_initializer='normal',activation='relu'))


model_b.add(Dense(256, kernel_initializer='normal',activation='relu'))

# The Output Layer :


model_b.add(Dense(1, kernel_initializer='normal',activation='linear'))
optimizer = SGD(lr = 0.0001)


model_b.compile(loss='mean_squared_error', optimizer= optimizer)


model_b.fit(X_train, y_train, batch_size=70,
              epochs=256,
              validation_data=(X_test, y_test),callbacks = [es])


predb = model_b.predict(X_test)

If anyone has free time, may answer.

Best

$\endgroup$
2
  • 1
    $\begingroup$ Regardless of where you post this though, could you explain why you have set a target MSE of 0.5? For instance, is it something you know can be achieved with the given data (e.g. other people are reporting error of 0.5 MSE or better using exact same test set)? Have you done some analysis and have reason to believe 0.5 MSE is achievable? Please use edit to add details to the question. $\endgroup$ – Neil Slater Oct 27 '20 at 12:46
  • $\begingroup$ @Slater; I was asked at an interview $\endgroup$ – Mrinmay Oct 28 '20 at 17:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.