Applying Artificial neural network into kaggle's house prices data set gave bad predicted values

Question

I am trying to solve the kaggle's house prices using neural network. I've already made it with ensembling several models (XGBoost, GradientBooster and Ridge) and I've got a great score ranking me between the top 25%.

I imagined that by adding a new model to the ensembled models like ANN would increase prediction accuracy, so I did the following:

import keras

model = keras.models.Sequential()

model.add(keras.layers.Dense(235, activation='relu', input_shape=(235,)))
model.add(keras.layers.Dense(235, activation='relu'))
model.add(keras.layers.Dense(235, activation='relu'))
model.add(keras.layers.Dense(235, activation='relu'))
model.add(keras.layers.Dense(235, activation='relu'))
model.add(keras.layers.Dense(1))

model.compile(optimizer='adam', loss='mean_squared_error')
model.summary()
model.fit(dataset_ann, y, epochs=100, callbacks=[keras.callbacks.EarlyStopping(patience=3)])
y_pred = model.predict(X_test_ann)

I choosed 235 neurons for each layer, as the training set has 235 features.

For model ensembling:

y_p = (0.1*model.predict(X_test_ann)+0.2*gbr.predict(testset)+0.3*xgb.predict(testset)+0.1*regressor.predict(testset)+0.1*elastic.predict(testset)+0.2*ridge.predict(testset))

The shape of y_p is (1459, 1459) instead of (1459, ) where columns are all having the same values, so taking y_p[0] would be more than enough.

I submitted the result into kaggle and went from top 25% into bottom 60%.

Is it because the number of hidden layers with its input? Or because there is few data to train (1460 rows of train set) and the neural network needs more than that? Or is it because of the number of neurons in each layer?

I tried with epoch = 30, 100, 1000 and got nearly the same bad ranking.

Answer 1

Is it because the number of hidden layers with its input? Or because there is few data to train (1460 rows of train set) and the neural network needs more than that? Or is it because of the number of neurons in each layer?

I think you are onto something here. You have over a thousand nodes, almost as many as training samples you have. In my experience, the network will have no trouble overfitting, given you train long enough. If you dont train long enough on the other hand, your network will probably not learn anything.

I would try with a smaller network, so you "force" it to learn something, instead of memorizing.