0
$\begingroup$

I am using feedforward neural network for regression and what I get as a result of prediction is a constant value visible on the graph below: enter image description here

Data I use are typical standardised tabular numbers. The architecture is as follows:

model.add(Dropout(0.2))
model.add(Dense(units=512, activation='relu'))
model.add(Dropout(0.3))
model.add(Dense(units=256, activation='relu'))
model.add(Dropout(0.3))
model.add(Dense(units=128, activation='relu'))
model.add(Dense(units=128, activation='relu'))
model.add(Dense(units=1))

adam = optimizers.Adam(lr=0.1)

model.compile(loss='mean_squared_error', optimizer=adam)

reduce_lr = ReduceLROnPlateau(
    monitor='val_loss',
    factor=0.9,
    patience=10,
    min_lr=0.0001,
    verbose=1)

tensorboard = TensorBoard(log_dir="logs\{}".format(NAME))

history = model.fit(
    x_train,
    y_train,
    epochs=500,
    verbose=10,
    batch_size=128,
    callbacks=[reduce_lr, tensorboard],
    validation_split=0.1)

It seems to me that all weights are zeroed and only constant bias is present here, since for different data samples from a test set I get the same value, but I am not sure.

I understand that the algorithm has found smallest MSE for such a constant value, but is there a way of avoiding such situation, since straight line is not really good solution for my project?

$\endgroup$
  • $\begingroup$ Have you tried experimenting with a lower learning rate as a starting point? You're starting with 0.1 which is quite high for most of the cases, and only reduce by 0.9 which is not much. $\endgroup$ – razvanc92 Aug 27 at 7:31
  • 1
    $\begingroup$ I have now and it worked out! Lowering initial learning rate to 0.005 eliminated straight line as an output. You can post it as an answer :) Thanks! $\endgroup$ – Makintosz Aug 27 at 8:18
  • $\begingroup$ and if you have some idea why this happens so, with greater learning rate, please do share as well. $\endgroup$ – Makintosz Aug 27 at 8:26
0
$\begingroup$

You should try experimenting with a lower learning rate as a starting point. You're starting with 0.1 which is quite high for most of the cases, and only reduce by 0.9 which is not much.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.