-1
$\begingroup$

I am using TensorFlow's Adam optimizer and ReLu activation. But each time I train my model the cost becomes stagnant after some epochs. Can any one help me out with the possible reasons I have added an image of the cost: The learning rate is 0.001, the 1st 2 columns are the training data for input neurons and the 3rd column is the training data for output neurons, the nn has two input neurons, two hidden layers with 3 neurons each and one output neuron https://i.stack.imgur.com/Uw5Of.jpg

$\endgroup$
  • 1
    $\begingroup$ Welcome to ai.se...I have edited the question a little bit...I suggest you edit the question to add more relevant details like layers, learning rate, type of NN, the meaning of the columns in the pic $\endgroup$ – DuttaA Aug 5 '18 at 9:43
1
$\begingroup$

Without seeing your code and/or data, it's going to be difficult to give a concrete answer.

Possibly reasons might include:

  1. Your model isn't complex enough to fit the patterns in the data, and so it reaches a local minimum in the error function.

  2. The patterns in the data are not self-consistent, and so this is actually the global minimum in the error function.

  3. There's something wrong with your learning rate parameter, or with the gradient in the problem. Try increasing the learning rate, and ensure your model is being initialized with random weights.

$\endgroup$
  • $\begingroup$ I agree. The question needs to be built out a little. There's no way to tell why the convergence does not progress beyond the point shown in the image. About a dozen pieces of information are missing to make a reasonable guess at diagnosis. $\endgroup$ – FauChristian Aug 7 '18 at 4:01
1
$\begingroup$

In addition to John's answer, you may want to check how you initialize your weights and biases, and your activation functions.

ReLU is generally used to deal with gradient vanishing problem, which I believe doesn't exist in your problem with such a small scale. It may 'deactivate' some nodes when the node value is below zero, which might be a big problem because you have so little numbers of nodes.

From your picture I can see only positive inputs and negative outputs. So if you use normal distribution for weight initialization you'll get half of your weights negative, which will be deactivated by reLU. Besides, you definitely don't want to use reLU in your output nodes.

It seems like your output can be categorized, what about use classification methods instead of regression?

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.