I am using TensorFlow's Adam optimizer and ReLu activation. But each time I train my model the cost becomes stagnant after some epochs. Can any one help me out with the possible reasons I have added an image of the cost: The learning rate is 0.001, the 1st 2 columns are the training data for input neurons and the 3rd column is the training data for output neurons, the nn has two input neurons, two hidden layers with 3 neurons each and one output neuron
Without seeing your code and/or data, it's going to be difficult to give a concrete answer.
Possibly reasons might include:
Your model isn't complex enough to fit the patterns in the data, and so it reaches a local minimum in the error function.
The patterns in the data are not self-consistent, and so this is actually the global minimum in the error function.
There's something wrong with your learning rate parameter, or with the gradient in the problem. Try increasing the learning rate, and ensure your model is being initialized with random weights.
In addition to John's answer, you may want to check how you initialize your weights and biases, and your activation functions.
ReLU is generally used to deal with gradient vanishing problem, which I believe doesn't exist in your problem with such a small scale. It may 'deactivate' some nodes when the node value is below zero, which might be a big problem because you have so little numbers of nodes.
From your picture I can see only positive inputs and negative outputs. So if you use normal distribution for weight initialization you'll get half of your weights negative, which will be deactivated by reLU. Besides, you definitely don't want to use reLU in your output nodes.
It seems like your output can be categorized, what about use classification methods instead of regression?