# Behaviour of cost

In doing a project using neural networks with an input layer, 4 hidden layers and an output layer ,I used mini batch gradient descent. I noticed that the randomly initialised weights seemed to do a good performance and gave a low error. As the model started training after about 200 iterations there was large jump in error and then it came down slowly from there. I have also noticed that sometimes the cost just increases over a set of consecutive iterations. Can anyone explain why these happen? It is not like there are outliers or a new distribution as every iteration exposes it to the entire dataset. I used learning rate 0.01 and regularisation parameter 10. I also tried regularisation parameter 5 and also 1. And by the cost I mean, the sum of squared errors of all minibatches/2m plus regularisation term error. Further if this happens and my cost after the say 10000th iteration is more than my cost when I initialised with random weights (lol) can I just take the initial value? As those weights seem to be doing better.

The large jumps are the most puzzling.

This is the code

Any help would be greatly appreciated. Thanks

• Can you share your code and data? A guess would be that your data isn't continuous and thus gradient descent won't work. See stats.stackexchange.com/questions/253088/… – ubershmekel Aug 1 '18 at 7:06
• @ubershmekel i have added the code to the question. If you could go through it and figure something out, that would be really helpful. thanks – pranav Aug 1 '18 at 9:57
• @ubershmekel You're right the problem are the training data. If the data only contains an XOR-problem the network will make no problems. But a complicated dataset and random datasets are trouble makers for a neural network. So the hint is, to let all parameters of the training algorithm untouched and only make the data more adorable for machine learning. – Manuel Rodriguez Aug 1 '18 at 18:58
• @pranav please only paste the code as text, not screenshots. I don't want to have to transcribe your code to test it. – ubershmekel Aug 2 '18 at 22:28
• @ubershemekel thank you for your help. But I figured out the problem. The learning rate was too high. So I reduced it down to a really low value and it worked fine. I also used Xavier initialisation. Check out this article: thelaziestprogrammer.com/sharrington/math-of-machine-learning/… – pranav Aug 3 '18 at 2:41

I would say just don't go for using regularization at first, try with lower learning rate = 0.0001 see the behavior. Try to post entire architecture of your model so that one can better answer your problem.