# Why isn't the loss of my neural network reduced after 2500 iterations?

I have developed a basic feedforward neural network from scratch to classify whether image is of cat or not cat. It works fine, but after 2500 iterations, my cost function is not reducing properly.

The loss function which I am using is

$$L(\hat{y},y) = -ylog\hat{y}-(1-y)log(1-\hat{y})$$

Can you please point out where I am going wrong the link to the notebook is https://www.kaggle.com/sidcodegladiator/catnoncat-nn?

• It might be the vanishing gradient problem. – efedoganay Jul 11 at 17:45
• Hi and welcome! Please, read ai.stackexchange.com/help/on-topic so that you are familiar with our scope. Anyway, I think you should provide more details, such as the type of neural network (feedforward, recurrent, etc.), what task are you trying to solve, the loss function you are using, activation functions, etc., and maybe show the plots of the loss. – nbro Jul 11 at 18:36
• @nbro: I have edited the post thank you – Siddarth Jul 12 at 3:38

You may try to adjust the learning rate first. As the learning rate has a great effect on changing the weights and the bias value.

See if the results has changed after adjusting the learning rate.

• I tried that as well, What do you suggest my learning rate should be? – Siddarth Jul 16 at 8:13
• You can try to set the learning rate to 0.01 or 0.1 to see if the results of outcome is better or not – Oscar916 Jul 16 at 8:18
• I tried 0.01, 0.1 and even 1 what I have noticed is the rate at which the cost function is decreasing is good but the problem is it still getting plateaued at 0.64 after 2500 epochs – Siddarth Jul 16 at 8:49

Since after a number of iterations the cost function is not reducing, this may be able to be diagnosed as a vanishing gradient problem. A solution to this is the use of a Residual neural network.