I have developed a basic feedforward neural network from scratch to classify whether image is of cat or not cat. It works fine, but after 2500 iterations, my cost function is not reducing properly.

The loss function which I am using is

$L(\hat{y},y) = -ylog\hat{y}-(1-y)log(1-\hat{y})$

Can you please point out where I am going wrong the link to the notebook is https://www.kaggle.com/sidcodegladiator/catnoncat-nn?

  • $\begingroup$ It might be the vanishing gradient problem. $\endgroup$
    – efedoganay
    Commented Jul 11, 2020 at 17:45
  • 1
    $\begingroup$ This isn't a cnn, it's a basic MLP and that it performs poorly isn't surprising. $\endgroup$ Commented Dec 16, 2020 at 6:49
  • $\begingroup$ what optimizer are you using? I'd suggest you to try Adam $\endgroup$
    – SpiderRico
    Commented Apr 11, 2022 at 6:31
  • $\begingroup$ Which architectures are you using? Batch size? Learning rate? $\endgroup$ Commented Mar 30 at 19:34
  • $\begingroup$ What value are you using as a learning rate, what range are your weights/biases/kernels, and what is the range of your $x$ data? $\endgroup$ Commented Mar 31 at 0:16

3 Answers 3


You may try to adjust the learning rate first. As the learning rate has a great effect on changing the weights and the bias value.

See if the results has changed after adjusting the learning rate.

  • $\begingroup$ I tried that as well, What do you suggest my learning rate should be? $\endgroup$
    – Siddarth
    Commented Jul 16, 2020 at 8:13
  • $\begingroup$ You can try to set the learning rate to 0.01 or 0.1 to see if the results of outcome is better or not $\endgroup$
    – Oscar916
    Commented Jul 16, 2020 at 8:18
  • $\begingroup$ I tried 0.01, 0.1 and even 1 what I have noticed is the rate at which the cost function is decreasing is good but the problem is it still getting plateaued at 0.64 after 2500 epochs $\endgroup$
    – Siddarth
    Commented Jul 16, 2020 at 8:49

Since after a number of iterations the cost function is not reducing, this may be able to be diagnosed as a vanishing gradient problem. A solution to this is the use of a Residual neural network.

Another solution is that you carefully initialise your weights as throughout your neural network your gradient may exponentially explode or exponentially vanish.

Watch this video on how to initialise weights truly randomly: https://www.youtube.com/watch?v=s2coXdufOzE


Another possible cause for your issue is that your algorithm is having an high bias problem. This is due to your algorithm not performing well on the training set. In your case one of the best solutions would be to make your network deeper and so it shall be able to conduct more complex functions and so perform better on your training set.

  • $\begingroup$ Thanks for the tip after doing that I have initialized my weights like this parameters['W'+str(l)]=np.random.randn(layers_dims_vector[l],layers_dims_vector[l-1])*np.sqrt(2/layers_dims_vector[l-1]) but still my cost does not seem to get reduced, Can you point me if there is something else I can do? Or do you see any error in the algorithm? $\endgroup$
    – Siddarth
    Commented Jul 18, 2020 at 15:16
  • $\begingroup$ @Siddarth I have made an edit which should help in case that this is not a vanishing gradient issue, as seen from the random initialisation barely having an effect on the performance of your algorithm. $\endgroup$
    – jr235
    Commented Jul 19, 2020 at 4:59
  • $\begingroup$ Hi I tried with this configuration [12288,7000,4000,2000,10000,500,200,150,100,50,25,12,1] first is no of inputs and last is the output layer remaining are hidden layers. The network has become slow but still use, Can you check my code and let me know what I am missing? $\endgroup$
    – Siddarth
    Commented Jul 21, 2020 at 11:43
  • $\begingroup$ In order to reduce the time your algorithm spends training you should use another optimisation algorithm (e.g Adam optimisation algorithm, Gradient Descent with momentum, Root Mean Square Propagation etc...). In order to make my diagnostic more accurate split your data set into a training (70%) and development set (30% depending on how large your data set is, the larger the smaller the dev set size), and plot the costs over time on both sets. $\endgroup$
    – jr235
    Commented Jul 22, 2020 at 7:30
  • $\begingroup$ Hi I have observed inspite of increasing the height and depth of network, The cost is getting plateaued. Is the cost function getting stuck in local minima? If thats the case do you see any issue with the algorithm? $\endgroup$
    – Siddarth
    Commented Jul 23, 2020 at 15:26

I would try at least a dozen of random architectures, there's a interesting research paper suggesting randomization (can't find It right now, but believe me I read it) is the best bet to find optimal hyperparameters. And indeed in my opensource project it worked like a charm. The first attempt likes half of the neural networks performed at 10% accuracy on MNIST task( same as randomly guessing a digit), I removed from random selection parameters that seemed to not work (in example I removed layers with 9 and 16 neurons and Sigmoid activation function) and tried again the following time i obtained many networks above 75% accuracy. I even obtained a Small Fully Connected network that performs 87,1% accuracy with 5 layers and less than 200 neurons. And that with less that 60.000 samples (not even used the whole training set).. which Is remarkable results, but mostly due to.. Luck.

The thing is no one knows which architectures are going to work, you have to try. If after something like 20 random attempts you still fail then the problem is likely to lie elsewhere. Maybe in the data, maybe in the encoding, maybe in the implementation.

You mentioned nothing about your architecture by the way. It Is common to use Glorot init on tanh/Sigmoid layers and He initialization on ReLU layers.


I noticed now the link. Try to randomize Number of layers and Number of neurons.

Then randomize 1 to 3

  • If 1 all layers are same type
  • If 2 alternate tanh and relu layers
  • If 3 randomize type of layer

Make at least 10 random networks Tell me back the results. Don't get frustrated nobody knows what works in advance. Also note that on big images, It Is usually used some convolution layer to reduce train time.


You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .