# What could be the problem when a neural network with four hidden layers with the sigmoid activation function is not learning?

I have a large set of data points describing mappings of binary vectors to real-valued outputs. I am using TensorFlow, and would like to train a model to predict these relationships. I used four hidden layers with 500 neurons in each layer, and sigmoidal activation functions in each layer.

The network appears to be unable to learn, and has high loss even on the training data. What might cause this to happen? Is there something wrong with the design of my network?

When training our neural network, you need to scale your dataset in order to avoid slowing down the learning or prevent effective learning. Try normalizing your output. This Tutorial might help

• @AggrajGupta See me answer. You should also scale your data, but that's probably not the root of your problem. – John Doucette Nov 19 '19 at 1:56

Your code suggests a likely problem here: It looks like you are training a very deep neural network with sigmoidal activation functions at every layer.

The sigmoid has the property that its derivative (S*(1-S)) will be extremely small when the activation function's value is close to 0 or close to 1. In fact, the largest it can be is about 0.25.

The backpropigation algorithm, which is used to train a neural network, will propagate an error signal backwards. At each layer, the error signal will be multiplied by, among other things, the derivative of the activation function.

It is therefore the case that by the 4th layer your signal is at most $$0.25^4 = \frac{1}{256}$$ the size that it was at the start of the network. In fact, it is likely much smaller than this. With a smaller signal, your learning rates at the bottom of the nextwork will effectively be much smaller than the learning rates at the top, which will make it very difficult to pick a learning rate that is effective overall.

This problem is known as the vanishing gradient.

To fix this, if you want to use a deep architecture, consider using an activation function that does not suffer from a vanishing gradient. The Rectified Linear activation function, used in so-called "ReLU" units, is a non-linear activation that does not have a vanishing gradient. It is common to use ReLUs for the earlier layers in a network, and a sigmoid at the output layer, if you need outputs to be bounded between 0 and 1.