So I am training an ANN for classification between 3 classes. The ANN has an input layer, one hidden layer and a 3 node output layer.

The problem I am facing is that the output being produced by the 3 output nodes are so close to 1 (for the first few iterations at least and so i am assuming the problem propagates to future outputs as well) the weights are not being updated (or hardly updated) due to overflow (about 10^-11). Now I can fix the overflow problem (but I don't think it is the culprit). I think such low values of error is the main culprit, and I cannot figure what is causing such low values of error.

What will cause the network to behave more interactively like I will be actually able to grasp the weight updates and not something in the order of 10^-11?

Note: Data set contain values in the order of 10's and the weights randomly initialized are in the order of 0 < w < 1. I have tried feature normalization but it is not that effective.

Any help is highly appreciated.

EDIT: I did not know the term was called vanishing gradient, so I added it for better readability

NOTE: Experienced users can freely edit the question to cater to a more general problem, since I believe the problem has many other variations.