So I am training an ANN for classification between 3 classes. The ANN has an input layer, one hidden layer and a 3 node output layer. 

The problem I am facing is that the output being produced by the 3 output nodes are so close to 1 (for the first few iterations at least and so i am assuming the problem propagates to future outputs as well) the weights are not being updated (or hardly updated) due to overflow (about `10^-11`). Now I can fix the overflow problem (but I don't think it is the culprit). I think such low values of error is the main culprit, and I cannot figure what is causing such low values of error. 

What will cause the network to behave more interactively like I will be actually able to grasp the weight updates and not something in the order of `10^-11`?

Note: Data set contain values in the order of `10's` and the weights randomly initialized are in the order of `0 < w < 1`. I have tried feature normalization but it is not that effective.

Any help is highly appreciated.

EDIT: I did not know the term was called vanishing gradient, so I added it for better readability

NOTE: Experienced users can freely edit the question to cater to a more general problem, since I believe the problem has many other variations.