# Which solutions are there to the problem of having too large activations before the softmax (or sigmoid) layer?

I'm trying to build a neural network (NN) for classification using only N-bit integers for both the activations and weights, then I will train it with some heuristic algorithm, based only on the NN evaluation.

Currently, I'm using a non-linear activation function for hidden units. Because of its probability interpretation, I am forced to use the softmax (or the sigmoid for 2-class case) for the output layer. However, because of the use of integers, the linear combination of the activations and weights can easily be too large, and this causes a problem to the exponential in the softmax evaluation.

Any solution?

• Hello. Welcome to AI SE. I've edited your post in order to improve its clarity and, in particular, to add a more descriptive title. Let me know if I changed the meaning of your post with my edit. If yes, you can edit again your post to fix this. – nbro Jun 5 at 12:42