I'm trying to build a neural network (NN) for classification using only N-bit integers for both the activations and weights, then I will train it with some heuristic algorithm, based only on the NN evaluation.
Currently, I'm using a non-linear activation function for hidden units. Because of its probability interpretation, I am forced to use the softmax (or the sigmoid for 2-class case) for the output layer. However, because of the use of integers, the linear combination of the activations and weights can easily be too large, and this causes a problem to the exponential in the softmax evaluation.
Any solution?