0
$\begingroup$

I'm trying to build a neural network (NN) for classification using only N-bit integers for both the activations and weights, then I will train it with some heuristic algorithm, based only on the NN evaluation.

Currently, I'm using a non-linear activation function for hidden units. Because of its probability interpretation, I am forced to use the softmax (or the sigmoid for 2-class case) for the output layer. However, because of the use of integers, the linear combination of the activations and weights can easily be too large, and this causes a problem to the exponential in the softmax evaluation.

Any solution?

Improve this question
$\endgroup$
1
  • $\begingroup$ Hello. Welcome to AI SE. I've edited your post in order to improve its clarity and, in particular, to add a more descriptive title. Let me know if I changed the meaning of your post with my edit. If yes, you can edit again your post to fix this. $\endgroup$
    – nbro
    Jun 5, 2021 at 12:42

1 Answer 1

Reset to default
1
$\begingroup$

First of all, check out this question. Generally, you don't need to apply softmax and using raw logits leads to better numerical stability.

The numerical issue that you are talking about is well known and dealt with the so-called logsumexp trick. This usually is already incorporated in standard NN libraries. For example keras CategoricalCrossentropy loss can be configured to compute it from_logits.

Improve this answer
$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .