I think to have found a somewhat interesting connection between neural networks and another area of mathematics. However, it requires the activation functions in the network to have a bounded - ideally small - domain. For the sake of simplicity, I am restricting this to feedforward networks.

My approach has been the following: Assuming bounded input and weights, a maximal input can be derived. Before each application of an activation function, I thus just scale down the range from the maximal one to the one permitted.

This however causes nearly all weights that are not close (in absolute terms) to the maximal ones to lead to very small outputs of the network, meaning nearly all weight combinations lead to outputs near zero. The network thus has issues learning even simple tasks.

My question, therefore, is: Has anyone ever studied these issues and maybe found a network architecture that works well with this? Or another solution for bounded domains?

  • $\begingroup$ Might not be what you look for but I remember the paper "Neural Arithmetic Logic Units" bounding weights in the architecture with activation functions. arxiv.org/pdf/1808.00508.pdf $\endgroup$
    – hal9000
    May 14 at 22:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.