I think to have found a somewhat interesting connection between neural networks and another area of mathematics. However, it requires the activation functions in the network to have a bounded - ideally small - domain. For the sake of simplicity, I am restricting this to feedforward networks.
My approach has been the following: Assuming bounded input and weights, a maximal input can be derived. Before each application of an activation function, I thus just scale down the range from the maximal one to the one permitted.
This however causes nearly all weights that are not close (in absolute terms) to the maximal ones to lead to very small outputs of the network, meaning nearly all weight combinations lead to outputs near zero. The network thus has issues learning even simple tasks.
My question, therefore, is: Has anyone ever studied these issues and maybe found a network architecture that works well with this? Or another solution for bounded domains?