I've been writing a deep learning Java framework as a way for myself to learn how it all works and I have had a decent amount of success so far. Best performance is just over 90% accuracy with three fully connected sigmoid layers of 500, 300 and 10 nodes respectively using LeCun normalization for the initial weight values.

However, in trying to improve the accuracy of my model on the MNIST dataset I've been looking at alternative activation functions. I've tried ReLU with Kaiming normalization and I found that this works well when training on 10.000 images (about 85±5% accuracy), but loses accuracy (~40%) after training on all 60.000 training images, possibly due to more neurons dying over time.

I've been trying to get the SiLU (Sigmoid-weighted Linear Unit) activation function to work, but no matter what initialization I use I find the weights exploding fairly quickly. The SiLU activation function is defined as x*sigmoid(x) and essentially has a small wave just below 0, akin to GELU, to prevent dead neurons.

How should I normalize the initial weights of SiLU layers for them not to explode?

I tried LeCun-, Glorot- and Kaiming normalization. They all ended up exploding and getting 0% accuracy.



You must log in to answer this question.