I am beginner in deep learning and currently training a few neural networks (Pytorch) for problems in audio and speech. For my tasks, simple feed-forward networks are working well enough. I use basic layers like Linear, ReLU and Softmax with nll loss. I have tested a few initialization schemes provided by Pytorch and noticed that initialization has significant (but not high) effect in the speed of training and the final accuracy of the model. I am currently using
torch.nn.init.kaiming_uniform_ for initialization.
In my understanding, all these are data independent initialization schemes. I would like to try something that is data dependent. I saw a few pre-training strategies with unsupervised learning followed by supervised learning, but they seem overly complicated.
I am looking for something simple where I can use (preferably a fraction of the) training data to 'tweak' weights to better positions before the training. Are there any such strategies?
Addendum-1: Current initialization schemes (AFAIK) are mostly random values with control over range or energy to prevent values from dying down or blowing up. My aim is to further improve the starting point of training by taking account training data (or at least some of it). I am thinking of something like this. We pass a few batches of training data through the initial network and collect statistics on neuron outputs. Based on this, we identify the misbehaving neurons and tweak the weights and biases to reduce such issues so as to improve the training speed or accuracy. Is there anything of that kind?