I am beginner in deep learning and currently training a few neural networks (Pytorch) for problems in audio and speech. For my tasks, simple feed-forward networks are working well enough. I use basic layers like Linear, ReLU and Softmax with nll loss. I have tested a few initialization schemes provided by Pytorch and noticed that initialization has significant (but not high) effect in the speed of training and the final accuracy of the model. I am currently using torch.nn.init.kaiming_uniform_ for initialization.

In my understanding, all these are data independent initialization schemes. I would like to try something that is data dependent. I saw a few pre-training strategies with unsupervised learning followed by supervised learning, but they seem overly complicated.

I am looking for something simple where I can use (preferably a fraction of the) training data to 'tweak' weights to better positions before the training. Are there any such strategies?

Addendum-1: Current initialization schemes (AFAIK) are mostly random values with control over range or energy to prevent values from dying down or blowing up. My aim is to further improve the starting point of training by taking account training data (or at least some of it). I am thinking of something like this. We pass a few batches of training data through the initial network and collect statistics on neuron outputs. Based on this, we identify the misbehaving neurons and tweak the weights and biases to reduce such issues so as to improve the training speed or accuracy. Is there anything of that kind?

  • $\begingroup$ Hello. Can you clarify why you want to do this? Wouldn't this basically then bias the training process based on the data? Maybe you could use a separate dataset (from the training dataset) for initialising the weights: in this case, it would probably be less biased, but I still don't understand why you want to do this. $\endgroup$
    – nbro
    Sep 5 at 17:50
  • $\begingroup$ @nbro, kindly see Addendum-1. $\endgroup$
    – Axeon Thra
    Sep 5 at 22:14
  • $\begingroup$ Based on your addendum, there doesn't seem to be a big difference between what you seem to want and just training the neural network. $\endgroup$
    – nbro
    Sep 5 at 22:26
  • $\begingroup$ What you describe seems just like training (where tweaking problematic weights correspond to taking gradient descent steps) as @nbro has pointed out. Regardless, there's something called meta-learning, and it seems related to me. In meta-learning, you're trying to initialize weights in a way such that, the model can be adapted to different downstream tasks quickly. $\endgroup$
    – SpiderRico
    Sep 6 at 1:56
  • $\begingroup$ @nbro, The key difference is that we are not using back-propagation to minimize a 'differentiable' cost function. Instead, we can tweak the weights using a bunch of heuristics that may or may not be differentiable. $\endgroup$
    – Axeon Thra
    Sep 6 at 9:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.