I noticed that there are many studies in recent years on how to train/update neural networks faster/quicker with equal or better performance. I find the following methods(except the chips arms race):

  1. using few-shot learning, for instance, pre-taining and etc.
  2. using the minimum viable dataset, for instance using (guided) progressive sampling.
  3. model compression, for instance, efficent transformers
  4. Data echoing, or simply put let the data pass multiple times in the graph(or GPU)

Is there a systematic structure on this topic and how can we update or train a model faster without loss of its capacity?

  • $\begingroup$ use validation set maybe? $\endgroup$
    – Alex
    Jan 16 '21 at 11:27
  • $\begingroup$ @Alex What do you mean by validation set? I cannot get your point, sorry. Could you please flesh your comment out into an answer? $\endgroup$ Jan 16 '21 at 11:33
  • $\begingroup$ Use val data to check if you are overfitting (in addition to other tricks), this will prevent the degradation of the model's performance $\endgroup$
    – Alex
    Jan 16 '21 at 11:40
  • $\begingroup$ @Alex Yes, but let's consider the model that takes N epochs to get the best performance over the develop set, can we speed the training by other methods? Make the N smaller than that for the original model or less time? For instance by reducing the amount of training data or compressing the model size? $\endgroup$ Jan 16 '21 at 11:43
  • $\begingroup$ what do you mean by 'compressing the model size'? $\endgroup$
    – Alex
    Jan 16 '21 at 11:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.