I want to create a neural network and train it on some data, however I want to be able to create a new model without retraining it from the start.

An example, I have 1000 data points in my training data

  1. model - trained on 0-99
  2. model - trained on 1-100
  3. model - trained on 2-101
  4. and so forth

So I'm wondering if I can use the first model to train the second model, essentially forgetting the first data point.

You can view it as a sliding window over the 1000 data points, sliding one data point to the right for each new model.

Does it make sense? Is there any easy way to solve this problem?

  • 2
    $\begingroup$ As usually the main goal of training an NN is to generalise a function from example data, could you clarify what you mean by "forgetting the first data point"? Are you wanting to purposefully overfit so that predictions against the first data point are no better than random? If not, do you have any criteria for when the training for any specific model e.g. "model 2 trained on 1-100" is complete, or whether its results are acceptable to you? It is clearly possible to take model 1 and re-train it, but what is missing in the question is the goal for doing so . . . and that affects the answer here $\endgroup$ – Neil Slater Apr 10 '18 at 22:00
  • $\begingroup$ The idea is that the first data point is not relevant any more. I dont want overfit, so imagine continuous addition of data points. For a given situation, only the 100 most recent data points are relevant. When a data point is added, the most recent model is trained on a data point that is (in theory) not relevant any more. $\endgroup$ – norflow Apr 10 '18 at 22:15
  • $\begingroup$ Adding to my comment, essentially Im asking if it is possible for a model to only consider the most recent data points, without retraining the model from the bottom on the most recent data - only partially traning it in some way, making it "forget" the no longer relevant data form the previous training set. $\endgroup$ – norflow Apr 10 '18 at 23:16
  • $\begingroup$ If you're afraid of overfitting, there are much better ways, like dropout, l2 regularisation and just stochastic batching. $\endgroup$ – Andreas Storvik Strauman Apr 11 '18 at 7:03
  • $\begingroup$ Actually, your method looks a lot like cross validation (not the SE site, but the actual algorithm) that excludes some of the data for each iteration. $\endgroup$ – Andreas Storvik Strauman Apr 11 '18 at 7:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.