2
$\begingroup$

I want to create a neural network and train it on some data, however I want to be able to create a new model without retraining it from the start.

An example, I have 1000 data points in my training data

  1. model - trained on 0-99
  2. model - trained on 1-100
  3. model - trained on 2-101
  4. and so forth

So I'm wondering if I can use the first model to train the second model, essentially forgetting the first data point.

You can view it as a sliding window over the 1000 data points, sliding one data point to the right for each new model.

Does it make sense? Is there any easy way to solve this problem?

Improve this question
$\endgroup$
9
  • 2
    $\begingroup$ As usually the main goal of training an NN is to generalise a function from example data, could you clarify what you mean by "forgetting the first data point"? Are you wanting to purposefully overfit so that predictions against the first data point are no better than random? If not, do you have any criteria for when the training for any specific model e.g. "model 2 trained on 1-100" is complete, or whether its results are acceptable to you? It is clearly possible to take model 1 and re-train it, but what is missing in the question is the goal for doing so . . . and that affects the answer here $\endgroup$
    – Neil Slater
    Apr 10, 2018 at 22:00
  • $\begingroup$ The idea is that the first data point is not relevant any more. I dont want overfit, so imagine continuous addition of data points. For a given situation, only the 100 most recent data points are relevant. When a data point is added, the most recent model is trained on a data point that is (in theory) not relevant any more. $\endgroup$
    – norflow
    Apr 10, 2018 at 22:15
  • $\begingroup$ Adding to my comment, essentially Im asking if it is possible for a model to only consider the most recent data points, without retraining the model from the bottom on the most recent data - only partially traning it in some way, making it "forget" the no longer relevant data form the previous training set. $\endgroup$
    – norflow
    Apr 10, 2018 at 23:16
  • $\begingroup$ If you're afraid of overfitting, there are much better ways, like dropout, l2 regularisation and just stochastic batching. $\endgroup$
    – Andreas Storvik Strauman
    Apr 11, 2018 at 7:03
  • $\begingroup$ Actually, your method looks a lot like cross validation (not the SE site, but the actual algorithm) that excludes some of the data for each iteration. $\endgroup$
    – Andreas Storvik Strauman
    Apr 11, 2018 at 7:06

1 Answer 1

Reset to default
0
$\begingroup$

tl;dr Yes.

Question: Can I train starting with a trained network instead of from a newly initialized network.

Answer: You can, and should, save network weights when it is done training. You can also retain it in memory after writing a copy of it to disc. If you then train the network some more, you can save it again to a different filename. You can repeat this for the set of inputs.

At the end of the process you would have only one network in memory, you would have several files of "snapshots" of the weights, and each snapshot would be the result of training on a different parcel of the total data.

Improve this answer
$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .