I would like to know if it was possible to train a neural network on daily new data. Let me explain this more in detail. Let's say you have daily data from 2010 to 2019. You train your NN on all of it, but, from now on, every day in 2019 you get new data. Is it possible to "append" the training of the NN or do we need to retrain an entire NN with the data from $2010$ to $2019+n$ with $n$ the day for every new day?

I don't know if it is relevant but my work is on binary classification.

  • $\begingroup$ There is a similar question here, but it focuses on the case where new classes are introduced later. $\endgroup$
    – nbro
    Nov 10 '20 at 10:26

Yes, this is possible. Continuously extending your training data is known as incremental learning.

You might also want to take a look at transfer learning, in which you reuse a trained model for a different purpose. This is very useful if you have a smaller dataset.

In your particular case, you could train a NN once using your data from 2010 to 2019 and use it as a base model. Every time you get new data, you can use transfer learning to slightly re-train this model. Based on parameters such as the number of epochs and the learning rate, you can determine how much of an impact this new data will have.

  • $\begingroup$ This is interesting given that I tried to do it once with a for loop, when i = 0 I did a model.fit with the data from 2010 to 2019 and for i !=0 i just added the new data, but I had a problem with the learning when i != 0, even 100 epochs wasn't enough to make the NN change his prediction to make it correct Is it because I didn't do it properly with a save.model ? $\endgroup$ Jul 4 '19 at 9:54
  • $\begingroup$ Or maybe because the i = 0's X_train has a shape like (4000,3) and the i != 0's X_train has a shape like (1,3), is that relevant ? $\endgroup$ Jul 4 '19 at 18:17
  • 1
    $\begingroup$ There is not so much of a "transfer" if the learning is on the new data but of similar data type as the past data. Maybe "fine-tuning" is the term to use here. $\endgroup$ Apr 1 '21 at 11:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.