5
$\begingroup$

I would like to know if it was possible to train a neural network on daily new data. Let me explain this more in detail. Let's say you have daily data from 2010 to 2019. You train your NN on all of it, but, from now on, every day in 2019 you get new data. Is it possible to "append" the training of the NN or do we need to retrain an entire NN with the data from $2010$ to $2019+n$ with $n$ the day for every new day?

I don't know if it is relevant but my work is on binary classification.

Improve this question
$\endgroup$
1
  • $\begingroup$ There is a similar question here, but it focuses on the case where new classes are introduced later. $\endgroup$
    – nbro
    Nov 10, 2020 at 10:26

1 Answer 1

Reset to default
5
$\begingroup$

Yes, this is possible. Continuously extending your training data is known as incremental learning.

You might also want to take a look at transfer learning, in which you reuse a trained model for a different purpose. This is very useful if you have a smaller dataset.

In your particular case, you could train a NN once using your data from 2010 to 2019 and use it as a base model. Every time you get new data, you can use transfer learning to slightly re-train this model. Based on parameters such as the number of epochs and the learning rate, you can determine how much of an impact this new data will have.

Improve this answer
$\endgroup$
3
  • $\begingroup$ This is interesting given that I tried to do it once with a for loop, when i = 0 I did a model.fit with the data from 2010 to 2019 and for i !=0 i just added the new data, but I had a problem with the learning when i != 0, even 100 epochs wasn't enough to make the NN change his prediction to make it correct Is it because I didn't do it properly with a save.model ? $\endgroup$
    – neomatriciel
    Jul 4, 2019 at 9:54
  • $\begingroup$ Or maybe because the i = 0's X_train has a shape like (4000,3) and the i != 0's X_train has a shape like (1,3), is that relevant ? $\endgroup$
    – neomatriciel
    Jul 4, 2019 at 18:17
  • 1
    $\begingroup$ There is not so much of a "transfer" if the learning is on the new data but of similar data type as the past data. Maybe "fine-tuning" is the term to use here. $\endgroup$
    – Bikash Gyawali
    Apr 1, 2021 at 11:08

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .