1
$\begingroup$

I am using LSTM network for predicting IOT time-series data receiving from un-reliable devices and networks.
This results in several multiple sections [continuous streak of bad data for several days until the problem is fixed].
I need to exclude this bad data section before feeding it to model training.
Since I am using LSTM-RNN network, it requires to do an un-roll data based on the previous records.

How can I properly exclude this bad data?
I thought of an approach as training model separately using each batch of good data, and use subsequent good-data batch for fine-tuning the model.
Please let me know if this is a good approach? or is there a better method?

example data:
"1-01",266.0
"1-02",145.9
"1-03",183.1
"1-04",0  [bad data]
"1-05",0  [bad data]
"1-06",0  [bad data]
"1-07",0  [bad data]
"1-08",224.5
"1-09",192.8
"1-10",122.9
"1-11",0  [bad data]
"1-12",0  [bad data]
"2-01",194.3
"2-02",149.5
$\endgroup$
1
$\begingroup$

Your idea is a good one.

Another idea is to upsample or aggregate your data. For example, average by week if you generally have a couple of missing days in every week.

A similar question on Stack Exchange: https://stats.stackexchange.com/questions/374935/how-to-deal-with-really-sparse-time-series-data-for-a-binary-classification-task

| improve this answer | |
$\endgroup$
  • $\begingroup$ Thanks for the pointer $\endgroup$ – Remis Haroon Jul 6 at 14:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.