1
$\begingroup$

I am using LSTM network for predicting IOT time-series data receiving from un-reliable devices and networks.
This results in several multiple sections [continuous streak of bad data for several days until the problem is fixed].
I need to exclude this bad data section before feeding it to model training.
Since I am using LSTM-RNN network, it requires to do an un-roll data based on the previous records.

How can I properly exclude this bad data?
I thought of an approach as training model separately using each batch of good data, and use subsequent good-data batch for fine-tuning the model.
Please let me know if this is a good approach? or is there a better method?

example data:
"1-01",266.0
"1-02",145.9
"1-03",183.1
"1-04",0  [bad data]
"1-05",0  [bad data]
"1-06",0  [bad data]
"1-07",0  [bad data]
"1-08",224.5
"1-09",192.8
"1-10",122.9
"1-11",0  [bad data]
"1-12",0  [bad data]
"2-01",194.3
"2-02",149.5
$\endgroup$

1 Answer 1

1
$\begingroup$

Your idea is a good one.

Another idea is to upsample or aggregate your data. For example, average by week if you generally have a couple of missing days in every week.

A similar question on Stack Exchange: https://stats.stackexchange.com/questions/374935/how-to-deal-with-really-sparse-time-series-data-for-a-binary-classification-task

$\endgroup$
1

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .