1
$\begingroup$

I'm a student and writing my first paper for submission on conference. I have a question

there is a dataset below. this is temporal-spatial dataset.

Date         Hour   City       Sensor1  Sensor2  Sensor3 Sensor4 ...
21-06-10     0      Region1      0.12     0.52    0.33     0.44  ...
21-06-10     1      Region2      0.16     0.83    0.34     0.49  ...
21-06-10     2      Region1      0.21     0.44    0.57     0.5   ...
...

My Task is anomaly detection for each region

I want to use LSTM. So, I represent the temporal-spatial data to two time-series data. my dataset can be represented below.

City       Date       Hour     Sensor1  Sensor2  Sensor3 Sensor4 ...
Region1   21-06-10     0         0.12     0.52    0.33     0.44  ...
Region1   21-06-10     2         0.21     0.44    0.57     0.5   ...
...


City       Date       Hour     Sensor1  Sensor2  Sensor3 Sensor4 ...
Region2   21-06-10     1         0.16     0.83    0.34     0.49  ...
...

However, then, there is no a row with attribute 'Hour=1' in Region1 dataset (you can see the table below)

City       Date       Hour     Sensor1  Sensor2  Sensor3 Sensor4 ...
Region1   21-06-10     0         0.12     0.52    0.33     0.44  ...
Region1   21-06-10     1         NaN      NaN     NaN      NaN   ...
Region1   21-06-10     2         0.21     0.44    0.57     0.5   ...
...

Can I insert estimated values into the row with attribute 'Hour=1' in Region1 dataset? (for example, I want to insert average between the first row and the third row)

Can I claim to have utilized a real world dataset even with this missing value estimation?

$\endgroup$

1 Answer 1

2
$\begingroup$

You can claim to use a real-world dataset, you would just need to specify that some values were interpolated.

Do you have to have the inter-mediate values though? By the looks of it, each "region" was only measured every 2 hours, so I would just keep it that way and just have the resolution be 2 hours. It doesn't have to be hourly, and probably shouldn't since that isn't the resolution of the data by the looks of it.

If it does need to be hourly then it is fine to just linearly interpolate the data. Additionally, you can try and train the network to accept empty inputs (though It'd definitely be easier to just interpolate your dataset)

$\endgroup$
4
  • $\begingroup$ Thank you for your explanation. According to your advice, Is it ok that I use my dataset with missing value estimation to submit a paper? Can I say that this is not data manipulation? $\endgroup$ Jul 13, 2021 at 1:39
  • $\begingroup$ @Dae-YoungPark I can't say for sure because I don't have the full dataset, but it doesn't look like there is any missing data. It just looks like the measurements were taken every 2 hours (for each region), which is fine. It's the same as measurements being taken every minute, just because there isn't data at every second doesn't mean there's missing data - it just means the resolution is minutely. $\endgroup$
    – Recessive
    Jul 13, 2021 at 2:15
  • $\begingroup$ The measurements is not taken every 2 hours. Actually, About Region1, there is a row with Hour=3. Thus, I think there are some missing values. In such a case, can I claim to use a real-world dataset even though some data are interpolated in data science area? $\endgroup$ Jul 13, 2021 at 2:34
  • $\begingroup$ If you're using real-world data then it is a real-world dataset. If you can still draw representative conclusions about the data, what difference does it make? $\endgroup$
    – Recessive
    Jul 13, 2021 at 7:25

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .