Feature extraction timeseries, model compatibility

Question

I've got a timeseries with sensor data (e.g. accelerometer and gyroscope). I now want to extract the activity out of it (e.g. walking, standing, driving, ...). I Followed this Jupyter Notebook. But there are some issues left.

1. Why do they only pick 500 rows?
2. What's the point of re-arranging the rows/columns?
3. When they build their decicion tree learner with the train data, they build it upon extracted features. But how can we then use this tree for new sensor data? Should we extract the features of the new data and pass it as input for the tree? But new sensor data might not have as many features as the train data. Eg: (ValueError: Number of features of the model must match the input. Model n_features is 321 and input n_features is 312)

Answer 1

there's a lot to un-pack in this question.

Why do they only pick 500 rows?

my guess: in order to keep the example running quickly. tsfresh usually takes a while to calculate its features. note that when they evaluated their model, they took the last 500 samples.

What's the point of re-arranging the rows/columns?

answer: the data frame format that tsfresh requires in order to calculate features is that format. it is a bit of a pain...especially when you need to keep track of an id-column for other data.

When they build their decicion tree learner with the train data, they build it upon extracted features.

answer: yes

But how can we then use this tree for new sensor data? Should we extract the features of the new data and pass it as input for the tree?

answer: yes

But new sensor data might not have as many features as the train data. Eg: (ValueError: Number of features of the model must match the input. Model n_features is 321 and input n_features is 312)

answer: yes it will. I don't know where your copy-pasted error message came from. When you generate a set of features using tsfresh, you can do it in a couple of different ways. you can generate all of them or you can generate a subset of them---they generated a subset...but then subsetted it once again(using their importance stuff...whether you use the importance methods or other methods to select features you will end up with a bunch of features you need to calculate). If you follow the procedure for generating features based on a pre-determined subset (relevant_features in their case), you will end up with the same # of features. This needs to be stressed...don't generate tsfeatures that you don't need! as it will take foooorever.