I have a device with an accelerometer and gyroscope (6-axis). The device sends live raw telemetry data to the model 40 samples for each input, 6 values per sample (accelerometer xyz, gyroscope xyz). The model predicts between 12 different labels of unique motions, the dataset has a size of 120k x 40 x 6, ~10k/label.
It's not a small dataset but at 10k/label it's also not too big. I have a Conv1D model that ends with a Softmax layer and it works extremely well. I make sure to shuffle my training/test dataset for more variance before splitting it 80:20.
I get 0.998 validation accuracy and 0.031 validation loss. And in live application it performs well.
However, at times (quite often) motions that are just noise for instance person walking around or just waving the device around is predicted as one of these 12 unique labels with high probability due to Softmax since that one happens to be the closest to the input data but is in fact very different.
How, do you deal with this ? One option I see it to just train the model with at 13th label being basically NOTHING and recording noise. But there has to be a better solution.
I have also tried sigmoid and binary crossentropy but the dataset is small for this kind of application and I only get about ~ 0.5/0.84 on validation loss/accuracy which is not really usable.
In this case I would have to drastically increase the dataset.