1
$\begingroup$

I have a device with an accelerometer and gyroscope (6-axis). The device sends live raw telemetry data to the model 40 samples for each input, 6 values per sample (accelerometer xyz, gyroscope xyz). The model predicts between 12 different labels of unique motions, the dataset has a size of 120k x 40 x 6, ~10k/label.

It's not a small dataset but at 10k/label it's also not too big. I have a Conv1D model that ends with a Softmax layer and it works extremely well. I make sure to shuffle my training/test dataset for more variance before splitting it 80:20.

I get 0.998 validation accuracy and 0.031 validation loss. And in live application it performs well.

However, at times (quite often) motions that are just noise for instance person walking around or just waving the device around is predicted as one of these 12 unique labels with high probability due to Softmax since that one happens to be the closest to the input data but is in fact very different.

How, do you deal with this ? One option I see it to just train the model with at 13th label being basically NOTHING and recording noise. But there has to be a better solution.

I have also tried sigmoid and binary crossentropy but the dataset is small for this kind of application and I only get about ~ 0.5/0.84 on validation loss/accuracy which is not really usable.

In this case I would have to drastically increase the dataset.

Improve this question
$\endgroup$

1 Answer 1

Reset to default
2
$\begingroup$

I don't fully agree with the other answer, mainly on point 1.

First thing first, adding a "nothing" label, means that you also have to gather new data to train that "nothing" neuron.

Instead, in my opinion, you want a density estimation filter at the beginning of your process, which would mean to have a 2 stage pipeline:

  1. check if the incoming data comes from your training dataset
  2. if so, classify it

The nice part of the density estimation, is that you don't need any additional data to train it, and the first model that comes in my mind for this, is VAE.

What you want to do with VAE is this:

  1. train the VAE
  2. do a grid search on which KL divergence it's better to use as a threshold
  3. deploy VAE and classifier, and use the VAE as a first stage filtering

In addition to this, the You can try to do preprocessing i don't really see how it could help.

About the You may also want to estimate your model's uncertainty... if you use the entropy as he is explaining on his blogpost, you are relying on the assumption that the model is well calibrated, which means that you should have a model that predicts a uniform distribution if confused... however, even though this is theoretically meaningful, a neural network might or might not do that.

For example, think about if you only have 2 samples, 1 per class, binary classification. Any boundary boundary between the two can have a 0 loss, however what you want, is to have something like a SVM boundary, so that the space is evenly split between the two points (which is not in any way guaranteed or even encouraged to be done by a NN)
Alternatively, you can use Bayesian NN, but they also have some practical biases when implemented, so they don't work that well...

You want probably the simple/most performing model uncertainty technique? Use ensembles

Improve this answer
$\endgroup$
6
  • $\begingroup$ I will give a shot to VAE now and let you know how it goes. I did try adding a noise dataset but the results were so-so. $\endgroup$
    – Sterling Duchess
    Aug 16 at 21:47
  • $\begingroup$ If I understand you correctly you are suggesting a two layered model approach. To train both a VAE and my Conv1D classification model. Then first pass the data to VAE to remove remove the noise or to try to construct one of the 12 unique gestures than pass the output of VAE to classification model for prediction ? $\endgroup$
    – Sterling Duchess
    Aug 16 at 23:00
  • $\begingroup$ @SterlingDuchess idk what you mean with "2 layer", usually it's called "2 stage approach"... the VAE is used to assess if it's an outlier (one of those random events that you don't really care) and then if it's not, to use a classifier to classify it $\endgroup$
    – Alberto
    Aug 16 at 23:06
  • $\begingroup$ I mean basically have two model definitions one VAE and my softmax model and first pass the input to VAE then to Softmax or ? Do you maybe have an example I never worked with autoencoders before. $\endgroup$
    – Sterling Duchess
    Aug 17 at 8:30
  • $\begingroup$ @SterlingDuchess look up "anomaly detection with VAE", and then the code is like if vae_encoder.predict(data).norm() > threshold: return classifier(data) else: None $\endgroup$
    – Alberto
    Aug 17 at 23:06

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .