How to label unsupervised data for deep learning multi-classification

Question

I have unlabeled credit card transaction data that has the following columns:

Transaction_ID     Frequency       Amount    Fees
   192831             21            829       23
   382912             14            920       24
   483921            839           24059      87

Eventually, I'd like to build a deep learning model(e.g. LSTM) that can tell me whether a transaction(row) has a "high", "moderate", or "low" risk. However, since the data is unlabeled, I believe I need to label the data first before I feed the data into the deep learning model.

For example, transactions that have small frequency and amount values like the first two rows need to be labeled as "low (0)" while transactions that have large frequency and amount like the last row should be labeled as "high (2)". If both frequency and amount have moderate values, the row will be labeled as "moderate(1)".

I wonder if it is okay to use other machine learning techniques such as K-Means clustering to label the data before I feed the data into the deep learning model. Is it okay to use one Machine Learning algorithm (K-means) to label the data and feed the same labeled data into another Deep Learning model (LSTM)? Or is it a bad practice? For example, if the first model (K-means) is biased, will that bias(error) be carried over from the first model to the second model (LSTM)?

If it is a bad practice to use two different ML technologies, what else can I do to label the data?

Answer 1

"is it okay to use another machine learning technology such as K-Means clustering to label the data?"

In computer vision there's an entire branch called automatic image annotation dedicated to this topic. And after a 2 sec search online I found a tutorial that suggest precisely what you want to try. So yes, on the paper it's ok to try, the real question though should be:

Will it work?

And the unfortunate answer is: no. Or to be less harsh: only with toy datasets, not real ones.

Moreover take a step back and think about the possible outcomes of the approach and their implications. If you train a k-mean clustering on your data you have 2 possible scenarios:

• it works, then you you have an unsupervised model that does the job, why bothering training a supervise model that at best will perform as good as the one you already have?
• it doesn't work, you made your life harder and you're at the same point where you started.

what else can I do to label the data?

Even though it's not the answer you want to hear:

Labeling is like removing a plaster from a wounded cut, you spend hours thinking how to do it without suffering to just realize at the end that the best way is to just do it.

Answers to the most predictable counter argument, the data are too many:

• Rely on online crowdsourcing platforms: it seems that you already have rules to use as an annotation guideline. But it costs money.
• Label just a part of the data, train, and repeat till you get a good model. You can also leverage active learning to optimize and make the labeling process smarter.
• Dig the web till you find an annotated dataset.
• Move to unsupervised learning, but for the training itself. At the very least you'll rely on theories and model designed precisely for unlabelled data without having to train several models with a cascade of performance drops.

As a final note, there are several reasons to just start labeling the data yourself, the main one being that you'll have control over the quality of the annotations. Many people underestimate or don't even know about the importance of inter annotators agreement metrics like Cohen's Kappa that estimate not only the quality of a dataset but also how hard for a human the task you want to automatize is. Which is extremely relevant, cause if for humans the best agreement score is 80% instead of 100% then there's no way you'll obtain more than that from a model, due to the personal biases introduced by the annotators in the data. And by knowing that, you'll be happy about your 0.8 f-score without wasting hours thinking what you did wrong and why the model is not performing better. Hope this will be be somehow helpful and good luck with your data!