0
$\begingroup$

I'm fairly new to the field of deep learning and would like to ask which deep learning techniques can be used for anomaly detection in unlabeled data. For example, let's say I want to detect anomalous transactions within financial transaction data that shows the number of transactions and the amount in transactions. The data looks like this:

Transaction_ID        Count         Amount
     1241               9            425
     1563               7            354
     4456              142           45683
     7895              452           75358

As you can see, the data itself is not labeled and I'd like to use some deep learning techniques for anomaly detection within this data. I'd say a transaction is anomalous if it has a too large number in the Count or Amount columns like the third and last row.

My questions are:

  1. Do I need to label the data first to separate the normal transactions from the anomalous transactions before I apply any deep learning technique? For example, if I want to use AutoEncoder, I need to train the model with "normal" transaction data first. Then, how do I decide which data points are normal or not? Can I use another machine learning method like K-Means clustering to determine which transactions are normal? What are some other ways to differentiate normal transactions from anomalous transactions to feed the data into a deep learning model?

  2. I've tried the following approach: I labeled the data using K-means clustering first and built a classification model using Neural networks such as CNN, RNN, and LSTM. The labeled data has a new label column(1:anomalous, 0: normal) and I used this label as my "Y" in the deep learning model. I was able to get a high accuracy rate but I wonder if this approach has any issues!. Is it okay to use one machine learning method to label the data and feed that data into another deep learning model for classification? Or will bias from the first ML model be carried over into the second DL model?

  3. Besides what I've mentioned above, could you suggest any Deep Learning approach that is more suitable for this type of analysis?

$\endgroup$
1
  • $\begingroup$ Hello. Welcome to Artificial Intelligence SE. Ideally, a post should contain only 1 question. In this post, you're asking multiple questions. Although they are related, I would recommend that you edit your post to leave only one question, so that people can focus on that questions. If you have other questions, you can ask them in separate posts. $\endgroup$
    – nbro
    Oct 29 '21 at 10:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.