Questions tagged [anomaly-detection]

For questions related to anomaly detection (or outlier detection) algorithms, which is the identification of rare items, events or observations which raise suspicions by differing significantly from the majority of the data. There are unsupervised, supervised and semi-supervised anomaly detection algorithms.

Filter by
Sorted by
Tagged with
0
votes
0answers
13 views

Weight for Samples on SVM (Support Vector Machine)

There is a option sample_weight in fit(X[, y, sample_weight]) function (OneClassSVM, sklearn library). If I use the option ...
0
votes
0answers
17 views

How can I weight each point in one-class SVM?

I want to give weights to some data points Specifically, these are points related to anomalies (I'm implementing one-class SVM for anomaly detection) Exactly, I want to consider some data points that ...
0
votes
0answers
14 views

Determining the value of error threshold in summarization of differences in multidimensional aggregates

I have been implementing an OLAP related journal "iDiff : Informative summarization of differences in multidimensional aggregates". In this paper, The author have proposed a methodology ...
0
votes
0answers
12 views

Evaluation metric for time-series anomaly detection

My dataset is time-series sensor data and anomaly ratio is between 5% and 6% 1. For time-series anomaly detection evaluation, which one is better, precision/recall/F1 or ROC-AUC ? When empirically ...
1
vote
0answers
14 views

What is the best clustering method to detect anomalies for data with mostly categorical data?

I have a dataset with about 85 columns. Out of the 85 columns, 70+ are categorical. My goal is to identify the outliers in this dataset through clustering methods as I do not have a target column. ...
-1
votes
1answer
27 views

error while fitting auto encoder for anomaly detection on sensor data [closed]

I have sensor data like this and it's unsupervised and it has almost 800 features. my use-case is for anomaly detection. I applied Principal component analysis (PCA) for dimensionality reduction. now ...
0
votes
1answer
25 views

How to train a model for 1 image class to detect anomaly?

I want to train a model with python over the images, and these images are for a metal product. my aim is to detect the defects, to notice if a product is a failure. what kind of architecture do you ...
0
votes
0answers
21 views

How to identify segment/object that is anomaly using computer vision

I have an OpenCV script that can pull out the shape of different objects in an image. For example, here is the output from one script where it calculated the outline of 4 different shapes[ and here is ...
3
votes
1answer
590 views

What is the difference between out of distribution detection and anomaly detection?

I'm currently reading the paper Likelihood Ratios for Out-of-Distribution Detection, and it seems that their problem is very similar to the problem of anomaly detection. More precisely, given a neural ...
0
votes
0answers
68 views

How to compare multiple one-class variational autoencoders?

I have trained multiple one-class vanilla variational autoencoders that each learn the distribution of one class and have the same architecture. The classes are mostly discrete, but there are several ...
1
vote
0answers
19 views

Object Detection as a means of Anomaly Detection

Is it possible to train an Object Detector (e.g. SSD), to detect when something is not in the image. Imagine an assembly line that transports some objects. Each object needs to have 5 screws. If the ...
0
votes
0answers
16 views

How to classify anomalies between two sound datasets?

I have two sound datasets and each one has 80% normal and 20% anomalous data points. The first one is a rock song and the second one is a mellow indie song. I use half of the normal data as a baseline ...
0
votes
0answers
53 views

What are some scalable approaches to perform anomaly detection (for images with small cracks) with unsupervised learning?

I have some images with anomalies, like small cracks, but it's an imbalanced dataset. Please, suggest some effective scalable approaches. Should I consider convolutional auto-encoders? It's supposed ...
1
vote
0answers
77 views

How can a de-noising auto-encoder act as an anomaly detection model?

In some research papers, I have seen that, for training the autoencoders, instead of giving the non-anomalous input images, they add some anomalies to the normal input images, and train the auto-...
1
vote
0answers
29 views

What models will you suggest to use in Industrial Anomaly Detection and Predictive analysis on live streamed data?

I have been working on industrial data, that is fed live, I want to explore a few models which might suit for this the best. The data are KPI data from the manufacturing Industry.
2
votes
0answers
143 views

Is it possible to use entity embedding with autoencoder for anomaly detetction?

I'm trying to build autoencoder in keras in order to detect anomalies. However, most of the data is categorical and I have to encode it. When it comes to production, categorical features can take new ...
3
votes
0answers
26 views

Defect Detection System using Deep Learning

What is the general approach to defect detection in deep learning? Would the approach be better if we try to learn the positive images (defects in images) as much as possible or we try to learn the ...
1
vote
1answer
81 views

Understanding the reconstruction loss in the paper "Anomaly Detection using Deep Learning based Image Completion"

I would like to implement the approach represented in this paper. Here they used following reconstruction loss: $$ L(X)= \frac{\lambda \cdot || M \odot (X - F(\overline{M} \odot X)) ||_{1} + (1 - \...
3
votes
1answer
265 views

How can auto-encoders compute the reconstruction error for the new data?

Autoencoders are used for unsupervised anomaly detection by first learning the features of the data set with mainly "normal" data points. Then new data can be considered anomalous if the new ...
2
votes
0answers
112 views

Application of Blockchain in Fraud detection in stock market

I want to develop a fraud detection application in the stock market Using Blockchain technology, we have some pattern that defines the anomaly for use of supervised machine learning but there is one ...
4
votes
1answer
78 views

Which unsupervised learning algorithm can be used for peaks detection?

So, I have a dataset that has around 1388 unique products and I have to do unsupervised learning on them in order to find anomalies (high/low peaks). The data below just represents one product. The <...
8
votes
1answer
190 views

Which unsupervised learning technique can be used for anomaly detection in a time series?

I've started working on anomaly detection in Python. My dataset is a time series one. The data is being collected by some sensors which record and collect data on semiconductor-making machines. My ...
1
vote
1answer
72 views

Are there any advantages of using rules-based approaches versus models for detecting spam?

Suppose that we have unlabeled data. That is, all we have are a collection of emails and want to determine whether any of them is spam or not. Let's say we have $1,000$ rules to determine whether a ...
2
votes
0answers
121 views

How to perform unsupervised anomaly detection from log file with mostly textual data?

I have a log file of the format, Index, Date, Timestamp, Module, App, Context, Session, Verbosity level, Description The log file can be considered as a master log, which consists of individual ...
2
votes
2answers
87 views

Which unsupervised anomaly detection algorithms are there?

I need to create model which will find suspicious entries or anomalies in a network, whose characteristics or features are the asset_id, ...
5
votes
1answer
105 views

Find anomalies from records of categorical data

I have a data-set with $m$ observations and $p$ categorical variables (nominal), each variable $X_1, X_2,\dots, X_p$ has several different possible values. Ultimately, I am looking for a way to find ...
11
votes
1answer
19k views

How can I train an AI to find known and new patterns in a log file? [closed]

I'm developing an AI tool to find known equipment errors and find new patterns of failure. This log file is time-based and has known messages (information and error). I'm using a JavaScript library, ...