Questions tagged [anomaly-detection]
For questions related to anomaly detection (or outlier detection) algorithms, which is the identification of rare items, events or observations which raise suspicions by differing significantly from the majority of the data. There are unsupervised, supervised and semi-supervised anomaly detection algorithms.
30
questions
1
vote
2
answers
50
views
How to detect outlier images?
Before I describe my challenge, I want to point out that I have searched extensively online for "outlier image detection", "anomaly images detection", etc., but all returned ...
- 121
0
votes
0
answers
28
views
ML Network Traffic Classification Problem
I am hoping for suggestions or advice as to whether ML offers a suitable solution to the below problem. I am not so familiar with ML techniques so apologies if this is a straight forward question.
I ...
- 1
0
votes
0
answers
14
views
How many opposing users should be recorded exterior to the average data before being combined?
Outside of the programming toward AI, I am having difficulty putting together a plan on how this machine I hope to build would work.
The basic question is: How should it handle user reviews / ...
- 101
0
votes
0
answers
17
views
Very high ACC (ca. 95%) with 1DConvNet for Time Series
Does this sound legit, for people working with CNN and Time Series?
I have a Framework that applies Dynamic Tim Warping (DTW) on time series, using the DTW distance matrix, I cluster my data and ...
- 51
0
votes
0
answers
7
views
Surveys, Papers, Hand on Tutorials about training data generation for anomaly detection
I am searching for anything related to supervised, semi supervised or unsupervised anomaly detection w.r.t training data generation.
I am looking toward reading any work that tackles the issue how to ...
- 51
0
votes
1
answer
120
views
How to interpret binary classification metrics on an imbalanced data set?
I have an imbalanced dataset on intrusion detection. I have (attack class) 3668045 samples and (benign class) 477 samples. I made a 70:30 Train test split. My problem is to predict whether the given ...
- 7
1
vote
0
answers
28
views
Pseudo Label Generation for Generative Cooperative Learning
I am trying to implement this paper for unsupervised video anomaly detection.
The gist of the paper seems to be:
Create a dataset for an unsupervised setting, by mixing up the train and anomalous ...
- 141
0
votes
1
answer
30
views
How to handle anomaly detections with multiple different timeseries' from network traffic?
I would like to implement an anomaly detection algorithm on multiple timeseries' from different network users. Since each user has different behavior and network traffic usage, my question is how can ...
- 131
0
votes
1
answer
27
views
How is it possible to detect anomalies in batches of 2 minutes of web access logs?
I have data coming from web access logs in the following form:
...
- 161
2
votes
1
answer
305
views
How to calculate a meaningful distance between multidimensional tensors
TLDR: given two tensors $t_1$ and $t_2$, both with shape $(c,h,w),$ how shall the distance between them be measured?
More Info: I'm working on a project in which I'm trying to distinguish between an ...
- 351
1
vote
1
answer
117
views
ML model to predict timeouts
I am new to ML and am trying to build a model to predict timeouts for a website.
The website is being monitored once a minute and the data consists of a timestamp and the response time in seconds. E.g....
- 11
0
votes
0
answers
16
views
Unsupervised methodologies to detect collective anomalies in transaction data
I am researching various methodologies to detect collective anomalies in transactions data.
I have seen some supervised approaches, but not the unsupervised ones.
Please share any resources or ...
- 101
0
votes
0
answers
56
views
How can I weight each point in one-class SVM?
I want to give weights to some data points
Specifically, these are points related to anomalies
(I'm implementing one-class SVM for anomaly detection)
Exactly, I want to consider some data points that ...
1
vote
0
answers
19
views
What is the best clustering method to detect anomalies for data with mostly categorical data?
I have a dataset with about 85 columns. Out of the 85 columns, 70+ are categorical. My goal is to identify the outliers in this dataset through clustering methods as I do not have a target column.
...
- 111
0
votes
1
answer
48
views
How to train a model for 1 image class to detect anomaly?
I want to train a model with python over the images, and these images are for a metal product.
my aim is to detect the defects, to notice if a product is a failure.
what kind of architecture do you ...
- 1
3
votes
1
answer
3k
views
What is the difference between out of distribution detection and anomaly detection?
I'm currently reading the paper Likelihood Ratios for Out-of-Distribution Detection, and it seems that their problem is very similar to the problem of anomaly detection. More precisely, given a neural ...
- 223
1
vote
0
answers
77
views
Object Detection as a means of Anomaly Detection
Is it possible to train an Object Detector (e.g. SSD), to detect when something is not in the image. Imagine an assembly line that transports some objects. Each object needs to have 5 screws. If the ...
- 205
0
votes
0
answers
19
views
How to classify anomalies between two sound datasets?
I have two sound datasets and each one has 80% normal and 20% anomalous data points. The first one is a rock song and the second one is a mellow indie song. I use half of the normal data as a baseline ...
1
vote
0
answers
81
views
How can a de-noising auto-encoder act as an anomaly detection model?
In some research papers, I have seen that, for training the autoencoders, instead of giving the non-anomalous input images, they add some anomalies to the normal input images, and train the auto-...
- 11
1
vote
0
answers
32
views
What models will you suggest to use in Industrial Anomaly Detection and Predictive analysis on live streamed data?
I have been working on industrial data, that is fed live, I want to explore a few models which might suit for this the best.
The data are KPI data from the manufacturing Industry.
- 31
3
votes
0
answers
32
views
Defect Detection System using Deep Learning
What is the general approach to defect detection in deep learning?
Would the approach be better if we try to learn the positive images (defects in images) as much as possible or we try to learn the ...
- 131
1
vote
1
answer
95
views
Understanding the reconstruction loss in the paper "Anomaly Detection using Deep Learning based Image Completion"
I would like to implement the approach represented in this paper. Here they used following reconstruction loss:
$$
L(X)= \frac{\lambda \cdot || M \odot (X - F(\overline{M} \odot X)) ||_{1} + (1 - \...
- 205
3
votes
1
answer
575
views
How can auto-encoders compute the reconstruction error for the new data?
Autoencoders are used for unsupervised anomaly detection by first learning the features of the data set with mainly "normal" data points. Then new data can be considered anomalous if the new ...
- 131
2
votes
0
answers
125
views
Application of Blockchain in Fraud detection in stock market
I want to develop a fraud detection application in the stock market Using Blockchain technology, we have some pattern that defines the anomaly for use of supervised machine learning but there is one ...
- 121
4
votes
1
answer
153
views
Which unsupervised learning algorithm can be used for peaks detection?
So, I have a dataset that has around 1388 unique products and I have to do unsupervised learning on them in order to find anomalies (high/low peaks).
The data below just represents one product. The <...
- 225
8
votes
1
answer
242
views
Which unsupervised learning technique can be used for anomaly detection in a time series?
I've started working on anomaly detection in Python. My dataset is a time series one. The data is being collected by some sensors which record and collect data on semiconductor-making machines.
My ...
- 225
1
vote
1
answer
100
views
Are there any advantages of using rules-based approaches versus models for detecting spam?
Suppose that we have unlabeled data. That is, all we have are a collection of emails and want to determine whether any of them is spam or not. Let's say we have $1,000$ rules to determine whether a ...
- 11
2
votes
0
answers
147
views
How to perform unsupervised anomaly detection from log file with mostly textual data?
I have a log file of the format,
Index, Date, Timestamp, Module, App, Context, Session, Verbosity level, Description
The log file can be considered as a master log, which consists of individual ...
- 21
2
votes
2
answers
105
views
Which unsupervised anomaly detection algorithms are there?
I need to create model which will find suspicious entries or anomalies in a network, whose characteristics or features are the asset_id, ...
- 33
5
votes
1
answer
127
views
Find anomalies from records of categorical data
I have a data-set with $m$ observations and $p$ categorical variables (nominal), each variable $X_1, X_2,\dots, X_p$ has several different possible values.
Ultimately, I am looking for a way to find ...
- 51