Questions tagged [anomaly-detection]

For questions related to anomaly detection (or outlier detection) algorithms, which is the identification of rare items, events or observations which raise suspicions by differing significantly from the majority of the data. There are unsupervised, supervised and semi-supervised anomaly detection algorithms.

Filter by
Sorted by
Tagged with
0 votes
0 answers
28 views

Which main steps should I consider in order to successfully use a VAE for Anomaly Detection?

I am thinking about using the variational autoencoder model for anomaly detection . I have an Android Logs dataset. As the logs generated are a representative of time series type of data I thought ...
MLenthusiast's user avatar
0 votes
0 answers
30 views

VAE ( variational autoencoder) for timeseries anomaly detection ,

I am implementing VAE based anomaly detection for multivariate timeseries using keras, I have ELBO (Evidence lower bound) which is combination of $$-\ D_{KL}\left({\ q}_\varphi\left(z\middle| x^i\...
mmv_87's user avatar
  • 1
0 votes
0 answers
13 views

Zero-shot out of distribution text classification

I'm building out a pipeline that would allow me to filter out text based on whether or not the text belongs to any of the classes I've defined. I feel like one (albeit naive) approach would simply be ...
mehsheenman's user avatar
0 votes
1 answer
51 views

How can I combine unsupervised learning with supervised learning?

I am currently using an isolation forest (from sklearn library) to detect anomalies in a data frame (basically it's a dynamic data frame more of a kind of time series I am. But I have certain criteria ...
SUNITA GUPTA's user avatar
1 vote
0 answers
32 views

Replicating conv autoencoder for anomaly detection, very blurry reconstructions

I’m trying to train an autoencoder on the hazelnut dataset of MVTec AD for reconstruction to detect anomalies. I’m am trying to replicate the results of this study: https://arxiv.org/pdf/2008.12977....
JeanMi's user avatar
  • 165
0 votes
1 answer
92 views

CNN how to measure the amount of FPS that can be processed?

This is my first question in the AI stack exchange. I want to ask about how to measure how many FPS can a CNN model process during real time detection. I am working on a real time detection system ...
Jo Sky's user avatar
  • 3
0 votes
0 answers
26 views

Can AI-written text detection be made more accurate if you know the prompt?

Usually, genAI detection is of the form: Input: text. Output: was it generated by AI? Thus far, AI-written text detection is terribly inaccurate, and if you're a user of r/ChatGPT, you've probably ...
Rebecca J. Stones's user avatar
1 vote
2 answers
736 views

How to detect outlier images?

Before I describe my challenge, I want to point out that I have searched extensively online for "outlier image detection", "anomaly images detection", etc., but all returned ...
pookie's user avatar
  • 121
0 votes
0 answers
18 views

Very high ACC (ca. 95%) with 1DConvNet for Time Series

Does this sound legit, for people working with CNN and Time Series? I have a Framework that applies Dynamic Tim Warping (DTW) on time series, using the DTW distance matrix, I cluster my data and ...
Skobo Do's user avatar
0 votes
0 answers
7 views

Surveys, Papers, Hand on Tutorials about training data generation for anomaly detection

I am searching for anything related to supervised, semi supervised or unsupervised anomaly detection w.r.t training data generation. I am looking toward reading any work that tackles the issue how to ...
Skobo Do's user avatar
0 votes
1 answer
150 views

How to interpret binary classification metrics on an imbalanced data set?

I have an imbalanced dataset on intrusion detection. I have (attack class) 3668045 samples and (benign class) 477 samples. I made a 70:30 Train test split. My problem is to predict whether the given ...
Zal's user avatar
  • 7
1 vote
0 answers
36 views

Pseudo Label Generation for Generative Cooperative Learning

I am trying to implement this paper for unsupervised video anomaly detection. The gist of the paper seems to be: Create a dataset for an unsupervised setting, by mixing up the train and anomalous ...
satan 29's user avatar
  • 141
0 votes
1 answer
51 views

How to handle anomaly detections with multiple different timeseries' from network traffic?

I would like to implement an anomaly detection algorithm on multiple timeseries' from different network users. Since each user has different behavior and network traffic usage, my question is how can ...
V.Hunon's user avatar
  • 131
0 votes
1 answer
44 views

How is it possible to detect anomalies in batches of 2 minutes of web access logs?

I have data coming from web access logs in the following form: ...
Kosmylo's user avatar
  • 161
2 votes
1 answer
522 views

How to calculate a meaningful distance between multidimensional tensors

TLDR: given two tensors $t_1$ and $t_2$, both with shape $(c,h,w),$ how shall the distance between them be measured? More Info: I'm working on a project in which I'm trying to distinguish between an ...
Hadar Sharvit's user avatar
1 vote
1 answer
205 views

ML model to predict timeouts

I am new to ML and am trying to build a model to predict timeouts for a website. The website is being monitored once a minute and the data consists of a timestamp and the response time in seconds. E.g....
gwolter's user avatar
  • 11
0 votes
0 answers
76 views

How can I weight each point in one-class SVM?

I want to give weights to some data points Specifically, these are points related to anomalies (I'm implementing one-class SVM for anomaly detection) Exactly, I want to consider some data points that ...
Dae-Young Park's user avatar
1 vote
0 answers
21 views

What is the best clustering method to detect anomalies for data with mostly categorical data?

I have a dataset with about 85 columns. Out of the 85 columns, 70+ are categorical. My goal is to identify the outliers in this dataset through clustering methods as I do not have a target column. ...
user13074756's user avatar
0 votes
1 answer
70 views

How to train a model for 1 image class to detect anomaly?

I want to train a model with python over the images, and these images are for a metal product. my aim is to detect the defects, to notice if a product is a failure. what kind of architecture do you ...
Sina M's user avatar
  • 1
5 votes
1 answer
5k views

What is the difference between out of distribution detection and anomaly detection?

I'm currently reading the paper Likelihood Ratios for Out-of-Distribution Detection, and it seems that their problem is very similar to the problem of anomaly detection. More precisely, given a neural ...
mhdadk's user avatar
  • 243
1 vote
0 answers
138 views

Object Detection as a means of Anomaly Detection

Is it possible to train an Object Detector (e.g. SSD), to detect when something is not in the image. Imagine an assembly line that transports some objects. Each object needs to have 5 screws. If the ...
oezguensi's user avatar
  • 205
0 votes
0 answers
19 views

How to classify anomalies between two sound datasets?

I have two sound datasets and each one has 80% normal and 20% anomalous data points. The first one is a rock song and the second one is a mellow indie song. I use half of the normal data as a baseline ...
user14361718's user avatar
1 vote
0 answers
83 views

How can a de-noising auto-encoder act as an anomaly detection model?

In some research papers, I have seen that, for training the autoencoders, instead of giving the non-anomalous input images, they add some anomalies to the normal input images, and train the auto-...
ans_ak's user avatar
  • 11
1 vote
0 answers
35 views

What models will you suggest to use in Industrial Anomaly Detection and Predictive analysis on live streamed data?

I have been working on industrial data, that is fed live, I want to explore a few models which might suit for this the best. The data are KPI data from the manufacturing Industry.
ana's user avatar
  • 131
3 votes
0 answers
32 views

Defect Detection System using Deep Learning

What is the general approach to defect detection in deep learning? Would the approach be better if we try to learn the positive images (defects in images) as much as possible or we try to learn the ...
user1538798's user avatar
1 vote
1 answer
97 views

Understanding the reconstruction loss in the paper "Anomaly Detection using Deep Learning based Image Completion"

I would like to implement the approach represented in this paper. Here they used following reconstruction loss: $$ L(X)= \frac{\lambda \cdot || M \odot (X - F(\overline{M} \odot X)) ||_{1} + (1 - \...
oezguensi's user avatar
  • 205
3 votes
1 answer
796 views

How can auto-encoders compute the reconstruction error for the new data?

Autoencoders are used for unsupervised anomaly detection by first learning the features of the data set with mainly "normal" data points. Then new data can be considered anomalous if the new ...
Brian's user avatar
  • 131
2 votes
0 answers
138 views

Application of Blockchain in Fraud detection in stock market

I want to develop a fraud detection application in the stock market Using Blockchain technology, we have some pattern that defines the anomaly for use of supervised machine learning but there is one ...
R1-'s user avatar
  • 137
4 votes
1 answer
189 views

Which unsupervised learning algorithm can be used for peaks detection?

So, I have a dataset that has around 1388 unique products and I have to do unsupervised learning on them in order to find anomalies (high/low peaks). The data below just represents one product. The <...
some_programmer's user avatar
8 votes
1 answer
263 views

Which unsupervised learning technique can be used for anomaly detection in a time series?

I've started working on anomaly detection in Python. My dataset is a time series one. The data is being collected by some sensors which record and collect data on semiconductor-making machines. My ...
some_programmer's user avatar
1 vote
1 answer
116 views

Are there any advantages of using rules-based approaches versus models for detecting spam?

Suppose that we have unlabeled data. That is, all we have are a collection of emails and want to determine whether any of them is spam or not. Let's say we have $1,000$ rules to determine whether a ...
rulesguy's user avatar
2 votes
0 answers
159 views

How to perform unsupervised anomaly detection from log file with mostly textual data?

I have a log file of the format, Index, Date, Timestamp, Module, App, Context, Session, Verbosity level, Description The log file can be considered as a master log, which consists of individual ...
Kraken10's user avatar
2 votes
2 answers
115 views

Which unsupervised anomaly detection algorithms are there?

I need to create model which will find suspicious entries or anomalies in a network, whose characteristics or features are the asset_id, ...
Abishek's user avatar
  • 33
5 votes
1 answer
132 views

Find anomalies from records of categorical data

I have a data-set with $m$ observations and $p$ categorical variables (nominal), each variable $X_1, X_2,\dots, X_p$ has several different possible values. Ultimately, I am looking for a way to find ...
bat's user avatar
  • 51