Before I describe my challenge, I want to point out that I have searched extensively online for "outlier image detection", "anomaly images detection", etc., but all returned results are about finding anomalies or outliers within an image (e.g., a defect in a machined part, a tear in fabric, etc.).

This is not what I am looking for: I am looking to identify images (whole images) that depict an object that is different from the other images. For instance, I have a stream of images from a live camera. Each image depicts a single dog (different breeds) and each image is centered on the dog. A few images depict a cat, rather than a dog... I would like to detect these cat images (outliers).

This is not a simple classification problem, where I have a labeled data set of cats and dogs... because I don't want to define the 'outlier'. In other words, the approach must be adaptable to the data: if I were to suddenly add a few images of trash, the model must flag those trash images as novel/outliers because in comparison to the number of dog images, trash images (like cat images) are pretty rare (or new).

Would something like sequential k-means work for this problem? I thought about sequential k-means, since it operates on a data stream and will adapt to the number of clusters (i.e, k does not need to be pre-defined).

What other approaches would work? Any resources (e.g., links to tutorials, GitHub repositories, specific algorithms, methodologies) that might help?


2 Answers 2


Hi there @pookie you can approach this problem using unsupervised anomaly detection techniques. One such technique is to use an autoencoder neural network.

The idea is to train an autoencoder on a large set of dog images, and then use the trained network to encode (compress) each image into a lower-dimensional representation. The encoded representations should capture the key features of dog images. The network can then be used to reconstruct the original image from the encoded representation.

Now, when a new image (e.g., a cat image or a trash image) is presented, the network will not be able to reconstruct it well because it has not seen that kind of image before. The reconstruction error will be higher for the novel image, which can be used to identify it as an outlier.

You can use this approach to build a threshold-based anomaly detection system, where any image with a reconstruction error above a certain threshold is considered an outlier. Alternatively, you can use a more sophisticated anomaly detection algorithm like One-class SVM, Isolation Forest, or Local Outlier Factor.

Another option is to use clustering techniques like k-means or DBSCAN to cluster the dog images and identify clusters with very few points as outliers. However, this approach may not work well if the outlier images are visually similar to dog images.

Here are some resources that may help you get started:

If you found this helpful, don't forget to upvote :)

  • $\begingroup$ Thank you for the response -- very helpful! The autoencoder approach sounds like the right way to go. The autoencoder link you provided is broken, though. Also, do you know of any good (and relevant) OC-SVM tutorials? Also, welcome to AI SE! $\endgroup$
    – pookie
    Mar 12, 2023 at 18:14
  • $\begingroup$ You're welcome but I'm sorry to hear that the link I provided for autoencoder isn't working, here's another one [link] (machinelearningmastery.com/lstm-autoencoders). You may also consider taking an online course on machine learning or deep learning that covers autoencoders and anomaly detection: link. And yepp, here's a resource that covers the basics of OC-SVM and provide examples of how to use it for outlier detection link $\endgroup$
    – Triple S
    Mar 12, 2023 at 18:48
  • $\begingroup$ Thanks! Now, what if I want to update the model? For instance, after the model has seen loads of images of cats, cats should be considered "normal" (no longer an outlier). Can the model auto-update over time as it learns what is "normal" for the environment in which it is deployed? $\endgroup$
    – pookie
    Mar 12, 2023 at 19:05
  • $\begingroup$ Yes, u can update the model over time as it sees new data which is called online learning/incremental learning Ig, where the model is updated with new data as it arrives, rather than being trained on a fixed dataset. One approach is to use a variant of the algorithms like One-Class SVM or autoencoders, that support incremental learning. Eg. in case of One-Class SVM, you could use an algorithm such as online SVM or stochastic gradient descent SVM to update the model as new data arrives. $\endgroup$
    – Triple S
    Mar 13, 2023 at 7:44
  • $\begingroup$ Another approach is to use a streaming clustering algorithm such as CluStream or D-Stream, which can adapt to changing data distributions over time. The key is to manage the trade-off between stability and flexibility. U want the model to be stable enough to capture the underlying patterns in the data, but on the other hand, you also want it to be flexible enough to adapt to changes in the data distribution. The right balance depends on the specifics of ur problem and the nature of the data u are working with. $\endgroup$
    – Triple S
    Mar 13, 2023 at 7:45

I agree with Triple S' answer, but as a preprocessing step I suggest to use some pre-trained image classification network without the last fully-connected layer. This will give you robust features which represent characteristics of the input image. But since these representations are high-dimensional, I'd apply an autoencoder to these features.

Actually I have implemented and blogged about such approach, but I'm not sure whether links to personal blogs are discouraged or not. Well at least each answer should work as a stand-alone resource. In my case the images originated from videos, and I wanted to cluster either the frames themselves or video clips.

video processing pipeline

I used this for mountain biking videos, which resulted in this kind of 2D map:

clustered videos

Each line represents a different mountain biking / downhill clip. Anyway, in your case I'd use 2D coordinates for outlier detection. They seem to contain enough interesting data, and are so easy to visualize as well!

Link to my blog article: Image and video clustering with an autoencoder. It also has examples on clustering frames from either Arnold Schwarzeneg­ger or James Bond films. It could group conceptually similar images together in the 2D plane, from different films.


You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .