It exists networks built to learn how to differentiate between classes even if there are looking quite the same. Usually, a [triplet loss][1] is used in those networks to learn the difference between the target, a positive sample, and a negative one.

For example, those networks are used to perform identity check with face images, the algorithm learns the differences between different people instead of recognizing people.

Here are some keywords that are possibly relevant: discriminative function, triplet loss, siamese network, one-shot learning.

Theses papers are interesting:

- [FaceNet: A Unified Embedding for Face Recognition and Clustering][2]
- [Dimensionality Reduction by Learning an Invariant Mapping][3]




  [1]: https://towardsdatascience.com/siamese-network-triplet-loss-b4ca82c1aec8
  [2]: http://arxiv.org/abs/1503.03832
  [3]: http://yann.lecun.com/exdb/publis/pdf/hadsell-chopra-lecun-06.pdf