A technique of aggregating many input images to a single representation of the relevant features within

Question

I have a few thousand images and I would like to generate a representation of the foreground patterns within - a composition of all of its features, so to speak. In simple terms: take 10000 images of a dog and then draw the archetypical dog.

Does this task have a name, and is there a method out there specifically for such purposes?

The images have different sizes and neither scale nor rotation invariant, so simple averaging algorithms wouldn't work. I would guess that deep learning techniques could be capable - e.g., extracting the features from the first layers of a neural network - as hinted at here: ..."The original network can't be used to classify new identities, on which it wasn't trained. But, the kth layer may provide a good representation of faces in general....".

I just don't necessarily need a model for prediction afterwards, just the aggregate representation will do.

Answer 1

This idea is more or less what driven the development of Google Deep Dream.

They framed the problem in a much different way though, i.e. as a reversed task compared to classification. So given a trained network, we choose a final activation, and we modify a given input image till the chosen activation is maximized trough gradient ascent (n.b. not gradient descent).

You don't specify your definition of "archetypical" so maybe that's not exactly what you're looking for, but conceptually this is more sounded than combining feature maps. Reason being that a feature map is just an image modified with some filters, so the average of a feature maps is basically the same as the average of images themselves. Deep dream instead modify an image in such a way that this image will react to specific, already learned filters. So it's a generative approach. You can see it in the example below, where a sky image is modify to produce a "face" activation. All those faces are not training images nor part of the original sky image.

Note that if you're looking for a model that given thousands of images produce always the same output one, then the problem is simply ill posed. Even unsupervised energy based models like restricted boltzmann machines simply learn a mixture of features, not a single "archetypical" image (see pic below).