How can I train a neural network to describe the characteristics of a picture?

Question

I have collected a set of pictures of people with a text explaining the characteristics of the person on the picture, for example, "Big nose" or "Curly hair".

I want to train some type of model that takes in any picture and returns a description of the picture in terms of characteristics.

However, I have a hard time figuring out how to do this. It is not like labeling "dog" or "apple" because then I can create a set of training data and then evaluate its performance, now I can not. If so I would probably have used a CNN and probably also VGG-16 to help me out.

I only have two ML courses under my belt and have never really encountered a problem like this before. Can someone help me to get in the right direction?

As of now, I have a data set of 13000 labeled images I am very confident it is labeled well. I do not know of any pre-trained datasets that could be of help in this instance, but if you know of one it might help.

Worth noting is that every label is or should at least be unique. If for example there exist two pictures with the same label of "Big nose" it is purely coincidental.

Answer 1

The term you are looking for is multi-label classification, i.e. where you are making more than one classification on each image (one for each label). Most examples you'll find online are in the NLP domain but it is just as easy with CNNs since it's essentially defined by the structure of the output layer and the loss function used. It's not as complicated as it might sound if you are already familiar with CNNs.

The output layer of a neural network (for 3 or more classes) has as many units as there are targets. The network learns to associate each of those units with a corresponding class. A multi-class classifier normally applies a softmax activation function to the raw unit output, which yields a probability vector. To get the final classification, the max() of the probability vector is taken (the most probable class). The output would look like this:

                 Cat    Bird   Plane   Superman  Ball   Dog   
Raw output:      -1     2      3       6         -1     -1
Softmax:         0.001  0.017  0.046   0.934     0.001  0.001
Classification:  0      0      0       1         0      0

Multi-label classification typically uses a sigmoid activation function since the probabilities of a label occuring can be treated independently. The classification is then determined by the probability (>=0.5 for True). For your problem, this output could look like:

                 Big nose  Long hair  Curly hair  Superman  Big ears  Sharp Jawline
Raw output:      -1        -2         3           6         -1        10
Sigmoid:         0.269     0.119      0.953       0.998     0.269     1.000
Classification:  0         0          1           1         0         1

The binary crossentropy loss function is normally used for a multi-label classifier since a n-label problem is essentially splitting up a multi-class classification problem into n binary classification problems.

Since all you need to do to get from a multi-class classifier to a multi-label classifier is change the output layer, its very easy to do with pre-trained networks. If you get the pre-trained model from Keras its as simple as including include_top=False when downloading the model and then adding the correct output layer.

With 13000 images, I would recommend using Keras' ImageDataGenerator class with the flow_from_dataframe method. This allows you to use a simple pandas dataframe to label and feed in all your images. The dataframe would look like this:

Filename  Big nose  Long hair  Curly hair  Superman  Big ears  Sharp Jawline
0001.JPG  0         0          1           1         0         1
0002.JPG  1         0          1           0         1         1
   .      .         .          .           .         .         .

flow_from_dataframe's class_mode parameter can be set to raw or multi_output along with x_col to 'Filename' and y_col to ['Big nose', 'Long hair', 'Curly hair', 'Superman', 'Big ears', 'Sharp Jawline'] (in this example). Check out the documentation for more details.

The amount of data you need for each label depends on many factors and is essentially impossible to know without trying. 13000 sounds like a good start but it also depends on how many labels you have and how evenly distributed they are between the labels. A decent guide (one of many) on how to set up a multi-label classifier and how to implement it with Keras can be found here. It also covers imbalances on label frequency and is well worth a read. I'd highly recommend that you become as intimately familiar with your dataset as possible before you start tuning your neural network architecture.

Answer 2

You can try image captioning. You can train a CNN model for image, and then, on top of that, provide the model embedding to another LSTM model to learn the encoded characteristics. You can directly use the pre-trained VGG-16 model and use the second last layer to create your image embeddings.

Show and Tell: A Neural Image Caption Generator is a really nice paper to start with. There is an implementation of it in TensorFlow: https://www.tensorflow.org/tutorials/text/image_captioning. The paper focuses on generating caption, but you can provide your 'characteristics' to LSTM, so that it can learn it for each image.

Answer 3

You can use image captioning. Look at the article Captioning Images with CNN and RNN, using PyTorch. The idea is very profound. The model encodes the image to high dimensional space and then passes it through LSTM cells and LSTM cells produce linguistic output.

See also Image captioning with visual attention.

Answer 4

I would do as suggested in the comments. First select an encoding scheme. I think what is called a difference hash would work well for this application. Code for that is shown below. Now take your data set of images and run them through the encoder and save the result in a database. The database would contain the "labeling" text and the encoder result. Now for a new image you are trying to label, input the image into the encoder. Take the encoder result and compare it to the encoded values in the database. Search through the encoded values in the database and find the closest match. You can then use a "threshold" value to determine if you want to give a specific label for the image or if the distance is above the threshold declare there is no matching label. You can determine the best "threshold" value by running you data set images with the known labels and iterate the threshold level and select the threshold with the least errors. I would use something like a 56 or a 128 length hash.

import cv2
import os
# f_path is the full path to the image file, hash length is an integer specifies length of the hash
def get_hash(f_path, hash_length):    
    r_str=''    
    img=cv2.imread(f_path,0)        # read image as gray scale image
    img = cv2.resize(img, (hash_length+1, 1), interpolation = cv2.INTER_AREA)    
    # now compare adjacent horizontal values in a row if pixel to the left>pixel toright result=1 else 0
    for col in range (0,hash_length):
        if(img[0][col]>img[0][col+1]):
            value=str(1)
        else:
            value=str(0)
        r_str=r_str + value
    number=0
    power_of_two=1
    for char in r_str:        
        number = number + int(char) * power_of_two
        power_of_two=2 * power_of_two    
    return ( r_str, number) 
# example on an image of a bird
f_path=r'c:\Temp\birds\test\robin\1.jpg'
hash=get_hash ( f_path, 16) # 16 length hash on a bird image
print (' hash string ', hash[0], '   hash number ', hash[1])

> results is
 hash string  1111111100000000    hash number  255

Answer 5

From what you wrote, the problem sounds a bit like face recognition, where a camera takes a picture of your face and compares it with a bunch of pictures in a database, for example, one for each employee if its at a company's main gate. If you look "similar" to one of the pictures in the database, the door opens and your ID/Name is displayed on a terminal.

This kind of system generates an encoding for each picture and evaluates the distance between your encoded picture and the encoding of each picture in the database. If this is at most some minimum value, it's considered a match.

So, what you could do is figure out some way to encode your pictures (say sum the pixel values for a very simple example, ideally you would use some sort of vector here because distances make sense with vectors) and store this encoding together with the label of the picture.

Once your database is complete (i.e. you have a bunch of pictures saved as a pair of [encoding, label]), you can "scan" each new picture, calculate its encoding (using the same algorithm that calculated your database encodings) and find the one entry in your database which minimizes the "encoding-distance".

If this sounds like a way to solve your problem, you need to come up with a proper encoding (like "run my images through a CNN and save the output of my last fully connected layer") and apply this to all the images you want to use as "training data", before "testing" it on some of the leftover images.