# Why is this ResNet50 misclassifying objects?

I'm new to Deep Learning, and I have some conceptual problems. I followed a simple tutorial here, and trained a model in Keras to do image classification on 10 classes of logos. I prepared 10 classes with each class having almost 100 images. My trained Resnet50 model performs exceptionally great when the image is one of those 10 logos, with 1.00 probability. But the problem is if I pass a non-logo item, a random image totally unrelated visually, still it marks it as one of those logos with close to 1.00 probability!

I'm confused. Am I missing anything? Why is this happening? How to find a solution? I need to find logos in video frames. But right now, with a high possbility each frame is marked as a logo!

Here is my simple training code:

def build_finetune_model(base_model, dropout, fc_layers, num_classes):
for layer in base_model.layers:
layer.trainable = False

x = base_model.output
x = Flatten()(x)
for fc in fc_layers:
# New FC layer, random init
x = Dense(fc, activation='relu')(x)
x = Dropout(dropout)(x)

# New softmax layer
predictions = Dense(num_classes, activation='softmax')(x)
finetune_model = Model(inputs=base_model.input, outputs=predictions)
return finetune_model
finetune_model = build_finetune_model(base_model, dropout=dropout, fc_layers=FC_LAYERS, num_classes=len(class_list))
filepath="./checkpoints/" + "ResNet50" + "_model_weights.h5"
checkpoint = ModelCheckpoint(filepath, monitor=["acc"], verbose=1, mode='max')
callbacks_list = [checkpoint]

history = finetune_model.fit_generator(train_generator, epochs=NUM_EPOCHS, workers=8,
steps_per_epoch=steps_per_epoch,
shuffle=True, callbacks=callbacks_list)

plot_training(history)

• This sounds like a classic example of training data that is not representative of the input your network will be receiving. Since the network has only seen input data that is expected to produce a value of 1.00 at one of the nodes, after significant training that's what it will do for all inputs, because it would be punished if it ever did otherwise. This is a really common problem, that has many solutions, but this answer from a while back has some good suggestions on how to solve for input entirely out of the scope of the problem: stackoverflow.com/a/52831580/9546874 – Recessive Nov 4 at 23:55
• Too add to @Recessive's answer -- answer (2) in that link is good, but to make it work you also need some training data which doesn't contain logos! Since you have a softmax final layer, I guess your labels are (currently) vectors which are all 0 except for a single 1. In a training image with no logo, it will be all zeros. – jmmcd Nov 5 at 1:16
• Thanks guys. As I was thinking myself, looks like I should really add another class "unknown". But the problem is how big it should be? We can not put whatever on earth as a non-logo class :-/ – Tina J Nov 5 at 17:38
• @TinaJ Another alternative is to use an object detection model instead - one that answers the question "of the objects I know, how many appear in this image, and which?" compared to image classification, which instead asks "of the objects that I know, which is the one in this image most similar to?" – gmds Nov 5 at 23:50
• @gmds can you describe more?! What do you mean object detection model? – Tina J Nov 6 at 16:58

Your problem is a classification problem. If you are following the tutorial, it is using the ResNet50 network, which is a convolutional neural network with one fully connected layer at the end. At the end the activation function is softmax. Detailed description of the activation function can be found here: Softmax function explained

Basically, softmax increases the difference between the higher probability and lower probability. It also limits the output between 0 and 1.

## Problem Origin

Due to the nature of the softmax function, it always chooses the best one and enlarge the value to be a value near one, even if the range of the output predictions is very small like 0-0.1. Also you training data only have the data of the 10 logos labelled, so if the network see unseen images with no logo it recognizes, it predicts the one with the most similarity. If you want to classify images with no logos in it, you should add an extra class in the training dataset and also the code to train the network to learn to classify images which is not in the 5 class into a separate maybe called unlabeled. Hope I can help you and have a nice day.

• Assuming, that the wrong classification of the images is caused by the logits layer is a great choice, if the goal is to ignore the logo pictures which are feed as input into the neural network. Instead of explaining how to improve the accuracy, the description focus on the softmax function. Unfortunately, it has no advantage over a normal sigmoid function, which maps an input space into a normalized output. – Manuel Rodriguez Nov 5 at 15:33
• Thanks anyways. As I was thinking myself, looks like I should really add another class "unknown". But the problem is how big it should be? We can not put whatever on earth as a non-logo class :-/ – Tina J Nov 5 at 17:38
• You should try putting at least the same amount of unknown data as the other class each have. If it doesn't work, put more of the unknown data until it works. Experimenting and trying is the key. Hope it helps – Clement Hui Nov 6 at 0:08
• @TinaJ You could use a GAN like logic to produce random non-logo images (without training the generator of course) – Fnguyen Nov 6 at 14:34
• @TinaJ There are tons of examples if you Google GAN. I'd recommend machinelearningmastery because he has the whole Keras code. If you want it to be somewhat realistic you could train the generator a bit so it's not just random. – Fnguyen Nov 6 at 17:31