# Semantic issues with predictions made by my trained model

I'm new to Deep Learning. I used Keras and trained a inception_resnet_v2 model for my binary classification application (fire detection). As suggested from my previous question of a non-X class, I prepared a dataset of 8000 images of fire, and a larger dataset for non-fire (20,000 random images) to make sure the network also sees images of non-fire to perform classification.

I trained the model, but now when trying to load the model and pass images of fire and non-fire ones, it shows same result for all of them:

[[0. 1.]]
[[0. 1.]]
[[0. 1.]]
[[0. 1.]]
[[0. 1.]]


What is going wrong? Am I doing anything wrong? Should I get the result another way?

===============================================

I know it's not SO, but this is my prediction code in case it matters:

from __future__ import print_function
import cv2, os, glob
import numpy as np
from keras.preprocessing import image

if __name__ == '__main__':

os.chdir("test")
for file in glob.glob("*.jpg"):
img_path = file
x = image.img_to_array(img)
x = np.expand_dims(x, axis=0)

dictionary = {0: 'non-fire', 1: 'fire'}

results = model.predict(x)
print(results)
predicted_class= np.argmax(results)
acc = 100*results[0][predicted_class]
print("Network prediction is: file: "+ file+", "+dictionary[predicted_class]+", %{:0.2f}".format(acc))


And here is the training:

from keras.applications.inception_resnet_v2 import InceptionResNetV2, preprocess_input
from keras.preprocessing.image import ImageDataGenerator
from keras.layers import Dense, Activation, Flatten, Dropout
from keras.models import Sequential, Model
from keras.callbacks import ModelCheckpoint
from keras.metrics import binary_accuracy
import os
import json
#==========================
HEIGHT = 300
WIDTH = 300
TRAIN_DIR = "data"
BATCH_SIZE = 8 #8
steps_per_epoch = 1000 #1000
NUM_EPOCHS = 50 #50
lr= 0.00001
#==========================
FC_LAYERS = [1024, 1024]
dropout = 0.5

def build_finetune_model(base_model, dropout, fc_layers, num_classes):
for layer in base_model.layers:
layer.trainable = False

x = base_model.output
x = Flatten()(x)
for fc in fc_layers:
# New FC layer, random init
x = Dense(fc, activation='relu')(x)
x = Dropout(dropout)(x)

# New layer
predictions = Dense(num_classes, activation='sigmoid')(x)
finetune_model = Model(inputs=base_model.input, outputs=predictions)
return finetune_model

train_datagen =  ImageDataGenerator(preprocessing_function=preprocess_input, rotation_range=90, horizontal_flip=True, vertical_flip=True
,validation_split=0.2)
train_generator = train_datagen.flow_from_directory(TRAIN_DIR, target_size=(HEIGHT, WIDTH), batch_size=BATCH_SIZE
,subset="training")
#split validation manually
validation_generator = train_datagen.flow_from_directory(TRAIN_DIR, target_size=(HEIGHT, WIDTH), batch_size=BATCH_SIZE,subset="validation")

base_model = InceptionResNetV2(weights='imagenet', include_top=False, input_shape=(HEIGHT, WIDTH, 3))

root=TRAIN_DIR
class_list = [ item for item in os.listdir(root) if os.path.isdir(os.path.join(root, item)) ]
print ("class_list: "+str(class_list))

finetune_model = build_finetune_model(base_model, dropout=dropout, fc_layers=FC_LAYERS, num_classes=len(class_list))

# change to categorical_crossentropy for multiple classes

filepath="./checkpoints/" + "Resnet_{epoch:02d}_{acc:.2f}" +"_model_weights.h5"
checkpoint = ModelCheckpoint(filepath, monitor=["val_accuracy"], verbose=1, mode='max', save_weights_only=False)
callbacks_list = [checkpoint]

history = finetune_model.fit_generator(train_generator, epochs=NUM_EPOCHS, workers=BATCH_SIZE,
validation_data=validation_generator, validation_steps = validation_generator.samples,
steps_per_epoch=steps_per_epoch,
shuffle=True, callbacks=callbacks_list)


I think you may have a class imbalance problem here, if I am reading your output correctly. You have 20,000 negative examples, but only 8000 positive ones, and you are minimizing binary cross entropy without re-weighting the examples, so your model can achieve a low-ish loss just by consistently outputing a value close to 0. This forms a local optima in the search space for the model.

To fix this, you could try to optimize some other loss function that is more sensitive to class imbalances, or, likely more productively, you could just use an equal number of examples for each class.

• I've seen examples (cat vs non-cat) where they used 5K for non-cats and 1600 for cats. But what combo of activation/loss shall I use? Is it OK to change the sigmoid to softmax and still have the binary_crossentropy? (like the cat exmaple). Nov 19 '19 at 3:20
• @TinaJ whether you can handle class imbalance will really depend on the problem you are working on. If the classes are difficult to separate, then it will be a problem. If not, it probably won't be. I think the choice of activation function shouldn't make a big difference here. Nov 19 '19 at 3:27
• My impression was the more non-object I have, the better the accuracy. Didn't know the size imbalance can also be a negative factor! Nov 19 '19 at 3:31
• @TinaJ an extreme case that might help with the intuition: Imagine we have a dataset of blood test results, where 99.99% of the results are negative for a disease, and 0.01% are positive. A model can achieve an extremely low cross-entropy loss value by claiming everything is negative. It will be right almost always, but the model is not a useful one. Nov 19 '19 at 3:34