I'm new to Deep Learning. I used Keras and trained a inception_resnet_v2 model for my binary classification application (fire detection). As suggested from my previous question of a non-X class, I prepared a dataset of 8000 images of fire, and a larger dataset for non-fire (20,000 random images) to make sure the network also sees images of non-fire to perform classification.

I trained the model, but now when trying to load the model and pass images of fire and non-fire ones, it shows same result for all of them:

[[0. 1.]]
[[0. 1.]]
[[0. 1.]]
[[0. 1.]]
[[0. 1.]]

What is going wrong? Am I doing anything wrong? Should I get the result another way?


I know it's not SO, but this is my prediction code in case it matters:

from __future__ import print_function
from keras.models import load_model, model_from_json
import cv2, os, glob
import numpy as np
from keras.preprocessing import image

if __name__ == '__main__':
    model = load_model('Resnet_26_0.79_model_weights.h5')

    for file in glob.glob("*.jpg"):
        img_path = file
        img = image.load_img(img_path, target_size=(300, 300))
        x = image.img_to_array(img)
        x = np.expand_dims(x, axis=0)

        dictionary = {0: 'non-fire', 1: 'fire'}

        results = model.predict(x)
        predicted_class= np.argmax(results)
        acc = 100*results[0][predicted_class]
        print("Network prediction is: file: "+ file+", "+dictionary[predicted_class]+", %{:0.2f}".format(acc))

And here is the training:

from keras.applications.inception_resnet_v2 import InceptionResNetV2, preprocess_input
from keras.preprocessing.image import ImageDataGenerator
from keras.layers import Dense, Activation, Flatten, Dropout
from keras.models import Sequential, Model
from keras.optimizers import SGD, Adam
from keras.callbacks import ModelCheckpoint
from keras.metrics import binary_accuracy
import os
import json
HEIGHT = 300
WIDTH = 300
TRAIN_DIR = "data"
steps_per_epoch = 1000 #1000
NUM_EPOCHS = 50 #50
lr= 0.00001
FC_LAYERS = [1024, 1024]
dropout = 0.5

def build_finetune_model(base_model, dropout, fc_layers, num_classes):
    for layer in base_model.layers:
        layer.trainable = False

    x = base_model.output
    x = Flatten()(x)
    for fc in fc_layers:
        # New FC layer, random init
        x = Dense(fc, activation='relu')(x) 
        x = Dropout(dropout)(x)

    # New layer
    predictions = Dense(num_classes, activation='sigmoid')(x) 
    finetune_model = Model(inputs=base_model.input, outputs=predictions)
    return finetune_model

train_datagen =  ImageDataGenerator(preprocessing_function=preprocess_input, rotation_range=90, horizontal_flip=True, vertical_flip=True
train_generator = train_datagen.flow_from_directory(TRAIN_DIR, target_size=(HEIGHT, WIDTH), batch_size=BATCH_SIZE
#split validation manually
validation_generator = train_datagen.flow_from_directory(TRAIN_DIR, target_size=(HEIGHT, WIDTH), batch_size=BATCH_SIZE,subset="validation")

base_model = InceptionResNetV2(weights='imagenet', include_top=False, input_shape=(HEIGHT, WIDTH, 3))

class_list = [ item for item in os.listdir(root) if os.path.isdir(os.path.join(root, item)) ]
print ("class_list: "+str(class_list))

finetune_model = build_finetune_model(base_model, dropout=dropout, fc_layers=FC_LAYERS, num_classes=len(class_list))

adam = Adam(lr)
# change to categorical_crossentropy for multiple classes
finetune_model.compile(adam, loss='binary_crossentropy', metrics=['accuracy'])

filepath="./checkpoints/" + "Resnet_{epoch:02d}_{acc:.2f}" +"_model_weights.h5"
checkpoint = ModelCheckpoint(filepath, monitor=["val_accuracy"], verbose=1, mode='max', save_weights_only=False)
callbacks_list = [checkpoint]

history = finetune_model.fit_generator(train_generator, epochs=NUM_EPOCHS, workers=BATCH_SIZE, 
                                    validation_data=validation_generator, validation_steps = validation_generator.samples, 
                                       shuffle=True, callbacks=callbacks_list)

I think you may have a class imbalance problem here, if I am reading your output correctly. You have 20,000 negative examples, but only 8000 positive ones, and you are minimizing binary cross entropy without re-weighting the examples, so your model can achieve a low-ish loss just by consistently outputing a value close to 0. This forms a local optima in the search space for the model.

To fix this, you could try to optimize some other loss function that is more sensitive to class imbalances, or, likely more productively, you could just use an equal number of examples for each class.

  • $\begingroup$ I've seen examples (cat vs non-cat) where they used 5K for non-cats and 1600 for cats. But what combo of activation/loss shall I use? Is it OK to change the sigmoid to softmax and still have the binary_crossentropy? (like the cat exmaple). $\endgroup$
    – Tina J
    Nov 19 '19 at 3:20
  • $\begingroup$ @TinaJ whether you can handle class imbalance will really depend on the problem you are working on. If the classes are difficult to separate, then it will be a problem. If not, it probably won't be. I think the choice of activation function shouldn't make a big difference here. $\endgroup$ Nov 19 '19 at 3:27
  • $\begingroup$ My impression was the more non-object I have, the better the accuracy. Didn't know the size imbalance can also be a negative factor! $\endgroup$
    – Tina J
    Nov 19 '19 at 3:31
  • 1
    $\begingroup$ @TinaJ an extreme case that might help with the intuition: Imagine we have a dataset of blood test results, where 99.99% of the results are negative for a disease, and 0.01% are positive. A model can achieve an extremely low cross-entropy loss value by claiming everything is negative. It will be right almost always, but the model is not a useful one. $\endgroup$ Nov 19 '19 at 3:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.