I am training a combined model (fine-tuned VGG16 for images and shallow FCN for numerical data) to do a binary classification. However, the overall AUC score is not what I expected it to be.
Image-only mean AUC after 5-fold cross-validation is about 0.73 and numeric data only 5-fold mean AUC is 0.65. I was hoping to improve the mean AUC by combining the models into one and merging output layers using
concatenate in Keras.
img_output = Dense(256, activation="sigmoid")(x_1)
numeric_output = Dense(128, activation="relu")(x_2)
are the output layers of the two models. And,
concat = concatenate([img_output, numeric_output]) hidden1 = Dense(64, activation="relu")(concat) main_output = Dense(1, activation='sigmoid', name='main_output')(hidden1)
is the way I concatenated them.
Since image-only performance was better I decided that it might be reasonable to have more dense layers for
256) and ended up using
numeric_output.I could only reach up to mean AUC of 0.67 using a combined model. I think I should rearrange the concatenation of two outputs somehow (by introducing another learnable parameter (like the formula (10) at 3.3 section of this work?, bias?, or something else) to get more boost on mean AUC. However, I was not able to find what options were available.
Hope you have some ideas worth trying.