How can I merge outputs of two separate layers so that the overall performance improves?

Question

I am training a combined model (fine-tuned VGG16 for images and shallow FCN for numerical data) to do a binary classification. However, the overall AUC score is not what I expected it to be.

Image-only mean AUC after 5-fold cross-validation is about 0.73 and numeric data only 5-fold mean AUC is 0.65. I was hoping to improve the mean AUC by combining the models into one and merging output layers using concatenate in Keras.

img_output = Dense(256, activation="sigmoid")(x_1) 

and

numeric_output = Dense(128, activation="relu")(x_2) 

are the output layers of the two models. And,

concat = concatenate([img_output, numeric_output])
hidden1 = Dense(64, activation="relu")(concat)
main_output = Dense(1, activation='sigmoid', name='main_output')(hidden1)

is the way I concatenated them.

Since image-only performance was better I decided that it might be reasonable to have more dense layers for image_output (256) and ended up using 128 in numeric_output.I could only reach up to mean AUC of 0.67 using a combined model. I think I should rearrange the concatenation of two outputs somehow (by introducing another learnable parameter (like the formula (10) at 3.3 section of this work?, bias?, or something else) to get more boost on mean AUC. However, I was not able to find what options were available.

Hope you have some ideas worth trying.

Answer 1

I'm not sure it's possible to help much because this is an experimental question. I'm afraid the only answer comes with testing many different options.

I see a little thing that might be making your model a little worse, though:

• You're concatenating "relu" with "sigmoid".

Placing two different nature values in the same array may make it more difficult for updating weights properly.

A few independent suggestions:

• Make the output of the images model have "relu" activation as well before the concatenation. Preferrably, use a batch normalization before the image relu and before the number relu (this way you concatenate values that are in very similar ranges).
• Instead of concatenating, you can try Multiply()([img_output, numeric_output]), in this case, both outputs must have the same size, one of them uses "relu" or "linear", and the other uses "sigmoid".

---

Now, something important when using AUC: you need big batch sizes, because AUC is dependent on the whole data, it's not like usual metrics/losses that you can take the mean from the results of each batch.