1
$\begingroup$

I have a vanilla image classification problem. The image may optionally have some numerical metadata associated with it. We don't assume uniform availability of this metadata, i.e., the model should be able to produce an output even in its absence. What type of neural network should I explore for this problem? I have been trying various Graph Neural Networks such as Relational GCN [1] and Heterogeneous Graph Transformer [2] with the metadata feature representation obtained using an MLP. Unfortunately, it doesn't outperform the image-only model. Any ideas are much appreciated!

[1] Schlichtkrull, Michael, et al. "Modeling relational data with graph convolutional networks." The Semantic Web: 15th International Conference, ESWC 2018, Heraklion, Crete, Greece, June 3–7, 2018, Proceedings 15. Springer International Publishing, 2018. [2] Hu, Ziniu, et al. "Heterogeneous graph transformer." Proceedings of the web conference 2020. 2020.

$\endgroup$

2 Answers 2

1
$\begingroup$

You could use your existing CNN architecture and concatenate the metadata with the flattened last convolutional layer. Add a flag Boolean feature for whether the metadata is present, and if it is not, then set other values of the feature to neutral value, such as the mean (if you are using standard scaler to normalise features, then that will be all zeros)

In general it's ok to have inputs going to different layers within a network, or having it split, combine or skip between layers. This gives you a lot of flexibility to handle different input types without needing to explore more complex solutions such as graph networks

$\endgroup$
1
$\begingroup$

Train a self-supervised model to generate embeddings of your metadata

Train a model to generate embeddings of your images (or use/fine-tune a pretrained image model)

Train a prediction model that takes in concatenated [image_embedding, metadata_embedding] to make predictions

For images with no metadata, use a vector of zeros in place of the metadata embedding.

If your training data has "missing metadata" images at a rate similar to your inference environment, the model should work things out during training.

If your training dataset is enriched for images with metadata, add a dropout step to randomly zero out the metadata embedding so your model trains on a realistic proportion of "missing metadata" images.

$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .