BACKGROUND: To apply multimodal machine learning (ML), the various data modalities typically come from the same example (e.g., chest X-ray (modality 1) and cancer biomarkers (modality 2) come from the same patient (the example)). The problem is that we often don't have that in public datasets. Instead, datasets are more commonly independent (e.g., a dataset of chest X-rays from one set of patients and a dataset of cancer biomarkers from a second set of patients).

QUESTION: Is there any validity to artificially creating "co-registered" datasets from such independent datasets for the ultimate purpose of leveraging multimodal ML (e.g., generate all possible pairs of chest X-rays and cancer biomarkers from different patients with cancer and generate the same for different patients without cancer)?

NOTE: It is understood that this approach is non-canonical and has flaws, but that is not the question here. I am more interested in learning whether this could be a second best option for researchers who lack co-registered samples but still want to develop multimodal ML models. Please provide your response along with some justification as to why this would or would not be valid "second best option".


New NOTE: I thought it was implied in the question post that I am aware that the interaction between modalities is not available to be leveraged by multimodal ML since this is really the crux of the problem with fusing independent datasets. However, the question still stands: "Would this be a valid second best option?"

  • 1
    $\begingroup$ A keyword that may be useful to you is the concept of "semi-synthetic" data generation. It seems like that is what you are trying to do :) $\endgroup$
    – Scriddie
    Nov 23, 2022 at 11:09

1 Answer 1


One problem with such an approach is if you would know how chest X-ray and cancer biomarkers interact you would likely use that model or knowledge directly instead of using it to generate data. And aligning data among subjects without such knowledge might even give you less information than working with each of the two modalities separately since your model might pick up on some incidental relations. It's like having a regression problem but being unaware of the interaction term.


You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .