Are there any good ways of simultaneously incorporating object detection with speech recognition? For example, if you want to identify whether an animal is a dog or cat, we can obviously use visual features (e.g. YOLO, CNNs, etc.). But how would you incorporate speech and sound in this model?
Check out this paper. It deals with the problem of mixing input modalities! Here's the abstract.
This paper presents a novel model for multimodal learning based on gated neural networks. The Gated Multimodal Unit (GMU) model is intended to be used as an internal unit in a neural network architecture whose purpose is to find an intermediate representation based on a combination of data from different modalities. The GMU learns to decide how modalities influence the activation of the unit using multiplicative gates. It was evaluated on a multilabel scenario for genre classification of movies using the plot and the poster. The GMU improved the macro f-score performance of single-modality approaches and outperformed other fusion strategies, including mixture of experts models. Along with this work, the MM-IMDb dataset is released which, to the best of our knowledge, is the largest publicly available multimodal dataset for genre prediction on movies.