MMP: Towards Robust Multi-Modal Learning with Masked Modality Projection

In real-world applications, input modalities might be missing due to factors like sensor malfunctions or data constraints. Our recent paper addresses this challenge with a method called Masked Modality Projection (MMP), which is designed to maintain robust performance even when some input modalities are unavailable. This approach involves training a single model that can adapt to any missing modality scenario without needing specialized adjustments for each combination of missing inputs.

Summary of the Paper

Multimodal learning seeks to combine data from multiple input sources to enhance the performance of different downstream tasks. In real-world scenarios, performance can degrade substantially if some input modalities are missing. Existing methods that can handle missing modalities involve custom training or adaptation steps for each input modality combination. These approaches are either tied to specific modalities or become computationally expensive as the number of input modalities increases. In this paper, we propose Masked Modality Projection (MMP), a method designed to train a single model that is robust to any missing modality scenario. We achieve this by randomly masking a subset of modalities during training and learning to project available input modalities to estimate the tokens for the masked modalities. This approach enables the model to effectively learn to leverage the information from the available modalities to compensate for the missing ones, enhancing missing modality robustness. We conduct a series of experiments with various baseline models and datasets to assess the effectiveness of this strategy. Experiments demonstrate that our approach improves robustness to different missing modality scenarios, outperforming existing methods designed for missing modalities or specific modality combinations.

Our Contribution

Our Contribution and Its Significance Our primary contributions are:

• Introduction of MMP: A novel approach that enables a single multimodal model to be robust to any missing modality scenario.
• Performance Improvement: Through comprehensive experiments across multiple datasets, we demonstrated that MMP significantly outperforms baseline methods in scenarios with missing modalities, while maintaining competitive performance when all modalities are present.
• Versatility and Adaptability: MMP can be integrated with various existing multimodal architectures without requiring significant changes. This makes it suitable for a wide range of multimodal tasks, reducing the need for task-specific adjustments.

These contributions are significant as they provide a scalable solution to the common issue of missing modalities in real-world applications, saving time and computational resources that would otherwise be spent on fine-tuning or retraining models.

Future Directions

While MMP has proven effective in many scenarios, future work will explore its potential application to other complex multimodal tasks and datasets. We also aim to refine the approach further, focusing on reducing the computational cost of the projection step and improving the alignment between projected and real modality data.

More Details About the Paper

• Date First Available: 3 October 2024
• Authors: Niki Nezakati , Md Kaykobad Reza , Ameya Patil, Mashhour Solh, and M. Salman Asif
• Paper Link: arXiv