HCLIP: Beyond CLIP for Cost-Effective Multimodal Retrieval in Education

Abstract

Multimodal retrieval systems have gained significant attention due to their ability to process and cross-retrieve data containing images and text. However, the factors such as high cost of development, limitation on resources, and the proper addressing of the modality gap, the inherent representational differences between modalities pose a challenge to building effective and efficient retrieval models. In this work, we propose a low-resource, cost-efficient hybrid multimodal retrieval model that integrates Contrastive Language-Image Pre-training (CLIP) and All-MiniLM-L6-v2 to create a shared embedding space while storing raw images in an unstructured database. Our primary contributions include (1) the development of a hybrid model that outperforms CLIP-native retrieval, (2) a novel bidirectional neural network alignment technique that brings textual and visual modalities closer together, and (3) a comprehensive analysis of the modality gap's impact on downstream retrieval performance. Through proper evaluation using transparent techniques such as Mean Reciprocal Rank (MRR) and Cosine-Weighted MRR, our method demonstrates improved retrieval accuracy over baseline approaches. Experimental results exhibit that a lower modality gap does not always prove to be efficient on the downstream retrieval. Our findings pave the way for more efficient, adaptable, and cost-effective multimodal retrieval methodologies in low-resource environments, not limited to the education domain.