Abstract: Computed Tomography (CT) is a pivotal diagnostic tool in medical imaging. The presence of metallic objects within the scan field, however, leads to the formation of metal artifacts in reconstructed CT images, which can significantly impair the accuracy of diagnosis and treatment. In recent years, deep learning-based approaches to Metal Artifact Reduction (MAR) have been developed to mitigate these artifacts. Yet, current methods often fail to capitalize on the complementary information available from different modalities, resulting in the suboptimal reduction of artifacts. To overcome this limitation, a Multi-modal Information Fusion Network (MIFNet) is constructed and applied to a model-driven MAR framework in this study. Regularization constraints are imposed on metal artifacts and CT images, leveraging the additive nature of metal artifacts, to formulate and solve an optimization model for MAR. Each iterative sub-step of the proposed algorithm is expanded into a network, creating a model-driven network with enhanced interpretability. Additionally, a deep neural network is developed that integrates local and global image information along with textual prior information, serving as the CT image proximal network within the model-driven MAR framework. The Contrastive Language-Image Pretraining (CLIP) model is utilized to incorporate textual prior knowledge, thereby aiding in MAR. Experimental results indicate that the multi-modal information fusion method presented here surpasses existing deep learning-based MAR methods in terms of MAR performance.