Multimodal AI Framework for Personalized and Context-Aware Skin Disease Diagnosis, Monitoring, and Treatment Support

Abstract

Dermatoscopic assessment of skin diseases based on visual morphology may not provide sufficient discrimination due to differences in cutaneous appearance, the severity of disease symptoms and individual biological or environmental factors. Current artificial intelligence (AI)-based dermatological systems mainly integrate unimodal image-based data which is constrained by comparative diagnostic performance in visually ambiguous conditions and across different skin complexions. Image-only approaches also do not utilize patient-reported symptoms needed to tailor treatment plans. In this study, we explore the potential of a unified multimodal AI framework towards robust, context-aware and patient- centric skin disease diagnosis, monitoring and treatment support. The framework combines deep learning-based image analysis with symptom-aware inputs extracted from voice recordings and structured text, which allows for improved diagnostic reliability. The proposed framework also introduces an explainable severity assessment module which evaluates disease progression via interpretable features and rule-based score. Domain adaptation methods further employed lead to better generalization for out-of-distribution data originating from different populations and reduce model bias. A knowledge-driven recommendation module generates context-aware personalized treatment recommendations based on predicted disease categories and patient-related information. Experimental results demonstrate that the proposed multimodal framework improves contextual understanding and robustness in visually ambiguous cases while enhancing interpretability, improved generalization, and practical applicability in teledermatology environments, while adding interpretability, fairness and real-world applicability of teledermatology systems.