Hybrid Deep Learning Approach Using YOLO and Spatial Transformers for Kinesthetic Math Education in Low-Resource Settings

Abstract

This research presents a hybrid deep-learning system to enhance kinesthetic math education for Grade 4 and 5 students in low-resource Sri Lankan classrooms, addressing addition, subtraction, and time-telling skills in Sinhala. Integrating fine-tuned You Only Look Once version 11 (YOLOv11) for real-time abacus recognition and pre-trained Spatial Transformer Network (STN) with Residual Network 50 (ResNet50) for analog clock analysis, the system bridges physical manipulatives with digital feedback. Optimized for low-cost hardware via post-training quantization, it reduces model sizes by up to 56.6% and inference times by 45.9%, enabling deployment on standard PCs. The methodology employs YOLOv11 to detect abacus beads (mean Average Precision, mAP50-95: 0.898) and STN + ResNet50 to correct clock perspectives (95.2% accuracy on SynClock), delivering immediate Sinhala feedback through a culturally adapted interface. The system was evaluated on an Intel Core i5 PC with 8GB RAM and achieved sub-second inference (YOLOv11: 170.1 ms, STN + ResNet50: 32 ms post-quantization) while retaining accuracy. A pilot study with 20 students showed a 25% improvement in arithmetic scores and 30% in time-telling accuracy, with 85% reporting higher engagement. These findings demonstrate the system's efficacy in boosting numeracy and interaction in resource-scarce settings, aligning with the Visual, Aural, Read/Write, Kinesthetic (VARK) framework. Despite challenges like hardware constraints and real-world variability, this scalable, offline-capable solution offers a novel approach to educational technology, with potential for broader deployment in Sri Lanka and similar contexts, addressing gaps in artificial intelligence-driven kinesthetic learning tools. ©