Sri Lankan Sign Language Detection System for Service Robots

Abstract

Sinhala Sign Language (SSL) is the main way many Deaf and hard-of-hearing people in Sri Lanka communicate, but there are very few digital tools to recognize it automatically. This thesis addresses that problem by developing deep learning models that can recognize SSL gestures accurately while also working in real time. A dataset of 5,000 SSL videos covering 50 gestures was created and divided into training, validation, and test sets, with preprocessing steps such as frame resizing, normalization, and data augmentation. Four models were tested: a CNN to learn visual features, an LSTM to capture movement over time, a combined CNN-LSTM model, and YOLOv8 for fast, detection-based recognition. The CNN achieved the best overall performance with 98.32% accuracy, followed closely by the LSTM at 98.18%, while YOLOv8 reached 97.20% accuracy but delivered very high real-time speed of up to 90 frames per second. The CNN-LSTM model performed less well than expected. A working prototype was also built, allowing users to perform signs and see the recognized output in Sinhala or English, with optional speech, showing the system’s practical value for inclusive communication.This research establishes a foundational framework for SSL recognition, showing that deep learning models can achieve state of the art accuracy while highlighting the trade offs between accuracy, latency, and deployment feasibility in real world contexts.