Faster Than the Teacher, Smarter Than the Student: Classifying with Wisdom via Knowledge Distillation in LLMs

Abstract

Large language models (LLMs) have achieved remarkable success across various natural language processing (NLP) tasks, driven by their ability to capture complex language patterns through large-scale pretraining. However, their substantial computational demands limit their deployment in resourceconstrained environments. To address this, this research introduced Knowledge distillation-based framework for text classification using a multiclass approach across three domains: entertainment, sports, and politics. We utilize both hard labels (ground-truth categories) and soft labels (logits from a teacher model) to train a student and a distilled model. The teacher model is accurate but computationally expensive. The student model is lightweight and fast, yet less accurate. Through distillation, we derive a task-specific distilled model that balances speed and accuracy. We also compare the performance of these models against traditional classifiers such as LSTM, SVM, and Naive Bayes. Traditional models excel comparing to the LLMs. Considering only task agnostic language models, evaluation shows that the distilled model performs significantly better than the student and competitively against the teacher, offering a practical trade-off. Our study demonstrates the value of soft label transfer and semantic alignment for improving classification performance in resource-constrained environments. The text classification code can be found at: https://github.com/Abishethvarman/KD-Text-Classification