Story Point Estimation with Explainable AI

Abstract

Accurate story point estimation remains as a challenge in Agile software development because story point estimation depends on human intuition, experience, and subjectivity. Traditional story point estimation methods like planning poker and expert judgment most of the time lead to inconsistencies and biases, and it could impact resource allocation of the project and predictability of the project. This study addresses those limitations by seamless integration of transformer based natural language processing (NLP) models (BERT variants) and Explainable AI techniques (XAI) to interpret the estimation of story points. Four transformer-based models, BERT-base-uncased, DistilBERT, RoBERTa-base, and DistilRoBERTa-base were trained using TAWOS dataset with baseline and customized preprocessing pipelines. Advanced preprocessing techniques such as adaptive Fibonacci mapping, semantic T5 based data augmentation, and context injection, improved the model accuracy. The model trained with DistilBERT achieved the highest performance with 0.520 with 10 early stopping patience and 0.507 accuracy with early stopping patience 4 and the lowest mean absolute error 0.72 (MAE = 0.72). To improve the transparency and interpretability of trained models, XAI methods like SHAP and LIME explanations were applied. A survey with 8 agile practitioners showed strong alignment between XAI explanations and human explanations, with SHAP explanation achieving 72% and LIME explanation achieving 65% overlapping with agile practitioners identified keywords. The findings show that transformer models with XAI achieved accuracy comparable to human estimations (80% agreement) with interpretable predictions. This study contributes to a transparent, and data-driven framework for agile story point estimation, connecting the gap between human expertise and Artificial intelligence decisions.