CROSS-PROMPT TRANSFER AND SUBGROUP ERROR STABILITY IN AUTOMATED ESSAY SCORING

Authors

DOI:

https://doi.org/10.26577/JMMCS209110426

Keywords:

automated essay scoring, machine learning, cross-prompt generalization, fairness, writing assessment

Abstract

 Automated essay scoring (AES) can support large-scale assessment, but strong performance on a random test split does not establish that a model measures writing quality robustly across tasks or produces comparable errors for different student groups. This study evaluates these validity dimensions jointly on the public ASAP 2.0 source-based writing corpus. The analyzed training file contains 24,728 essays from seven prompts with human holistic scores and demographic metadata. A transparent word-count-only Ridge regressor was compared with a histogram-based gradient boosting regressor trained on seven interpretable textual features. Performance was evaluated using quadratic weighted kappa (QWK), mean absolute error (MAE), root mean squared error (RMSE), Spearman correlation, and adjacent agreement. Uncertainty was quantified with 500 bootstrap resamples. Generalization was examined through leave-one-prompt-out validation, supplemented by a prompt-normalized score-scale ablation. Subgroup error differences were tested using Mann Whitney U tests, rank-biserial effect sizes, and Holm correction. The engineered-feature model achieved QWK=0.732 (95% CI: 0.718–0.746), MAE=0.488, RMSE=0.641, and Spearman ρ=0.788, outperforming the length-only baseline (QWK=0.651). However, word count alone correlated strongly with human score (ρ=0.733, p < 0.001). Leave-one-prompt-out QWK ranged from 0.520 to 0.768, and score normalization did not remove the weakest transfer result. After Holm correction, essays from English language learners showed a statistically significant but small increase in absolute error (adjusted p < 0.001, |r| = 0.099). The study contributes an interpretable evaluation of how strong aggregate scoring agreement can coexist with variable cross-prompt transfer and subgroup-specific error differences, using a transparent length-only baseline and a seven-feature nonlinear model. Practically, the framework can be used to screen AES models before educational deployment and to identify conditions requiring additional human review. The findings indicate that aggregate agreement should be supplemented with task-transfer, uncertainty, and subgroup error analyses before AES models are considered for educational use.

Author Biographies

  • Aigerim Sultanali, Khoja Akhmet Yassawi International Kazakh-Turkish University, Turkistan, Kazakhstan

    Aigerim Sultanali – PhD student at the Khoja Akhmet Yassawi International Kazakh-Turkish University (Turkistan, Kazakhstan, e-mail: aigerim.sultanali@ayu.edu.kz).

  • Gulnur Kazbekova, Khoja Akhmet Yassawi International Kazakh-Turkish University, Turkistan, Kazakhstan

    Gulnur Kazbekova – Candidate of Technical Sciences, Associate Professor at the Khoja Akhmet Yassawi International Kazakh-Turkish University (Turkistan, Kazakhstan, e-mail: gulnur.kazbekova@ayu.edu.kz).

Published

2026-11-18

How to Cite

CROSS-PROMPT TRANSFER AND SUBGROUP ERROR STABILITY IN AUTOMATED ESSAY SCORING. (2026). Journal of Mathematics Mechanics and Computer Science, 132(4), 157-171. https://doi.org/10.26577/JMMCS209110426