CROSS-PROMPT TRANSFER AND SUBGROUP ERROR STABILITY IN AUTOMATED ESSAY SCORING
DOI:
https://doi.org/10.26577/JMMCS209110426Keywords:
automated essay scoring, machine learning, cross-prompt generalization, fairness, writing assessmentAbstract
Automated essay scoring (AES) can support large-scale assessment, but strong performance on a random test split does not establish that a model measures writing quality robustly across tasks or produces comparable errors for different student groups. This study evaluates these validity dimensions jointly on the public ASAP 2.0 source-based writing corpus. The analyzed training file contains 24,728 essays from seven prompts with human holistic scores and demographic metadata. A transparent word-count-only Ridge regressor was compared with a histogram-based gradient boosting regressor trained on seven interpretable textual features. Performance was evaluated using quadratic weighted kappa (QWK), mean absolute error (MAE), root mean squared error (RMSE), Spearman correlation, and adjacent agreement. Uncertainty was quantified with 500 bootstrap resamples. Generalization was examined through leave-one-prompt-out validation, supplemented by a prompt-normalized score-scale ablation. Subgroup error differences were tested using Mann Whitney U tests, rank-biserial effect sizes, and Holm correction. The engineered-feature model achieved QWK=0.732 (95% CI: 0.718–0.746), MAE=0.488, RMSE=0.641, and Spearman ρ=0.788, outperforming the length-only baseline (QWK=0.651). However, word count alone correlated strongly with human score (ρ=0.733, p < 0.001). Leave-one-prompt-out QWK ranged from 0.520 to 0.768, and score normalization did not remove the weakest transfer result. After Holm correction, essays from English language learners showed a statistically significant but small increase in absolute error (adjusted p < 0.001, |r| = 0.099). The study contributes an interpretable evaluation of how strong aggregate scoring agreement can coexist with variable cross-prompt transfer and subgroup-specific error differences, using a transparent length-only baseline and a seven-feature nonlinear model. Practically, the framework can be used to screen AES models before educational deployment and to identify conditions requiring additional human review. The findings indicate that aggregate agreement should be supplemented with task-transfer, uncertainty, and subgroup error analyses before AES models are considered for educational use.











