LLM Critiques Automate Item Evaluation for Standardized Tests.

Evaluating the quality of items for standardized tests is a labor-intensive process, typically requiring extensive manual expert review and field testing. This study introduces an Automated Item Evaluation (AIE) model designed to predict whether an item will be accepted or rejected based on historical data. The model leverages both the raw text of the test items and critiques generated by a Large Language Model (Qwen3). A DeBERTaV3-large classifier was fine-tuned on raw item text, another on the LLM-generated critiques, and a fusion model combined representations from both. The fusion model demonstrated the strongest overall performance, achieving an accuracy of 0.75 and an F1 score of 0.64. Performance was notably higher for mathematics items compared to English language arts. While the model shows promise in reducing the burden of manual review, especially when prioritizing sensitivity (identifying problematic items), it struggled to accurately identify items flagged for bias, sensitivity, fairness, or accessibility concerns, particularly in ELA. This highlights the continued importance of human oversight for ethical considerations.