WrAFT: New Automated Writing Evaluation System for Essays.

The study introduces WrAFT, a novel Automated Writing Evaluation (AWE) system specifically designed for argumentative essays. WrAFT stands out due to its modular architecture, which separates the complex tasks of essay evaluation into distinct components: scoring, surface-level feedback, and deep-level feedback. This design allows for a more targeted and effective approach to assessment. In developing WrAFT, the researchers rigorously evaluated several Large Language Models (LLMs), including LLaMA-3.3-70B-Instruct, GPT-4o, and Claude 3.7, employing both direct prompting and supervised fine-tuning techniques. Using a proprietary dataset of 480 TOEFL Independent Writing essays, WrAFT demonstrated state-of-the-art performance in scoring, achieving a quadratic weighted kappa of 0.84 and an RMSE of 0.44 against official scores. Furthermore, human evaluations confirmed the high quality of its generated feedback, with approval ratings exceeding 93% for all feedback types. An interactive user interface for WrAFT is publicly available and free to use.