Paper deep dive
Beyond Score Prediction: LLM-Based Essay Scoring and Feedback Generation via Reinforcement Learning with Rubric Rewards
Xuefeng Jin, Jiashuo Zhang, Teng Cao, Bin Yang
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 95%
Last extracted: 7/25/2026, 1:28:34 AM
Summary
The paper introduces RLAES, a unified framework for Automated Essay Scoring (AES) and Automated Feedback Generation (AFG) using Reinforcement Learning (RL). It proposes Rubric-based Feedback Evaluation (RFE) with 166 binary rubric items to measure feedback quality, Adaptive Gated Feedback Optimization (AGFO) to optimize reward computation efficiency, and Adjacent Contrastive Reasoning (ACR) to improve ordinal score calibration. RLAES-AGFO achieves state-of-the-art performance on the ASAP benchmark with a QWK of 0.803.
Entities (9)
Relation Signals (8)
RLAES-AGFO → evaluatedon → ASAP
confidence 99% · On the ASAP benchmark, RLAES-AGFO achieves the best scoring performance
RFE → comprises → 166 binary rubric items
confidence 98% · Rubric-based Feedback Evaluation (RFE), an essay-grounded feedback evaluation framework comprising 166 fine-grained binary rubric items
RLAES-AGFO → achieves → QWK 0.803
confidence 97% · On the ASAP benchmark, RLAES-AGFO achieves the best scoring performance among LLM-based methods (QWK = 0.803)
RLAES → uses → RFE
confidence 95% · Building on RFE, we propose Adaptive Gated Feedback Optimization (AGFO)... RFE supplies rfeedback by evaluating f^ against Rfb.
RLAES → uses → AGFO
confidence 95% · We propose RLAES, a unified LLM framework... Building on RFE, we propose Adaptive Gated Feedback Optimization (AGFO)
RLAES → uses → ACR
confidence 95% · We also propose Adjacent Contrastive Reasoning (ACR) to improve ordinal score calibration... RLAES-AGFO achieves the best scoring performance
RLAES → usesbasemodel → Qwen3.5-9B
confidence 92% · In our post-training experiments, we use Qwen3.5-9B (Qwen Team 2026a) as the base model
RLAES → basedon → GRPO
confidence 90% · We use GRPO to jointly optimize AES and AFG.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Large language models (LLMs) have been widely applied to automated essay scoring (AES) and automated feedback generation (AFG). However, existing studies rely primarily on prompt engineering or supervised fine-tuning, while systematic research on reinforcement learning (RL) post-training and automated evaluation of feedback quality remains limited. We propose RLAES, a unified LLM framework that jointly optimizes essay scoring and feedback generation through RL. To make feedback quality measurable, interpretable, and usable for training, we introduce Rubric-based Feedback Evaluation (RFE), an essay-grounded feedback evaluation framework comprising 166 fine-grained binary rubric items and an LLM-as-judge. Building on RFE, we propose Adaptive Gated Feedback Optimization (AGFO), which activates rubric-based feedback rewards on demand during RL, reducing evaluation overhead while improving feedback quality. We also propose Adjacent Contrastive Reasoning (ACR) to improve ordinal score calibration by explicitly contrasting adjacent score levels. Experimental results show that the RFE framework captures essay-feedback consistency, exhibits strong pairwise discriminative power, and closely aligns with expert preferences. On the ASAP benchmark, RLAES-AGFO achieves the best scoring performance among LLM-based methods (QWK = 0.803), while maintaining feedback quality comparable to GPT-5.5 and avoiding the feedback degradation observed under score-only RL. Code and datasets are publicly available at this https URL.
Tags
Links
- Source: https://arxiv.org/abs/2607.19219v1
- Canonical: https://arxiv.org/abs/2607.19219v1
Trouble viewing inline? Open PDF directly →
Full Text
52,987 characters extracted from source content.
Expand or collapse full text
Beyond Score Prediction: LLM-Based Essay Scoring and Feedback Generation via Reinforcement Learning with Rubric Rewards Xuefeng Jin, Jiashuo Zhang, Teng Cao, Bin Yang Contact: myjinxf@163.com Abstract Large language models (LLMs) have been widely applied to automated essay scoring (AES) and automated feedback generation (AFG). However, existing studies rely primarily on prompt engineering or supervised fine-tuning, while systematic research on reinforcement learning (RL) post-training and automated evaluation of feedback quality remains limited. We propose RLAES, a unified LLM framework that jointly optimizes essay scoring and feedback generation through RL. To make feedback quality measurable, interpretable, and usable for training, we introduce Rubric-based Feedback Evaluation (RFE), an essay-grounded feedback evaluation framework comprising 166 fine-grained binary rubric items and an LLM-as-judge. Building on RFE, we propose Adaptive Gated Feedback Optimization (AGFO), which activates rubric-based feedback rewards on demand during RL, reducing evaluation overhead while improving feedback quality. We also propose Adjacent Contrastive Reasoning (ACR) to improve ordinal score calibration by explicitly contrasting adjacent score levels. Experimental results show that the RFE framework captures essay-feedback consistency, exhibits strong pairwise discriminative power, and closely aligns with expert preferences. On the ASAP benchmark, RLAES-AGFO achieves the best scoring performance among LLM-based methods (QWK = 0.803), while maintaining feedback quality comparable to GPT-5.5 and avoiding the feedback degradation observed under score-only RL. Code and datasets are publicly available at https://github.com/hellomuyi/RLAES. 1 Introduction Automated Essay Scoring (AES) and Automated Feedback Generation (AFG) are core applications of artificial intelligence in education (Abudalfa and Barrot 2026). These systems use natural language processing, machine learning, and large language models (LLMs) to automatically score student essays and generate constructive feedback. Together, AES and AFG can reduce educators’ grading workload while providing students with timely, personalized diagnostic feedback that helps them identify weaknesses and improve their writing over time (Li and Collins 2026). Research on AES dates back to the 1960s (Sun et al. 2025). Traditional AES approaches have evolved from regression models relying on hand-crafted features (Choi et al. 2026) to deep learning paradigms that formulate AES as numerical classification or regression with automatic representation learning. This evolution has progressed from convolutional neural networks (CNNs) and recurrent neural networks (RNNs) (Taghipour and Ng 2016) to more advanced Transformer-based pretrained language models, most notably the BERT family (Li and Pan 2025b; Xie et al. 2022). Recent LLM-based approaches to AES can be broadly divided into prompt engineering and fine-tuning. Prompt engineering studies have extensively explored zero-shot (Seßler et al. 2025; Su et al. 2025) and few-shot learning (Huang et al. 2026; Mansour et al. 2024), prompts augmented with scoring rubrics (Mansour et al. 2024), chain-of-thought (CoT) reasoning (Xiao et al. 2025), pairwise comparisons (Choi et al. 2026; Cai et al. 2025; Shibata and Miyamura 2025), and multi-agent frameworks (Wang et al. 2026). Fine-tuning studies focus mainly on supervised fine-tuning (SFT) (Xiao et al. 2025; Ormerod and Kwako 2024; Johnsi and Kumar 2025). These studies have shown that fine-tuned LLMs, even smaller open-source ones, substantially outperform their extremely large proprietary counterparts equipped with a range of enhanced prompting strategies. Beyond score prediction, less attention has been paid to AFG, including the evaluation and optimization of feedback, despite the pedagogical importance of formative feedback (Liu et al. 2024). Small models that perform competitively on AES typically use BERT-based encoder-only architectures and cannot generate feedback without an additional decoder. Existing LLM-based feedback generation methods either directly prompt an LLM (Sasaki et al. 2026; Xiao et al. 2024; Ormerod and Kwako 2024) or perform SFT on feedback generated by strong reasoning models (Xiao et al. 2025; Li and Pan 2025a). For feedback evaluation, a widely accepted framework for automated evaluation is still lacking. Traditional feedback evaluation relies on annotated feedback datasets (Sasaki et al. 2026; Liu et al. 2024). However, such annotations typically cover only a limited set of categories and are therefore ill-suited to evaluating the rich feedback generated by LLMs. LLM-generated feedback is generally evaluated manually (Xiao et al. 2025; Ormerod and Kwako 2024), but human evaluation is costly, difficult to scale, and difficult to convert into a reward signal for optimizing feedback quality. Li and Pan (2025a) propose an automated evaluation method that assesses feedback quality by computing the semantic similarity between the generated feedback and the scoring rubric descriptions associated with the gold score. Although LLMs have created new technical possibilities for AES and AFG, two challenges remain. On the one hand, reinforcement learning (RL) post-training remains underexplored in LLM-based AES and AFG. Current LLM approaches focus primarily on prompt engineering and SFT. Prompt engineering is flexible but sensitive to prompt design, whereas SFT directly imitates gold scores. RL may be better suited to score reasoning and alignment with scoring rubrics, but its effects on both scoring accuracy and feedback quality have not been systematically studied. On the other hand, AFG lacks effective and scalable methods for feedback evaluation and optimization. LLM-generated feedback is often rich and structurally complex, leading existing work to rely heavily on human assessment (Xiao et al. 2025; Ormerod and Kwako 2024). Automated alternatives such as BERTScore measure the semantic similarity between generated feedback and scoring rubric descriptions associated with the gold score (Li and Pan 2025a). However, the gap between general, high-level scoring rubrics and essay-specific feedback limits the effectiveness of this evaluation strategy. The lack of an effective feedback evaluation method directly limits AFG optimization. To address these challenges, we propose RLAES, a unified LLM-based RL post-training framework for AES and AFG. First, to fill the gap in effective feedback evaluation, we introduce Rubric-based Feedback Evaluation (RFE), a prompt-specific, essay-grounded framework comprising 166 binary rubric items and an LLM-as-judge. RFE provides a fine-grained automated measure of feedback quality that can serve as an RL reward. Building on RFE, we propose Adaptive Gated Feedback Optimization (AGFO), which activates rubric-based feedback rewards on demand during RL. AGFO reduces LLM-as-judge overhead while jointly optimizing scoring accuracy and feedback quality, thereby preventing the feedback degradation caused by score-only RL. The resulting RLAES-AGFO model achieves the best performance among LLM-based AES methods. In addition, we propose Adjacent Contrastive Reasoning (ACR), a general strategy that explicitly guides the model to contrast adjacent score levels. ACR improves sensitivity to fine-grained quality differences and alleviates adjacent confusion in ordinal classification. Our main contributions are summarized as follows: • We extend LLM-based AES and AFG methodology beyond prompt engineering and SFT to RL post-training. The resulting RLAES-AGFO jointly optimizes scoring accuracy and feedback quality and achieves the best performance among LLM-based methods. • We introduce RFE, an essay-grounded framework that fills the gap in fine-grained feedback evaluation for AFG. • We propose ACR, a general strategy for ordinal classification tasks. ACR enables models to perceive and distinguish adjacent score levels, improving scoring accuracy. 2 Preliminaries 2.1 Task Definition Let =(ei,si)i=1ND=\(e_i,s_i)\_i=1^N denote a dataset, where eie_i represents the i-th essay together with its scoring context, such as the writing prompt, source text, and scoring rubric guidelines, and sis_i is its corresponding holistic score. The objective is to train a model on D that generalizes well to unseen essays, enabling accurate score prediction for AES and reliable feedback generation for AFG. We focus on prompt-specific AES and AFG, where the writing prompts used for training and testing are drawn from the same set. We define feedback (or explanation) as all components of an AES system’s output other than the score prediction. We structure the output such that the model first generates feedback and then predicts a score. In this setting, the feedback can be viewed as a CoT-like intermediate rationale before the final score. 2.2 RL Post-Training Reinforcement learning with verifiable rewards (RLVR) relies on explicit, verifiable binary rules as reward signals, such as output correctness. DeepSeek-R1 (Guo and others 2025) demonstrated its effectiveness in improving reasoning capabilities. Related algorithms such as Group Relative Policy Optimization (GRPO) (Shao et al. 2024) have been widely applied to domains with checkable answers, including mathematics and code. For difficult-to-verify domains such as essay feedback evaluation, however, feedback quality cannot be determined by a single binary rule and instead requires multidimensional assessment. Directly applying RLVR in such settings would yield an overly coarse reward signal. Conversely, reinforcement learning from human feedback (RLHF) can use human preferences to handle open-ended outputs, but preference judgments are highly subjective and may induce reward hacking. Rubrics as Rewards (RaR) (Gunjal et al. 2025) decomposes evaluation criteria into individually assessable rubric items, bridging the gap between strictly verifiable rewards and subjective human preferences. It provides structured rewards for open-ended tasks and is directly compatible with on-policy RL algorithms such as GRPO. Specifically, given an input prompt x and a model response y y, the rubric reward is defined as r(x,y^)=∑j=1kwjcj(x,y^)∑j=1kwj,r(x, y)= _j=1^kw_jc_j(x, y) _j=1^kw_j, (1) where k is the number of rubric items, wj>0w_j>0 is the weight of item j, and cj:(x,y^)↦0,1c_j:(x, y) \0,1\ indicates whether the response y y to the model input x satisfies that item. An LLM-as-judge assigns each indicator cjc_j independently, and all items (wj,cj)j=1k\(w_j,c_j)\_j=1^k define the rubrics. 3 Approach 3.1 Rubric-Based Feedback Evaluation (RFE) Framework Inspired by Deep Research Bench I (Li et al. 2026) and RaR (Gunjal et al. 2025), we propose the Rubric-based Feedback Evaluation (RFE) framework, which supports automated feedback evaluation and converts feedback quality into a scalar reward for RL training. RFE comprises prompt-specific, checklist-style rubrics and an LLM-as-judge. We will release these resources with the code. Figure 1 illustrates the construction pipeline. Figure 1: Construction of the RFE rubrics. Reference Feedback Extraction. We use Claude-Opus-4.7 to extract handwritten essays and teacher feedback from official ASAP materials (Hamner et al. 2012). Manual verification yields 92 essay–feedback pairs across eight writing prompts. We remove feedback that is overly brief, merely repeats scoring rubric guidelines without essay-specific evidence, or omits too many trait domains, retaining 50 high-quality samples as pseudo-gold feedback. Because teacher feedback has practical limitations, such as omitting revision suggestions or providing only a single high-level assessment, we refine these samples through LLM rewriting, manual calibration, and expert validation, producing 50 gold feedback samples. To construct negative examples, we order the samples by prompt and score and pair each essay with feedback from an adjacent-scoring essay under the same prompt, yielding 88 perturbed feedback samples. Rubric Generation and Iteration. The Main Agent generates initial rubrics with manually adjusted top-level dimensions and fine-grained rubric items. It then evaluates the rubrics on the gold feedback and produces an analysis report. Five Rubric Refinement Agents—GPT-5.5, Gemini-3.1-Pro, Claude-Opus-4.8, DeepSeek-V4-Pro, and GLM-5.1—independently propose revisions. The Main Agent aggregates these proposals and updates the rubrics until the agents’ evaluations converge. After manual revision, the final rubrics, denoted by ℛfbR_fb, contain 166 binary items across four dimensions: Coverage, Evidence, Faithfulness, and Safety. LLM-as-Judge Evaluation. Given an essay–feedback pair (ei,fi)(e_i,f_i), an LLM-as-judge independently determines whether each rubric item is satisfied. The RFE score is then computed using Equation 1. We assign equal weight to all items (wj=1w_j=1); thus, each dimension’s contribution is proportional to its number of items. 3.2 Adaptive Gated Feedback Optimization (AGFO) We use GRPO to jointly optimize AES and AFG. The reward function consists primarily of a score reward and a feedback reward. Given a student essay and its context e, the gold score s, the model-predicted score s s, the model-generated feedback f f, and the feedback rubrics ℛfbR_fb, the total reward rtotalr_total is defined as rtotal=rscore(s,s^)+λf⋅rfeedback(e,ℛfb,f^),r_total=r_score(s, s)+ _f· r_feedback(e,R_fb, f), (2) where λf _f is a hyperparameter controlling the weight of the feedback reward. The score reward is defined as rscore(s,s^)=−|s−s^|αM,r_score(s, s)=- |s- s |α\,M, (3) where M=smax−sminM=s_ -s_ is the prompt-specific score range and α is a scaling factor. RFE supplies rfeedbackr_feedback by evaluating f f against ℛfbR_fb. However, invoking an LLM-as-judge at every optimization step incurs prohibitive time and computational costs. We therefore propose Adaptive Gated Feedback Optimization (AGFO), which adaptively controls the activation of the feedback reward during training. Specifically, every n steps, the system invokes the LLM-as-judge to compute the current model’s feedback reward rfeedbackr_feedback on the batch and jointly optimizes scoring and feedback. If this reward is less than or equal to the predefined threshold τ, the gate remains open, and the judge is queried and the resulting feedback reward is included in the total reward at every subsequent step until rfeedbackr_feedback exceeds τ. Otherwise, the gate is closed between periodic checks, and the total reward contains only the score reward. Pseudocode for AGFO is provided in Algorithm 1. Notably, AGFO unifies two special cases via the threshold τ. When τ=0τ=0, the feedback reward is activated only every n steps, reducing AGFO to periodic feedback optimization. When τ=1τ=1, it is activated at every step, reducing AGFO to full feedback optimization. We denote the RL configuration using only the score reward as RLAES-Score and the configuration incorporating AGFO as RLAES-AGFO. Algorithm 1 Adaptive Gated Feedback Optimization (AGFO) 0: policy πθ _θ; dataset D; rubrics ℛfbR_fb; judge J; period n; threshold τ; weight λf _f; total steps T 0: optimized policy πθ _θ 1: r¯fb←0 r_fb← 0 2: for t←1t← 1 to T do 3: Sample mini-batch ℬ⊂B 4: Generate responses (f^i,s^i)∼πθ(⋅∣ei)\( f_i, s_i)\ _θ(· e_i), ei∈ℬe_i 5: if tmodn≠0t n≠ 0 and r¯fb>τ r_fb>τ then 6: rifb←0,∀ir^fb_i← 0,\ ∀ i gate closed 7: else 8: rifb←rfeedback(ei,ℛfb,f^i)r^fb_i← r_feedback(e_i,R_fb, f_i) gate open, query J 9: Update r¯fb r_fb with rifb\r^fb_i\ 10: end if 11: for all i∈ℬi do 12: ri←riscore+λfrifbr_i← r^score_i+ _f\,r^fb_i 13: end for 14: Update πθ _θ via GRPO using ri\r_i\ 15: end for 16: return πθ _θ 3.3 Adjacent Contrastive Reasoning (ACR) Automated essay scoring can be formulated as an ordinal classification problem: score levels are ordered labels rather than independent classes. The main difficulty is determining whether an essay belongs to a given score level or an adjacent lower or higher level. Standard prompting methods ask an LLM to produce a holistic evaluation and score but do not explicitly require comparisons at score boundaries. This omission may make the model more prone to predicting a score level adjacent to the gold score—a phenomenon we call adjacent confusion. To alleviate this issue, we propose Adjacent Contrastive Reasoning (ACR), inspired by the anchoring-and-adjustment heuristic in human judgment (Tversky and Kahneman 1974). As illustrated in Figure 2, ACR treats the adjacent lower and higher scores around a predicted score as local hard-negative labels and requires the model to explicitly answer two contrastive questions before producing its final score: Why should the essay not receive the adjacent lower/higher score? This local counterfactual comparison encourages the model to identify evidence that distinguishes adjacent score levels instead of producing only a generic holistic assessment. ACR improves scoring in three ways. First, it transforms scoring from direct score mapping into local boundary discrimination, thereby focusing the model’s attention on fine-grained differences between adjacent scores. Second, treating adjacent levels as hard negatives directs attention toward more discriminative textual evidence. Third, considering both adjacent directions provides a bidirectional calibration signal, helping reduce systematic overestimation or underestimation in the ordinal score space. ACR differs from existing few-shot prompting and pairwise-comparison methods, which introduce external reference essays and assist scoring by comparing the relative quality of a target essay against the reference. In contrast, ACR requires no external reference samples and adds only local comparisons with adjacent score levels. We also evaluate ACR-Lower, a one-sided variant that retains only the question of why the essay should not receive the adjacent lower score. As a general strategy, ACR may be extended to other ordinal classification problems. Figure 2: ACR for ordinal score calibration. 4 Experimental Setup 4.1 Dataset We evaluate on the Automated Student Assessment Prize (ASAP) dataset (Hamner et al. 2012), a widely adopted benchmark for English AES. The dataset comprises 12,978 essays from three writing types and eight writing prompts (Table 1). Following Taghipour and Ng (2016), we conduct five-fold cross-validation using their standardized train-dev-test splits. With one minor modification, we merge the training data from all eight prompts within the same fold and perform a single joint training run across prompts, rather than training a separate model for each prompt. This design simplifies the experimental procedure, avoids model fragmentation, and aligns with efficient parameter sharing and unified model serving in the LLM era. We provide detailed results in Appendix A, showing that prompt joint training incurs negligible performance degradation across model families, with QWK decreasing by no more than 0.005. Prompt Type Grade # Essays Avg. Len. Range 1 Arg. 8 1783 350 2–12 2 Arg. 10 1800 350 1–6 3 SD 10 1726 150 0–3 4 SD 10 1772 150 0–3 5 SD 8 1805 150 0–4 6 SD 10 1800 150 0–4 7 Narr. 7 1569 250 0–30 8 Narr. 10 723 650 0–60 Table 1: Statistics of the ASAP dataset (Avg. Len.: average essay length; Arg.: argumentative; SD: source-dependent; Narr.: narrative). 4.2 Evaluation Metrics We use quadratic weighted kappa (QWK), the most widely used metric in AES and the sole evaluation metric of the ASAP competition, to evaluate score prediction. QWK measures agreement between model predictions and human-assigned gold scores. For AFG evaluation, we employ our proposed RFE framework with GPT-5.5 as the LLM judge. Evaluating one essay–feedback pair against all 166 rubric items costs approximately $0.12. Due to budget constraints, the feedback evaluation is conducted on fold 0. Each fold covers the complete ASAP dataset, and folds differ only in how the dataset is split. Therefore, evaluation on a single fold remains sufficiently representative and valid. 4.3 Baseline Models We group the baselines into two categories: LLM prompt engineering and LLM post-training. LLM Prompt Engineering. These methods directly prompt LLMs to perform scoring without updating model parameters. We include GPT-3.5 with rubric-guided one-shot prompting (Mansour et al. 2024), GPT-4 with rubric prompting (Li and Pan 2025a), GPT-4 with few-shot prompting (Xiao et al. 2025), and Llama-3-8B-Instruct (Ormerod and Kwako 2024). We also evaluate vanilla prompting with DeepSeek-V4-Pro (DeepSeek-AI 2026) and Qwen3.7-Max (Qwen Team 2026b), which serve as the base settings for our ACR variants. LLM Post-Training. These methods adapt LLMs to AES by updating model parameters. Existing baselines in this category are primarily based on SFT, including fine-tuned GPT-3.5 (Xiao et al. 2024), the Dual-Process Model (Xiao et al. 2025), fine-tuned Llama-3.2-1B-Instruct (Johnsi and Kumar 2025), fine-tuned Llama-3-8B-Instruct (Ormerod and Kwako 2024), and RTS (Cai et al. 2025). For AFG, prior work remains limited, and unavailable code and differing tasks hinder fair comparison. We therefore use GPT-5.5, prompted with role-playing instructions and scoring rubric guidelines, as a strong feedback-generation baseline. 4.4 Implementation Details In our post-training experiments, we use Qwen3.5-9B (Qwen Team 2026a) as the base model and adopt low-rank adaptation (LoRA) for parameter-efficient fine-tuning. RLAES is trained for 25 epochs with a learning rate of 2.0×10−52.0× 10^-5 and a warmup ratio of 0.1. We use a cosine learning-rate scheduler with a minimum learning-rate ratio of 0.1 and set the per-device training batch size to 8. For GRPO, we set the Kullback–Leibler (KL) regularization coefficient to 0 and apply a length penalty similar to DAPO (Yu et al. 2025). During rollout generation, we sample 8 responses for each query at a temperature of 0.9. All experiments are conducted on eight NVIDIA H20/A100 GPUs with DeepSpeed ZeRO-3. The feedback reward weight λf _f is set to 1, and the score-interval scaling factor α is set to 0.1. For AGFO, the gating period n is set to 30, and the feedback reward threshold τ is set to 0.8 based on an offline estimate. For the final feedback evaluation, we use GPT-5.5 (OpenAI 2026) as the LLM-as-judge. During RL training, feedback rewards are computed using DeepSeek-V4-Pro (DeepSeek-AI 2026) to reduce computational and financial costs. The temperature is set to 0 for all judge models. Unless otherwise noted, GPT-5.5 refers to gpt-5.5-0424-global. 5 Results and Analysis 5.1 Results on Score Prediction Type Model P1 P2 P3 P4 P5 P6 P7 P8 AVG LLM Prompt Engineering GPT-3.5 (rubric,1-shot) (2024) 0.120 0.193 0.198 0.416 0.576 0.606 0.123 0.276 0.313 GPT-4 (w/ rubric) (2025a) 0.272 0.481 0.478 0.503 0.557 0.529 0.123 0.384 0.415 GPT-4 (few-shot) (2025) 0.280 0.338 0.331 0.784 0.623 0.728 0.257 0.454 0.474 Llama-3-8B-Instruct (2024) 0.255 0.463 0.432 0.557 0.653 0.608 0.283 0.362 0.452 DeepSeek-V4-Pro (vanilla) 0.165 0.519 0.403 0.608 0.503 0.638 0.215 0.392 0.430 + ACR 0.186 0.524 0.441 0.600 0.538 0.683 0.230 0.497 0.462 + ACR-Lower 0.190 0.519 0.413 0.610 0.545 0.724 0.242 0.526 0.471 Qwen3.7-Max (vanilla) 0.231 0.517 0.396 0.483 0.545 0.650 0.184 0.565 0.446 + ACR 0.248 0.500 0.487 0.594 0.593 0.668 0.217 0.478 0.473 + ACR-Lower 0.275 0.525 0.499 0.633 0.660 0.716 0.230 0.508 0.506 LLM Post-Training GPT-3.5 (2024) 0.741 0.618 0.704 0.859 0.796 0.848 0.727 0.614 0.738 Dual-Process Model (2025) 0.761 0.652 0.724 0.809 0.812 0.776 0.707 0.489 0.716 Llama-3.2-1B-Instruct (2025) 0.711 0.778 0.682 0.723 0.769 0.692 0.776 0.631 0.720 Llama-3-8B-Instruct (2024) 0.821 0.727 0.717 0.824 0.815 0.829 0.837 0.752 0.789 RTS (2025) 0.835 0.710 0.730 0.840 0.821 0.839 0.838 0.740 0.794 SFTAES 0.824 0.702 0.707 0.819 0.800 0.820 0.827 0.770 0.784 RLAES-Score 0.840 0.739 0.732 0.829 0.821 0.839 0.846 0.770 0.802 RLAES-AGFO 0.839 0.734 0.735 0.829 0.820 0.837 0.853 0.775 0.803 Table 2: Scoring performance (QWK) across the eight ASAP prompts. LLM Prompt Engineering and ACR. Table 2 reports QWK for LLM prompt engineering and post-training methods. GPT-4 (w/ rubric) and GPT-4 (few-shot) achieve average QWK values of 0.415 and 0.474, respectively, substantially below post-trained LLMs. This gap highlights the difficulty of directly prompting LLMs for AES. For DeepSeek-V4-Pro, average QWK increases from 0.430 with vanilla prompting to 0.462 with ACR and 0.471 with ACR-Lower. For Qwen3.7-Max, the corresponding values are 0.446, 0.473, and 0.506. As shown in Figure 3, both models systematically underestimate essay scores under vanilla prompting, whereas the one-sided ACR-Lower mitigates this bias by prompting them to identify the essay’s strengths. These results demonstrate that ACR improves the scoring accuracy of base models. Comparison between SFT and RLAES. SFTAES, our SFT baseline that outputs only the predicted score, achieves an average QWK of 0.784, outperforming all prompt engineering baselines and several LLM post-training baselines. With RL using only the score reward, RLAES-Score raises average QWK to 0.802, an improvement of 0.018 over SFTAES, and surpasses RTS (0.794), the strongest existing LLM post-training baseline. RLAES-AGFO achieves 0.803, indicating no observed reduction in scoring accuracy after adding AGFO. Moreover, RLAES-Score and RLAES-AGFO obtain the two highest average QWK values among the LLM post-training methods and come closest to the performance of a single human rater (0.805) (Uto 2026). Together with their improvements over SFTAES, these results further support the effectiveness of RL post-training for AES. Figure 3: Mean predicted scores and one-standard-deviation ranges under different ACR configurations. Dashed markers show prompt-level gold means. 5.2 Results on Feedback Generation Using the RFE framework, we compare the feedback quality (RFE score) of four models: GPT-5.5 as a reference; RLAES-Step 0 (the Qwen3.5-9B initialization before RLAES post-training); RLAES-Score, trained solely with score rewards; and RLAES-AGFO, trained with AGFO. Overall Feedback Quality Comparison. As shown in Table 3, AGFO improves the model’s feedback-generation capability: the average RFE score of RLAES-AGFO reaches 0.8399, comparable to that of GPT-5.5 (0.8334). Meanwhile, RLAES-AGFO achieves a slightly higher QWK than RLAES-Score (0.8082 vs. 0.8043), suggesting that AGFO does not trade scoring accuracy for feedback quality. By contrast, the RFE score of RLAES-Score decreases from 0.6978 at Step 0 to 0.5608 after score-only RL, indicating that optimizing only the score reward may degrade feedback quality. We examine a possible explanation by analyzing the training dynamics below. In terms of training efficiency, we normalize the training time of RLAES-Score to 1×1×. Full feedback optimization (τ=1τ=1) costs approximately 12×12×, whereas RLAES-AGFO costs 2×2× (about 160 hours per fold), representing a reduction of approximately 83%. RLAES-AGFO substantially reduces the overhead of frequent LLM-as-judge evaluations while achieving a favorable trade-off between training efficiency and model performance. Model P1 P2 P3 P4 P5 P6 P7 P8 AVG QWK GPT-5.5 0.7650 0.7843 0.8654 0.9064 0.8767 0.8904 0.8217 0.7576 0.8334 0.5615 RLAES-Step 0 0.6400 0.6075 0.8028 0.8389 0.7614 0.7597 0.6325 0.5396 0.6978 0.3343 RLAES-Score 0.4508 0.4590 0.6546 0.7042 0.7108 0.7126 0.4096 0.3846 0.5608 0.8043 RLAES-AGFO 0.8670 0.7870 0.9119 0.9242 0.9133 0.8804 0.7552 0.6798 0.8399 0.8082 Table 3: Feedback performance (RFE score) across the eight ASAP prompts, with overall QWK. Training Dynamics of Feedback Quality. Figure 4 tracks QWK and feedback quality (rubric reward, i.e., RFE score) on Prompt 1 test set at selected training checkpoints. For RLAES-Score, QWK rises from 0.052 to above 0.83, whereas rubric reward briefly increases from 0.640 to 0.702 before falling to 0.433. Thus, improvements in scoring accuracy do not necessarily translate into better feedback. A plausible explanation is that the model initially uses feedback as an intermediate reasoning path for score prediction, temporarily improving feedback quality. Once it learns to predict scores directly from input features, feedback no longer contributes to score optimization and, without direct optimization, begins to deteriorate. By contrast, at the final checkpoint, RLAES-AGFO achieves an RFE score of 0.866 versus 0.433 for RLAES-Score while maintaining comparable QWK, indicating that AGFO prevents feedback collapse without sacrificing scoring performance. Figure 4: QWK and rubric reward during RLAES-Score and RLAES-AGFO training. 5.3 Evaluation of the RFE Framework RFE Scores Across Feedback Types. Table 4 compares RFE scores for pseudo-gold, gold, and perturbed feedback. Gold feedback achieves an overall score of 0.969, whereas pseudo-gold feedback scores 0.694, with particularly low scores for Coverage (0.463) and Evidence (0.668). When each essay is paired with feedback from an adjacent-scoring essay, the overall score falls to 0.379, accompanied by sharp declines in Evidence (0.108) and Faithfulness (0.318). These results provide initial evidence that RFE detects essay–feedback inconsistency. Type Overall Cov. Evid. Faith. Safety Pseudo-gold 0.694 0.463 0.668 0.849 0.880 Gold 0.969 0.974 0.939 0.994 0.947 Perturbed 0.379 0.507 0.108 0.318 0.667 Table 4: Overall and dimension-level RFE scores of different feedback types (Cov.: Coverage; Evid.: Evidence; Faith.: Faithfulness). Pairwise Discriminative Power. We compare RFE with the BERTScore-based rubric-similarity method proposed by Li and Pan (2025a). As shown in Table 5, RFE ranks gold feedback above its perturbed counterpart for all 88 pairs (100.0%), whereas BERTScore does so for 40 of 78 applicable pairs (51.3%; Prompts 7 and 8 are excluded because their holistic scores cannot be uniquely mapped to trait-specific rubric descriptions). This contrast highlights a limitation of rubric-similarity evaluation: because BERTScore does not condition on the target essay, it cannot directly assess whether the feedback is faithful to that essay. RFE instead achieves stronger pairwise discrimination in this setting by using essay-grounded rubric items. Method Pairs G >> P G << P Tie BERTScore 78 40 38 0 RFE Score 88 88 0 0 Table 5: Pairwise discriminative performance on gold–perturbed feedback pairs. Agreement with Expert Preferences. To assess agreement with expert preferences, we sample 43 essays and use an LLM to generate two feedback responses of comparable overall quality but with subtle differences for each essay. We limit the within-pair length difference to 10% to control for length bias. A senior evaluator familiar with ASAP and proficient in English identifies the preferred response in each pair. We score both responses with RFE and BERTScore and compare their rankings with the expert preferences. Table 6 reports expert agreement and score separation on the 43 challenging feedback pairs. RFE matches the expert preferences on 40 pairs (93.0%), whereas BERTScore matches on 23 pairs (53.5%). RFE also yields larger score separation: its mean and maximum absolute within-pair differences are 0.1163 and 0.2857, versus 0.0133 and 0.0453 for BERTScore. These results indicate that RFE not only aligns more closely with expert preferences but also provides more discriminative feedback-quality scores. Method Agree. Avg. Abs. Δ Max. Abs. Δ BERTScore 23/43 (53.5%) 0.0133 0.0453 RFE Score 40/43 (93.0%) 0.1163 0.2857 Table 6: Agreement with expert preferences and score separation (Avg./Max. Abs. Δ : mean/maximum within-pair absolute score differences). 6 Conclusion We present RLAES-AGFO, an RL framework for jointly optimizing essay scoring and feedback generation. We introduce RFE for fine-grained feedback evaluation, yielding scores usable as RL rewards, and demonstrate that its rankings closely align with expert preferences. Building on RFE, AGFO adaptively activates feedback rewards during RL, reducing training time relative to full feedback optimization while avoiding the feedback degradation of score-only RL. We also introduce ACR to mitigate adjacent confusion in AES. Experiments show that RLAES-AGFO achieves the highest QWK among the LLM-based methods and an RFE score comparable to that of GPT-5.5, while ACR improves scoring performance across both evaluated base models. RLAES remains limited by its dependence on LLM judges, whose differing preferences can cause reward mismatch, particularly for complex scoring rubrics. Evaluation is also restricted to prompt-specific English essays from ASAP. Future work should examine cross-prompt, cross-domain, and multilingual settings, as well as alternative strategies for joint scoring and feedback optimization. References S. I. Abudalfa and J. S. Barrot (2026) Generative artificial intelligence for automated writing evaluation: a systematic review of trends, efficacy, and challenges. Assessing Writing 68, p. 101041. External Links: ISSN 1075-2935, Document, Link Cited by: §1. Y. Cai, K. Liang, S. Lee, Q. Wang, and Y. Wu (2025) Rank-Then-Score: Enhancing Large Language Models for Automated Essay Scoring. External Links: 2504.05736, Link Cited by: §1, §4.3, Table 2. H. Choi, M. Kang, J. Seong, and J. Huang (2026) Exploring zero-shot essay scoring: from feature-based to LLM-based approaches. Data Mining and Knowledge Discovery 40 (3), p. 35. External Links: Document, Link, ISSN 1573-756X Cited by: §1. DeepSeek-AI (2026) DeepSeek-v4: towards highly efficient million-token context intelligence. Cited by: §4.3, §4.4. A. Gunjal, A. Wang, E. Lau, V. Nath, Y. He, B. Liu, and S. Hendryx (2025) Rubrics as rewards: reinforcement learning beyond verifiable domains. External Links: 2507.17746, Link Cited by: §2.2, §3.1. D. Guo et al. (2025) DeepSeek-R1 incentivizes reasoning in LLMs through reinforcement learning. Nature 645 (8081), p. 633–638. External Links: Document, Link, ISSN 1476-4687 Cited by: §2.2. B. Hamner, J. Morgan, lynnvandev, M. Shermis, and T. V. Ark (2012) The hewlett foundation: automated essay scoring. Note: https://kaggle.com/competitions/asap-aesKaggle Cited by: §3.1, §4.1. Y. Huang, C. Palermo, and J. Wilson (2026) Accuracy and fairness of generative AI in automated essay scoring: Comparing GPT-4o, feature-based models, and human raters. Assessing Writing 69, p. 101047. External Links: Document, Link, ISSN 1075-2935 Cited by: §1. R. Johnsi and G. B. Kumar (2025) Enhancing automated essay scoring by leveraging LSTM networks with hyper-parameter tuned word embeddings and fine-tuned LLMs. Engineering Research Express 7 (2), p. 025272. External Links: Document, Link, ISSN 2631-8695 Cited by: §1, §4.3, Table 2. A. W. Li and P. Collins (2026) Formative feedback across sources: Student perceptions and writing outcomes with instructor, peer, and AI-generated feedback. Reading and Writing. External Links: Document, Link, ISSN 1573-0905 Cited by: §1. R. Li, M. Du, B. Xu, C. Zhu, X. Wang, and Z. Mao (2026) DeepResearch bench i: diagnosing deep research agents via rubrics from expert report. arXiv. External Links: Document, Link Cited by: §3.1. X. Li and W. Pan (2025a) CEAES: bidirectional reinforcement learning optimization for consistent and explainable essay assessment. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Vienna, Austria, p. 26267–26279. External Links: Link, Document Cited by: §1, §1, §4.3, §5.3, Table 2. X. Li and W. Pan (2025b) KAES: Multi-aspect Shared Knowledge Finding and Aligning for Cross-prompt Automated Scoring of Essay Traits. Proceedings of the AAAI Conference on Artificial Intelligence 39 (23), p. 24476–24484. External Links: Document, Link, ISSN 2159-5399 Cited by: §1. Y. Liu, J. Han, A. Sboev, and I. Makarov (2024) GEEF: a neural network model for automatic essay feedback generation by integrating writing skills assessment. Expert Systems with Applications 245, p. 123043. External Links: ISSN 0957-4174, Link, Document Cited by: §1. W. A. Mansour, S. Albatarni, S. Eltanbouly, and T. Elsayed (2024) Can Large Language Models Automatically Score Proficiency of Written Essays?. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), Torino, Italia, p. 2777–2786. External Links: Link Cited by: §1, §4.3, Table 2. OpenAI (2026) GPT-5.5 System Card. Note: https://openai.com/index/gpt-5-5-system-card/Accessed: 2026-07-14 Cited by: §4.4. C. M. Ormerod and A. Kwako (2024) Automated Text Scoring in the Age of Generative AI for the GPU-poor. External Links: 2407.01873, Link Cited by: §1, §1, §1, §4.3, §4.3, Table 2, Table 2. Qwen Team (2026a) Qwen3.5: towards native multimodal agents. External Links: Link Cited by: §4.4. Qwen Team (2026b) Qwen3.7: the agent frontier. External Links: Link Cited by: §4.3. T. Sasaki, R. Conijn, and M. C. Willemsen (2026) The Blind Spots in Automated Feedback Generation for Academic Writing. In Proceedings of the LAK26: 16th International Learning Analytics and Knowledge Conference (LAK 2026), p. 610–620. External Links: Document, Link Cited by: §1. K. Seßler, M. Fürstenberg, B. Bühler, and E. Kasneci (2025) Can AI grade your essays? A comparative analysis of large language models and teacher ratings in multidimensional essay scoring. In Proceedings of the 15th International Learning Analytics and Knowledge Conference (LAK 2025), p. 462–472. External Links: Document, Link Cited by: §1. Z. Shao, P. Wang, Q. Zhu, R. Xu, J. Song, X. Bi, H. Zhang, M. Zhang, Y. K. Li, Y. Wu, and D. Guo (2024) DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models. External Links: 2402.03300, Link Cited by: §2.2. T. Shibata and Y. Miyamura (2025) LCES: zero-shot automated essay scoring via pairwise comparisons using large language models. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, C. Christodoulopoulos, T. Chakraborty, C. Rose, and V. Peng (Eds.), Suzhou, China, p. 29988–30001. External Links: Link, Document, ISBN 979-8-89176-332-6 Cited by: §1. J. Su, Y. Yan, F. Fu, Z. Han, J. Ye, X. Liu, J. Huo, H. Zhou, and X. Hu (2025) EssayJudge: a multi-granular benchmark for assessing automated essay scoring capabilities of multimodal large language models. In Findings of the Association for Computational Linguistics: ACL 2025, Vienna, Austria, p. 6363–6389. External Links: Link, Document Cited by: §1. J. Sun, T. Song, W. Peng, and J. Song (2025) A survey of automated essay scoring: challenges, advances, and future. Neurocomputing 650, p. 130916. External Links: ISSN 0925-2312, Link, Document Cited by: §1. K. Taghipour and H. T. Ng (2016) A neural approach to automated essay scoring. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, J. Su, K. Duh, and X. Carreras (Eds.), Austin, Texas, p. 1882–1891. External Links: Link, Document Cited by: §1, §4.1. A. Tversky and D. Kahneman (1974) Judgment under uncertainty: heuristics and biases. Science 185 (4157), p. 1124–1131. External Links: Document Cited by: §3.3. M. Uto (2026) Has automated essay scoring reached sufficient accuracy? deriving achievable qwk ceilings from classical test theory. In International Conference on Artificial Intelligence in Education, p. 555–570. Cited by: §5.1. Y. Wang, Z. Ding, X. Wu, S. Sun, N. Liu, and X. Zhai (2026) AutoSCORE: Enhancing Automated Scoring with Multi-Agent Large Language Models via Structured Component Recognition. Proceedings of the AAAI Conference on Artificial Intelligence 40 (48), p. 40898–40906. External Links: Document, Link Cited by: §1. C. Xiao, W. Ma, Q. Song, S. X. Xu, K. Zhang, Y. Wang, and Q. Fu (2025) Human-AI Collaborative Essay Scoring: A Dual-Process Framework with LLMs. In Proceedings of the 15th International Learning Analytics and Knowledge Conference (LAK 2025), p. 293–305. External Links: Document, Link Cited by: §1, §1, §1, §4.3, §4.3, Table 2, Table 2. C. Xiao, W. Ma, S. X. Xu, K. Zhang, Y. Wang, and Q. Fu (2024) From Automation to Augmentation: Large Language Models Elevating Essay Scoring Landscape. CoRR abs/2401.06431. External Links: 2401.06431, Document, Link Cited by: §1, §4.3, Table 2. J. Xie, K. Cai, L. Kong, J. Zhou, and W. Qu (2022) Automated Essay Scoring via Pairwise Contrastive Regression. In Proceedings of the 29th International Conference on Computational Linguistics, Gyeongju, Republic of Korea, p. 2724–2733. External Links: Link Cited by: §1. Q. Yu, Z. Zhang, R. Zhu, Y. Yuan, X. Zuo, Y. Yue, W. Dai, T. Fan, G. Liu, J. Liu, L. Liu, X. Liu, H. Lin, Z. Lin, B. Ma, G. Sheng, Y. Tong, C. Zhang, M. Zhang, R. Zhang, W. Zhang, H. Zhu, J. Zhu, J. Chen, J. Chen, C. Wang, H. Yu, Y. Song, X. Wei, H. Zhou, J. Liu, W. Ma, Y. Zhang, L. Yan, Y. Wu, and M. Wang (2025) DAPO: an open-source llm reinforcement learning system at scale. In Advances in Neural Information Processing Systems, D. Belgrave, C. Zhang, H. Lin, R. Pascanu, P. Koniusz, M. Ghassemi, and N. Chen (Eds.), Vol. 38, p. 113222–113244. External Links: Link Cited by: §4.4. Appendix A Joint Training Across Prompts ASAP is commonly evaluated by performing five-fold cross-validation independently for each of its eight prompts. Consequently, this “single-prompt, single-model” paradigm requires 8×5=408× 5=40 training runs for a complete evaluation, complicating the experimental workflow and prolonging the evaluation cycle, particularly for computationally intensive LLM post-training. This model fragmentation also increases deployment and maintenance costs and conflicts with the efficient parameter sharing and unified model serving expected of LLM-based systems. We instead pool the training splits of all eight prompts within each fold and train a single model, which is evaluated on the corresponding test splits. Because each input includes prompt-specific scoring guidelines, the model can handle different score ranges without normalization. This protocol reduces a complete evaluation from 40 to 5 training runs while preserving the original data partitions: no test examples are included in the pooled training set, thereby eliminating the risk of data leakage from the test sets. Evaluation of Joint Training Across Prompts. We evaluate prompt joint training with NPCR, SFTAES, and RLAES-Score, representing models spanning three method classes: the BERT-based AES model, LLM SFT, and LLM RL, respectively. NPCR is the most competitive open-source BERT-based baseline. The evaluation is conducted on fold 0. As shown in Table 7, joint training changes average QWK by −0.0048-0.0048, −0.0027-0.0027, and −0.0041-0.0041, respectively. All decreases are below 0.005 for any of the three model classes, indicating only a negligible loss in scoring performance. Overall, prompt joint training reduces the number of training runs from 40 to 5 with negligible performance loss, while simplifying the experimental workflow and enabling unified modeling and efficient deployment. Appendix B Additional Ablation Studies Ablation on Feedback Output. We compare two RLAES-Score configurations that use the same output format during training and inference: feedback-then-score and outputting only the score. As shown in Table 8, removing feedback from both training and inference decreases average QWK only marginally, from 0.8043 to 0.8036 (Δ=−0.0007 =-0.0007). This negligible difference indicates that, when optimization uses only the score reward, including feedback in the model output has little effect on scoring accuracy. Ablation on ACR in RL Post-Training. We examine whether ACR remains beneficial during RL post-training by training RLAES-Score with the ACR response format. As shown in Table 9, ACR increases average QWK only marginally, from 0.802 to 0.805 (Δ=+0.003 =+0.003). This gain is substantially smaller than that observed in LLM prompt engineering, where ACR raises average QWK from 0.430 to 0.462 for DeepSeek-V4-Pro and from 0.446 to 0.473 for Qwen3.7-Max. One explanation is that ACR provides base models with an explicit inductive bias for adjacent-score discrimination, whereas RL post-training can internalize the same logic through direct optimization of the score reward, leaving limited additional benefit from ACR. Model P1 P2 P3 P4 P5 P6 P7 P8 AVG QWK QWK NPCR w/o joint 0.8260 0.7170 0.7270 0.8650 0.8400 0.8340 0.7920 0.7410 0.7928 – NPCR 0.8334 0.7301 0.7328 0.8170 0.8267 0.8462 0.7880 0.7292 0.7879 −0.0048-0.0048 SFTAES w/o joint 0.8183 0.7006 0.7287 0.8303 0.8260 0.8466 0.8132 0.7401 0.7880 – SFTAES 0.8240 0.7044 0.7158 0.8244 0.8014 0.8319 0.8139 0.7663 0.7853 −0.0027-0.0027 RLAES w/o joint 0.8487 0.7392 0.7466 0.8509 0.8455 0.8498 0.8410 0.7452 0.8084 – RLAES 0.8331 0.7529 0.7620 0.8355 0.8332 0.8462 0.8333 0.7386 0.8043 −0.0041-0.0041 Table 7: Effect of prompt joint training on scoring performance (QWK). RLAES denotes RLAES-Score trained using only the score reward. Model P1 P2 P3 P4 P5 P6 P7 P8 AVG QWK RLAES-Score (w/ FB) 0.8331 0.7529 0.7620 0.8355 0.8332 0.8462 0.8333 0.7386 0.8043 RLAES-Score (w/o FB) 0.8282 0.7348 0.7485 0.8573 0.8291 0.8366 0.8354 0.7590 0.8036 Table 8: Effect of feedback output on scoring performance (QWK). “w/ FB” and “w/o FB” denote feedback-then-score and score-only outputs, respectively. Model P1 P2 P3 P4 P5 P6 P7 P8 AVG QWK RLAES-Score 0.840 0.739 0.732 0.829 0.821 0.839 0.846 0.770 0.802 – RLAES-Score (w/ ACR) 0.835 0.721 0.730 0.839 0.820 0.837 0.853 0.803 0.805 +0.003+0.003 Table 9: Effect of ACR-formatted RL post-training on scoring performance (QWK). Δ denotes the change in average QWK relative to RLAES-Score. Appendix C Experimental Prompts Listing 1: LLM-as-judge prompt template for RFE. ⬇ ### I. Role & Objective You are a senior educational assessment expert evaluating the quality of feedback produced by an Automated Essay Scoring (AES) system. You apply rigorous, high-standard evaluation criteria. Your task: For each rubric item below, judge whether the AES feedback satisfies that item (1 = yes, 0 = no). Base your judgment strictly on the provided materials. Apply strict standards - partial or borderline satisfaction should be scored 0. ### I. Guiding Principles - An AES system’s output consists of two parts: **Explanations** (textual feedback/commentary) and **Score** (numeric rating). In this evaluation, "feedback" refers exclusively to the Explanations portion. Evaluate ONLY the Explanations text; ignore the score the AES system assigned. - Each rubric item must be judged independently. Do not let one item’s judgment influence another. - **Anonymization placeholders**: ‘@CAPS1‘, ‘@LOCATION1‘, ‘@PERCENT1‘, ‘@DATE1‘, ‘@ORGANIZATION1‘, ‘@PERSON1‘, ‘@MONEY1‘, ‘@TIME1‘, etc., are de-identified placeholders inserted by Named Entity Recognition (NER) during data preprocessing. Assume they represent correct entities. The AES feedback should NOT interpret, comment on, or explain these placeholders. - **OCR / typo handling**: Student essays may contain OCR errors. If the AES feedback quotes an OCR artifact as a student error, that is an incorrect citation. If the AES feedback quotes a genuine student spelling/grammar error, that is valid. - **Evidence Grounding**: When judging whether the feedback provides "specific references," accept all of the following as valid evidence: (a) Direct quotation from the essay (verbatim text in quotes or clearly set off) (b) Specific paraphrase that points to identifiable, locatable content in the essay (c) Reference to a specific, identifiable detail (e.g., a particular argument, example, paragraph, or narrative event) Only reject vague generalizations that could apply to any essay (e.g., "the essay has good details" without specifying which). **Important**: For items requiring "at least two" references, count each distinct quotation/paraphrase/detail reference separately. Two references to the same passage count as one. - **Reasoning chain**: When an item asks whether the feedback "explains WHY" or provides a "reasoning chain," check that the feedback explicitly connects specific textual evidence to its evaluative conclusion (e.g., "The essay states X, which demonstrates Y because Z"). Merely juxtaposing a quote next to a judgment without explaining the connection does NOT satisfy this requirement. - **Conditional items**: Some items have conditions (e.g., "If the feedback does not address conventions, score 1."). If the condition is not met, the item is automatically scored 1. - **Reasoning-first**: For each item, first cite specific evidence from the AES feedback or student essay, then derive your judgment (0 or 1). - **When uncertain, default to 0** (strict evaluation). ### I. Input Materials <essay_prompt> essay_prompt </essay_prompt> <student_essay> student_essay </student_essay> <rubric_guidelines> rubric_guidelines </rubric_guidelines> <AES_feedback_to_evaluate> model_answer </AES_feedback_to_evaluate> ### IV. Rubric Items to Evaluate rubric_items_text ### V. Output Requirements 1. Output ONLY a valid JSON array. No markdown code block markers. 2. No additional explanatory text outside the JSON. 3. Each object in the array corresponds to one rubric item above. <output_format> [ "item_id": "<string>", "dimension": "<string>", "reason": "<evidence-driven explanation, cite specific content from the AES feedback or student essay>", "judgment": <0 or 1> , … ] </output_format> Now, please begin your evaluation. Listing 2: Prompt Template for AES and AFG ⬇ # Role As a virtual evaluator with expertise in English composition, your role is to critically analyze and grade student essays according to a predetermined set of rubrics. You are to act as an impartial judge and evaluate the essays based on the quality of the writing and adherence to the essay prompt. # Sample Essay Prompt essay_prompt # Student’s Essay to Evaluate essay # Please carefully read the following rubric guidelines. The scoring range is from min_score to max_score: scoring_rubric_guidelines Note: The essay contains anonymized tags such as @CAPS1, @LOCATION1, etc. Please treat these as normal names, places, or capitalized words, and DO NOT penalize the student for their presence. # Please strictly follow the format below when scoring. The explanations must be at least 100 words: - **Explanations**: x - **Score**: x/max_score Listing 3: Prompt Template for ACR ⬇ # Please strictly follow the format below when scoring. - **Explanations**: [Provide detailed analysis according to the rubric guidelines.] - **Why not the adjacent lower score**: [If your Score x is the minimum score min_score, simply state "This is the lowest possible score" and summarize why it falls into the lowest tier. Otherwise, explicitly identify the adjacent lower score x-1, and explain why this essay is scored x instead of x-1.] - **Why not the adjacent higher score**: [If your Score x is the maximum score max_score, simply state "This is the highest possible score" and summarize why it falls into the top tier. Otherwise, explicitly identify the adjacent higher score x+1, and explain why this essay is scored x instead of x+1.] - **Score**: x/max_score Listing 4: Prompt Template for ACR-Lower ⬇ # Please strictly follow the format below when scoring. - **Explanations**: [Provide detailed analysis according to the rubric guidelines.] - **Why not the adjacent lower score**: [If your Score x is the minimum score min_score, simply state "This is the lowest possible score" and summarize why it falls into the lowest tier. Otherwise, identify the adjacent lower score x-1, briefly recall the rubric description for x-1, and explain why this essay clearly exceeds that description.] - **Score**: x/max_score