Paper deep dive
Optimizing In-Context Demonstrations for LLM-based Automated Grading
Yucheng Chu, Hang Li, Kaiqi Yang, Yasemin Copur-Gencturk, Kevin Haudek, Joseph Krajcik, Jiliang Tang
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 91%
Last extracted: 7/20/2026, 5:39:04 AM
Summary
The paper introduces GUIDE (Grading Using Iteratively Designed Exemplars), a framework for optimizing in-context learning (ICL) for automated grading of open-ended student responses. GUIDE addresses the limitations of standard semantic similarity retrieval by employing a boundary-focused optimization problem. It uses contrastive operators within a Bayesian optimization loop to select 'boundary pairs'—semantically similar examples with different grades—and generates discriminative rationales to explain score distinctions. This approach improves rubric adherence and performance on borderline cases across physics, chemistry, and pedagogical datasets.
Entities (10)
Relation Signals (8)
GUIDE → appliedto → Physics
confidence 95% · Extensive experiments across datasets in physics... demonstrate that GUIDE significantly outperforms standard retrieval baselines.
GUIDE → appliedto → Chemistry
confidence 95% · Extensive experiments across datasets in... chemistry... demonstrate that GUIDE significantly outperforms standard retrieval baselines.
GUIDE → uses → In-Context Learning
confidence 95% · GUIDE operates on a continuous loop of selection and refinement... employing novel contrastive operators... to identify 'boundary pairs'.
GUIDE → outperforms → Standard Retrieval Baselines
confidence 93% · Extensive experiments... demonstrate that GUIDE significantly outperforms standard retrieval baselines.
GUIDE → optimizes → Boundary Pairs
confidence 92% · GUIDE reframes exemplar selection and refinement in automated grading as a boundary-focused optimization problem.
GUIDE → generates → Discriminative Rationales
confidence 90% · We enhance exemplars by generating discriminative rationales that explicitly articulate why a response receives a specific score.
Standard Retrieval Methods → failstocapture → Subtle Decision Boundaries
confidence 88% · Standard retrieval methods typically select examples based on semantic similarity, which often fails to capture subtle decision boundaries required for rubric adherence.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Automated assessment of open-ended student responses is a critical capability for scaling personalized feedback in education. While large language models (LLMs) have shown promise in grading tasks via in-context learning (ICL), their reliability is heavily dependent on the selection of few-shot exemplars and the construction of high-quality rationales. Standard retrieval methods typically select examples based on semantic similarity, which often fails to capture subtle decision boundaries required for rubric adherence. Furthermore, manually crafting the expert rationales needed to guide these models can be a significant bottleneck. To address these limitations, we introduce GUIDE (Grading Using Iteratively Designed Exemplars), a framework that reframes exemplar selection and refinement in automated grading as a boundary-focused optimization problem. GUIDE operates on a continuous loop of selection and refinement, employing novel contrastive operators to identify "boundary pairs" that are semantically similar but possess different grades. We enhance exemplars by generating discriminative rationales that explicitly articulate why a response receives a specific score to the exclusion of adjacent grades. Extensive experiments across datasets in physics, chemistry, and pedagogical content knowledge demonstrate that GUIDE significantly outperforms standard retrieval baselines. By focusing the model's attention on the precise edges of rubric, our approach shows exceptionally robust gains on borderline cases and improved rubric adherence. GUIDE paves the way for trusted, scalable assessment systems that align closely with human pedagogical standards.
Tags
Links
- Source: https://arxiv.org/abs/2603.00465v1
- Canonical: https://arxiv.org/abs/2603.00465v1
Trouble viewing inline? Open PDF directly →
Full Text
43,965 characters extracted from source content.
Expand or collapse full text
11institutetext: Michigan State University 22institutetext: University of Southern California Optimizing In-Context Demonstrations for LLM-based Automated Grading Yucheng Chu Hang Li Kaiqi Yang Yasemin Copur-Gencturk Kevin Haudek Joseph Krajcik Jiliang Tang Abstract Automated assessment of open-ended student responses is a critical capability for scaling personalized feedback in education. While large language models (LLMs) have shown promise in grading tasks via in-context learning (ICL), their reliability is heavily dependent on the selection of few-shot exemplars and the construction of high-quality rationales. Standard retrieval methods typically select examples based on semantic similarity, which often fails to capture subtle decision boundaries required for rubric adherence. Furthermore, manually crafting the expert rationales needed to guide these models can be a significant bottleneck. To address these limitations, we introduce GUIDE (Grading Using Iteratively Designed Exemplars), a framework that reframes exemplar selection and refinement in automated grading as a boundary-focused optimization problem. GUIDE operates on a continuous loop of selection and refinement, employing novel contrastive operators to identify “boundary pairs” that are semantically similar but possess different grades. We enhance exemplars by generating discriminative rationales that explicitly articulate why a response receives a specific score to the exclusion of adjacent grades. Extensive experiments across datasets in physics, chemistry, and pedagogical content knowledge demonstrate that GUIDE significantly outperforms standard retrieval baselines. By focusing the model’s attention on the precise edges of rubric, our approach shows exceptionally robust gains on borderline cases and improved rubric adherence. GUIDE paves the way for trusted, scalable assessment systems that align closely with human pedagogical standards. 1 Introduction Automated assessment is a central goal in artificial intelligence in education (AIED), offering the potential for timely feedback and scalable support in large, diverse classrooms [3]. Within this domain, automatically grading open-ended short answers remains technically demanding. Unlike keyword matching, valid assessment requires interpreting semantic nuances, evaluating causal reasoning and handling the immense linguistic variability students use to express identical concepts. Traditional supervised models, which rely on task-specific features and large annotated datasets often struggle to transfer effectively across changing curricula or distinct student populations due to overfitting [3]. Large language models (LLMs) have transformed this landscape by enabling in-context learning (ICL), allowing models to perform complex assessments given only a rubric and labeled demonstrations [2, 11]. This paradigm is particularly appealing for educational assessments as it leverages the model’s pre-trained world knowledge to generalize across different topics and question types without extensive, task-specific feature engineering. Despite these advantages, the reliability of LLM-based in-context graders is often compromised by their sensitivity to the selection and ordering of exemplars [20, 13]. This sensitivity is exacerbated by the noise inherent in authentic classroom data, where student responses exhibit various writing styles and partial reasoning that are difficult to separate [3, 7]. This in part reflects developmental learning, as learners take up new ideas and gradually develop more sophisticated understanding of disciplinary topics. To systematically enhance reliability, we approach the problem through two complementary perspectives: enriching the informational content of individual exemplars and optimizing the discriminative composition of the exemplar set. The first perspective addresses the ambiguity inherent in mapping noisy student text to a score. Augmenting examples with high-quality chain-of-thought (CoT) rationales can significantly improve model performance by explicitly breaking down the grading process [19]. In educational contexts, expert-written rationales are particularly valuable, as they not only justify the score but also model the specific pedagogical tone and rubric adherence required for valid assessment. However, manual curation of such expert explanations is labor-intensive and often prohibitively expensive to scale. To address this bottleneck, we seek to synthesize these high-quality rationales. By pairing existing responses with generated reasoning, we can construct robust exemplars without sole reliance on human experts. The second pathway focuses on defining the precise decision boundaries within the context window. We posit that the reliability issue often stems from suboptimal context composition, where the selected examples fail to clarify edge cases. Standard selection strategies, such as similarity-based retrieval [10], often fail in this domain because they prioritize semantic proximity over pedagogical utility. In a grading context, retrieving an example that is semantically similar to the student’s response is insufficient. Effective grading prompts require not merely representative examples, but contrastive pairs that appear semantically similar but receive different scores. This forces the model to attend to the nuanced differences that define the grade. While research in other domains has attempted to enhance ICL [18, 16], they have significant limitations when applied to automated grading. For example, BRIDGE [18] treats exemplars as independent discrete units and optimizes a set to maximize global accuracy. While effective for tasks like mathematical problem solving where answers are definitive, this approach often fails to capture subtle distinctions required for rubric adherence in educational settings. These challenges raise a pivotal question regarding how we can automatically construct and refine in-context exemplars that attend to boundary cases to maximize grading reliability under realistic constraints. To address this challenge, we present GUIDE (Grading Using Iteratively Designed Exemplars), an automated framework that reframes exemplar optimization in grading as a boundary definition problem. Unlike prior work that relies solely on global accuracy signals to select independent demonstrations, GUIDE enhances optimization process with the focus on discriminative interaction between examples. The framework operates on an iterative loop of “selection” and “refinement”. First, to select the most effective set of exemplars, we introduce novel contrastive operators within a constrained Bayesian optimization framework. Instead of randomly mutating the exemplar set, the contrastive operators actively seek boundary pairs that are semantically similar but possess different grades. The operators attempt to swap or insert these boundary cases into context window, forcing the optimizer to select exemplars that clearly delineates the decision boundary rather than just illustrating typical cases. Since selecting boundary examples is insufficient if the model does not understand why they differ, we adopt discriminative rationale generation. We prompt the LLM to generate contrastive rationales that explicitly articulate why a response receives a specific score and not its adjacent neighbors (for instance, explaining why a response is a score of 1 rather than 0 or 2). By automating the construction of these pedagogical boundary pairs, GUIDE not merely shows what the typical examples are, but teaches where the precise boundaries between distinct score levels are drawn. The optimized boundary pairs and their rationales are used as seeds to generate new synthetic rationales and select the ideal boundary cases for the next iteration. Finally, this optimal exemplar set with synthetic rationales can clarify the decision space while significantly reduce manual effort. We validate our approach through extensive empirical analysis on diverse educational datasets spanning the domains of physics science, chemistry, and pedagogical knowledge. Our research demonstrates that GUIDE consistently outperforms baselines by prioritizing the resolution of borderline cases, thereby offering a more robust and pedagogically valid foundation for automated grading. 2 Related Work 2.1 Automated Grading of Open-Ended Responses Automated grading for short-answer and essay scoring has evolved significantly over the past decades. Early systems relied heavily on hand-crafted features, which are fed into traditional supervised classifiers [3]. The deep learning era shifted the paradigm toward pre-trained language models like BERT and RoBERTa, which achieved state-of-the-art performance by fine-tuning on labeled datasets [5, 12]. Most recently, the field has adopted LLMs via ICL, where the model grades responses given only a rubric and a few examples [2]. While ICL reduces the labeling burden, recent studies highlight that off-the-shelf LLMs can be inconsistent, struggling with specific rubric constraints or exhibiting unsatisfactory performance based on student writing styles [14]. 2.2 Prompt and Exemplar Optimization in ICL In-context learning (ICL) performance is highly sensitive to the formatting and ordering of demonstrations, often requiring careful engineering to maximize model accuracy [20]. Dominant selection heuristics typically retrieve examples from a fixed pool based on semantic similarity or distribution coverage [10, 17]. Similarity metrics in grading often fail to distinguish between correct reasoning and plausible-sounding misconceptions that share high lexical overlap [16]. Furthermore, relying on a static pool prevents the correction of noisy or unclear student responses [3]. To address the fragility of fixed retrieval, recent research has shifted toward automating the design of instructions and demonstrations. Techniques like APE and the “optimize-generate” paradigm treat the prompt as a variable to be optimized, iteratively synthesizing or refining exemplars to improve performance [21, 18]. 3 Problem Statement We formulate automated grading as a classification task that maps a student response x∈x to a discrete score y∈y (e.g., =0,1,2Y=\0,1,2\). We utilize a frozen large language model ℳM to perform in-context learning, where predictions are conditioned on a context window composed of a task instruction ℐI, a grading rubric ℛR, and a set of demonstrations ℰ=e1,…,ekE=\e_1,…,e_k\. Each demonstration is a tuple ej=(xj,yj,rj)e_j=(x_j,y_j,r_j), containing a reference response, its ground-truth score, and a rationale explaining the grading logic. The model predicts the score for a query response x according to: y^=argmaxy∈Pℳ(y∣x,ℐ,ℛ,ℰ) y= *argmax_y P_M(y x,I,R,E) In practice, we assume access to a training dataset =(xi,yi)i=1NS=\(x_i,y_i)\_i=1^N. Optionally, a small subset of training examples exp⊆S_exp is associated with expert-annotated rationales (where expS_exp may be ∅ ), while the remainder lacks reasoning annotations. Our objective is to optimize the demonstration set ℰE for grading performance on a held-out validation set V. Specifically, we seek to select a subset of training examples and generate effective rationales for them, forming an optimal demonstration set ℰ∗=(xj,yj,rj)j=1kE^*=\(x_j,y_j,r_j)\_j=1^k where (xj,yj)∈(x_j,y_j) , to maximize the prediction accuracy of ℳM. 4 Method We propose GUIDE (Grading Using Iteratively Designed Exemplars), an iterative framework that constructs optimal in-context demonstration sets by targeting decision boundaries where nuanced grading errors occur. As illustrated in Fig. 1 and Algorithm 1, GUIDE alternates between two phases over T rounds. In the selection phase, constrained Bayesian optimization identifies a boundary-focused demonstration set ℰ∗E^* from a candidate pool P. In the generation phase, discriminative rationale generation synthesizes rubric-compliant rationales via contrastive infilling, thereby enriching P for the subsequent iteration. Figure 1: An overview of the GUIDE algorithm. 4.1 Phase 0: Candidate Pool Initialization GUIDE operates over a dynamic candidate pool P, which augments the training set S with generated rationales. Each exemplar in the pool is a triplet e=(x,y,r)e=(x,y,r), where r is a rationale explaining why input x warrants label y. Unlike the static training set, the pool evolves across rounds as new rationales are generated. We initialize the pool by retaining expert exemplars with their human-authored rationales (if available) and generating initial rationales for all remaining training examples (Algorithm 1 Line 1): 0←exp∪(xi,yi,ri(0)):(xi,yi)∈∖expP_0 _exp∪\(x_i,y_i,r_i^(0)):(x_i,y_i) _exp\ where exp=(xi,yi,riexp)(xi,yi)∈expP_exp=\(x_i,y_i,r_i^exp)\_(x_i,y_i) _exp denotes the expert exemplars with their human-authored rationales, and ri(0)r_i^(0) is generated by the LLM conditioned on the expert exemplars as context. 4.2 Phase 1: Bayesian Optimization with Contrastive Selection Given the pool tP_t at round t, we seek an optimal subset ℰt∗⊆tE_t^* _t that, when used as the few-shot context, yields the highest alignment with the ground truth rubric. Direct optimization is computationally intractable due to the exponential search space of potential subsets and the high cost of executing an LLM inference pass for every evaluation. To address this, we employ Bayesian optimization (BO) to efficiently navigate the search space (Lines 4–18). This approach maintains a probabilistic surrogate model to approximate the mapping between an exemplar subset and its grading quality, allowing us to identify promising candidates with minimal computational expense. Algorithm 1 GUIDE Framework 1:S, expS_exp, V, rounds T, budget nevaln_eval 2:ℰT∗E^*_T 3:Initialize 0←exp∪(xi,yi,ri(0))(xi,yi)∈∖expP_0 _exp∪\(x_i,y_i,r_i^(0))\_(x_i,y_i) _exp 4:for t=1t=1 to T do 5: Phase 1: Selection 6: ←∅D← ⊳ Observation history 7: for i=1i=1 to ninitn_init do ⊳ GP Initialization 8: ℰ←RandomSubset(t)E← RandomSubset(P_t) 9: ←∪(ℰ,Evaluate(ℰ,))D ∪\(E, Evaluate(E,V))\ 10: end for 11: Fit GP on D 12: for j=1j=1 to neval−ninitn_eval-n_init do ⊳ BO Iterations 13: Generate candidates C via contrastive operators 14: ℰnext←argmaxℰ∈EI(GP,ℰ)E_next← _E EI(GP,E) 15: ←∪(ℰnext,Evaluate(ℰnext,))D ∪\(E_next, Evaluate(E_next,V))\ 16: Refit GP on D 17: end for 18: a∗←max(ℰ,⋅)∈Acc(ℰ)a^*← _(E,·) Acc(E) ⊳ Best accuracy 19: ∗←(ℰ,⋅)∈:Acc(ℰ)=a∗D^*←\(E,·) :Acc(E)=a^*\ ⊳ Tie candidates 20: ℰt∗←argmaxℰ∈∗(Contrastive(ℰ),−|ℰ|)E^*_t← _E ^* (Contrastive(E),-|E| ) ⊳ Lexicographic 21: if t<Tt<T then 22: Phase 2: Generation 23: for (xi,yi)∈(x_i,y_i) do 24: ri(t)←ContrastiveInfill(xi,yi,ℰt∗)r_i^(t)← ContrastiveInfill(x_i,y_i,E^*_t) 25: end for 26: t+1←Merge&Cap(t,(xi,yi,ri(t)))P_t+1← Merge\&Cap(P_t,\(x_i,y_i,r_i^(t))\) 27: end if 28:end for 29:return ℰT∗E^*_T Optimization Objective To operationalize the concept of grading quality, we define a multi-objective function g(ℰ)g(E) that balances global accuracy with the structural properties of the context window. While high validation accuracy is the primary signal of success, relying on it alone can lead to unstable results due to the stochastic nature of LLMs. Therefore, we incorporate regularization terms that prioritize sparsity and pedagogical discriminability. Overall, we optimize three objectives simultaneously: (1) validation accuracy Acc(ℰ,)Acc(E,V), (2) sparsity −|ℰ|-|E| to encourage the selection of concise subsets that reduce token usage and inference latency, and (3) contrastive density Contrastive(ℰ)Contrastive(E) to reward boundary pair coverage. Rather than manually tuning fixed weights, we adopt Tchebycheff scalarization with randomized weights [1], which converts the multi-objective problem into a sequence of single-objective problems: g(ℰ)=max(w1⋅(Acc(ℰ,)−Acc∗),w2⋅(−|ℰ|))+w3⋅Contrastive(ℰ)g(E)= (w_1·(Acc(E,V)-Acc^*),\;w_2·(-|E|) )+w_3·Contrastive(E) where Acc∗Acc^* is the best accuracy observed so far, and the weights (w1,w2,w3)(w_1,w_2,w_3) are randomly sampled at each BO iteration. This randomization enables exploration of diverse trade-offs on the Pareto front without requiring manual hyperparameter tuning. The crucial component for enhanced reliability is the Contrastive(ℰ)Contrastive(E) term, which quantifies the subset’s ability to define decision boundaries. We calculate this score based on the density of “boundary pairs” within the selected subset. A pair of exemplars (ei,ej)(e_i,e_j) is considered a boundary pair if they possess high semantic similarity yet carry distinct ground truth labels. Formally, this is defined where sim(ei,ej)≥τsim(e_i,e_j)≥τ and ℓ(ei)≠ℓ(ej) (e_i)≠ (e_j) (sim(⋅)sim(·)111Cosine similarity between embeddings from text-embedding-3-small. Embeddings are computed for each exemplar by concatenating the input response and rationale. denotes cosine similarity and ℓ(⋅) (·) represents the ground truth label). By maximizing this term, the objective function explicitly rewards subsets that present the model with boundary scenarios (i.e., instances where adjacent scores are separated by subtle linguistic or logical nuances) thereby teaching the model to distinguish between grade levels rather than merely recognizing surface patterns. 4.2.1 Candidate Generation via Contrastive Operators To effectively navigate the combinatorial search space, we approach candidate generation from two complementary perspectives: greedy exploitation and global exploration. The greedy component focuses on iteratively refining the currently best-performing subset, ℰbestE_best, by generating slightly different candidates yet with structurally significant modifications. This local refinement is driven by our novel Contrastive Operators, which are designed to actively manipulate the decision boundaries within the context window. For each exemplar ei∈ℰbeste_i _best, we first identify a set of boundary candidates iC_i from the broader pool tP_t defined by: i=ej∈t∖ℰbest∣sim(ei,ej)≥τ∧|ℓ(ei)−ℓ(ej)|=1C_i=\e_j _t _best (e_i,e_j)≥τ | (e_i)- (e_j)|=1\. This constraint ensures that we focus exclusively on candidates that are semantically proximal to existing demonstrations yet belong to a strictly adjacent label. We utilize this set through two distinct operations that alter the decision space in different ways. First, the Contrastive-Add operator inserts a candidate ej∈ie_j _i into the existing subset alongside eie_i. By placing two semantically similar but distinctly labeled examples side-by-side, this operation increases the resolution of the context, forcing the model to attend to the specific micro-features that distinguish the grades, effectively sharpening the decision boundary. However, relying solely on addition would cause the subset to grow monotonically, eventually exceeding the context window or inflating inference costs. To mitigate this, we employ the Contrastive-Swap operator, which replaces eie_i with eje_j. This operation recalibrates the location of the decision boundary within the semantic space, allowing the optimizer to test whether shifting the local standard for a specific concept improves global accuracy. We also allow standard One-Flip perturbations to further fine-tune the subset [8]. Conversely, the exploration component prevents the algorithm from stagnating in local optima through Random Sampling, which introduces subsets drawn independently from the entire pool tP_t. Finally, to ensure the search is continually anchored by high-quality pedagogical priors, the manually curated expert subset is invariably added to the candidate pool in every iteration. 4.2.2 Efficient Evaluation and Selection To select the best candidate from the generated pool without running expensive inferences on all of them, we utilize a Gaussian process (GP) surrogate. Since exemplar subsets are discrete combinatorial objects, we encode them as binary membership vectors and project them into a continuous latent space. The GP is fitted on historically observed pairs of (subset, objective value), allowing it to predict the mean quality μ and uncertainty σ for all new candidates. We employ the Expected Improvement (EI) acquisition function to select the most promising candidate for actual LLM evaluation: EI(ℰ)=(μ(ℰ)−g+)Φ(z)+σ(ℰ)ϕ(z)EI(E)=(μ(E)-g^+) (z)+σ(E)φ(z), where g+g^+ represents the best objective value observed so far, and Φ , ϕφ are the standard normal CDF and PDF. The EI criterion naturally balances the exploitation of high-performing regions with the exploration of uncertain configurations. Once the chosen candidate is evaluated via the LLM, the result is used to update the GP, and the cycle repeats (Lines 10–15). Upon the exhaustion of the evaluation budget, the final subset is selected using a lexicographic criterion (Lines 16–18) that prioritizes validation accuracy first, followed by the contrastive score, ensuring that the final output is not only accurate but pedagogically robust. 4.3 Phase 2: Discriminative Rationale Generation Figure 2: Illustrative examples of demonstrations (student response, expert label, and synthetic rationale) generated by GUIDE. The second phase creates a feedback loop to improve the quality of the candidate pool tP_t itself. The logic driving this iteration is that the quality of a generated rationale is strictly dependent on the quality of the context provided during its generation. The optimal subset ℰt∗E^*_t identified in Phase 1 is, by definition, highly effective at distinguishing grade boundaries. We utilize this subset to regenerate rationales for the training data S, thereby creating higher-quality candidates for subsequent optimization rounds (Lines 21–24). Discriminative Rationale Generation GUIDE employs discriminative rationale generation via contrastive infill. For each training example (xi,yi)∈(x_i,y_i) , we perform teacher-forced infilling where the LLM is prompted to generate the rationale rir_i conditioned on the correct label yiy_i and the optimal exemplar set ℰt∗E^*_t as context. As shown in Fig. 3, the prompt explicitly requires the model to distinguish the true score from adjacent grades. This forces the articulation of specific rubric constraints and produces the high-quality semantic vectors necessary for contrastive operators. Size Regulation After each round t, the generated rationales form a new batch of exemplars that are merged into the existing pool: t+1←t∪(xi,yi,ri(t)):(xi,yi)∈P_t+1 _t∪\(x_i,y_i,r_i^(t)):(x_i,y_i) \ New exemplars are merged via deduplication, where two exemplars are considered duplicates only if they share identical input, label, and rationale triplets after normalization. This allows the pool to contain multiple rationale variants for the same input-label pair, enabling the optimizer to discover the most effective phrasing. To maintain tractable optimization, we enforce a maximum pool size NmaxN_ . If |t+1|>Nmax|P_t+1|>N_ , we apply a reduction procedure (Line 24) that partitions the pool into protected expert exemplars expP_exp and generated exemplars genP_gen. We retain all expert exemplars and sample Nmax−|exp|N_ -|P_exp| generated exemplars uniformly at random. This ensures that pedagogical priors are never discarded while maintaining diversity among the generated candidates. Instruction for Score 0: “Explain why this deserves a 0 (not a 1 or 2). Specifically mention what is missing that would be needed for a higher score.” Instruction for Score 1: “Explain why this deserves a 1 (not a 0 or 2). Mention what prevents it from being a 0, and what is missing for a 2." Instruction for Score 2: “Explain why this deserves a 2 (not a 0 or 1). Specifically mention what makes this sufficient for the highest score.” Figure 3: Sample prompts for discriminative rationale generation. The model is prompted to articulate specific boundary conditions based on the label. 4.4 Inference After T rounds of optimization, the framework converges on the final optimal demonstration set ℰT∗E^*_T. During inference, this frozen set serves as the static in-context prompt for all incoming test instances: y^=argmaxy∈Pℳ(y∣xtest,ℐ,ℛ,ℰT∗) y= *argmax_y P_M(y x_test,I,R,E^*_T) Unlike dynamic retrieval methods that select different examples for each query, GUIDE relies on the optimized ℰT∗E^*_T to provide a generalized decision surface. This ensures that the discriminative criteria learned during optimization are consistently applied to grading new submissions. 5 Experiments This section evaluates whether GUIDE improves automated grading. Our primary focus is on rubric boundaries where a student response falls between two adjacent score levels. These are the most difficult cases to grade and are most likely to be misgraded by exactly one point. 5.1 Experimental Setup 5.1.1 Datasets We conduct experiments on three datasets that represent different grading contexts in science education and teacher education. The first dataset ID_I contains 314 high school student responses to one assessment item associated with 11 rubrics. This dataset tests students’ knowledge of electrical interactions in high school physical science [9]. ID_I is assessed with binary labels 0,1\0,1\, where 0 denotes requirement ‘not met’ and 1 denotes requirement being ‘met’ for each of the 11 ideas targeted in the rubric. As ID_I lacks expert rationales, GUIDE initializes by synthesizing rationales from scratch. The second dataset CD_C is a chemistry education dataset developed within the 3DLP framework [6]. CD_C contains two tasks, each assessed along two dimensions: DCI (disciplinary core ideas) and SEP (science and engineering practices). CD_C is graded with ordinal labels 0,1,2\0,1,2\, where 0 denotes the requirement is ‘not met’, 1 denotes requirement is ‘partially met’, and 2 denotes requirement is ‘fully met’. The third dataset TD_T is a more complex teacher education dataset, which assesses teacher’s nuanced interpretations regarding pedagogical knowledge in teaching mathematics [4]. TD_T contains four grading tasks, each using ordinal labels 0,1,2\0,1,2\. Overall, we divide each dataset into train, validation, and test sets using a ratio of 3:1:1. The full statistical details of these datasets are in Table 1. Table 1: A summary of statistics of all datasets. Dataset Domain #Samples Scores #Expert Examples ID_I Interaction 314 0,1 0 TD_T Teacher Education 229∼ 236 0,1,2 3∼ 5 per category CD_C Chemistry 163∼ 184 0,1,2 3 per category 5.1.2 Model and Prompt We utilize GPT-4o-mini 222https://platform.openai.com/docs/models/gpt-4o-mini as the backbone for all grading, rationale generation, and inference tasks. To support the contrastive selection mechanism described in Section 4, we employ text-embedding-3-small 333https://platform.openai.com/docs/models/text-embedding-3-small to compute the pairwise similarity matrix required for the contrastive operators and the Contrastive(ℰ)Contrastive(E) objective. The optimization process is configured to run for T=5T=5 rounds. During Bayesian optimization in each round, the Gaussian process evaluates Neval=32N_eval=32 candidate subsets. To maintain a balance between context richness and a concise demonstration subset, we constrain the demonstration set size |E||E| to [4,16][4,16]. The semantic similarity threshold for constituting a boundary pair τ=0.7τ=0.7. The high threshold ensures that the Contrastive(ℰ)Contrastive(E) objective only rewards pairs that are semantically very close yet possess divergent labels. The candidate pool for the acquisition function is set to 256, and the maximum size of global exemplar pool P is capped at Nmax=512N_ =512 to ensure computational tractability. For GUIDE experiments, we use Tchebycheff scalarization with weights sampled at each iteration: w1∼Uniform(0.25,1.0)w_1 (0.25,1.0), w2=0.8⋅(1−w1)w_2=0.8·(1-w_1), and w3=0.2⋅(1−w1)w_3=0.2·(1-w_1). This eliminates the need for manual weight tuning while enabling exploration of diverse accuracy-parsimony-contrastive trade-offs. The temperature is set to 0.2 for all generation and evaluation calls to ensure stability while allowing for slight variations. All inputs are formatted using a fixed prompt template across all experiments. The prompt structure begins with system instruction and full rubric, followed by the selected in-context demonstrations, and ends with the target student response. Each demonstration contains student response, expert-assigned score, and an expert or generated rationale. All methods use the same task instruction and rubric and only differ in the selection of in-context demonstrations. 5.1.3 Baseline Methods We compare our framework with five representative exemplar selection baselines, including static strategies (fixed sets for all queries), dynamic strategies (query-specific retrieval), and optimization-based methods. All baselines use the same initial candidate pool and underlying LLM as our main method. The baseline details are as follows: (a) Naive uses a fixed demonstration set without optimization or selection beyond directly including the provided exemplars. This represents the standard “few-shot” or “zero-shot” performance. (b) Random randomly samples a fixed set of k exemplars from the training pool, keeping the subset constant for all test queries. This verifies whether sophisticated selection strategies outperforms chance selection. (c)KNN SBERT dynamically retrieves the top-k most semantically similar exemplars for each query using all-MiniLM-L6-v2 (SBERT) [15] and cosine similarity. This tests the hypothesis that semantic similarity alone yields the most relevant rubric context for grading specific inputs. (d)Vote-k constructs a diverse static set of k exemplars by maximizing the minimum distance between selected examples, ensuring broad coverage of the semantic space rather than clustering around the query. (e) BRIDGE [18] uses an optimize-generate loop to find demonstrations that maximizes general validation performance. Unlike GUIDE, it does not explicitly target rubric boundaries or generate discriminative rationales. Table 2: Dataset-level averages (arithmetic mean over categories/tasks). ID_I (11 categories) CD_C (4 tasks) TD_T (4 tasks) Method Acc QWK AdjErr Acc QWK AdjErr NonAdjErr Acc QWK AdjErr NonAdjErr Random 0.75 0.43 0.25 0.58 0.32 0.40 0.03 0.59 0.54 0.38 0.03 KNN SBERT 0.78 0.44 0.21 0.58 0.26 0.40 0.05 0.52 0.52 0.42 0.02 Vote-K 0.73 0.45 0.27 0.62 0.38 0.36 0.03 0.60 0.58 0.38 0.01 Naive 0.74 0.42 0.26 0.69 0.39 0.31 0.00 0.59 0.54 0.37 0.04 BRIDGE 0.90 0.57 0.19 0.76 0.53 0.24 0.00 0.66 0.65 0.32 0.02 GUIDE 0.92 0.62 0.08 0.80 0.59 0.20 0.00 0.71 0.67 0.28 0.02 Notes: red / green / blue denote the best / second-best / third-best method for each metric. 5.1.4 Metrics To ensure a comprehensive evaluation, we use three specific metrics. First, accuracy (exact match) measures the percentage of times the model predicts the exact correct score. Second, the quadratic weighted kappa (QWK) is particularly relevant for ordinal scales (like 0, 1, 2) because it penalizes disagreements based on their distance. In this metric, confusing a 0 with a 2 is penalized much more heavily than confusing a 0 with a 1. Thirdly, to directly diagnose boundary failures, we compute confusion matrix-based error rates. For ordinal labels 0,1,2\0,1,2\ (TD_T and CD_C), we define the adjacent error rate (AdjErr) as the function of predictions that are off by exactly one level (e.g., instances where the model predicts a 1 when the truth is 2). The non-adjacent error rate (NonAdjErr) measures predictions that skip a level, such as predicting 0 when the truth is 2, which represents a severe failure of logic. For the binary dataset ID_I, NonAdjErr is always zero by definition, and AdjErr captures all misclassifications. 5.2 Results 5.2.1 Overall grading quality Table 2 presents the aggregate performance across all datasets. On the ordinal datasets CD_C and TD_T, GUIDE consistently outperforms the baselines. As shown in the dataset-level averages, GUIDE achieves the highest accuracy and QWK across the board. On TD_T, Naive baseline struggles with an accuracy of 0.590.59 and QWK of 0.540.54, while GUIDE achieves substantial gains with an accuracy of 0.710.71 and QWK of 0.670.67. This represents a relative accuracy improvement of approximately 20%20\% over the naive baseline. On CD_C, GUIDE similarly dominates with an accuracy of 0.800.80 and QWK of 0.590.59, compared to the Naive accuracy of 0.690.69 and QWK of 0.390.39. On the interaction dataset ID_I, GUIDE demonstrates robust generalization to authentic student language, achieving the highest accuracy of 0.920.92 and QWK of 0.620.62, significantly surpassing standard baselines like Random (accuracy=0.750.75) and Naive (accuracy=0.740.74). 5.2.2 Boundary error decomposition A central hypothesis of GUIDE is that boundary-focused exemplars reduce confusion between adjacent scores. The error decomposition in Table 2 confirms this. Across all three datasets, GUIDE yields the lowest AdjErr among all methods. On ID_I, GUIDE reduces AdjErr to 0.080.08, a massive reduction compared to 0.260.26 for Naive. On TD_T and CD_C, GUIDE similarly achieves the lowest adjacent error rates of 0.280.28 and 0.200.20 respectively. Crucially, this reduction in AdjErr does not come at the cost of increased gross errors. The NonAdjErr remains negligible (0.000.00 and 0.020.02) for GUIDE, indicating that the primary challenge in these datasets is indeed boundary disambiguation, which the method effectively addresses. 5.2.3 Performance comparison vs. BRIDGE Comparing GUIDE directly to BRIDGE highlights the value of boundary-targeted optimization over general metric optimization. While BRIDGE consistently improves over Naive, GUIDE achieves further gains across all metrics. For TD_T, BRIDGE reaches an accuracy of 0.660.66, whereas GUIDE advances this to 0.710.71. Similarly, on CD_C, GUIDE outperforms BRIDGE with an accuracy of 0.800.80 compared to 0.760.76. The distinction is also evident in boundary errors; on ID_I, GUIDE achieves an AdjErr of 0.080.08 compared to 0.190.19 for BRIDGE. This suggests that explicitly contrasting adjacent labels and generating discriminative rationales provides a stronger learning signal for the LLM than simply optimizing for validation accuracy. 6 Discussion 6.0.1 The Importance of Boundary-Focused Learning Our findings suggest that one key challenge in automated grading lies in distinguishing between adjacent scores. Standard retrieval methods prioritize semantic similarity, which retrieve keyword-heavy examples that fail to clarify the grading criteria. By explicitly optimizing for “contrastive density,” GUIDE forces the model to confront “boundary pairs” that look similar but earn different scores. Experimental results demonstrate that focusing on these specific edge cases significantly sharpens its ability to apply the rubric correctly. This implies that effective ICL for grading requires examples that act as counterpoints rather than just representatives. 6.0.2 The Role of Discriminative Rationales A critical component of GUIDE is the generation of discriminative rationales. Our ablation studies reveal that standard expert rationales are not always sufficient for disambiguation. The effectiveness of GUIDE lies in its “boundary” reasoning capability. By prompting the model to explain not only why a score is appropriate but also why adjacent scores are inappropriate, we provide the grader with a clearer decision surface. 6.0.3 Efficiency and Scalability in Educational Contexts From a practical perspective, GUIDE offers a solution to the “cold-start” problem in automated grading bypassing the thousands of labeled examples required by traditional supervised learning. GUIDE can achieve high reliability with a small, optimized context window of 4 to 16 examples. Furthermore, the framework’s ability to synthesize rationales releases educators from manually writing detailed feedback for every training example. This allows the system to bootstrap from a small set of grades, making high-quality automated assessment accessible for new courses or changing curricula with minimal human effort. 6.0.4 Budget Although optimization relies on an iterative loop with multiple LLM calls during training, the financial cost remains highly accessible. Utilizing the cost-effective GPT-4o-mini model minimizes the optimization budget, with single-item optimization typically costing $5-8. This represents a manageable one-time investment, as the resulting optimized prompt can be deployed indefinitely for inference at a minimal marginal cost. 6.0.5 Limitations and Future Directions Future work should explore this boundary-focused optimization to multimodal tasks, such as grading diagrams or mathematical derivations, where the definition of a semantic neighbor may be complex. 7 Conclusion In this work, we addressed the reliability in LLM-based automated grading by shifting the focus from general accuracy to boundary precision with GUIDE. This framework iteratively selects and refines in-context demonstrations using novel contrastive operators, targeting semantically similar examples with different grades to clarify decision boundaries. Experiments across physics, chemistry, and teacher education datasets demonstrate that GUIDE significantly reduces adjacent grading errors and shows potential for high-quality, human-aligned automated assessment. References [1] V. J. Bowman Jr (1976) On the relationship of the tchebycheff norm and the efficient frontier of multiple-criteria objectives. In Multiple Criteria Decision Making: Proceedings of a Conference, Cited by: §4.2. [2] T. Brown et al. (2020) Language models are few-shot learners. NeurIPS. Cited by: §1, §2.1. [3] S. Burrows, I. Gurevych, and B. Stein (2015) The eras and trends of automatic short answer grading. IJAIED. Cited by: §1, §2.1, §2.2. [4] Y. Copur-Gencturk and T. Tolar (2022) Mathematics teaching expertise: a study of the dimensionality of content knowledge, pedagogical content knowledge, and content-specific noticing skills. Teaching and Teacher Education. Cited by: §5.1.1. [5] J. Devlin et al. (2019) Bert: pre-training of deep bidirectional transformers for language understanding. In NAACL, Cited by: §2.1. [6] P. He et al. (2023) Predicting student science achievement using post-unit assessment performances in a coherent high school chemistry project-based learning system. Journal of Research in Science Teaching. Cited by: §5.1.1. [7] A. Horbach and T. Zesch (2019) The influence of variance in learner answers on automatic content scoring. In Frontiers in education, Cited by: §1. [8] D. S. Johnson, C. H. Papadimitriou, and M. Yannakakis (1988) How easy is local search?. Journal of computer and system sciences. Cited by: §4.2.1. [9] L. Kaldaras, H. Akaeze, and J. Krajcik (2021) Developing and validating next generation science standards-aligned learning progression to track three-dimensional learning of electrical interactions in high school physical science. Journal of Research in Science Teaching. Cited by: §5.1.1. [10] J. Liu et al. (2021) What makes good in-context examples for gpt-33?. arXiv preprint arXiv:2101.06804. Cited by: §1, §2.2. [11] P. Liu et al. (2023) Pre-train, prompt, and predict: a systematic survey of prompting methods in natural language processing. ACM computing surveys. Cited by: §1. [12] Y. Liu et al. (2019) Roberta: a robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692. Cited by: §2.1. [13] S. Min et al. (2022) Rethinking the role of demonstrations: what makes in-context learning work?. arXiv preprint arXiv:2202.12837. Cited by: §1. [14] A. Mizumoto and M. Eguchi (2023) Exploring the potential of using an ai language model for automated essay scoring. RMAL. Cited by: §2.1. [15] N. Reimers and I. Gurevych (2019) Sentence-bert: sentence embeddings using siamese bert-networks. arXiv preprint arXiv:1908.10084. Cited by: §5.1.3. [16] O. Rubin, J. Herzig, and J. Berant (2022) Learning to retrieve prompts for in-context learning. In NAACL, Cited by: §1, §2.2. [17] H. Su et al. (2022) Selective annotation makes language models better few-shot learners. arXiv preprint arXiv:2209.01975. Cited by: §2.2. [18] X. Wan et al. (2025) From few to many: self-improving many-shot reasoners through iterative optimization and generation. arXiv preprint arXiv:2502.00330. Cited by: §1, §2.2, §5.1.3. [19] J. Wei et al. (2022) Chain-of-thought prompting elicits reasoning in large language models. NeurIPS. Cited by: §1. [20] Z. Zhao et al. (2021) Calibrate before use: improving few-shot performance of language models. In ICML, Cited by: §1, §2.2. [21] Y. Zhou et al. (2022) Large language models are human-level prompt engineers. In ICLR, Cited by: §2.2.