Paper deep dive

SEFORA: Student Essays with Feedback Corpus and LLM Feedback Evaluation Framework

Shayan Peyghambari Oskoui, Norah Almousa, Zhaoyi Joey Hou, Carolina Gustafson, Gayle Rogers, Raquel Coelho, Diane Litman, Xiang Lorraine Li

Year: 2026Venue: arXiv preprintArea: cs.CLType: PreprintEmbeddings: 97

Intelligence

Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 99%

Last extracted: 7/5/2026, 8:50:11 AM

Summary

The paper introduces SEFORA, a public corpus of student essays paired with authentic, span-anchored instructor feedback, and UNIMATCH, a reference-based evaluation framework. SEFORA contains 564 drafts and 8,240 instructor annotations across various college writing genres. UNIMATCH evaluates LLM-generated feedback by segmenting it into 'feedback units' and using optimal bipartite matching to calculate precision, recall, and F1 based on semantic correspondence. Experimental results across 74 configurations show that LLMs struggle to match instructor priorities, with performance (F1 < 0.4) often degrading as model verbosity increases.

Entities (7)

Feedback Unit · concept · 100%Gemini 3.1 Flash-Lite · model · 100%gpt-5-nano · model · 100%LLM · technology · 100%SEFORA · datasetcorpus · 100%UNIMATCH · evaluation-framework · 100%University of Pittsburgh · organization · 100%

Relation Signals (4)

SEFORA → contains → instructor feedback

confidence 100% · SEFORA is a public corpus pairing instructor inline feedback with assignment prompts

SEFORA → developedat → University of Pittsburgh

confidence 100% · Shayan Peyghambari Oskoui... University of Pittsburgh

UNIMATCH → evaluates → LLM feedback

confidence 100% · UNIMATCH is a reference-based evaluation framework for open-ended generation

Gemini 3.1 Flash-Lite → usedas → similarity scorer

confidence 100% · We adopt gemini-3.1-flash-lite-preview as the similarity scorer in our main pipeline

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:Effective writing feedback is among the strongest drivers of student learning, yet producing it at scale is labor-intensive. LLMs offer a natural path to scaling writing support, but two gaps stand in the way: few public corpora capture how instructors actually deliver feedback in real classrooms, and no reliable method measures whether generated feedback aligns with what an instructor would write. We address both. SEFORA is a public corpus pairing instructor inline feedback with assignment prompts, rubrics, scores, and multi-draft revisions across various college writing genres, comprising 564 drafts and 8,240 instructor annotations. UniMatch is a reference-based evaluation framework for open-ended generation: it segments feedback into feedback units, scores their semantic correspondence under instructor-derived criteria, and aligns them via optimal matching to yield interpretable precision, recall, and F1. Across 74 experimental configurations spanning multiple LLMs, no setting exceeds 0.4 F1. UniMatch reveals that models struggle to identify the feedback instructors would prioritize, and performance degrades as models generate more.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

96,601 characters extracted from source content.

Expand or collapse full text

SEFORA: Student Essays with Feedback Corpus and LLM Feedback Evaluation Framework Shayan Peyghambari Oskoui, Norah Almousa, Zhaoyi Joey Hou, Carolina Gustafson, Gayle Rogers, Raquel Coelho, Diane Litman, Xiang Lorraine Li University of Pittsburgh shayan.p, xianglli@pitt.edu Abstract Effective writing feedback is among the strongest drivers of student learning, yet pro- ducing it at scale is labor-intensive. LLMs of- fer a natural path to scaling writing support, but two gaps stand in the way: few public corpora capture how instructors actually de- liver feedback in real classrooms, and no reli- able method measures whether generated feed- back aligns with what an instructor would write. We address both. SEFORA is a public corpus 1 pairing instructor inline feedback with assign- ment prompts, rubrics, scores, and multi-draft revisions across various college writing gen- res, comprising 564 drafts and 8,240 instruc- tor annotations. UNIMATCH is a reference- based evaluation framework for open-ended generation: it segments feedback into feedback units, scores their semantic correspondence un- der instructor-derived criteria, and aligns them via optimal matching to yield interpretable pre- cision, recall, and F 1 . Across 74 experimen- tal configurations spanning multiple LLMs, no setting exceeds0.4F 1 . UNIMATCH reveals that models struggle to identify the feedback instructors would prioritize, and performance degrades as models generate more. 1 Introduction Feedback plays a vital role in student learning in writing. It helps students correct misunderstand- ings and refine how they apply knowledge (Ahea et al., 2016; Banihashem et al., 2024), and it is consistently identified as one of the strongest in- fluences on learning and achievement (Hattie and Timperley, 2007). But effective feedback is not one- size-fits-all. Students report that the most useful feedback is specific to their own writing (Lipnevich and Smith, 2009), and effective feedback accounts for the draft’s position within the revision process (Carless and Boud, 2018); poorly targeted feedback can even be detrimental (Kluger and DeNisi, 1996). 1 https://github.com/ShayanPey/SEFORA ... ... Generate feedback Gold Segments LLM Segments Student's essay UNIMATCH Segmentation Retrieve gold feedback SEFORA Figure 1: Overview of the UNIMATCH evaluation pipeline. For each paragraph, LLM-generated feedback is segmented into units and compared with instructor feedback units. The resulting semantic similarity scores are used to compute an optimal matching between in- structor and model feedback units, producing the final evaluation metrics. In writing instruction, producing such feedback is labor-intensive, and its cost at scale discourages the sustained practice effective instruction requires (Applebee and Langer, 2011; Graham, 2019). This opens a natural opportunity for NLP: systems that generate useful feedback on drafts could help scale writing support. Progress on this problem is constrained by two bottlenecks. First, few public datasets preserve how instructor feedback is actually delivered in coursework (Table 1): multifaceted comments (of- ten addressing several points at once) anchored to specific spans (a paragraph, sentence, or word) and interpretable alongside the assignment prompt, rubric, and revision history. Some resources substi- tute structured labels (error tags or analytic scores) for formative commentary (Crossley et al., 2024; Mathias and Bhattacharyya, 2018; Dahlmeier et al., 2013; Lee et al., 2015), restrict coverage to a sin- gle prompt or narrow genre (Kashefi et al., 2022; Zyska et al., 2026), or forgo expert annotation for crowd- or model-generated feedback (Behzad et al., 2024). Without datasets that capture feedback as instructors deliver it, evaluating whether LLMs pro- arXiv:2607.00274v1 [cs.CL] 30 Jun 2026 duce what an instructor would write remains out of reach. Second, feedback evaluation is difficult for three reasons. (i) Reference-free evaluation is conceptu- ally appealing, but ‘good’ writing feedback has no single operational definition, as its quality depends on many interacting dimensions weighted differ- ently across raters and settings (Pearson, 2022; Stahl et al., 2024); LLM-as-judge proxies inherit this ambiguity while adding biases of their own (Zheng et al., 2023; Deutsch et al., 2022). (i) Reference-based evaluation mitigates this by com- paring generated feedback against instructor feed- back, but existing metrics fall short: n-gram overlap is unreliable for open-ended generation (Novikova et al., 2017; Reiter, 2018), and embedding-based metrics, while less surface-bound, correlate only weakly with human judgments (Liu et al., 2016), as we also find in our setting (§4.2). (i) A deeper, often overlooked issue is granularity: a feedback message typically bundles several distinct feedback units (Yen et al., 2020; Zou et al., 2024) – self- contained statements about specific aspects of the writing – where even a single sentence may contain several (e.g., ‘This paragraph is well-written, but try to make it more concise’). Holistic compari- son cannot tell which observations are recovered or missed, so reliable evaluation must compare feedback at the unit level and by its underlying meaning. We address both. SEFORA (Student Essays with Feedback Corpus) is a public corpus pairing au- thentic instructor-authored span-anchored feedback with assignment prompts, rubrics, analytic scores, and multi-draft revisions across diverse college writing genres. UNIMATCH is a reference-based framework that evaluates LLM feedback at the unit level rather than holistically, segmenting feedback into feedback units and aligning them via opti- mal matching under instructor-derived similarity criteria; the resulting similarity-weighted F 1 cap- tures both what the model omits and what it over- generates (Figure 1). Together they enable system- atic study of LLM-generated writing feedback at the granularity instructors work in. Our experiments across 74 configurations span- ning multiple LLMs show that no setting exceeds 0.4F 1 . Models struggle to produce the comments instructors would emphasize. Feedback verbosity, in the sense of producing more comments rather than longer ones, is the dominant driver of poor alignment. What UNIMATCH penalizes is ped- agogically meaningful: overly verbose feedback overwhelms the student, leading to counterproduc- tive results (Kluger and DeNisi, 1996). 2 Related Work Essay datasets. Many existing writing datasets target Automatic Essay Scoring (AES), predicting numerical scores rather than formative feedback (Ramesh and Sanampudi, 2022; Ke and Ng, 2019; Hou et al., 2025), or emphasize structured annota- tions (e.g., error tags or analytic scores) over for- mative text, including PERSUADE 2.0 (Crossley et al., 2024), ASAP/ASAP++ 2 (Mathias and Bhat- tacharyya, 2018), and NUCLE (Dahlmeier et al., 2013). While valuable for large-scale assessment, scores rate a draft but do not tell a student how to revise it (Ke et al., 2019); feedback genera- tion instead requires localized, actionable, context- sensitive comments tied to the writer’s text and the assignment goals. Efforts to collect feedback-based essay datasets have grown in recent years, yet each captures only part of the setting we target. They pair expert feed- back with a narrow slice of writing: Insta-Reviewer (Jia et al., 2022) with graduate project reports, Ar- gRewrite V.2 (Kashefi et al., 2022) with a single argumentative prompt, and Exposía (Zyska et al., 2026) with research exposés, while CityU (Lee et al., 2015) 3 offers short tutor comments on ES- L/EFL writing. Each is limited on at least one axis central to classroom feedback, lacking multi- ple drafts or revisions (Jia et al., 2022; Lee et al., 2015) or covering only a single genre (Jia et al., 2022; Kashefi et al., 2022; Zyska et al., 2026). LEAF (Behzad et al., 2024) scales to more essay- feedback pairs but sources them from online users and AI rather than instructors. SEFORA comple- ments these resources with authentic, instructor- authored feedback, both span-anchored and overall, at larger scale across diverse college writing genres (Table 1), paired with assignment prompts, rubrics, analytic scores, and a multi-draft revision structure, enabling analyses that prior corpora do not directly support. Feedback evaluation. Evaluating generated feedback is difficult, as wording and delivery can vary widely even for the same underlying issue. 2 https://w.kaggle.com/c/asap-aes 3 The original paper described the corpus as not yet publicly available at the time, and we could not verify a current public release. Dataset nameSize (# drafts) Feedback word count Writing genre(s) Total (corpus) Inline /draft Overall /draft Exposía (Zyska et al., 2026)5517.6K107213Proposal ArgRewrite V.2 (Kashefi et al., 2022)8612.9K–150Essay Insta-Reviewer (Jia et al., 2022)48426.6K–55Research Report SEFORA (ours)564136.4K14795Essay, Narrative, Explanation, Empathy Table 1: Comparison of related datasets of student writing with instructor feedback. The table counts only instructor-authored feedback (peer, user, and AI-augmented excluded), with per-draft word counts averaged over each corpus’s total draft count. Writing genres follow the BAWE classification of Gardner and Nesi (2013). Inline denotes span-anchored feedback; other features (e.g., error tags, scores) are omitted. Reference-free evaluation raises the issues dis- cussed previously: effective feedback has no sin- gle operational definition (Pearson, 2022; Stahl et al., 2024), and LLM-as-judge proxies inherit this ambiguity while adding biases of their own (Zheng et al., 2023; Deutsch et al., 2022). We thus focus on reference-based and human-centered evaluation, which fall into three groups (Celikyil- maz et al., 2020). Content-overlap metrics such as BLEU (Papineni et al., 2002) and ROUGE (Lin, 2004) reward surface similarity and are brittle for open-ended feedback. Model-based metrics such as BERTScore (Zhang et al., 2019), BARTScore (Yuan et al., 2021), and BLEURT (Sellam et al., 2020) capture general semantics better but are hard to interpret and miss whether LLM feedback iden- tifies the same substantive points as the instructor. Human-centered evaluation is reliable but costly and hard to scale (Jia et al., 2022). UNIMATCH tar- gets both reliability and interpretability by compar- ing feedback as individual units and scoring their correspondence under feedback-specific similarity criteria. 3 Dataset SEFORA was collected over two semesters from a range of first-year undergraduate courses in the university’s English Department, spanning compo- sition, ESL, narrative and creative writing, and a senior seminar. It includes assignment prompts, grading rubrics, multi-draft student essays, and two forms of instructor feedback: span-anchored (in- line) comments (Figure 2) and overall assessments. The dataset is publicly available. 4 Scope, scale, and genres.SEFORA contains 371 essays by 155 students across 34 assignments from 9 classes, comprising 564 drafts (220 single-draft, 4 https://github.com/ShayanPey/SEFORA 109 two-stage, 42 three-stage) that total 8,186 para- graphs. Under the BAWE classification (Gardner and Nesi, 2013), these span four genre families: Essay, Narrative Recount, Explanation, and Empa- thy Writing. 5 Instructors contributed 5,684 inline (span-anchored) annotations – highlights, sticky notes, and strikeouts, nearly all with comment – av- eraging 10 annotations and 147 feedback words per draft. Additionally, 437 drafts carry overall com- ments (121 words avg.), 295 carry analytic trait scores (6.5 per draft avg.), and 205 a holistic score, for 8,240 instructor annotations in total. Annotation content.SEFORA preserves the orig- inal paragraph structure and aligns each annotation to the text it targets: most are paragraph-level and span-anchored – tied to the exact span the instruc- tor marked, whether a phrase, sentence, or word – with document-level comments and scores retained when present (details in §A.2). What distinguishes SEFORA from error-tag and score-only corpora is the substance of its feedback: it is overwhelmingly facilitative (inviting the writer’s own reflection) rather than directive (prescribing changes) (Straub, 1996). Rather than flagging surface errors, instruc- tors engage substantive aspects of writing – narra- tive development, idea elaboration, reader engage- ment, voice, revision guidance – and frequently pose open questions back to the writer. Collection, Privacy, and Ethics.Data collection was approved by the university’s IRB. The research team contacted instructors, who emailed students a recruitment notice outlining the study, its objec- tives, risks, and voluntary participation. Materi- als were gathered from instructors and consenting students across two semesters. All IRB-related 5 For each assignment we map its writing prompt to the genre family it most closely matches; BAWE’s explicit defini- tions and examples make this a direct mapping. That experience made me realize I don't really have a physical place in the world, my place in the world is where I am surrounded by people that make me feel comfortable. And in my case, that's two places, back home in Germany and my new home here. I think that can be a lesson for anyone reading this because I think many people are scared to make a step like that because they don't want to go too far away from home and rather stay in their comfort zone. I encourage everyone to step out of their comfort zone because that is something that will shape your personality and ultimately make you a greater person. You will probably face adversity and I am not saying that you are going to feel at home right away but if you trust the process it is going to be a great experience. A lot of my friends go to college 35 Minutes by car ... I can say that you can create a home for yourself everywhere in the world as long as you have the right people around you that make you feel like you are home. Interesting! referent? What does "that" refer to? You're noticing some interesting things, but what is the "so what" of this idea for your reader? This idea is still pretty vague and inconclusive. How are you a "greater person," for example? who are those people? Why do they make you feel like you're at home? Perhaps this needs to be developed into a whole big idea... :) minutes: lowercase Figure 2: An annotated draft from SEFORA, illustrating the dataset’s span-anchored, fine-grained instructor feedback: sticky-note comments paired with color-coded highlights. Pink highlights mark especially effective passages, yellow highlights indicate issues needing attention, and green highlights mark ideas worth developing in a subsequent draft. Full annotation conventions are documented in §A.2. materials are provided in §A.1. All essays were released with student consent and processed un- der strict privacy constraints. We remove direct identifiers (student and instructor names, course identifiers, submission dates) and replace names within essay bodies with consistent within-essay pseudonyms while preserving grammatical and nar- rative coherence; references to public figures are retained. We further screen the corpus for indi- rect identifying information (e.g., social media or sandbox handles), and check for abusive or harm- ful content usingomni-moderation 6 exclusively via the API under OpenAI’s data-handling policy. 7 We do not mask vulgar or strong language, as do- ing so would alter the writer’s voice and distort the feedback context; the dataset is released with appropriate content warnings. Parsing. Original submissions in PDF or.docx were converted to a JSON format that preserves both paragraph segmentation and annotation an- choring. We release the deterministic parser 8 to support future work on similar materials. Full pars- ing details are in §A.3. 4 Evaluation Framework Beyond a dataset, scaling instructor-quality feed- back requires a way to evaluate it; both reference- free and reference-based approaches face chal- 6 https://developers.openai.com/api/docs/ models/omni-moderation-latest 7 https://developers.openai.com/api/docs/ guides/your-data/ 8 https://github.com/ShayanPey/SEFORA lenges. We introduce UNIMATCH, a reference- based framework that compares model and instruc- tor feedback at the level of feedback units rather than as monolithic texts. Instructor feedback serves as a principled reference for two reasons. First, it encodes pedagogical priorities – what to flag, what to praise, and how to phrase guidance – per cur- riculum rubrics and teaching goals. Second, good writing feedback has no single operational defini- tion (Pearson, 2022; Stahl et al., 2024); instruc- tor judgment, while not the only valid target, is the most defensible anchor for a given draft, espe- cially for the open-ended writing in SEFORA where LLM feedback remains far from human quality (Chakrabarty et al., 2024; Gómez-Rodríguez and Williams, 2023). UNIMATCH compares model and instructor feedback in three stages (Figure 1): (i) segment each into feedback units, (i) score semantic corre- spondence between instructor-model unit pairs, and (i) apply maximum bipartite matching over those scores to derive interpretable unit-level alignments and aggregate precision, recall, and F 1 . Stages (i) and (i) are independently verifiable against hu- man annotation, enabling robustness checks on the pipeline. 4.1 Feedback Segmentation Prior work has typically treated feedback as a monolithic message rather than a set of separable units (Wu and Schunn, 2020; Lyu et al., 2024), but holistic comparison cannot tell which individual observations are recovered or missed. Other work has recognized the importance of feedback units, though with varying definitions: one or more sen- tences expressing a coherent thought (Yen et al., 2020), or a self-contained message targeting a spe- cific issue (Zou et al., 2024). Building on these (Yen et al., 2020; Zou et al., 2024), we adopt feed- back unit – a self-contained statement addressing one specific aspect of the student’s writing – as the basic evaluation granularity, and develop an annota- tion guideline (§B.1) for segmenting instructor and LLM-generated feedback (examples in Table 2). This task is related to discourse parsing (Li et al., 2014; Soricut and Marcu, 2003; Marcu, 2000), but differs in two key ways: we omit inter-segment relations, and place emphasis on sub-sentence seg- mentation, as a single sentence may contain mul- tiple feedback units, while a single unit may span multiple sentences. We therefore draw on prior work on sub-sentence segmentation in discourse (Li et al., 2022; Stab and Gurevych, 2017; Hua et al., 2019; Wang et al., 2018). Formalization and agreement metrics. Let T = t 1 ,...,t n be the sequence of words of a feed- back message. A segmentation is represented by a boundary vectorQ ∈ 0, 1 n−1 whereQ i = 1 marks a boundary betweent i andt i+1 . We mea- sure segmentation agreement using WindowDiff (Pevzner and Hearst, 2002), a standard discourse segmentation metric, with window sizekset to half the average reference segment length (k=10in our samples): WD(r,h) = 1 n− k n−k X i=1 1[b(r i:i+k )̸= b(h i:i+k )], whererandhare the two segmentations being compared, one designated the reference and the other the hypothesis (the second annotator, or the model when validating automation), andb(x i:j )is the boundary count betweent i andt j in segmenta- tionx. Because WindowDiff tolerates small bound- ary shifts, we also report boundary precision, recall, and F 1 calculated over boundary indices while ex- cluding true negatives to avoid inflated agreement. Automatic segmentation validation.Following the segmentation guideline (§B.1), two graduate students independently annotated more than 50 feedback instances, containing both instructor and LLM feedback. Agreement was high: WD =0.073, precision =0.99, recall =0.94, F 1 =0.96. The an- notators then resolved disagreements to produce # Segmented feedback 1:This is well-written,but try to make it more concise. 2: Nice! So much authenticity in your voice. 3:Try to add more sensory details. Like what was special about that night? Also, why did he suddenly leave? Table 2: Examples of feedback segmented into feedback units. a consensus segmentation, which we treat as the reference. Using the same segmentation guideline given to human annotators as prompt, GPT-5-nano reaches comparable agreement against this consensus ref- erence (WD =0.036, precision =1.00, recall =0.95, F 1 =0.97), indistinguishable from inter-annotator agreement. 4.2 Feedback Unit Similarity We then compare the similarity between feedback units. This task is related to Semantic Textual Sim- ilarity (STS) (Bär et al., 2012; Majumder et al., 2016), but differs in two key ways: the texts being compared are feedback units rather than arbitrary sentences, and both address the same paragraph of the same essay. We thus introduce a task-specific guideline (§B.2) for scoring feedback unit similarity along two dimensions of feedback: the target (the as- pect, span, or idea addressed) and the comment (the evaluation, critique, or suggestion about that target). Drawing on integer-scale scoring practice in STS (Agirre et al., 2012; Xu et al., 2015; Agirre et al., 2015), we use a 0–4 scale chosen to suit the feedback setting: anchored at 4 (near-equivalent as feedback) and 0 (irrelevant or contradictory), with an explicit middle value (2) for pairs that share an important point but differ on another (intermedi- ate levels and details in §B.2). The guideline was developed in collaboration with course instructors and refined iteratively by two graduate student an- notators, who piloted it, discussed disagreements, and refined wording and examples until agreement stabilized. They then independently scored 120 feedback unit pairs spanning the full scale; we re- port inter-annotator Pearson and Spearman corre- lations ofr = 0.7572andρ = 0.7558. We use the mean as the reference similarity for these 120 pairs. ModelVersionSizeReason r ρ GPT 5.1–low 0.805 0.794 5 † minilow 0.723 0.730 5nanolow 0.598 0.604 Gemini 3.1flash-lite ‡ minimal 0.813 0.818 2.5lite– 0.701 0.711 Table 3:The Pearsonrand Spearmanρcor- relationsbetweenLLM-generatedandhuman- annotated similarity scores for paired feedback units (withbatch_size= 50) over120samples. † Snapshotgpt-5-mini-2025-08-07. ‡ Model name gemini-3.1-flash-lite-preview. annot=4 annot=3 annot=2 annot=1 annot=0 pred=0pred=1pred=2pred=3pred=4 Figure 3: Agreement between annotator-averaged scores (rows, rounded to the model’s scale) and gemini-3.1-flash-lite-previewscores (columns); darker cells hold more pairs. Intensity along the diag- onal indicates strong agreement; the low-score corner is darkest as most feedback-unit pairs are unrelated and score low. Automatic similarity scorer validation. Exist- ing similarity methods fall short on this task: lexical-overlap metrics yield Pearson correlations below0.1and embedding-based methods reach at mostr ≈ 0.67, both too low to be reliable (details in §C.1). We therefore experimented with several LLMs using the defined guideline as prompt, with several exceedingr = 0.7and the strongest exceed- ingr = 0.8. However, as scoring every pair inde- pendently at our scale would be expensive, we in- troduce batching (details in §C.2), which preserves performance at a fraction of the cost (Table 3 9 ). We adoptgemini-3.1-flash-lite-previewas the similarity scorer in our main pipeline (r = 0.813 andρ = 0.818against the human reference). Fig- ure 3 visualizes its agreement on a sample with the pipeline’s natural score distribution. 9 Model performance can shift over time; results were collected in May 2026. Result using the snapshot model gpt-5-mini-2025-08-07 is also reported for stability. 4.3 Feedback Unit Cross Matching Given pairwise similarity scores between instructor and model-generated feedback units, UNIMATCH computes a one-to-one alignment between the two sets. Intuitively, each feedback unit is matched to its most similar counterpart on the other side, subject to a globally optimal overall matching (Fig- ure 4 illustrates). Specifically, for instructor feed- back unitsG = G i |G| i=1 and predicted feedback unitsP = P j |P| j=1 , letφ(G i ,P j ) ≥ 0denote the similarity scorer. We define the optimal alignment as the maximum-weight bipartite matching M ∗ = arg max M∈M X (G i ,P j )∈M φ(G i ,P j ), whereMis the set of 1-to-1 matchings between GandP. ComputingM ∗ is a standard maximum- weight bipartite matching problem, and we use the Hungarian algorithm (Kuhn, 1955) for efficiency. LetΦ(M ∗ ) = P (G i ,P j )∈M ∗ φ(G i ,P j )denote the total weight of this alignment. To turnΦ(M ∗ )into interpretable precision and recall, we normalize it against per-side up- per bounds, each set’s summed within-set self- similarity ( P j φ(P j ,P j )predicted, P i φ(G i ,G i ) instructor), which equals its unit count times the maximum score (4) since each unit matches itself. Following coreference resolution (Luo, 2005), we define soft precision and recall as p = Φ(M ∗ ) P j φ(P j ,P j ) , r = Φ(M ∗ ) P i φ(G i ,G i ) , and the soft F 1 as F 1 = 2pr p + r . The result is an interpretable, soft precision/recall view of feedback quality: precision captures how much of the model’s feedback aligns with what an instructor wrote, and recall captures how much of the instructor’s feedback the model recovers. 10 End-to-end validation.UNIMATCH’s compo- nents – segmentation (§4.1) and similarity scoring (§4.2) – are validated against human judgment in isolation, but this does not guarantee that the com- posed metric aligns with instructor judgment. To 10 The pipeline also admits a threshold-based instantiation with hard matches above a chosenτ, exposing strictness as an explicit knob (§D). A: B: So creative! Nice!But try to make it more concise. This is well-written,however, it’s a little verbose. Figure 4: Matching feedback A and B: boxes mark segments; solid lines = high similarity, dashed = low. test the pipeline end to end, we selected 20 para- graphs, each with three raw LLM feedback texts (§5) that UNIMATCH scores at well-separated lev- els (F 1 ≈ 0 ,0.5,1), and had a writing expert inde- pendently rank the three by quality against the refer- ence feedback. 11 Across all 60 texts, UNIMATCH F 1 correlates strongly with the instructor ranking (Pearsonr = 0.836, Spearmanρ = 0.837), show- ing that the full pipeline recovers expert judgments. 5 Experiments We evaluate paragraph-level LLM-generated feed- back under UNIMATCH, varying model choice and prompting setup to study what task framing and generation choices drive feedback quality. Task framing and evaluation data.We generate feedback at the paragraph level, matching the gran- ularity of inline instructor annotations in SEFORA. For each target paragraph, the model receives the paragraph plus as much surrounding context as fits within a fixed 8,192-token budget, enough to in- clude the entire essay for most drafts. We evaluate on the 3,499 paragraphs in SEFORA with at least one instructor annotation (full-corpus); of these, 2,324 carry exactly one (the single-reference sub- set). We report metrics on both. Models. To support broad ablations across settings, we primarily use small open-weight instruction-tuned models, namely, Llama-3.1-8B (Grattafiori et al., 2024) (Llama), DeepSeek-R1- Distill-Llama-8B (Guo et al., 2025) (DeepSeek), and Mistral-7B (Jiang et al., 2023) (Mistral). To probe whether the effects extend to a larger regime, we also evaluate Qwen2.5-72B-Instruct (Qwen et al., 2025) (Qwen) and closed model GPT-5.1 12 (GPT) on a subset of configurations. Prompt template.We use two prompt templates. The default constrained prompt asks the model to identify the single most important focus in the 11 Ties were permitted but occurred for only two paragraphs. 12 https://developers.openai.com/api/docs/ models/gpt-5.1 target paragraph and produce one feedback unit un- der a prescribed output format; the unconstrained prompt places no limit on the number of feedback units it may return. On top of the template, we vary four further dimensions of task specification: zero- shot vs. few-shot prompting (5 exemplars) (Brown et al., 2020); guided vs. unguided prompting, where the guided setting supplies a list of feedback cate- gories (Narciss, 2008; Keuning et al., 2018) in the prompt; inclusion vs. omission of the assignment prompt and rubric; and chain-of-thought prompting (Wei et al., 2022; Lyu et al., 2023). The guided cat- egories cover the common functional roles of writ- ing feedback: Task Constraints, Concepts, Elabora- tion, Clarification, Mistakes, and Praise. Few-shot exemplars are 5 instructor-authored feedback sam- ples covering each category. Full prompt templates and guided variants are in §F. Generation settings. We allocate up to 250 out- put tokens per target paragraph – well above the typical length of instructor feedback on a single paragraph in SEFORA (≈ 20tokens 13 ) – to avoid truncating multi-point responses. In total, the model and prompt combination yields 74 experimental configurations. All prompts (§F), inference pipelines, the UNIMATCH frame- work, and code are publicly available. 14 6 Results Table 4 reports UNIMATCH scores on the full cor- pus under the zero-shot setting (with rubrics and category guidance), where most models perform best; full results are in Tables 5–7 of §E. No con- figuration exceeds0.4F 1 , and both precision and recall stay low: even the strongest GPT model reaches only0.38precision and0.27recall, so models miss most of the feedback instructors pri- oritize while spending much of their output on points instructors do not raise. Among the small models (≤ 8B parameters), Mistral is consistently strongest, ahead of Llama 15 . The larger Qwen and GPT lead overall – GPT on the shared sub- set, Qwen on the single best configuration (0.32 F 1 ) – but only narrowly, and all remain far from matching instructor feedback. Table 4 additionally presents ablations: each varies a single variable relative to the base setting <zero-shot, rubric, guided, default prompt>. We 13 Exact number of tokens depends on the tokenizer. 14 https://github.com/ShayanPey/SEFORA 15 Sometimes DeepSeek as shown in §E. ModelPrecisionRecallF 1 Ablations (F 1 ) Single-referenceUnconstrainedUnguidedNo rubricsFew-shot Llama_8B0.2750.2360.2540.2900.1130.2030.2680.271 Mistral_7B0.2880.2670.2770.3110.1590.2510.2780.258 Qwen_72B0.3740.2680.3120.3870.1790.2710.2850.317 GPT-5.10.3760.2730.3160.3710.1360.2960.2880.307 Table 4: UNIMATCH results for the configuration <zero-shot, rubric, guided, default prompt> on the full corpus. Left: precision, recall, F 1 . Right: F 1 under single-axis variations – Single-reference subset (paragraphs with exactly one instructor annotation), Unconstrained prompt, Unguided, No rubrics, and Few-shot. Full per-configuration results in Tables 5–7 (§ E). 4k5k7k12k19k 0.1 0.2 0.3 0.4 Prompt Number of generated feedback units Score Precision Recall F 1 Figure 5: Precision, recall, and F 1 across the 74 config- urations of §5, sorted by number of generated feedback units (x-axis is ordinal and not to scale). The strip marks the prompt template: the constrained template (dark) and the unconstrained (light). Precision and F 1 fall as volume rises while recall does not compensate. show a subset due to space constraints, but the pattern matches the full results. On the single- reference subset (fourth column), scores are higher as expected, since our constrained default prompt emits one feedback unit per paragraph and thus naturally aligns more with single-reference para- graphs. Verbosity. The dominant signal in our experi- ments is verbosity. Across all models, the con- strained prompt (our default) beats the uncon- strained alternative, which yields substantially lower F 1 (the unconstrained column of Table 4). Removing the single-feedback constraint lets mod- els emit more than one feedback unit per paragraph, and the resulting over-generation sharply lowers precision. This holds even on the full corpus, where paragraphs often carry multiple instructor units and the unconstrained prompt should in principle have the edge: its extra units add low-similarity matches that depress precision more than they lift recall. Figure 5 confirms the pattern across all evaluated configurations – the number of generated feedback units is strongly negatively correlated with both precision and F 1 (r = −0.850andr = −0.840, respectively), while the accompanying recall gains are too modest to compensate (full results in §E). Guiding the model with feedback categories. Supplying explicit feedback categories (the guided setting) is the next strongest factor: it improves F 1 by about+0.059without rubric and+0.035with rubric on the full-corpus evaluation in matched comparisons (the Unguided column of Table 4; per-configuration numbers in Tables 5–7, and a per-model breakdown in Figure 14, §E). Rubric and few-shot prompting. Two further levers contribute little (Table 4, No rubrics and Few-shot columns). Rubric inclusion is essentially neutral, with small, unstable differences: in our paragraph-level setting well-framed prompts al- ready supply most of the structure a rubric would contribute, and rubrics largely encode whether a draft is on-task – which, in authentic coursework, it usually is, leaving little for a rubric to anchor given that SEFORA’s feedback is facilitative (§3), engaging execution rather than rubric conformance. Few-shot prompting is beneficial but secondary, improving F 1 by about+0.017on the full corpus, mainly through precision; its gains are small un- der the default prompt, as exemplars help most when task framing is weak, and once the prompt already induces the desired behavior, demonstra- tions add little. Larger few-shot gains under the unconstrained alternative are in Table 6 (§E); a small chain-of-thought study on the three small models follows the same pattern (§E). Per-model breakdowns of both effects appear in Figures 15a and 15b (§E). 7 Conclusion We introduce SEFORA, a corpus of authentic in- structor feedback on student writing, and UNI- MATCH, a reference-based framework for evaluat- ing generated feedback. Across 74 configurations, no setting exceeds0.4F 1 : the central challenge for LLMs is not producing feedback but produc- ing the comments an instructor would prioritize. Over-generation is consistently costly, with pre- cision dropping sharply as models generate more while recall rarely compensates, a pattern that is pedagogically meaningful, since overwhelming a student with feedback is itself counterproductive (Kluger and DeNisi, 1996). Validated end-to-end against expert judgment, UNIMATCH provides a reliable automatic metric for this task, and together with SEFORA lays a foundation for future work on instructor-aligned feedback generation across diverse college writing genres. 8 Limitations Although UNIMATCH enables fine-grained analy- sis, its evaluation operates on feedback units as seg- mented, without further decomposing compound statements. Some units express multiple claims (e.g., ‘this paragraph is detailed and smooth’) and could in principle be split further. Future work could incorporate automatic claim decomposition to isolate individual feedback propositions at the atomic level. Our evaluation operates at the paragraph level: the model is given a target paragraph, and UNI- MATCH assesses the feedback it produces rather than whether the model identifies which span war- rants comment in the first place. SEFORA’s span- anchored annotations make span identification di- rectly studiable, but we leave this setting to future work. Our experiments also evaluate against the inline (span-anchored) feedback only, leaving the document-level overall assessments and analytic scores in SEFORA for future work. Our alignment uses maximum-weight bipartite matching over all eligible unit pairs. Alternative alignment constraints (e.g., prioritizing globally highest-scoring pairs before enforcing one-to-one matching) may produce different match structures and could be explored as alternative objectives. Our pipeline relies on closed-source mod- els(GPT-5-nanoforsegmentationand gemini-3.1-flash-lite-previewforsimi- larity scoring), chosen because they meet the human-agreement bar required for reliable eval- uation in our setting. This reliance has practical downsides: hosted models can change or be deprecated over time, and their use introduces cost and reproducibility constraints. A natural next step is to distill or fine-tune smaller open-source models for both the segmentation and similarity stages, using our human-annotated subsets as supervision. Because each stage is independently validated against human annotation (§4.1, §4.2), open-source replacements can be substituted into the pipeline whenever they reach comparable agreement, without re-validating UNIMATCH as a whole. UNIMATCH treats instructor feedback as ground truth. Reference-free evaluation may also be pos- sible by leveraging methods from the automated essay scoring (AES) literature. For example, gen- erated feedback could be applied to the relevant essay span and the resulting revision evaluated with a quality scorer to estimate feedback effectiveness. Developing such metrics is beyond the scope of this work. Finally, our study relies primarily on automatic evaluation under the proposed framework and does not include a complementary human evaluation of overall feedback quality. While UNIMATCH is de- signed to better reflect semantic, point-level align- ment with instructor feedback than whole-text over- lap metrics, it does not directly assess dimensions such as helpfulness, pedagogical appropriateness, or actionability as perceived by human readers. A targeted human study, or an LLM-as-judge evalu- ation calibrated against human judgments, would provide a useful complementary perspective. We leave such validation to future work. References Eneko Agirre, Carmen Banea, Claire Cardie, Daniel Cer, Mona Diab, Aitor Gonzalez-Agirre, Weiwei Guo, Inigo Lopez-Gazpio, Montse Maritxalar, Rada Mi- halcea, and 1 others. 2015. Semeval-2015 task 2: Semantic textual similarity, english, spanish and pilot on interpretability. In Proceedings of the 9th inter- national workshop on semantic evaluation (SemEval 2015), pages 252–263. Eneko Agirre, Daniel Cer, Mona Diab, and Aitor Gonzalez-Agirre. 2012. Semeval-2012 task 6: A pi- lot on semantic textual similarity. in* sem 2012: The first joint conference on lexical and computational semantics–volume 1: Proceedings of the main confer- ence and the shared task, and volume 2: Proceedings of the sixth international workshop on semantic eval- uation (semeval 2012). In Proceedings of the Sixth International Workshop on Semantic Evaluation (Se- mEval 2012), Montréal, QC, Canada, pages 7–8. Md Mamoon-Al-Bashir Ahea, Md Rezaul Kabir Ahea, and Ismat Rahman. 2016. The value and effective- ness of feedback in improving students’ learning and professionalizing teaching in higher education. Jour- nal of Education and Practice, 7(16):38–41. Arthur N Applebee and Judith A Langer. 2011. Ej extra: A snapshot of writing instruction in middle schools and high schools [free access]. English jour- nal, 100(6):14–27. Seyyed Kazem Banihashem, Nafiseh Taghizadeh Ker- man, Omid Noroozi, Jewoong Moon, and Hendrik Drachsler. 2024. Feedback sources in essay writing: peer-generated or ai-generated feedback? Interna- tional Journal of Educational Technology in Higher Education, 21(1):23. Daniel Bär, Chris Biemann, Iryna Gurevych, and Torsten Zesch. 2012. Ukp: Computing semantic textual similarity by combining multiple content sim- ilarity measures. In * SEM 2012: The First Joint Conference on Lexical and Computational Semantics– Volume 1: Proceedings of the main conference and the shared task, and Volume 2: Proceedings of the Sixth International Workshop on Semantic Evalua- tion (SemEval 2012), pages 435–440. Shabnam Behzad, Omid Kashefi, and Swapna Somasun- daran. 2024. Leaf: Language learners’ english essays and feedback corpus. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 2: Short Papers), pages 433–442. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, and 1 others. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901. David Carless and David Boud. 2018. The development of student feedback literacy: enabling uptake of feed- back. Assessment & Evaluation in Higher Education, 43(8):1315–1325. Asli Celikyilmaz, Elizabeth Clark, and Jianfeng Gao. 2020. Evaluation of text generation: A survey. arXiv preprint arXiv:2006.14799. Tuhin Chakrabarty, Philippe Laban, Divyansh Agar- wal, Smaranda Muresan, and Chien-Sheng Wu. 2024. Art or artifice? large language models and the false promise of creativity. In Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems, pages 1–34. Scott A Crossley, Y Tian, P Baffour, Abigail Franklin, Margaret Benner, and Ulrich Boser. 2024. A large- scale corpus for assessing written argumentation: Per- suade 2.0. Assessing Writing, 61:100865. Daniel Dahlmeier, Hwee Tou Ng, and Siew Mei Wu. 2013. Building a large annotated corpus of learner english: The nus corpus of learner english. In Pro- ceedings of the eighth workshop on innovative use of NLP for building educational applications, pages 22–31. Daniel Deutsch, Rotem Dror, and Dan Roth. 2022. On the limitations of reference-free evaluations of gen- erated text. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Process- ing (EMNLP), Abu Dhabi, United Arab Emirates. Association for Computational Linguistics. Guillaume Endignoux, Olivier Levillain, and Jean-Yves Migeon. 2016. Caradoc: A pragmatic approach to pdf parsing and validation. In 2016 IEEE Security and Privacy Workshops (SPW), pages 126–139. Ieee. Mark Everingham, Luc Van Gool, Christopher K. I. Williams, John Winn, and Andrew Zisserman. 2010. The PASCAL Visual Object Classes (VOC) Chal- lenge. International Journal of Computer Vision, 88(2):303–338. Jared Fernandez, Clara Na, Vashisth Tiwari, Yonatan Bisk, Sasha Luccioni, and Emma Strubell. 2025. En- ergy considerations of large language model infer- ence and efficiency optimizations. arXiv preprint arXiv:2504.17674. Sheena Gardner and Hilary Nesi. 2013. A classifica- tion of genre families in university student writing. Applied linguistics, 34(1):25–52. Carlos Gómez-Rodríguez and Paul Williams. 2023. A confederacy of models: a comprehensive evaluation of llms on creative writing. In Findings of the Associ- ation for Computational Linguistics: EMNLP 2023, pages 14504–14528. Steve Graham. 2019. Changing how writing is taught. Review of Research in Education, 43(1):277–303. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al- Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, and 1 others. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783. Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shi- rong Ma, Peiyi Wang, Xiao Bi, and 1 others. 2025. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948. John Hattie and Helen Timperley. 2007. The power of feedback. Review of educational research, 77(1):81– 112. Zhaoyi Joey Hou, Alejandro Ciuba, and Xiang Lor- raine Li. 2025. Improve llm-based automatic es- say scoring with linguistic features. arXiv preprint arXiv:2502.09497. Xinyu Hua, Mitko Nikolov, Nikhil Badugu, and Lu Wang. 2019. Argument mining for understanding peer reviews. arXiv preprint arXiv:1903.10104. Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qianglong Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, and 1 oth- ers. 2025. A survey on hallucination in large lan- guage models: Principles, taxonomy, challenges, and open questions. ACM Transactions on Information Systems, 43(2):1–55. Qinjin Jia, Mitchell Young, Yunkai Xiao, Jialin Cui, Chengyuan Liu, Parvez Rashid, and Edward Gehringer. 2022. Insta-reviewer: A data-driven ap- proach for generating instant feedback on students’ project reports. International Educational Data Min- ing Society. Albert Q. Jiang, Alexandre Sablayrolles, Arthur Men- sch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guil- laume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. 2023. Mistral 7b. Preprint, arXiv:2310.06825. Omid Kashefi, Tazin Afrin, Meghan Dale, Christopher Olshefski, Amanda Godley, Diane Litman, and Re- becca Hwa. 2022. Argrewrite v. 2: an annotated ar- gumentative revisions corpus. Language Resources and Evaluation, 56(3):881–915. Zixuan Ke, Hrishikesh Inamdar, Hui Lin, and Vincent Ng. 2019. Give me more feedback i: Annotating thesis strength and related attributes in student es- says. In Proceedings of the 57th annual meeting of the association for computational linguistics, pages 3994–4004. Zixuan Ke and Vincent Ng. 2019. Automated essay scoring: A survey of the state of the art. In IJCAI, volume 19, pages 6300–6308. Hieke Keuning, Johan Jeuring, and Bastiaan Heeren. 2018. A systematic literature review of automated feedback generation for programming exercises. ACM Transactions on Computing Education (TOCE), 19(1):1–43. Avraham N Kluger and Angelo DeNisi. 1996. The effects of feedback interventions on performance: a historical review, a meta-analysis, and a preliminary feedback intervention theory. Psychological bulletin, 119(2):254. Harold W Kuhn. 1955. The hungarian method for the assignment problem. Naval research logistics quar- terly, 2(1-2):83–97. John Lee, Chak Yan Yeung, Amir Zeldes, Marc Reznicek, Anke Lüdeling, and Jonathan Webster. 2015. Cityu corpus of essay drafts of english lan- guage learners: a corpus of textual revision in second language writing. Language Resources and Evalua- tion, 49(3):659–683. Jiaqi Li, Ming Liu, Bing Qin, and Ting Liu. 2022. A survey of discourse parsing. Frontiers of Computer Science, 16(5):165329. Jiwei Li, Rumeng Li, and Eduard Hovy. 2014. Recur- sive deep models for discourse parsing. In Proceed- ings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 2061– 2069. Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pages 74–81. Anastasiya A Lipnevich and Jeffrey K Smith. 2009. “i really need feedback to learn:” students’ perspec- tives on the effectiveness of the differential feedback messages. Educational Assessment, Evaluation and Accountability, 21(4):347–367. Chia-Wei Liu, Ryan Lowe, Iulian Vlad Serban, Mike Noseworthy, Laurent Charlin, and Joelle Pineau. 2016. How not to evaluate your dialogue system: An empirical study of unsupervised evaluation metrics for dialogue response generation. In Proceedings of the 2016 conference on empirical methods in natural language processing, pages 2122–2132. Xiaoqiang Luo. 2005. On coreference resolution perfor- mance metrics. In Proceedings of human language technology conference and conference on empirical methods in natural language processing, pages 25– 32. Qianru Lyu, Wenli Chen, Junzhu Su, and Kok Hui John Gerard Heng. 2024. Steps to implementation: the role of peer feedback inner structure on feedback implementation. Assessment & Evaluation in Higher Education, 49(4):572–585. Qing Lyu, Shreya Havaldar, Adam Stein, Li Zhang, Delip Rao, Eric Wong, Marianna Apidianaki, and Chris Callison-Burch. 2023.Faithful chain-of- thought reasoning. In The 13th International Joint Conference on Natural Language Processing and the 3rd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics (IJCNLP- AACL 2023). Goutam Majumder, Partha Pakray, Alexander Gelbukh, and David Pinto. 2016. Semantic textual similarity methods, tools, and applications: A survey. Com- putación y Sistemas, 20(4):647–665. Daniel Marcu. 2000. The theory and practice of dis- course parsing and summarization. MIT press. Sandeep Mathias and Pushpak Bhattacharyya. 2018. Asap++: Enriching the asap automated essay grading dataset with essay attribute scores. In Proceedings of the eleventh international conference on language resources and evaluation (LREC 2018). Susanne Narciss. 2008. Feedback strategies for inter- active learning tasks. In Handbook of research on educational communications and technology, pages 125–143. Routledge. Jekaterina Novikova, Ond ˇ rej Dušek, Amanda Cercas Curry, and Verena Rieser. 2017. Why we need new evaluation metrics for nlg. In Proceedings of the 2017 conference on empirical methods in natural language processing, pages 2241–2252. Kishore Papineni, Salim Roukos, Todd Ward, and Wei- Jing Zhu. 2002. Bleu: a method for automatic evalu- ation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computa- tional Linguistics, pages 311–318. William S Pearson. 2022. A typology of the char- acteristics of teachers’ written feedback comments on second language writing. Cogent Education, 9(1):2024937. Lev Pevzner and Marti A Hearst. 2002. A critique and improvement of an evaluation metric for text segmentation. Computational Linguistics, 28(1):19– 36. Maja Popovi ́ c. 2015. chrf: character n-gram f-score for automatic mt evaluation. In Proceedings of the tenth workshop on statistical machine translation, pages 392–395. Qwen, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, and 24 oth- ers. 2025. Qwen2.5 technical report. arXiv preprint arXiv:2412.15115. Dadi Ramesh and Suresh Kumar Sanampudi. 2022. An automated essay scoring systems: a system- atic literature review. Artificial Intelligence Review, 55(3):2495–2527. Ehud Reiter. 2018. A structured review of the validity of bleu. Computational Linguistics, 44(3):393–401. Siddharth Samsi, Dan Zhao, Joseph McDonald, Baolin Li, Adam Michaleas, Michael Jones, William Berg- eron, Jeremy Kepner, Devesh Tiwari, and Vijay Gade- pally. 2023. From words to watts: Benchmarking the energy costs of large language model inference. In 2023 IEEE High Performance Extreme Computing Conference (HPEC), pages 1–9. IEEE. Thibault Sellam, Dipanjan Das, and Ankur Parikh. 2020. Bleurt: Learning robust metrics for text generation. In Proceedings of the 58th annual meeting of the as- sociation for computational linguistics, pages 7881– 7892. Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and Tie- Yan Liu. 2020. Mpnet: Masked and permuted pre- training for language understanding. Advances in neural information processing systems, 33:16857– 16867. Radu Soricut and Daniel Marcu. 2003. Sentence level discourse parsing using syntactic and lexical informa- tion. In Proceedings of the 2003 Human Language Technology Conference of the North American Chap- ter of the Association for Computational Linguistics, pages 228–235. Christian Stab and Iryna Gurevych. 2017. Parsing argu- mentation structures in persuasive essays. Computa- tional Linguistics, 43(3):619–659. Maja Stahl, Leon Biermann, Andreas Nehring, and Hen- ning Wachsmuth. 2024. Exploring llm prompting strategies for joint essay scoring and feedback gen- eration. In Proceedings of the 19th workshop on innovative use of NLP for building educational appli- cations (BEA 2024), pages 283–298. Richard Straub. 1996. The concept of control in teacher response: Defining the varieties of “directive” and “facilitative” commentary. College Composition & Communication, 47(2):223–251. Wenhui Wang, Furu Wei, Li Dong, Hangbo Bao, Nan Yang, and Ming Zhou. 2020. Minilm: Deep self- attention distillation for task-agnostic compression of pre-trained transformers. Advances in neural in- formation processing systems, 33:5776–5788. Yizhong Wang, Sujian Li, and Jingfeng Yang. 2018. To- ward fast and accurate neural discourse segmentation. arXiv preprint arXiv:1808.09147. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, and 1 others. 2022. Chain-of-thought prompting elic- its reasoning in large language models. Advances in neural information processing systems, 35:24824– 24837. Yong Wu and Christian D Schunn. 2020. From feed- back to revisions: Effects of feedback features and perceptions. Contemporary Educational Psychology, 60:101826. Wei Xu, Chris Callison-Burch, and William B Dolan. 2015. Semeval-2015 task 1: Paraphrase and seman- tic similarity in twitter (pit). In Proceedings of the 9th international workshop on semantic evaluation (SemEval 2015), pages 1–11. Yu-Chun Grace Yen, Joy O Kim, and Brian P Bailey. 2020. Decipher: an interactive visualization tool for interpreting unstructured design feedback from multiple providers. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems, pages 1–13. Weizhe Yuan, Graham Neubig, and Pengfei Liu. 2021. Bartscore: Evaluating generated text as text gener- ation. Advances in neural information processing systems, 34:27263–27277. Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. 2019. Bertscore: Eval- uating text generation with bert. arXiv preprint arXiv:1904.09675. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023. Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. In Advances in Neural Information Processing Sys- tems 36 (NeurIPS 2023), Datasets and Benchmarks Track. Shaoyan Zou, Kai Guo, Jun Wang, and Yu Liu. 2024. Investigating students’ uptake of teacher-and chatgpt- generated feedback in efl writing: A comparison study. Computer Assisted Language Learning, pages 1–30. Dennis Zyska, Alla Rozovskaya, Ilia Kuznetsov, and Iryna Gurevych. 2026. Exposía: Academic writing assessment of exposés and peer feedback. arXiv preprint arXiv:2601.06536. A Dataset A.1 Data Collection Materials The study was approved by the university’s IRB (§3). The recruitment materials distributed to stu- dents and instructors are reproduced below. A.2 Annotation Representation Details Paragraph-level annotations are stored within the corresponding paragraph, while document-level comments and scores are stored outside the para- graph structure. Sticky notes are stored with their text and ex- act positions within the paragraph body, reflecting the original placement. Highlights are stored with their exact spans and associated comments; in our data, nearly all highlights contain instructor-written comment text. For all paragraph-level annotations we also retain the immediate left and right context of the marked span to facilitate locating the anno- tation within the paragraph, especially when the highlighted text is short or when a sticky note is tied to a specific local position in the essay. When instructors use non-generic highlight colors, we preserve these as structured signals. In this dataset, color usage follows a consistent instructor-defined legend: • Pink: “great job, this is an awesome line” •Green: “there is something really interest- ing here . . . expand on this point in your next draft” • Yellow: “there is some problem with this sen- tence” A.3 Parsing Details A central preprocessing challenge is preserving both paragraph structure and annotation anchor- ing across annotated PDF and.docxsubmissions. For.docx, we extract standard highlights and com- ments with their anchored spans. For PDFs, how- ever, annotations must be reconstructed from page- level geometry, including highlight quadrilaterals and note positions, rather than from a linear text stream (Endignoux et al., 2016). Naive PDF text extraction can interleave head- ers and footers and scramble paragraph boundaries, which are often conveyed by layout rather than explicit markup. We also explored LLM-based re- paragraphization of extracted text, but observed deviations from the source text, including missing or added tokens and altered boundaries, consistent with known hallucination behavior (Huang et al., 2025); it also incurs nontrivial computational cost (Fernandez et al., 2025; Samsi et al., 2023). We therefore implemented a deterministic parser that converts annotated PDF and.docxsubmissions into a unified JSON format while preserving para- graph segmentation and annotation anchoring. We release this parser with the dataset. The remainder of this section describes the PDF pipeline, which is the more involved of the two; the stages are sum- marized in Figure 9. Body text extraction. The parser first extracts the essay body from the page-level text blocks. This stage is fully deterministic: every character present in the source text blocks is retained, and no content is dropped, rewritten, or reordered by a model. The extracted body text serves as the canonical reference string to which all subsequent annotations are anchored. Highlight anchoring. Highlights are recovered from their page-level geometry rather than from a linear text stream. For each highlight, the parser collects the highlight quadrilaterals, determines the text covered by those quads, and matches the cov- ered text back to the corresponding span in the ex- tracted body, anchoring the highlight to that span. The associated highlight comment, when present, is stored alongside the anchored span. Because nearly all highlights in SEFORA carry instructor- written comment text (§A.2), this step preserves both the marked span and its commentary. Highlight color recovery.Highlight color is not reliably available as a structured attribute, so the Email to Students. Hello everyone, my name is [team member], and I am part of a research team led by [PI’s name] at [x University]. We are seeking participants for a research study. Your instructor is allowing us to recruit participants in this class. This research study is funded by the [grant number] and aims to investigate how using generative AI, specifically large language models (LLMs) to improve the first-year college student writing class experience. We aim to build GenAI-based tools to help you understand the important elements of writing. To build the tool, we need your written essays on a certain topic to understand the common patterns of student writing and to evaluate the tool's effectiveness. We will also be collecting self-reported demographic and gender information to help us build fairer tools. To protect your confidentiality, when the data is analyzed and reported, you will only be identified by a number, never by your name. Your research data may be shared with investigators in the wider research community for research and development purposes; however, this information will be shared in a de-identified manner (without identifiers). In addition to being used in a confidential manner as research data, data may also be used in limited academic settings (e.g., small paper excerpts shown in an academic presentation at a conference). Your participation is voluntary. If you choose to participate, you can stop at any time. Your consent is required to participate in this study. If you have any additional questions or are NOT interested in participating, please reply to this email. If you consent to participate, no action is needed. Thank you very much for your time! Figure 6: Recruitment email for students. Dear Instructors, Thank you for your interest in participating in our project on building socially responsible machine writing tutors. Below, we briefly introduce the project and describe the data we hope to collect with your help. The purpose of this study is to develop a language model-based writing tutor that can analyze college student writing. This tool is designed to provide more accurate evaluations and feedback on students’ writing and to support the assessment of student essays. As part of our research during this [time frame], we would ask you to help with the following activities: First, we ask that you provide the writing assignment instructions and grading rubrics. Assuming these materials already exist, we would appreciate receiving electronic copies. Second, we ask that you collect student essays and the corresponding grades for the writing assignments. If all student essays are submitted through Canvas, we would appreciate receiving electronic copies of those materials as well. Third, we ask that you collect students’ in-class writing. This part is optional. If students completed their in-class writing through the same platform, such as Google Docs, we would appreciate receiving an electronic version. Fourth, we ask that you introduce the project to your students and inform them that their essays may be collected for research purposes. We estimate that this would take about 20 minutes of class time. We would be happy to attend and, if helpful, provide more detailed information about large language models. For completing the first four activities, you will receive a stipend of $150. Fifth, we ask that you provide detailed feedback on student essays, such as individual scores for each rubric criterion. Specifically, we ask for detailed feedback for each participating student across all assignments. Completion of this activity includes an additional stipend of $100. For research activities 1 and 2, the easiest approach would be for you to share your Canvas site with us so that we can download the materials directly and minimize the effort required on your side. Otherwise, we would ask you to download the assignments and student submissions and then share them with us, which may be more time-consuming. Figure 7: Recruitment email for instructor part 1. For research activity 3): if, by default, the students write on their own computer or piece of paper, it’l be hard for us to collect the data. So, we ideally ask about the possibility of using a uniform tool like Google Docs to collect student responses to in-class writing exercises. All data we receive from you and your students will be given an ID#. All stored data will have this number on it and not anyone’s real name. All responses are confidential, and data will be stored securely. Your research data may be shared with investigators in the wider research community for research and development purposes; however, this information will be shared in a de-identified manner (without identifiers). In addition to being used in a confidential manner as research data, data may also be used in limited academic settings (e.g. small paper excerpts shown in an academic presentation at a conference). Your participation is voluntary, and you may withdraw from the study at any time. There are no foreseeable risks associated with this project except for a possible breach of confidentiality. There are no direct benefits. You will receive a stipend of $250 per term for completing the [time frame] data collection activities. Specifically, we will disburse $150 upon completion of all four activities #1-4 above. We will disburse an additional $100 for the completion of activity #5 (i.e., the detailed feedback for student essays). We hope that you decide to be part of this research study! To sign up, ask questions, OR decline, please email us [PI’s email] This study is funded by the [grant name] and is approved by the IRB for data collection. The study is being led by [PI] at the [x department in y university] and is conducted in collaboration with the Department of English. Figure 8: Recruitment email for instructor part 2. parser recovers it visually: it performs a localized rendering of the highlighted region, reads the ren- dered color, and maps the resulting HEX value to the nearest entry in our set of named highlight colors. This yields the color names used as struc- tured signals in the corpus, following the instructor- defined legend documented in §A.2 (e.g., pink, green, and yellow), as illustrated in Figure 2. Sticky-note anchoring. Sticky notes are posi- tioned by page coordinates rather than attached to a text span, so the parser anchors them by location. For each note, it identifies the surrounding text within a coordinate radius of the note’s placement, locates that surrounding text within the extracted body, and injects the note at the corresponding po- sition. This reproduces the original placement of the note within the paragraph body (§A.2). Paragraph structure. Text blocks do not nec- essarily correspond to paragraphs, so paragraph boundaries cannot be read directly from the block structure. Instead, the parser infers paragraph struc- ture from visual layout cues, primarily the spac- ing between text blocks, recovering the paragraph segmentation that layout conveys without explicit markup. Header, footer, and identifier removal. Recur- ring page elements such as top-of-page names and page numbers are removed based on their position on the page. Submission metadata such as the sub- mission date is detected using regular-expression patterns. To ensure that student and instructor names do not leak into the released corpus, posi- tional and regex-based removal was complemented by additional scanning tools and human inspection, consistent with the privacy procedures described in §3. B Guidelines B.1 Feedback Segmentation Annotation Guideline The guideline used by annotators to segment in- structor and LLM-generated feedback into feed- back units (§4.1) is shown in Figure 10; segmented examples appear in Table 2. B.2 Feedback Unit Similarity Annotation Guideline The guideline used to score similarity between feedback-unit pairs on the 0–4 scale (§4.2) is shown in Figures 11–13. 1. Body text extraction Deterministic. Extracts every character in the box. 2. Highlight anchoring Quads→covered text→matched span; comment stored. 3. Color recovery Local render→ HEX→ named color. 4. Sticky-note anchoring Coordinate-radius context→located in body→ injected. 5. Paragraph structure Inferred from inter-block spacing (visual cues). 6. Identifier removal Positional headers/footers; regex metadata; human- checked. Figure 9: Stages of the deterministic PDF parsing pipeline (§A.3). Each annotated submission is converted to a unified JSON representation preserving paragraph segmentation and annotation anchoring. C Feedback Unit Similarity Pipeline C.1 Automatic similarity metrics. In our experiment and setting, lexical-overlap mea- sures such as BLEU (Papineni et al., 2002) and chrF (Popovi ́ c, 2015) yield Pearson correlations be- low0.1with annotator scores. Embedding-based approaches (MiniLM (Wang et al., 2020), MPNet (Song et al., 2020), and BERTScore (Zhang et al., 2019)) perform better but still show low correla- tion with human scores, with Pearson correlations below0.3. Cosine similarity using OpenAI’s em- bedding modeltext-embedding-3-large 16 is the strongest automatic baseline, but its performance depends heavily on the context provided to the embedding model. Our best result (r = 0.6696, ρ = 0.6343) is obtained by prefixing each feed- back with “Semantic content of instructor feedback on a student-written paragraph: ” before embed- ding. Even so, it remains insufficiently aligned with human judgments for reliable evaluation. C.2 Batch Similarity Scoring Details Batched requests.Since the similarity guideline is long, scoring every pair independently for each paragraph, draft, and experimental setting would be expensive and inefficient. We therefore score multiple pairs per request (batch_size= b), in- 16 https://developers.openai.com/api/docs/ models/text-embedding-3-large Feedback Segmentation Annotation Guideline Guideline: A single feedback point (also referred to as a feedback unit) is defined as a distinct, self-contained statement that addresses one specific aspect of a student’s writing. We treat a feedback point as one or more sentences that convey a coherent thought and can be understood independently. Importantly, a single feedback message may contain multiple such points, each corresponding to a different issue or suggestion. Characteristics of a single feedback point: Specificity: Focuses on one aspect of writing (e.g., grammar, word choice, structure, content, tone, a character in the story, etc.). Coherence: Forms a complete thought that is interpretable without requiring additional context. Actionability: It may offer an identifiable issue or suggestion that the student could address. If a span elaborates on the same aspect of writing (e.g., further explaining or justifying the same critique or suggestion), it remains part of the same feedback point. If the span shifts to a different aspect, comment, or suggestion, it should be marked as a new feedback point with a delimiter. Task: - Insert a delimiter ($) when the feedback moves to a new aspect of the student’s writing. - Each segment should be able to stand alone as a complete feedback point. - Do not insert delimiters for mere elaboration, clarification, or examples of the same point. - Do not use any other characters (spaces, line breaks, tabs). Only insert $. - If there is a need to place a delimiter in the middle of a sentence, and if the feedback points are separated by a transition word (e.g. "and", "but", "also", etc.), then place the delimiter before the transition word. - Every delimiter used must be either at the end of the feedback, or adjacent to a space (or tab or a linebreak character). The only case where a delimiter may be placed between two non-space characters is when in the original sample, there is no space after the punctuation of the previous feedback point. Sample 1: I love this smell! Great sensory detail :)$ Sample 2: 1) Your opening sentence is engaging – nice job!$ 2) Consider varying your transitions between sentences (e.g., instead of using “and”, try using words like however, meanwhile, etc.)$ 3) What does the phrase “newly waxed floors” mean? Is it necessary to describe the floor in such detail?$ Sample 3: Your vivid descriptions set the stage beautifully. Keep focusing on using sensory details to engage readers further.$ Great job showing how the simple drill captured your interest. Consider expanding on why it resonated with you specifically.$ Figure 10: Feedback segmentation annotation guideline (page 1/1). Feedback Similarity Annotation Guideline Problem Statement Given two feedback units targeted to a paragraph of an essay, how similar are they? A feedback unit is defined as a self-contained statement that addresses one specific aspect of a student’s writing. Guideline Given a pair of feedback units, A and B, annotate their similarity on a scale of 0 to 4. Note 0: We assume that both feedback units are given on the same essay and the same paragraph. If one unit provides a comment without explicitly specifying which part of the story or essay it targets, while the other does, it is safe to assume they refer to the same part, as paragraphs typically cover only a specific portion of the story, essay, or a specific scene. Note 1: This is not literal semantic equivalence. Similarity is driven primarily by whether the units target the same aspect/issue and/or convey the same comment about the writing. Targeting the same aspect establishes a high baseline (~2), and alignment in what the feedback recommends/implies pushes the score up; opposition/contradiction pushes it down toward 0. And vice versa: if they share the same comment, that establishes a high baseline (~2-3), and alignment in the targeted aspect increases the score toward 4. Score 4 is the highest score, indicating that A and B are almost equivalent as feedback units. This represents a looser standard than literal semantic equivalence, as the feedback units may be phrased differently, but the underlying “point” they convey is the same. Score 0 is the lowest score, indicating that A and B are completely irrelevant to each other or entirely contradictory as feedback units. Conceptually, scores 1, 2, and 3 are defined as equally spaced points between the two ends of the spectrum: score 0 (completely irrelevant) and score 4 (almost equivalent). Score 3 indicates that A and B are very close as feedback units, but a detail differs. For instance, if they convey the same point (a similar comment), but one of them targets a more specific aspect that falls under the more general aspect the other unit is focusing on. Score 2 indicates that A and B share some important/main points, but also differ on other important/main points. For example, if both focus on the same aspect of the writing but provide different comments, they share an important point (same important aspect) yet differ on the comments (different important points). Similarly, if A and B express the same comment or point but focus on different aspects of the writing, they again share one important point while differing on another. Score 1 indicates that A and B are mostly irrelevant as feedback units, but share a minor common detail and are not completely unrelated. For instance, both may mention an issue (shared detail), yet the issues themselves are entirely different (differing important points), resulting in a similarity score of 1. If the aspects they are targeting are somewhat related, but one is praising, the other one is mentioning an issue with it, it would not receive a score of 1; it would receive a score of 0. Figure 11: Feedback similarity annotation guideline (page 1/3). Examples: Score 4: (Almost) semantically equivalent as feedback units. A: Great job! B: Wow, this paragraph is really well-written! -> They both convey generally/all aspects are good. As feedback, neither convey anything else. A: Too many repeated words. B: Try to broaden the vocabulary used. -> They are rephrasings of the same exact point/comment A: This opening is captivating! B: Nice job setting the scene and creating suspense. -> Opening ≈ Setting the Scene (Same aspect). Captivating ≈ Nice job creating suspense (Same point/comment) Score 3: Share the same important/main point(s), some details differ. A: Add more details about the scene. B: What did he see exactly? -> Both feedback units are asking for more details (shared important point), A is asking for general details, B is specifically asking for a specific missing detail (differing detail). A: The story is a little choppy. B: I couldn’t follow; why didn’t she leave then? -> They both convey that the story is abrupt and lacks proper flow (shared important point), B is addressing a specific part of the story that is abrupt. A: This is well-written. Nice job. B: The sensory details are just perfect. -> They both convey the point of being well-written (shared important point) but one specifies exactly what is making it well-written (different detail). A: This is good... B: Excellent! -> Both feedback units are conveying the point of being well-written (shared important point), but their intensity is different (different detail) Score 2: Share some important point(s), some important points differ. A: Such a balanced and well-written paragraph! B: You developed the characters well. -> They both convey the point of being well-written (shared important point), but B specifically mentions a main aspect of the essay, the characters (different important point). Why not 3: Characters of a story cannot be considered as a detail. Characters are part of the paragraph, but the difference is not negligible. Also, the comment of “balanced and well- written” and “well-developed” are also different. Figure 12: Feedback similarity annotation guideline (page 2/3). A: Clarify what is exactly happening in this scene! B: Please clarify what you mean by “blue smell”. -> Both are asking for clarification (shared important point), but B is asking for clarification of a specific phrase (different important point). Why not 3: They are asking for clarification, which is very general (but is enough to set the baseline on score 2), but A asks for clarification of what is happening, B is asking for clarification of a phrase or a sensory detail (which is completely different). A: Very nice voice and tone. B: Very well-developed characters! -> They are both saying that an element of the essay is good (important shared point), but both specify a specific aspect of the essay (important different point). That is, the aspect of voice/tone is largely unrelated to character development. Score 1: Share a few details, but important points differ. (Not COMPLETELY irrelevant, or they share the same general feedback category) General category: Mistake, Elaboration/Clarification, Praise A: It should be “wear”, not “where” B: Too much repetition... -> They share the same feedback category -- “mistake” A: Interesting conversation between you and Sara! B: Nice job with the sensory details. -> Same feedback category -- “praise”, but the focus and main substance of the feedback are irrelevant to each other: sensory details of the scene, and the conversation between characters. Feedback A also only conveys the point that the conversation itself is interesting, and is not specifically saying this is well-written. Score 0: Completely irrelevant or even opposite. (They neither express the same content nor target the same aspect, or they contradict each other.) A: This paragraph is great! B: This paragraph is a little choppy -> They somewhat contradict each other. They also don’t share the same feedback category. A: Very well-written. B: So why did she remain on the couch? -> Irrelevant: A is praising, B is asking for clarification about some part of the story. Figure 13: Feedback similarity annotation guideline (page 3/3). structing the model to evaluate each pair indepen- dently (implementation details in §C.2). Batching reduces the cost in inverse proportion, but since it alters the prompting setting, we reverify the agree- ment with the human reference. Correlation re- sults for multiple models under the batching setup are reported in Table 3 (§4.2), where we adopt gemini-3.1-flash-lite-previewfor the main pipeline on the strength of its agreement with hu- man scores. Batching strategy.The similarity guideline used for scoring feedback-unit pairs is nearly 2K tokens long (although the exact token count depends on the tokenizer), making repeated independent calls inefficient. To reduce token cost, we evaluate mul- tiple pairs in a single request (batch_size= b), instructing the model to assign one score to each labeled pair independently. Token cost. LetAdenote the irreducible per- request token cost (system instructions, formatting, and response overhead), andBthe guideline length. When scoringbpairs per request, the total token cost per pair can be approximated as T (b) = A + B b , so batching reduces the amortized guideline over- head by a factor of O(1/b). Effect on agreement.Batching slightly changes the prompting context because the model evaluates multiple pairs within the same request. We there- fore verify that this does not materially affect simi- larity scoring. Experiments withbatch_size= 50 show agreement with the human reference com- parable to the single-pair setting reported in §4.2, indicating that batching does not meaningfully alter model judgments. Implementation notes. We avoided the Batch API (i.e., offline requests used to reduce API cost) in order to preserve the ability to retry malformed outputs.In auxiliary runs with gpt-5-mini-2025-08-07, which was not the pri- mary model used in our main evaluation, we did not observe any syntactically invalid outputs. gemini-3.1-flash-lite-preview, which was used at much larger scale, produced a small number of syntactically invalid outputs, and in those cases, maximum of two retries were sufficient to obtain a syntactically valid output. D Threshold-Based Cross Matching UNIMATCH’s default cross-matching stage (§4.3) aggregates similarity scores into soft precision and recall by treating each unit’s full self-similarity as its maximum possible contribution. We comple- ment this with a threshold-based variant that mir- rors detection-style evaluation in computer vision (Everingham et al., 2010): given a thresholdτon the annotator-aligned 0–4 similarity scale, a pair counts as a match only if its similarity is at least τ, and the standard hard precision, recall, and F 1 follow. Formalization. LetG = G i |G| i=1 andP = P j |P| j=1 denote instructor and predicted feed- back units for a single paragraph, with simi- larity scorerφ(G i ,P j ) ∈ 0, 1, 2, 3, 4.For τ ∈ 1, 2, 3, 4, define the binarized adjacency a ij (τ ) = 1[φ(G i ,P j )≥ τ ]and compute the maxi- mum bipartite matching on a: M ∗ τ = arg max M∈M X (G i ,P j )∈M a ij (τ ), whereMis the set of 1-to-1 matchings betweenG andP. The match count and complementary error counts are TP τ =|M ∗ τ |, FP τ =|P|− TP τ , FN τ =|G|− TP τ , yielding the hard metrics p τ = TP τ |P| , r τ = TP τ |G| , F 1,τ = 2p τ r τ p τ + r τ . Interpretation ofτ.Because the similarity scale is grounded in the annotation guideline (§B.2),τ has a direct semantic reading:τ =1requires the pair to share at least a general feedback category,τ =2 requires sharing an important point (same targeted aspect or same comment),τ =3requires that the units differ only in a minor detail, andτ =4requires near-equivalence as feedback. The threshold thus exposes how strict an “alignment” must be to count as feedback recovery. Sensitivity to over-generation. In the soft vari- ant, each additional predicted unit contributes its full self-similarityφ max to the precision denomi- nator P j φ(P j ,P j ), so a weak-but-nonzero match still inflates the denominator without comparably increasing the numerator. The threshold-based vari- ant attenuates this: predictions affect precision only through their count, and any pair aboveτreceives full credit. Weak-but-above-threshold matches are therefore not partially discounted, and the per-extra- unit cost is smaller. The variant offers a less strin- gent view when over-generation is unavoidable; the soft variant remains preferred when fine-grained similarity differences carry signal. Ranking robustness across the 74 configura- tions.Despite the qualitative difference in metric form, the relative ordering of the 74 configurations from §5 is preserved under thresholding: for every τ ∈1, 2, 3, 4, the Spearman rank correlation be- tween configuration F 1 under the soft variant and F 1,τ isρ = 1.0. The dominant findings reported in §6 – the negative effect of verbosity, the gains from feedback-category guidance, the limited con- tribution of rubric inclusion, and the secondary role of few-shot prompting – therefore do not depend on the choice of matching metric. We omit per- configurationF 1,τ tables, as the rank-identity re- sult subsumes them; the implementation is released alongside the soft variant for reproduction. Implementation. Threshold-basedmatching reuses the same similarity matrices produced for the soft variant; no additional model calls are required. We binarize the matrix atτand run a bipartite matching solver on the resulting graph. Note that the Hungarian algorithm handles the general case of weighted edges (similarity scores), whereas in this setting edges are unweighted. E Full Evaluation Results This appendix presents per-configuration UNI- MATCH scores for all 74 settings described in §5. Tables 5, 6, and 7 report precision, recall, F 1 , and number of generated feedback units under zero- shot, few-shot, and chain-of-thought (CoT) prompt- ing, respectively. Each table reports both the single reference evaluation (paragraphs with exactly one instructor feedback unit) and the full corpus eval- uation (every paragraph with at least one instruc- tor annotation); the two regimes track each other closely on every comparison reported in §6, and we therefore discuss only the full-corpus numbers throughout the main text. Figure 16 summarizes the F 1 effect of the prompt template across repre- sentative configurations. Per-condition ablation visualizations. Fig- ures 14, 15a, and 15b provide per-model, per-condition visualizations of the guidance, rubric, and few-shot effects summarized in Table 4 (§6). Prompt template. The default constrained prompt (V 2 , Figure 18) substantially outperforms the unconstrained alternative (V 1 , Figure 17) across models, conditions, and shot settings (Figure 16). Across 28 matched pairs (same model, guidance, rubric, and shot),V 2 yields a mean F 1 gain of +0.090overV 1 , and wins in 27 of 28 pairs (96%). The mechanism is verbosity:V 1 averages roughly 12K generated feedback units per configuration vs. 4.6K forV 2 , and the resulting precision deficit over- whelms any recall benefit. This is consistent with the broader verbosity finding in §6: the constrained prompt enforces a per-paragraph output budget that the unconstrained prompt does not. Few-shot prompting underV 1 . Few-shot ex- emplars contribute much more under the uncon- strained prompt than under the constrained one. In matched comparisons, few-shot improves F 1 by +0.028underV 1 but only+0.005underV 2 (Ta- ble 6). This supports the interpretation in §6 that ex- emplars are most useful when task framing is weak- est:V 2 already specifies the target output structure, leaving little additional ground for exemplars to cover. The few-shot gain is precision-driven across all 30 matched (zero, few) pairs: precision rises by+0.028on average while recall falls slightly (−0.004). Chain-of-thought.CoT prompting (Table 7) was evaluated on the three small models in the zero-shot setting, using the unconstrainedV 1 prompt. Under matched conditions, CoT improves F 1 by+0.041 overV 1 but underperformsV 2 by−0.056. CoT set- tings produce 6.6K–8.4K generated feedback units per configuration, betweenV 1 andV 2 , and the qual- itative patterns from the main results hold: Mistral is the strongest small model, feedback-category guidance contributes a small positive effect, and rubric inclusion is essentially neutral. CoT does not alter the overall story; like other settings, its position in the F 1 ordering is largely determined by how many feedback units it generates. F Prompt Templates We use two prompt templates, both with two tog- gles (Figures 17, 18): Guided? controls whether ShotGuidedRubricVModel Single referenceFull corpus PRF1#Seg.PRF1#Seg. Zero Un- guided No Rubric 1 Llama0.0610.2600.099103090.0800.2320.11915580 DeepSeek0.0660.2160.10177430.0860.1900.11811571 Mistral0.0730.2760.11592730.0970.2480.13913779 2 Llama0.1450.2160.17436290.1830.1830.1835363 DeepSeek0.1360.2270.17040200.1720.1890.1805851 Mistral0.2320.3010.26231610.2510.2210.2354740 Rubric 1 Llama0.0580.2670.095112300.0790.2450.11916763 DeepSeek0.0680.2390.10682250.0950.2200.13211924 Mistral0.0880.2980.13682270.1230.2800.17112212 Qwen0.0930.2700.13870750.1120.2390.15311448 GPT0.1950.3540.25244200.2260.2850.2536764 2 Llama0.1900.2530.21732510.2160.1920.2034766 Mistral0.2250.3490.27337810.2410.2610.2515798 Qwen0.3020.3460.32228000.3120.2390.2714112 GPT0.3440.3570.35025260.3640.2500.2963681 Guided No Rubric 1 Llama0.0570.2430.093103510.0810.2340.12015583 DeepSeek0.0640.2500.10290540.0870.2280.12513208 Mistral0.0940.3060.14479380.1270.2880.17712127 2 Llama0.2790.3540.31230940.2930.2470.2684529 DeepSeek0.1960.3160.24236190.2170.2390.2275326 Mistral0.2900.3650.32330690.3010.2590.2784620 Qwen0.3220.3460.33426230.3410.2450.2853859 GPT0.3270.3460.33625730.3470.2460.2883792 Rubric 1 Llama0.0520.2490.087115930.0740.2390.11317316 DeepSeek0.0610.2400.09789870.0860.2270.12513467 Mistral0.0810.2770.12583590.1140.2630.15912408 Qwen0.1150.3030.16764070.1360.2610.17910305 GPT0.0720.3150.118106160.0890.2940.13617739 2 Llama0.2560.3340.29031840.2750.2360.2544592 Mistral0.2710.3650.31132860.2880.2670.2774962 Qwen0.3730.4030.38726290.3740.2680.3123838 GPT0.3560.3870.37126500.3760.2730.3163892 Table 5: UNIMATCH results for all zero-shot configurations on the single reference subset and the full corpus, as defined in §5.V 1 denotes the unconstrained prompt (Figure 17);V 2 denotes the default single-unit prompt (Figure 18), used throughout the main results. P and R denote precision and recall as formulated in §4.3; #Seg. denotes the number of generated segments. Model names are abbreviated for space; full names are given in §5. ShotGuidedRubricVModel Single referenceFull corpus PRF1#Seg.PRF1#Seg. Few Un- Guided No Rubric 1 Llama0.0770.2310.11572780.1020.2030.13610653 DeepSeek0.0700.2370.10879200.0930.2200.13112097 Mistral0.1100.2640.15658190.1480.2410.1838719 2 Llama0.1900.2590.21933270.2180.1980.2084873 DeepSeek0.1940.2760.22834190.2210.2140.2185113 Mistral0.2740.3180.29428180.2930.2270.2564151 Rubric 1 Llama0.0660.2290.10284610.0890.2010.12412062 DeepSeek0.0500.2490.083121420.0680.2290.10517996 Mistral0.1000.2820.14768830.1360.2650.18010440 Qwen0.2220.3140.26034430.2570.2490.2535199 GPT0.2600.3880.31136410.2990.3100.3055560 2 Llama0.2240.2580.24028080.2350.1800.2044103 Mistral0.2410.3440.28334840.2630.2600.2615305 Qwen0.2760.3310.30129190.2920.2340.2604305 GPT0.3480.3660.35725570.3690.2560.3023721 Guided No Rubric 1 Llama0.0850.2380.12568470.1170.2210.15310123 DeepSeek0.0500.2630.084128150.0690.2430.10718947 Mistral0.0740.3270.121107200.1030.3030.15415755 2 Llama0.3020.3480.32328120.3040.2310.2624079 DeepSeek0.1980.3040.24034420.2200.2340.2275293 Mistral0.2950.3500.32028910.3090.2490.2754316 Rubric 1 Llama0.0660.2120.10177770.0930.1940.12611183 DeepSeek0.0570.2420.092102860.0810.2280.11915056 Mistral0.0980.3050.14975800.1350.2810.18211198 Qwen0.2170.2990.25233380.2510.2370.2445055 GPT0.2890.3400.31228670.3180.2590.2854371 2 Llama0.2900.3570.32029930.3020.2450.2714355 Mistral0.2530.3410.29032890.2740.2440.2584774 Qwen0.3750.4090.39126570.3770.2740.3173903 GPT0.3520.3760.36326040.3690.2630.3073814 Table 6: UNIMATCH results for all few-shot (5-shot) configurations, evaluated on the same subsets and under the same prompt variants (V 1 , V 2 ) as the zero-shot setting. Column definitions follow Table 5. ShotGuidedRubricVModel Single referenceFull corpus PRF1#Seg.PRF1#Seg. Zero Un- Guided No Rubric CoT Llama0.1140.2140.14944640.1490.1870.1666641 DeepSeek0.0980.2210.13647080.1290.2010.1577322 Mistral0.1490.2850.19646530.1820.2400.2077065 RubricCoT Llama0.1130.2290.15148570.1450.1950.1677140 DeepSeek0.0930.2060.12843250.1300.1970.1576894 Mistral0.1370.3080.18954840.1730.2680.2108312 Guided No Rubric CoT Llama0.1130.2160.14846150.1430.1840.1616848 DeepSeek0.0960.2000.12942710.1270.1810.1496629 Mistral0.1370.2750.18348760.1820.2450.2097214 RubricCoT Llama0.1150.2380.15550010.1460.1980.1687260 DeepSeek0.0910.2040.12645340.1280.1950.1547111 Mistral0.1270.2940.17856310.1650.2590.2028431 Table 7: UNIMATCH results for chain-of-thought configurations under the unconstrained prompt (V 1 ). Column definitions follow Table 5. 0.230.280.260.320.260.32 0.180.240.210.260.200.30 G BaseR Few DeepSeekMistralLlamaQwenLlamaGPT Figure 14: Effect of guidance on F 1 . Each column is one model under a given condition (Base: zero-shot, no rubric; Few: few-shot; R: rubric); the top row adds feedback-category guidance (G) and the bottom row omits it. Guidance improves F 1 in every column. Darker cells indicate higher F 1 . 0.200.260.28 0.210.280.27 R B Few G LlamaMistralMistral (a) Rubric 0.260.320.37 0.270.310.36 F Z G R LlamaQwenGPT (b) Few-shot Figure 15: Effect of rubric and few-shot prompting on F 1 , each panel on its own color scale (darker = higher F 1 ). Columns indicate a model and a generation setting; rows toggle the studied factor. (a) Rubric present (R, the assignment rubric) vs. absent (B, base), across few- shot (Few), few-shot-with-guidance, and guidance (G, feedback categories in the prompt). (b) Few-shot (F) vs. zero-shot (Z), across guidance, guidance-with-rubric, and rubric; overlapping header lines mark the column with both settings. Both factors shift F 1 only slightly and inconsistently, unlike the uniform gain from guidance (Figure 14). the feedback categories (§5) are listed, and Rubric? whether the assignment prompt and rubric are in- serted viarubric. The unconstrained template (V 1 , Figure 17) places no limit on the number of feedback units; the constrained template (V 2 , Fig- ure 18) asks for a single most-important point under a fixed output format.V 2 is the default in the main experiments (§5);V 1 is used only in the prompt- design comparison (§E). 0.180.230.250.320.32 0.120.130.170.140.24 V 2 V 1 BaseGRR+GFew+R+G LlamaDeepSeekMistralGPTQwen Figure 16: F 1 across condition (top header) and prompt variant (rows).V 1 denotes the unconstrained prompt (§F);V 2 denotes the default single-unit prompt used throughout the main experiments. Prompt 1 Template You are a professional writing instructor providing very concise, useful feedback. You should spot obvious errors and mention surface-level improvements, but your main focus is to offer incisive, deep, insightful, and actionable advice that helps the student improve. When a passage is especially strong and offers little room for improvement, acknowledge that briefly. Your response should be concise, to the point, and limited to a few sentences, each addressing a specific improvement or commendation. While keeping the full essay in mind, focus only on the target paragraph. If something is wrong with a specific part, sentence, or word, explicitly identify that part. **Feedback categories/options:** *Task Constraints* – Clarifies goals, requirements, or rubric expectations. *Concepts* – Defines or explains key terms or conceptual context. *Elaboration* – Encourages expansion, depth, or development of ideas/scenes. *Clarification* – Flags confusion and asks for clearer meaning or context. *Mistakes* – Identifies and explains errors (grammar, logic, organization, repetition). *Praise* – Pure encouragement or positive reaction only. *None* – Personal opinion, symbols only, or mixed feedback types. ### Assignment Rubric: rubric ### Essay: context ### Target Paragraph: paragraph ### Provide feedback to the student on the target paragraph: Guided? Rubric? Figure 17: Unconstrained prompt template (free-form paragraph-level feedback with no constraint on the number of feedback units). Used in the supplementary results comparing prompt designs (§E); the guided variant is introduced in §5. Prompt 2 Template ### Instruction ### You are an experienced English teacher. Your task is to: 1) Identify the SINGLE most important feedback focus in the TARGET PARAGRAPH. 2) Determine which ONE feedback category best matches that focus. 3) Write ONE short piece of constructive feedback that reflects that focus. ### Feedback Categories ### Choose exactly ONE category based on the feedback focus you identify. - Task Constraints: Feedback about how the writing aligns with the assignment or rubric. - Concepts: Feedback that supports understanding or development of a key concept. - Elaboration: Feedback that encourages adding detail, examples, personal insight, or depth. - Clarification: Feedback that addresses meaning, coherence, or reader understanding. - Mistakes: Feedback that identifies and explains a specific error. - Praise: Feedback that offers encouragement without suggesting change. ### Output Format (MUST FOLLOW EXACTLY) ### Feedback Type: <one category name from the list above> Feedback: <ONE paragraph of feedback> ### Feedback Rules ### - The feedback paragraph must be 2–4 sentences. - Address the student directly using “you”. - Focus on ONE feedback focus only. - Do NOT summarize the essay or paragraph. - Do NOT list strengths and weaknesses. - Do NOT include multiple feedback moves. - Do NOT include meta-commentary. ### Essay ### context ### Target Paragraph ### paragraph ### Rubric ### rubric ### Output ### Rubric? Guided? Figure 18: Default single-unit prompt template used throughout the main experiments (§5); the guided variant is also introduced §5.

Instant research discovery