Paper deep dive

Grammar Concept Annotation at Scale: Deployed Fine-Tuned Small Language Models Outperform Prompted Frontier Models

Marjan Celikik, Ana Peleteiro Ramallo, Javier Morales

Year: 2026Venue: arXiv preprintArea: cs.CLType: PreprintEmbeddings: 68

Intelligence

Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 93%

Last extracted: 10/9/2026, 7:56:29 AM

Summary

This paper presents a scalable grammar mastery tracking system deployed on an online tutoring platform. The authors fine-tuned Qwen3.5 small language models (SLMs), specifically a 0.8B parameter model, on teacher-generated supervision data to perform grammar concept annotation. The fine-tuned 0.8B model outperforms prompted frontier models (GPT-5.4 and GPT-5.6 Sol) in precision and recall on human-curated benchmarks, while reducing serving costs by approximately 16x. An online A/B experiment demonstrated significant improvements in learner engagement (+15.8%) and business metrics such as scheduled hours and GMV.

Entities (9)

Preply · organization · 99%Qwen3.5-0.8B · model · 98%GPT-5.4 · model · 95%GPT 5.6 Sol · model · 95%LoRA · technique · 94%vLLM · softwarelibrary · 93%Grammar Concept Annotation · task · 92%Golden2 Benchmark · dataset · 90%Golden Benchmark · dataset · 90%

Relation Signals (9)

Qwen3.5-0.8B → deployedby → Preply

confidence 98% · Marjan Celikik Affiliation: Preply... deploying an efficient 0.8B model in an end-to-end grammar mastery tracker for all English learners on our platform.

Qwen3.5-0.8B → usedinsystem → Grammar Concept Annotation

confidence 96% · At the core of the system is the grammar concept annotation task... We fine-tuned Qwen3.5 small language models... then deploying an efficient 0.8B model

Qwen3.5-0.8B → outperforms → GPT-5.4

confidence 95% · On two human-curated benchmarks, both the deployed 0.8B model and a 4B reference comparator outperform prompted GPT-5.4 and GPT-5.6 Sol in precision and recall

Qwen3.5-0.8B → outperforms → GPT 5.6 Sol

confidence 95% · On two human-curated benchmarks, both the deployed 0.8B model and a 4B reference comparator outperform prompted GPT-5.4 and GPT-5.6 Sol in precision and recall

Qwen3.5-0.8B → finetunedwith → LoRA

confidence 94% · We adapted Qwen3.5... base models with completion-only LoRA

Qwen3.5-0.8B → servedby → vLLM

confidence 93% · A production serving architecture pairing the task-specialized 0.8B SLM with prompt compaction, native vLLM, and autoscaling

Qwen3.5-0.8B → reducescostby → 1.6x

confidence 92% · The deployed 0.8B SLM reduces serving cost by approximately 16× relative to GPT-5.4

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:Corrective feedback is among the best-evidenced drivers of second-language acquisition, yet corrections delivered during lessons rarely accumulate into an actionable view of grammar mastery. Prompted frontier models can provide such a view from learner--tutor lesson transcripts, but they are costly at scale. We close this gap by fine-tuning Qwen3.5 small language models (SLMs) on filtered and rebalanced teacher-generated supervision, then deploying an efficient 0.8B model in an end-to-end grammar mastery tracker for all English learners on our platform. Internalizing the annotation contract into adapter weights enables pairing the 0.8B model with a compact matched prompt rather than verbose instructions. On two human-curated benchmarks, both the deployed 0.8B model and a 4B reference comparator outperform prompted GPT-5.4 and GPT-5.6 Sol in precision and recall under nested matching criteria of increasing strictness: concept, evidence span, and correctness. The deployed 0.8B SLM reduces serving cost by approximately 16$\times$. A feature-level online experiment shows significant gains in learner engagement ($+15.8\%$) and key business metrics, including scheduled hours ($+2.1\%$) and GMV from new lessons ($+13.2\%$).

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

64,936 characters extracted from source content.

Expand or collapse full text

Grammar Concept Annotation at Scale: Deployed Fine-Tuned Small Language Models Outperform Prompted Frontier Models Marjan Celikik Affiliation: Preply Email: marjan.celikik@preply.com Ana Peleteiro Ramallo Affiliation: Preply Email: ana.peleteiro@preply.com Javier Morales Affiliation: Preply Email: javier.morales@preply.com Abstract Corrective feedback is among the best-evidenced drivers of second-language acquisition, yet corrections delivered during lessons rarely accumulate into an actionable view of grammar mastery. Prompted frontier models can provide such a view from learner–tutor lesson transcripts, but they are costly at scale. We close this gap by fine-tuning Qwen3.5 small language models (SLMs) on filtered and rebalanced teacher-generated supervision, then deploying an efficient 0.8B model in an end-to-end grammar mastery tracker for all English learners on our platform. Internalizing the annotation contract into adapter weights enables pairing the 0.8B model with a compact matched prompt rather than verbose instructions. On two human-curated benchmarks, both the deployed 0.8B model and a 4B reference comparator outperform prompted GPT-5.4 and GPT-5.6 Sol in precision and recall under nested matching criteria of increasing strictness: concept, evidence span, and correctness. The deployed 0.8B SLM reduces serving cost by approximately 16×. A feature-level online experiment shows significant gains in learner engagement (+15.8%+15.8\%) and key business metrics, including scheduled hours (+2.1%+2.1\%) and GMV from new lessons (+13.2%+13.2\%). 1 Introduction Corrective feedback is one of the best-evidenced drivers of second-language acquisition (Li, 2010). In live tutoring, however, corrections are often given in the moment rather than retained as structured evidence of progress. This makes it difficult for learners and tutors to track which grammatical concepts have been mastered, which have merely been avoided, and where the largest learning gaps remain. An explicit, longitudinal record bridges this gap: learners see which concepts to practice, while tutors see what to revisit in future lessons. We present a grammar mastery tracking system deployed to all English learners on our platform. It processes speech-to-text transcripts of lesson audio 11 1 Lesson transcripts are analyzed exclusively from sessions for which we have a lawful basis to do so (see Ethical Considerations). and aggregates correct and incorrect concept evidence into overall and per-concept grammar mastery scores (Figure 1). Evidence is weighted by concept frequency, CEFR level (Council of Europe, 2020), and a forgetting curve that discounts older evidence, consistent with trainable memory models for language learning (Settles and Meeder, 2016). Every piece of evidence is traceable to a highlighted span in the learner’s own sentences; incorrect uses also receive a short explanation (Figure 2c). At the core of the system is the grammar concept annotation task that identifies which curricular concepts a learner attempts, extracts the corresponding evidence spans, and judges each use as correct or incorrect. For incorrect uses, it also generates learner-facing explanations. The model returns these predictions in a single structured pass per learner turn, capturing both correct mastery and actionable errors. Our initial product pilot prompted GPT-5.4 for grammar concept annotation in learner utterances extracted from lesson transcripts. While it achieved the desired annotation quality, prompting GPT-5.4 at production scale would have far exceeded the feature’s serving budget (Section 7). Each request repeated the complete concept inventory, tagging rules, examples, and span conventions. Therefore, the pilot was initially rolled out to a small fraction of learners. The same pattern is widely reported for enterprise AI: many products remain in the pilot phase, and only about 5% reach production (Challapally et al., 2025). To make full rollout viable and improve quality, we fine-tuned Qwen3.5 SLMs on filtered and rebalanced teacher-generated supervision, internalizing the task contract through demonstrations rather than verbose instructions. We deployed an efficient 0.8B SLM with a compact matched prompt, using a 4B model as a reference comparator to confirm that aggressive prompt compaction and parameter reduction incur no quality penalty. On human-curated benchmarks, both models surpass prompted GPT-5.4 and GPT-5.6 Sol. The contributions of this work are as follows: 45%Grammarmastery score Concept Used Mastery Relative clauses 2k 10% Modal probability 320 25% Some / any / no 650 25%+5 Past perfect 240 40%+5 Present perfect cont. 1k 70% … Figure 1: Product view of the aggregated grammar mastery score (left) and per-concept breakdown (right; teal values indicate recent mastery gains). 1. A scalable end-to-end grammar mastery tracking system, deployed and evaluated in a feature-level online experiment showing significant uplift in learner engagement, tutoring hours, and gross merchandise value (GMV) from new lessons. 2. A formulation of learner grammar feedback as grammar concept annotation (Section 2), with two human-curated benchmarks and nested criteria for concept, exact evidence span, and correctness. On both benchmarks, our deployed 0.8B model and 4B reference comparator outperform prompted GPT-5.4 and GPT-5.6 Sol at every matching level in precision and recall. 3. A data quality pipeline combining evidence-grounded review, deterministic validation, filtering, and synthetic enrichment. Relative to matched 4B and 0.8B models trained on raw teacher annotations, the full pipeline raises recall by 36.9–40.7 percentage points when concept, exact span, and correctness must all match, with further gains on rare concepts and incorrect uses. 4. A production serving architecture pairing the task-specialized 0.8B SLM with prompt compaction, native vLLM, and autoscaling, reducing serving cost by approximately 16× relative to GPT-5.4 based on measured production traffic and load tests. 2 Grammar Concept Annotation Task The production pipeline turns speech-to-text transcripts of lesson audio into structured evidence for concept-level mastery tracking. It extracts learner utterances and relevant context, then an upstream LLM produces a corrected sentence that represents the learner’s intended meaning. The annotation model receives the original utterance and this correction, rather than the full transcript. For an input pair (ui,ci)(u_i,c_i), where uiu_i is the learner utterance and cic_i its correction, the system produces a set of annotations A⁡(ui,ci)=(kj,sj,yj,ej)j=1mi,mi≥0,A(u_i,c_i)=\(k_j,s_j,y_j,e_j)\_j=1^m_i, m_i≥ 0, where kj∈k_j is one of the ||=34|K|=34 grammar concepts,22 2 The 34 concepts correspond to an initial version of the system; the architecture extends and scales to a much larger number of concepts, with the next iteration containing more than 300 fine-grained concepts. sjs_j is an exact contiguous substring of uiu_i, yj∈correct,incorrecty_j∈\ correct, incorrect\ judges the use of kjk_j in sjs_j, and eje_j is a learner-facing explanation that is non-empty if and only if yj=incorrecty_j= incorrect. The model emits this set in a single pass: one utterance may yield several concepts or none. Because sjs_j must be an exact contiguous substring of uiu_i, the same span can be highlighted in the learner’s sentence and compared with gold spans by string identity at evaluation. Figure 2 shows an example input, model output, and learner-facing rendering. Appendix Table 11 states the same contract field by field. The runtime representation is one pipe-delimited annotation per line. The format was chosen during the product pilot to balance quality and cost. (a) Input Original: I have finish almost all. Corrected: I have finished almost all. (b) Model output 1|PRESENT_PERFECT|N|have finish|Use ‘finished’. 1|QUANTIFIERS_BASIC|Y|almost all| (c) Learner-facing rendering Present perfect I have finish almost all. → I have finished almost all. Use the past participle “finished” after “have”. Figure 2: Illustrative synthetic example. (a) Learner utterance and upstream correction. (b) Model output: sentence index, concept, correctness (Y/N), exact source span, and explanation for incorrect use only. (c) Learner-facing rendering of one incorrect annotation; correct annotations contribute to mastery without appearing as feedback. The nested structure of the tuple defines the evaluation hierarchy in Section 5: matching on kjk_j, on (kj,sj)(k_j,s_j), and on (kj,sj,yj)(k_j,s_j,y_j) yields progressively stricter criteria, while eje_j is assessed separately because explanations are free text. 3 Data and Quality Pipeline 3.1 Training Data Collection, Cleaning, and Enrichment Directly fine-tuning on raw production logs is hindered by teacher annotation noise, span boundary drift, and severe class imbalance favoring frequent, error-free structures. We address these challenges through a three-stage curation pipeline: (i) evidence-grounded review and programmatic validation of production annotations; (i) structured synthetic generation to rebalance rare concepts and pedagogical errors; and (i) corpus consolidation, deduplicating across sources (1.4% overlap) and strictly removing any utterance present in either golden benchmark. The resulting training corpus contains approximately 187k utterance groups (learner utterance, correction, and annotations), consolidated from 100k reviewed production records and 90k synthetic records prior to deduplication. Production data and cleaning. Each production group pairs a learner utterance and upstream correction with span-level annotations from the product pilot. We sample 50k groups uniformly to preserve production traffic distributions, and 50k via heuristic sampling prioritizing utterances suspected of incomplete or excessive annotations (anomalous counts relative to length, or low span coverage). To clean these records, we adopt a hybrid neuro-symbolic strategy: deterministic linguistic anchors extracted via spaCy (Honnibal et al., 2020) are converted into a plain-text inventory of grammatical elements (articles, pronouns, prepositions, modals, verb forms, and tense patterns), alongside hand-crafted candidate rules. In the prompt to GPT-5.4, these cues are combined with word-level edit differences from the upstream correction (pinpointing where the learner diverged from target grammar) and the pilot annotations. Rather than filtering alone, GPT-5.4 performs a full review and re-annotation pass (retaining, correcting, removing, or adding tuples). A deterministic post-validator then enforces strict contract invariants: rejecting non-substring spans, pruning invalid labels, and enforcing explanation consistency (Appendix B). Synthetic enrichment. Unconstrained synthetic generation often hallucinates labels or perpetuates natural class skews. We instead decouple syntactic scaffolding from lexical realization using structural backbones—sequences of concept labels and correctness flags harvested from quality-filtered production utterances. To address long-tail sparsity, we stratify backbone sampling across three 30k subsets: uniform sampling, error-weighted sampling (favoring templates with natural mistakes), and smoothed inverse concept-frequency sampling (boosting rare curricular targets). Unlike post-hoc token corruption, which produces disfluent or unnatural errors, we inject pedagogical errors directly into the generation plan: with p=0.05p=0.05, a correct slot is assigned an error type (within-concept misuse, omission, or confusable structure substitution). Prompted with the backbone and matching production exemplars, GPT-5.4 realizes the plan into learner-style sentences, exact spans, and targeted explanations. Deterministic post-validation then enforces span substring constraints and label alignment (Appendix B). 3.2 Golden Evaluation Benchmarks Evaluating an educational tracking system requires measuring both overall operational accuracy and robustness on critical failure modes. We curate two independent human-adjudicated benchmarks over the 34 concepts (Table 1). Golden (689 utterances) mirrors the natural production distribution, providing an unskewed evaluation of daily traffic. In contrast, Golden2 (2,977 utterances, constructed via model-assisted candidate retrieval followed by double-annotator curation and adjudication) is an error-enriched diagnostic benchmark (95.6% error-bearing utterances vs. 50.1% in Golden). Golden2 stress-tests the system against the product’s costliest failure mode—failing to detect a genuine learner error—while providing ample support to evaluate explanation fidelity. In both benchmarks, two annotators worked independently and a third adjudicated disagreements. Statistic Golden Golden2 Utterances 689 2,977 Grammar concepts 34 34 Concept annotations 3,598 17,400 Annotations / utterance 5.22 5.84 Annotations marked incorrect 592 (16.5%) 6,075 (34.9%) Utterances with ≥ 1 error 345 (50.1%) 2,845 (95.6%) Table 1: Composition of the two golden evaluation benchmarks under the common 34-concept evaluation scope. Annotation counts are computed after scope filtering. Learner ASR transcripts, annotations, and internal prompts are proprietary and privacy-sensitive, so we report aggregate statistics and methodological details but do not release records. Appendix E.4 reports an additional teacher-synthetic distribution-shift probe. 4 Fine-Tuning Method We adapted Qwen3.5 (Qwen Team, 2026) and Qwen3 (Yang et al., 2025) base models with completion-only LoRA (Hu et al., 2022). At training and serving time, packing five utterance–correction pairs into one conversation (Section 7) balanced throughput and memory; scaling further yielded diminishing returns while increasing activation memory. We used a 90/10 train–validation split and discarded examples exceeding the maximum sequence length rather than truncating targets. Checkpoints were selected by minimum teacher-forced validation loss evaluated four times per epoch with early-stopping patience of three evaluations (Appendix A). Prompt internalization and model sizing. Our primary deployment target is an efficient, sub-1B parameter model (Qwen3.5-0.8B) capable of serving high-concurrency traffic on single-GPU commodity instances (g6.2xlarge). In the production pilot, GPT-5.4 required a 3,810-token prompt detailing all 34 concepts, edge cases, and examples; at scale, repeatedly ingesting this verbose prompt induces severe prefill latency and dominates KV-cache memory in vLLM (Kwon et al., 2023), restricting concurrency. However, task fine-tuning enables prompt internalization (Zou et al., 2024; Shin et al., 2025): the model absorbs concept definitions and span conventions directly into adapter weights from demonstrations. We therefore designed a compact prompt (375 Qwen3.5 tokens, a 90.2% reduction; Appendix E.2) specifying only concept names, the schema contract, and brief rules, shrinking the maximum sequence length from 7,000 to 2,500 tokens (used for both 0.8B and 0.6B; Appendix A). To provide sufficient capacity for these rules, the 0.8B model used a higher adapter rank (r=128r=128 across all linear layers vs. r=16r=16 for 4B; Appendix Table 5). The larger Qwen3.5-4B model retained the detailed prompt strictly as a reference comparator to confirm that prompt compaction and parameter reduction incur no quality penalty. Differences between sizes thus reflect distinct operational recipes rather than an isolated parameter ablation. 5 Evaluation Methodology GPT-5.4 GPT-5.6 Sol Qwen3.5-4B Qwen3.5-0.8B Prompted (prod.) Prompted Base Fine-tuned Base Fine-tuned† Criterion P (%) R (%) P (%) R (%) P (%) R (%) P (%) R (%) P (%) R (%) P (%) R (%) Golden L1: concept 82.34 76.35 87.69 89.66 6.12 33.69 91.26 93.75 0.20 4.53 91.68 94.36 L2: + span 73.20 67.87 79.59 81.38 3.72 20.46 85.82 88.16 0.02 0.47 86.88 89.41 L3: + correctness 67.81 62.87 74.37 76.04 3.01 16.56 81.28 83.49 0.01 0.19 82.74 85.16 Golden2 L1: concept 84.89 74.28 89.65 86.55 9.04 31.31 92.53 90.30 0.20 4.10 93.48 91.59 L2: + span 73.17 64.02 78.98 76.25 4.35 15.07 86.09 84.02 0.02 0.47 87.04 85.28 L3: + correctness 67.04 58.66 74.97 72.39 3.10 10.73 80.84 78.89 0.01 0.21 82.55 80.88 Table 2: Micro precision and recall (%) on Golden and Golden2. GPT-5.4 is the production prompted model; GPT-5.6 Sol is a quality-only comparator. Bold marks the best value in each row. Base denotes untuned Qwen3.5 models; fine-tuned denotes selected full-pipeline checkpoints (raw-supervision controls in Table 3). The two fine-tuned models use independently tuned recipes; the 0.8B checkpoint, marked †, combines a compact prompt with separately tuned hyperparameters, so differences between sizes are not a controlled comparison. Bootstrap 95% confidence intervals appear in Appendix Table 7; key-macro results in Appendix Table 8. All fine-tuned Qwen–GPT-5.4 differences are significant under paired randomization tests (p<0.0001p<0.0001). Annotation evaluation. Each system predicts a set A^(ui,ci) A(u_i,c_i) over the 34-concept inventory, scored against gold A⁡(ui,ci)A(u_i,c_i) from Section 2. We benchmark against two reference classes: frontier models establish operational parity (production GPT-5.4) and a strong quality comparator (GPT-5.6 Sol), while base-model comparisons isolate task-adaptation gains against untuned counterparts. Unparseable outputs are recorded as format-adherence failures (parse errors) and treated as empty predictions, measuring syntactic reliability under autoregressive generation. We score the nested criteria of Section 2 at L1 (kjk_j), L2 ((kj,sj)(k_j,s_j)), and L3 ((kj,sj,yj)(k_j,s_j,y_j)), reporting micro precision and recall across utterances (key-macro averages in Appendix C.2). Test-set uncertainty is estimated via percentile 95% confidence intervals from 20,000 paired nonparametric cluster bootstrap resamples of packed five-utterance inference batches, preserving output dependencies. Differences from GPT-5.4 are assessed via two-sided paired randomization tests with 20,000 permutations (Koehn, 2004; Dror et al., 2018) (Appendix C.4). Explanation evaluation. Because surface n-gram metrics correlate poorly with pedagogical feedback quality, we adopt an LLM-as-a-judge framework (Zheng et al., 2023; Kobayashi et al., 2024): GPT-5.5 evaluates explanation acceptability given the utterance, span, reference correction, and concept definition (rubric in Appendix E.3). To counter potential model leniency, a human reviewer audits the judgments, and an independent second reviewer reconciles disagreements. Pipeline ablation. To isolate pipeline contributions, we trained raw-supervision controls using unreviewed GPT-5.4 annotations with identical prompts, splits, hyperparameters, and matched optimizer-update budgets. Because model sizes use independently tuned recipes, each control is compared only against its same-size counterpart. To evaluate how well the pipeline addresses specific distribution skews, we report equal-weight L3 recall across all 34 concepts (Mean-34; curricular balance), micro L3 recall restricted to gold incorrect uses (Incorrect; error sensitivity), and equal-weight L3 recall on Rare-9 (the nine least frequent concepts in raw training data, validating synthetic tail-rebalancing; Appendix C.3). 6 Results Annotation accuracy and specialization. Table 2 reports precision and recall on both benchmarks. Both fine-tuned models outperform GPT-5.6 Sol across all criteria, with the largest uplift under strict L3 matching (concept, exact span, and correctness). Base-model controls and format adherence. Untuned 4B and 0.8B models fail to parse on 26.9–32.8% and 75.3–92.0% of examples, respectively, demonstrating a wide adaptation gap. An earlier-generation Qwen3-0.6B checkpoint trained with the 0.8B recipe produced unparseable output on 57.4% of Golden2 versus 0.03% for 0.8B despite smooth validation convergence (generational differences prevent attributing this gap solely to parameter count; Appendix A). Model Training data L3 P L3 R Mean-34 L3 R Rare-9 L3 R Incorrect L3 R Golden2 Qwen3.5-4B Raw supervision 52.47 41.98 34.83 22.04 29.35 Full pipeline 80.84 78.89 69.44 67.35 67.56 Qwen3.5-0.8B Raw supervision 52.62 40.93 35.25 21.05 29.68 Full pipeline 82.55 80.88 72.64 66.86 69.37 Golden Qwen3.5-4B Raw supervision 57.34 46.16 44.70 38.67 22.64 Full pipeline 81.28 83.49 75.99 70.96 52.70 Qwen3.5-0.8B Raw supervision 56.59 44.50 44.21 40.55 20.61 Full pipeline 82.74 85.16 76.89 71.97 55.57 Table 3: Pipeline ablation (%) on Golden2 and Golden. Bold marks the best value in each column within each benchmark. Mean-34 is equal-weight recall over all concepts; Rare-9 is equal-weight recall over the nine concepts fixed from the raw training split; Incorrect restricts recall to gold incorrect-use annotations. Full-pipeline rows are the selected checkpoints in Table 2. Data-pipeline ablation. Matched within-size controls isolate the combined supervision pipeline (Table 3). On Golden2 it raises L3 precision by 28.37 (4B) and 29.93 (0.8B) points and L3 recall by 36.91 (95% CI [35.93, 37.87]) and 39.95 ([39.01, 40.89]) points; on Golden, L3 recall rises by 37.33 ([35.15, 39.48]) and 40.66 ([38.44, 42.81]) points. Gains extend across all subsets, notably Rare-9 (up 45.31 and 45.81 points on Golden2). On Golden2, raw-supervision recall drops by 23.64–23.78 points from L1 to exact-span L2, compared to 6.28–6.31 points for the full pipeline, underscoring an acute exact-span degradation under raw supervision. Explanation quality. On Golden2, GPT-5.5 judged 99.06% of GPT-5.4 explanations acceptable versus 94.91% (4B) and 93.88% (0.8B). However, following human audit and reconciliation, all three systems achieved similar acceptable rates (≈ 95%) with no significant differences. This disparity indicates that the uncalibrated LLM judge favored its frontier sibling, underscoring the necessity of human calibration. Online product impact. We deployed the complete system with Qwen3.5-0.8B (Section 7) in a 35-day feature-level A/B experiment randomized at the learner level with 50% exposure. At the fixed pre-scaling snapshot (Table 4), learner engagement with progress and all three business metrics improved significantly. These gains apply to the end-to-end product feature rather than the SLM in isolation. Metric Effect p Engagement with their progress +15.80% <0.001<0.001 Scheduled hours (CUPED) +2.06% 0.0210.021 Confirmed hours from new tutoring +10.87% 0.0200.020 GMV from new lessons +13.20% 0.0390.039 Table 4: Relative effects in the online experiment at the fixed pre-scaling evaluation snapshot (all corresponding 95% CIs exclude zero; bold marks p<0.05p<0.05). 7 Deployment and Operational Considerations Our production serving architecture prioritizes high concurrent throughput and cost-efficient horizontal scaling. We deploy Qwen3.5-0.8B with vLLM (Kwon et al., 2023) on autoscaled, single-GPU g6.2xlarge SageMaker instances that allow fine-grained capacity scaling. Before inference, the application packs prepared utterance–correction pairs into five-pair requests to an internal inference service. Requests are sequential within each transcript chunk, and vLLM batches concurrent requests arriving at each replica. The service applies the compact prompt (Appendix E.2), parses the structured output, and returns annotations aligned to input indices. Validation or service failures route work to GPT-5.4 (topology in Appendix Figure 3; autoscaling and load testing in Appendix D.1). The warmed load test exercised this packed-request path: 8 g6.2xlarge instances sustained 4,993.5 pairs per minute, exceeding the observed post-rollout peak. During a 31-day production APM snapshot, p95 latency was 4.3 seconds and the request error rate was 0.0015%. Combining measured GPT-5.4 production cost, observed demand, load-tested SLM capacity, and public SageMaker prices, recurring detection cost per lesson is 6.2% of the GPT-5.4 baseline—an approximately 16× reduction (Appendix D.2). 8 Related Work Task formulation. Grammatical error detection and correction localize or repair deviations (Rei and Yannakoudakis, 2016; Bryant et al., 2023), label edits, or generate learner feedback (Bryant et al., 2017; Nagata, 2019; Liang et al., 2023; Song et al., 2024). Curriculum-grounded methods often target predefined activities or dialogue contrasts (Quixal et al., 2021; Akef et al., 2026; Ribeiro-Flucht et al., 2026). In contrast, we identify concepts from a 34-concept inventory in unconstrained transcribed speech turns, jointly predicting evidence spans, correctness, and explanations. Teacher-generated and synthetic supervision. Prior work generates synthetic GEC data via tagged corruption or LLM prompting and adapts distributions via type-aware augmentation or reweighting (Stahlberg and Kumar, 2021; Li and Lan, 2025; Vainikko et al., 2025; Zhang et al., 2025). Our pipeline instead filters production annotations and samples concept–correctness backbones to rebalance the curriculum, benchmarked against matched raw-supervision controls. SLM specialization and deployment. Prior work internalizes prompts (Zou et al., 2024; Shin et al., 2025), distills teacher data into lightweight models (Wang et al., 2025), or supplies LLM explanations to SLM GEC systems (Li et al., 2025). We combine prompt internalization, a compact schema prompt, and completion-only LoRA in a deployed sub-1B SLM, jointly evaluating accuracy, serving cost, and online impact. 9 Conclusion We presented a deployed grammar tracker jointly predicting curricular concepts, exact evidence spans, correctness, and explanations for errors. Fine-tuned Qwen3.5 models outperformed GPT-5.4 and GPT-5.6 Sol across all matching levels on two human-curated benchmarks. Ablations confirmed our data pipeline yields 36.9–40.7 points of strict L3 recall over raw teacher supervision. The deployed 0.8B model cut serving cost by ≈ 16×, while an online experiment demonstrated significant engagement and business gains, showing that data quality, prompt internalization, and efficient serving make accurate feedback viable at scale. Limitations Modeling and training controls. Our pipeline ablation benchmarks raw supervision against the combined pipeline (review, deterministic validation, filtering, and synthetic enrichment), but does not isolate the individual contribution of each component. Furthermore, we report results from a single training run per configuration and do not estimate across-seed variance, meaning fine-grained metric differences should not be over-interpreted. Finally, because the 4B and 0.8B models rely on independently tuned training recipes and prompt lengths, and the 0.6B comparison belongs to an earlier Qwen generation, differences across model sizes reflect distinct operational configurations rather than an isolated parameter ablation. Operational and evaluation scope. The online A/B experiment evaluates the end-to-end product feature rather than the SLM or serving infrastructure in isolation. Similarly, our serving cost model isolates recurring grammar-detection compute and excludes upstream transcription, correction, and fallback calls to frontier APIs. Regarding explanation evaluation, while human audit reconciled acceptability rates, the leniency gap in the uncalibrated LLM judge demonstrates the need for domain-specific judge calibration; moreover, explanations were evaluated against gold rubrics rather than through direct perceptual studies with teachers or learners. Data scope, speech, and supervision biases. While our deployed checkpoints target an English-language curriculum of 34 concepts, the underlying pipeline—curated supervision, prompt internalization, and lightweight serving—is framework-agnostic and designed to scale to a substantially broader curriculum. Because learner turns are transcribed via automatic speech recognition (ASR), transcription errors can cascade into upstream correction and downstream concept tracking; we do not isolate robustness to ASR noise. Finally, because both reviewed production supervision and synthetic sentences derive from GPT-5.4, systematic teacher biases may persist in ways our benchmarks cannot fully reveal. Ethical Considerations We analyze lesson transcripts only from sessions for which we have a lawful basis to do so. Learner utterances, annotations, and internal prompts are not released; the illustrative example in Figure 2 is synthetic. The paper reports aggregate evaluation and online-experiment results only. References Akef et al. (2026) S. Akef, A. Mendes, P. Rebuschat, and D. Meurers LLM-powered but rule-grounded: pedagogically relevant grammatical error characterization for learner model construction. In Proceedings of the 21st Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2026), San Diego, California, USA, p. 851–862. External Links: Document, Link Cited by: §8. Amazon Web Services (2026) Amazon Web Services AWS price list for amazon SageMaker. Note: Published 4 September 2026; accessed 7 September 2026 External Links: Link Cited by: §D.2. Bryant et al. (2017) C. Bryant, M. Felice, and T. Briscoe Automatic annotation and evaluation of error types for grammatical error correction. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Vancouver, Canada, p. 793–805. External Links: Document, Link Cited by: §8. Bryant et al. (2023) C. Bryant, Z. Yuan, M. R. Qorib, H. Cao, H. T. Ng, and T. Briscoe Grammatical error correction: a survey of the state of the art. Computational Linguistics 49 (3), p. 643–701. External Links: Document, Link Cited by: §8. Challapally et al. (2025) A. Challapally, C. Pease, R. Raskar, and P. Chari The GenAI divide: state of AI in business 2025. Research report MIT Project NANDA. External Links: Link Cited by: §1. Chen et al. (2016) T. Chen, B. Xu, C. Zhang, and C. Guestrin Training deep nets with sublinear memory cost. arXiv preprint arXiv:1604.06174. External Links: Document, Link Cited by: Appendix A. Council of Europe (2020) Council of Europe Common european framework of reference for languages: learning, teaching, assessment—companion volume. Council of Europe Publishing, Strasbourg. External Links: ISBN 978-92-871-8621-8, Link Cited by: §1. Dror et al. (2018) R. Dror, G. Baumer, S. Shlomov, and R. Reichart The hitchhiker’s guide to testing statistical significance in natural language processing. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Melbourne, Australia, p. 1383–1392. External Links: Document, Link Cited by: §5. Honnibal et al. (2020) M. Honnibal, I. Montani, S. Van Landeghem, and A. Boyd spaCy: industrial-strength natural language processing in python. External Links: Document, Link Cited by: §3.1. Hu et al. (2022) E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen LoRA: low-rank adaptation of large language models. In International Conference on Learning Representations, External Links: Link Cited by: §4. Kobayashi et al. (2024) M. Kobayashi, M. Mita, and M. Komachi Large language models are state-of-the-art evaluator for grammatical error correction. In Proceedings of the 19th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2024), Mexico City, Mexico, p. 68–77. External Links: Link Cited by: §5. Koehn (2004) P. Koehn Statistical significance tests for machine translation evaluation. In Proceedings of the 2004 Conference on Empirical Methods in Natural Language Processing, Barcelona, Spain, p. 388–395. External Links: Link Cited by: §5. Kwon et al. (2023) W. Kwon, Z. Li, S. Zhuang, Y. Sheng, L. Zheng, C. H. Yu, J. E. Gonzalez, H. Zhang, and I. Stoica Efficient memory management for large language model serving with PagedAttention. In Proceedings of the 29th Symposium on Operating Systems Principles, p. 611–626. External Links: Document, Link Cited by: §4, §7. Li (2010) S. Li The effectiveness of corrective feedback in SLA: a meta-analysis. Language Learning 60 (2), p. 309–365. External Links: Document, Link Cited by: §1. Li and Lan (2025) X. Li and Y. Lan Large language models are good annotators for type-aware data augmentation in grammatical error correction. In Proceedings of the 31st International Conference on Computational Linguistics, Abu Dhabi, UAE, p. 199–213. External Links: Link Cited by: §8. Li et al. (2025) Y. Li, S. Qin, J. Ye, H. Huang, Y. Li, S. Guo, L. Qin, X. Hu, W. Jiang, H. Zheng, and P. S. Yu Rethinking the roles of large language models in Chinese grammatical error correction. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 6: Industry Track), Vienna, Austria, p. 553–567. External Links: Document, Link Cited by: §8. Liang et al. (2023) K. Liang, S. Davidson, X. Yuan, S. Panditharatne, C. Chen, R. Shea, D. Pham, Y. Tan, E. Voss, L. Fryer, and Z. Yu ChatBack: investigating methods of providing grammatical error feedback in a GUI-based language learning chatbot. In Proceedings of the 18th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2023), Toronto, Canada, p. 83–99. External Links: Document, Link Cited by: §8. Nagata (2019) R. Nagata Toward a task of feedback comment generation for writing learning. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), Hong Kong, China, p. 3206–3215. External Links: Document, Link Cited by: §8. Quixal et al. (2021) M. Quixal, B. Rudzewitz, E. Bear, and D. Meurers Automatic annotation of curricular language targets to enrich activity models and support both pedagogy and adaptive systems. In Proceedings of the 10th Workshop on NLP for Computer Assisted Language Learning, Online, p. 15–27. External Links: Link Cited by: §8. Qwen Team (2026) Qwen Team Qwen3.5: towards native multimodal agents. External Links: Link Cited by: §4. Rajbhandari et al. (2020) S. Rajbhandari, J. Rasley, O. Ruwase, and Y. He ZeRO: memory optimizations toward training trillion parameter models. In SC20: International Conference for High Performance Computing, Networking, Storage and Analysis, p. 1–16. External Links: Document, Link Cited by: Appendix A. Rei and Yannakoudakis (2016) M. Rei and H. Yannakoudakis Compositional sequence labeling models for error detection in learner writing. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Berlin, Germany, p. 1181–1191. External Links: Document, Link Cited by: §8. Ribeiro-Flucht et al. (2026) L. Ribeiro-Flucht, L. Huang, and X. Chen From dialogue to learner modeling: identifying candidate signals of productive use in LLM-based grammar practice. In Proceedings of the 21st Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2026), San Diego, California, USA, p. 933–940. External Links: Document, Link Cited by: §8. Settles and Meeder (2016) B. Settles and B. Meeder A trainable spaced repetition model for language learning. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Berlin, Germany, p. 1848–1858. External Links: Document, Link Cited by: §1. Shin et al. (2025) H. Shin, L. Ji, Y. Gong, S. Kim, E. Choi, and M. Seo Generative prompt internalization. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), Albuquerque, New Mexico, p. 7338–7363. External Links: Document, Link Cited by: §4, §8. Song et al. (2024) Y. Song, K. Krishna, R. Bhatt, K. Gimpel, and M. Iyyer GEE! grammar error explanation with large language models. In Findings of the Association for Computational Linguistics: NAACL 2024, Mexico City, Mexico, p. 754–781. External Links: Document, Link Cited by: §8. Stahlberg and Kumar (2021) F. Stahlberg and S. Kumar Synthetic data generation for grammatical error correction with tagged corruption models. In Proceedings of the 16th Workshop on Innovative Use of NLP for Building Educational Applications, Online, p. 37–47. External Links: Link Cited by: §8. Vainikko et al. (2025) M. Vainikko, T. Kamarik, K. Kert, K. Liin, S. Maine, K. Allkivi, A. Kaivapalu, and M. Fishel Paragraph-level error correction and explanation generation: case study for Estonian. In Proceedings of the 20th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2025), Vienna, Austria, p. 953–967. External Links: Document, Link Cited by: §8. Wang et al. (2025) C. Wang, J. Yan, Y. Yue, and J. Huang DistilQwen2.5: industrial practices of training distilled open lightweight language models. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 6: Industry Track), Vienna, Austria, p. 32–42. External Links: Document, Link Cited by: §8. Yang et al. (2025) A. Yang, A. Li, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Gao, C. Huang, C. Lv, C. Zheng, D. Liu, F. Zhou, F. Huang, F. Hu, H. Ge, H. Wei, H. Lin, J. Tang, J. Yang, J. Tu, J. Zhang, J. Yang, J. Yang, J. Zhou, J. Zhou, J. Lin, K. Dang, K. Bao, K. Yang, L. Yu, L. Deng, M. Li, M. Xue, M. Li, P. Zhang, P. Wang, Q. Zhu, R. Men, R. Gao, S. Liu, S. Luo, T. Li, T. Tang, W. Yin, X. Ren, X. Wang, X. Zhang, X. Ren, Y. Fan, Y. Su, Y. Zhang, Y. Zhang, Y. Wan, Y. Liu, Z. Wang, Z. Cui, Z. Zhang, Z. Zhou, and Z. Qiu Qwen3 technical report. arXiv preprint arXiv:2505.09388. External Links: Document, Link Cited by: §4. Zhang et al. (2025) Y. Zhang, Z. Xu, S. Wu, Y. Zhang, and D. Ramage Synthesizing and adapting error correction data for mobile large language model applications. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 6: Industry Track), Vienna, Austria, p. 1102–1112. External Links: Document, Link Cited by: §8. Zheng et al. (2023) L. Zheng, W. Chiang, Y. Sheng, S. Zhuang, Z. Wu, Y. Zhuang, Z. Lin, Z. Li, D. Li, E. P. Xing, H. Zhang, J. E. Gonzalez, and I. Stoica Judging LLM-as-a-judge with MT-Bench and chatbot arena. In Advances in Neural Information Processing Systems, Vol. 36, p. 46595–46623. External Links: Link Cited by: §5. Zou et al. (2024) J. Zou, M. Zhou, T. Li, S. Han, and D. Zhang PromptIntern: saving inference costs by internalizing recurrent prompt during large language model fine-tuning. In Findings of the Association for Computational Linguistics: EMNLP 2024, p. 10288–10305. External Links: Document, Link Cited by: §4, §8. Appendix A Training Configurations and Convergence All runs used bfloat16, gradient checkpointing (Chen et al., 2016), DeepSpeed ZeRO-2 (Rajbhandari et al., 2020), completion-only loss, and long-sample filtering. Validation was performed four times per epoch with early-stopping patience of three evaluations. Table 5 records the selected run configurations, and Table 6 reports training convergence and structured-output reliability for the three fine-tuned models. Setting Qwen3.5 4B Qwen3.5 0.8B Qwen3 0.6B Prompt design Detailed Compact Compact Max. sequence length 7,000 2,500 2,500 LoRA r/αr/α 16 / 32 128 / 64 128 / 64 LoRA dropout — 0.05 0.05 LoRA targets — All linear All linear Learning rate 2×10−52× 10^-5 1×10−41× 10^-4 1×10−41× 10^-4 LR schedule — Cosine Cosine Warm-up — 3% 3% Batch / device 1 1 1 Grad. accumulation 16 32 32 Effective batch 16 256 256 Selected epoch 15.00 7.50 8.70 Table 5: Recorded fine-tuning configurations for Qwen3.5-4B, Qwen3.5-0.8B, and Qwen3-0.6B. Effective batch size (256 for the compact runs) is the product of per-device batch size (1), gradient accumulation steps (32), and 8 distributed GPU workers (1×32×8=2561× 32× 8=256); the 4B run used 1 worker with accumulation 16 (1×16×1=161× 16× 1=16). Dashes indicate settings not retained in the available run record. Model Val. loss Epoch Parse errors Qwen3.5-4B† 0.2201 15.00 0 / 2,977 (0.00%) Qwen3.5-0.8B 0.2245 7.50 1 / 2,977 (0.03%) Qwen3-0.6B 0.2885 8.70 1,710 / 2,977 (57.44%) Table 6: Minimum teacher-forced loss on each run’s training-validation split and structured-output parse errors on Golden2. Bold highlights format adherence (≤0.03%≤ 0.03\% parse errors) and the lowest validation loss within each distinct recipe (0.2201 for detailed 4B; 0.2245 for compact 0.8B). The 0.8B and 0.6B runs shared prompt and optimization hyperparameters but selected different early-stopping checkpoints. † The 4B checkpoint used a different prompt, training target, and schedule; its loss is descriptive rather than a controlled cross-model comparison. The matched-recipe comparison between the 0.8B and 0.6B runs reveals a failure mode not visible in the loss curve. Although the 0.8B and 0.6B runs shared prompt and optimization hyperparameters, the 0.6B model converged smoothly to a minimum validation loss 28.5% higher, and 57.4% of its evaluation outputs failed schema validation (parse errors), compared with 0.03% for the 0.8B model: token-level optimization did not guarantee reliable autoregressive generation of the required structure. Because the models also belong to different Qwen generations, this comparison does not isolate parameter count as the sole cause. Appendix B Data Pipeline Details Error-prioritized sampling. The heuristic used to select the second 50k production groups flags annotation density d (annotations per non-noun word) or annotated-span coverage c outside fixed bounds (d≤0.75d≤ 0.75 or d≥1.25d≥ 1.25; c≤0.75c≤ 0.75). Flagged utterances are prioritized by (|d−1|+max⁡(0,1−c))×N+5(|d-1|+ (0,1-c))× NN+5, where N is utterance length in words, dampening extreme ratios on very short utterances. Deterministic evidence and validation. Beyond part-of-speech tags and rule-based construction detection, coherence checks and high-confidence rules add warnings and candidate concepts to the review checklist. In addition to the checks listed in Section 3.1, the validator resolves disfluent repetitions omitted by the correction before checking span containment, and falls back to the pilot annotations if all proposed tuples are rejected. Review prompt contract. Review calls query GPT-5.4 at temperature 0 with medium reasoning effort (one utterance per request). The prompt contract is executed in the standard chat JSON format below, specifying system instructions, user input context, and the expected assistant audit response; proprietary extraction notes are omitted. Data Cleaning Review Prompt Contract (Chat JSON) [ "role": "system", "content": "You are a professional linguist reviewing existing grammar annotations. For each preliminary row: KEEP, FIX, REMOVE, or ADD. - Span & Intent: Every text_span must be an exact substring. When a corrected line is present, label by intended target grammar, not surface form in isolation. - Correctness & Explanation: correct is true iff the concept is used properly. Non-empty error_explanation requires correct=false (REPETITION is the sole exception: false with empty explanation). - Output Format: Return a JSON object mapping utterance to [[text_span, concept_label, correct, error_explanation], ...]." , "role": "user", "content": "Utterance: She don’t like swimming in the pool. Error hints: don’t -> doesn’t Linguistic evidence: Pronouns: She; Prepositions: in; Articles: the; Verbs: don’t, like, swimming Prior analysis: [["don’t", "PRESENT_SIMPLE", true, ""], ["swimming", "PRESENT_CONTINUOUS", true, ""], ["the pool", "ARTICLES", true, ""]]" , "role": "assistant", "content": " "She don’t like swimming in the pool.": [ ["She", "PRONOUNS", true, ""], ["don’t like", "PRESENT_SIMPLE", false, "Use doesn’t with third-person She."], ["swimming", "VERB_PATTERNS", true, ""], ["in the pool", "PREPOSITIONS", true, ""], ["the pool", "ARTICLES", true, ""] ] " ] Synthetic backbone generation. Seed backbones are harvested from production records by stripping lexical tokens from annotated utterances, after discarding degenerate sequences (e.g., word repetitions or comma-heavy enumerations). The 90k synthetic corpus is produced across three 30k strata: (i) uniform sampling from production backbones; (i) error-weighted sampling, where templates with k natural error slots are sampled with weight proportional to c×(2+k)0.75c×(2+k)^0.75 (c is template frequency); and (i) inverse-frequency sampling, where anchor concepts are drawn with probability P⁡(gi)∝1/(ni+1000)1.0P(g_i) 1/(n_i+1000)^1.0 (nin_i is concept span count). Pedagogical error injection and synthesis. Each originally correct slot is mutated with probability p=0.05p=0.05 into one of three error kinds: (a) same-label misuse (e.g., wrong verb inflection); (b) omission (dropping an obligatory article or preposition); or (c) confusion swap, where the slot is temporarily projected to a confusable substitute label (e.g., present continuous instead of present perfect continuous) and paired with discourse cues (e.g., duration adverbials) that make the substitute form erroneous in context. GPT-5.4 (medium reasoning effort, 5 sentence variations per call) receives concept definitions for the projected sequence, numbered correct/not_correct slot lines with inline [Error: ...] instructions, and 1–3 retrieved production exemplars in the required JSON shape. Confusion-swap slots are shown to the model as the substitute label marked correct; post-processing restores intent labels, coerces incorrect spans to the planned intent, drops sentences with out-of-scope labels, and enforces substring validity. The sanitized chat exchange is structured under the standard message contract below. Synthetic Backbone Realization Contract (Chat JSON) [ "role": "system", "content": "You are an expert in SLA and synthetic data generation. Generate num_sentences distinct learner-style variations from the example structure. - Slots: Keep each correct slot fully grammatical; for each not_correct slot, follow the inline [Error: ...] instruction and keep the named intent label with correct=false. - Coverage: Annotate every grammar block in the sentence, including naturally occurring ones not in the sequence (e.g., pronouns, articles). - Output Format: Return JSON mapping each generated sentence to "corrected": ..., "annotations": [...]." , "role": "user", "content": "Definitions: PRESENT_SIMPLE: main verb in present simple; VERB_PATTERNS: -ing complement only. Sequence & labels: 1. PRESENT_SIMPLE: not_correct [Error: same-label misuse] Subject-verb agreement (e.g., "She don’t like ..."). 2. VERB_PATTERNS: correct Exemplar (reference): "She doesn’t like swimming in the park.": "corrected": null, "annotations": [["She","PRONOUNS",true,""], ["doesn’t like","PRESENT_SIMPLE",true,""], ["swimming","VERB_PATTERNS",true,""], ["in the park","PREPOSITIONS",true,""], ["the park","ARTICLES",true,""]] " , "role": "assistant", "content": " "He don’t enjoy running at the gym.": "corrected": "He doesn’t enjoy running at the gym.", "annotations": [ ["He", "PRONOUNS", true, ""], ["don’t enjoy", "PRESENT_SIMPLE", false, "Use doesn’t with third-person He."], ["running", "VERB_PATTERNS", true, ""], ["at the gym", "PREPOSITIONS", true, ""], ["the gym", "ARTICLES", true, ""] ] " ] Appendix C Extended Experimental Results and Statistical Methodology C.1 Confidence Intervals for Annotation Results Table 7 provides the complete percentile 95% confidence intervals corresponding to the primary evaluation in Table 2, derived from 20,000 paired cluster bootstrap resamples. Model L1 P L1 R L2 P L2 R L3 P L3 R Golden GPT-5.4 [80.28,84.19] [74.36,78.29] [70.99,75.22] [65.69,70.00] [65.62,69.87] [60.68,65.07] GPT-5.6 Sol [86.58,88.76] [88.29,90.94] [78.31,80.87] [79.88,82.83] [72.74,75.98] [74.28,77.81] Qwen3.5-4B Base [4.47,8.73] [30.67,36.71] [2.68,5.37] [18.19,22.76] [2.15,4.38] [14.44,18.74] Qwen3.5-4B FT [90.22,92.28] [92.81,94.66] [84.51,87.12] [86.95,89.34] [79.86,82.71] [82.06,84.93] Qwen3.5-0.8B Base [0.17,0.24] [3.86,5.22] [0.01,0.03] [0.25,0.74] [0.00,0.02] [0.06,0.37] Qwen3.5-0.8B FT [90.63,92.69] [93.45,95.23] [85.72,88.01] [88.28,90.53] [81.44,84.03] [83.85,86.48] Golden2 GPT-5.4 [84.26,85.52] [73.52,75.03] [72.35,73.99] [63.17,64.88] [66.18,67.91] [57.79,59.54] GPT-5.6 Sol [89.15,90.14] [86.02,87.09] [78.29,79.66] [75.57,76.95] [74.24,75.72] [71.67,73.11] Qwen3.5-4B Base [7.88,10.45] [30.06,32.56] [3.77,5.06] [14.24,15.93] [2.67,3.61] [10.04,11.44] Qwen3.5-4B FT [92.11,92.95] [89.87,90.73] [85.54,86.64] [83.46,84.57] [80.16,81.51] [78.21,79.57] Qwen3.5-0.8B Base [0.18,0.22] [3.81,4.40] [0.02,0.03] [0.36,0.59] [0.01,0.01] [0.14,0.29] Qwen3.5-0.8B FT [93.09,93.85] [91.16,92.00] [86.51,87.57] [84.71,85.85] [81.91,83.20] [80.21,81.55] Table 7: Percentile 95% confidence intervals for micro precision and recall in Table 2, based on 20,000 paired nonparametric cluster bootstrap resamples of packed five-utterance inference batches. Surplus generated utterances remain assigned to their originating batch. Bold marks the best interval in each column within each benchmark. Base and FT denote untuned and fine-tuned models, respectively. All values are percentages; all GPT-5.4 and GPT-5.6 Sol outputs parsed successfully. C.2 Key-Macro Annotation Results Table 8 presents key-macro-averaged precision and recall, assigning equal weight to each distinct target key at each evaluation level (target concepts at L1, concept–span pairs at L2, and concept–span–correctness triples at L3), complementing the class-level Mean-34 recall metric in Table 3. Macro recall and precision are computed as the unweighted mean of per-key scores across all distinct target and predicted keys, respectively. Criterion GPT-5.4 P / R (%) Qwen3.5-4B FT P / R (%) Qwen3.5-0.8B FT P / R (%) Golden (selected checkpoints) Macro L1: concept 74.08 / 75.34 84.21 / 91.43 85.60 / 89.82 Macro L2: + span 52.44 / 51.65 72.62 / 72.93 75.47 / 75.70 Macro L3: + correctness 45.77 / 45.08 64.48 / 64.82 67.84 / 68.07 Golden2 (selected checkpoints) Macro L1: concept 72.26 / 68.31 84.41 / 84.85 86.29 / 87.65 Macro L2: + span 47.64 / 46.13 67.57 / 67.16 68.90 / 68.68 Macro L3: + correctness 41.65 / 40.26 59.91 / 59.37 62.06 / 61.79 Table 8: Key-macro precision and recall (%) assigning equal weight to each distinct target matching key at each level, corresponding to the micro results in Table 2. Bold marks the best value for each metric in each row. C.3 Pre-Fixed Rare-Concept Subset We selected the nine concepts with the fewest annotations in the raw 83,815-group training split after restricting labels to the 34-concept inventory. Selection used no benchmark outcomes; there was no tie at the cutoff. Table 9 reports the selection counts and gold support on the primary Golden2 evaluation benchmark (Golden serves as an additional robustness check). Concept Raw train Golden2 Present perfect continuous 187 5 Causative / permissive 259 12 Past perfect 334 9 Zero conditional 335 13 Second conditional 490 17 Reported speech 574 19 Possessive ’s / whose 825 46 One / ones 1,008 20 Past continuous 1,151 30 Total 5,163 171 Table 9: Pre-fixed Rare-9 concepts and annotation support. Raw-train counts determined subset membership; Golden2 counts are reported only as evaluation support. C.4 Bootstrap Resampling and Statistical Significance Bootstrap resampling operates on packed five-utterance inference batches, so surplus generated utterances remain assigned to the batch that produced them and are scored as false positives. For paired ablation intervals, metric differences are computed within each aligned cluster resample, taking the 2.5 and 97.5 percentiles across 20,000 draws. For fixed-set Mean-34 and Rare-9 intervals, bootstrap draws without gold support for every included concept were redrawn conditionally (Golden2: 138 draws for 4B and 153 for 0.8B; Golden: 769 draws for each size). For pairwise comparisons against GPT-5.4, two-sided paired randomization tests permute model predictions across systems at the five-utterance batch cluster level using B=20,000B=20,000 Monte Carlo permutations. Empirical p-values are computed as p=(b+1)/(B+1)p=(b+1)/(B+1), where b is the number of permuted absolute test statistic differences exceeding or equaling the observed difference; tests yielding zero exceedances (b=0b=0) are reported as p<0.0001p<0.0001 (p=1/20,001p=1/20,001). Appendix D Serving Capacity, Autoscaling, and Cost Figure 3 provides the detailed topology omitted from the page-limited main paper. SageMaker distributes invocations across replicas; within each replica, vLLM batches concurrent sequences using chunked prefill, prefix caching, and an FP8 KV cache. Figure 3: Simplified production g6.2xlarge native-vLLM topology. The application backend and internal inference API are collapsed into the inference-client block. SageMaker distributes calls across independently provisioned one-GPU containers. Workload and measured capacity. Before inference, the application partitions prepared utterance–correction pairs into five-pair requests (averaging 4.27 pairs in production). To determine serving capacity, concurrent clients replayed five-pair requests through a warmed inference service. Eight g6.2xlarge instances sustained 4,993.5 pairs per minute, which equals 4,993.5/(8×5×60)=2.084,993.5/(8× 5× 60)=2.08 five-pair requests per second (RPS) per instance. Using five pairs per request is conservative relative to the production mean. D.1 Autoscaling Policy and Load Test The production endpoint uses AWS Application Auto Scaling targeting four concurrent in-flight requests per instance, with 10-minute scale-out and 20-minute scale-in cooldowns across a range of one to eight instances. We validated this policy by deploying the model to a staging endpoint and replaying a two-hour staged load test ramped from 1 to 6.4 RPS before declining (Table 10). Phase RPS Requests Errors p50 (s) p99 (s) Instances p1 0.9 1,070 0 4.43 6.87 0→10→ 1 p2 1.6 1,955 0 4.71 7.66 1→21→ 2 p3 3.2 3,817 2 4.53 20.07 2→52→ 5 p4 6.4 7,738 0 4.90 7.64 5→55→ 5 p5 1.9 2,246 0 4.23 6.46 5→55→ 5 p6 1.0 1,168 0 4.06 6.25 5→45→ 4 Table 10: Predeployment autoscaling test. “Instances” gives the actual transition during each traffic phase; the initial zero denotes endpoint startup before the configured one-instance minimum became active. Latency is measured per request. Across the test, the endpoint processed 17,994 requests with an overall error rate of 0.011% (two transient errors during the initial scale-out from two to five instances). At the 6.4 RPS peak, five instances sustained traffic with zero errors and a 7.64-second p99 latency; the resulting 1.281.28 RPS per instance (61.5%61.5\% of measured saturation capacity) operated safely within target concurrency thresholds. As load dropped, the scale-in cooldown retained capacity before returning to the single-instance baseline. D.2 Cost Projection To project monthly serving expenditure under live traffic patterns, we analyzed 26 days of hourly production request volume following full rollout. For each production hour, required capacity was computed as observed request rate divided by planning capacity (2.08×0.70=1.462.08× 0.70=1.46 RPS, derating measured saturation capacity to provide a 30% headroom margin, consistent with the stable peak rate observed in Table 10), rounded up, and clamped to the deployed 1–8 instance range. Average provisioned capacity was multiplied by 730 monthly hours and public on-demand SageMaker real-time pricing for g6.2xlarge instance-hours (Amazon Web Services, 2026). Compared against measured May 2026 production costs for the low-reasoning GPT-5.4 configuration, the resulting SLM serving cost is 6.2% of baseline per lesson—an approximately 16× reduction. This projection isolates recurring detection serving: the GPT-5.4 baseline is measured in production, while the SLM estimate combines observed demand, load-tested throughput, and public cloud pricing. Common upstream filtering and correction, fallback API calls, and one-time training and deployment work are excluded. Absolute spend, per-lesson costs, and eligible-lesson volumes are withheld. Appendix E Task Specifications and Auxiliary Probes E.1 Task Schema and Logical Contract Table 11 states the grammar concept annotation contract of Section 2 field by field. Field Type Definition Original Text Learner utterance extracted from the ASR lesson transcript. Corrected Text Upstream LLM correction preserving the intended meaning. Concept Enum One of 34 target grammar concepts. Span Text Exact contiguous substring of the original utterance. Correct Boolean Whether the concept is used correctly in that span. Explanation Text Corrective feedback when usage is incorrect; empty otherwise. Table 11: Logical input and output schema for grammar concept annotation. The first two fields form the input; the remaining fields describe each output annotation. E.2 Prompt Contracts: Verbose Baseline and Compact Inference We document the operational prompt contracts governing grammar concept annotation in the standard chat JSON format below. The verbose baseline prompt (3,810 Qwen3.5 tokens; excerpted in the first contract box) was executed during the production pilot with GPT-5.4 and served as the comparator prompt for Qwen3.5-4B. It specifies comprehensive category definitions, extraction boundaries, and edge-case rules for all 34 concepts in our curriculum. In contrast, the compact inference prompt (375 Qwen3.5 tokens, a 90.2% prompt reduction; presented in full in the second contract box) was designed for the deployed Qwen3.5-0.8B SLM. Because completion-only LoRA internalizes concept definitions and span boundaries directly into model weights (Section 4), runtime serving requires only the list of allowed concept names, the schema contract, and concise formatting constraints, minimizing prefill latency and KV-cache footprint in vLLM. Verbose Baseline Prompt Contract (Chat JSON) [ "role": "system", "content": "You are a deterministic grammar annotation engine. Given an ORIGINAL sentence (student version) and a CORRECTED sentence (professionally edited), extract, label, and compare all grammar-relevant native text spans using the Grammar Categories Table. # Instructions 1. Parse both ORIGINAL and CORRECTED sentences, identifying grammar-relevant spans. 2. Align each span by grammatical role and order, using syntax and structure. 3. For each alignment, set correct usage to Yes if identical; if not, set to No. 4. Strictly follow Output Format---no extra text, explanation, or commentary. # Grammar Categories Table (34 concepts; representative excerpt) - PRESENT_SIMPLE: Main verb in present simple. Include auxiliary (do/does/don’t/doesn’t) in negatives/questions. Extract copula be (am/is/are) and main verb have. Include contractions (it’s, I’m). - ARTICLES: Article + head noun. Extract article spans independently even inside prepositional phrases (e.g., in ’in the morning’, extract ’in the morning’ as PREPOSITIONS and ’the morning’ as ARTICLES). - PREPOSITIONS: Preposition + object. For phrasal verbs, extract verb/adj + preposition. [... 31 concept specifications and extraction notes omitted ...] # Span Extraction Rules - Exact contiguous substring; minimal span; no character overlap. - Report every detectable grammar feature---both correct and incorrect usage. # Output Format Output one annotation per line separated by |: sentence_index|grammar_concept|correct|text_span|explanation - correct: Y if correct, N if error. - explanation: For N, 6-20 words describing the fix from CORRECTED; leave empty for Y. For REPETITION, explanation must be empty." , "role": "user", "content": "Sentence pairs to analyze: 1|ORIGINAL: She don’t like swimming in the pool. 1|CORRECTED: She doesn’t like swimming in the pool." , "role": "assistant", "content": "1|PRESENT_SIMPLE|N|don’t like|Use doesn’t with third-person She. 1|VERB_PATTERNS|Y|swimming| 1|PREPOSITIONS|Y|in the pool| 1|ARTICLES|Y|the pool|" ] Compact Inference Prompt Contract (Chat JSON) [ "role": "system", "content": "Grammar annotation task. Input: numbered ORIGINAL / CORRECTED English sentence pairs. Output one pipe-delimited annotation per grammar-relevant span: sentence_index|grammar_concept|correct|text_span|explanation Allowed grammar_concept values (34 total): PRESENT_SIMPLE, PRESENT_CONTINUOUS, PAST_SIMPLE, PAST_CONTINUOUS, PRESENT_PERFECT, PRESENT_PERFECT_CONTINUOUS, PAST_PERFECT, FUTURE_SIMPLE, PASSIVE_VOICE, MODAL_ABILITY, MODAL_OBLIGATION, MODAL_REQUEST_SUGGEST, MODAL_PROBABILITY, VERB_PATTERNS, IMPERATIVES, ZERO_CONDITIONAL, FIRST_CONDITIONAL, SECOND_CONDITIONAL, THERE_IS_THERE_ARE, ARTICLES, PREPOSITIONS, ADVERBS, ADJECTIVES_BASIC, COMPARATIVES, SUPERLATIVES, QUANTIFIERS_BASIC, POSSESSIVE_ADJECTIVES, POSSESSIVE_S_WHOSE, DEMONSTRATIVES, SOME_ANY_NO_COMPOUNDS, ONE_ONES, RELATIVE_CLAUSES, REPORTED_SPEECH, CAUSATIVE_PERMISSIVE. Rules: - grammar_concept: Classify based on the grammar used in CORRECTED, even if ORIGINAL used a different structure. - text_span: Exact text from ORIGINAL. - correct: Y if ORIGINAL uses the feature correctly, N if CORRECTED fixes an error in it. - explanation: For Y, leave empty (line must end with a pipe |). For N, write a fix under 15 words using words from CORRECTED. Do not use pipes (|) or newlines. - Output only annotation lines. No headers, blank lines, or code fences." , "role": "user", "content": "Sentence pairs to analyze: 1|ORIGINAL: She don’t like swimming in the pool. 1|CORRECTED: She doesn’t like swimming in the pool." , "role": "assistant", "content": "1|PRESENT_SIMPLE|N|don’t like|Use doesn’t with third-person She. 1|VERB_PATTERNS|Y|swimming| 1|PREPOSITIONS|Y|in the pool| 1|ARTICLES|Y|the pool|" ] E.3 Explanation Judgment Rubric The LLM judge assigns accept to correct and actionable feedback and pass to useful but pedagogically weak feedback. It assigns reject when feedback targets the wrong locus, states an incorrect rule, proposes an invalid fix, is tautological, or is empty. Explanation acceptability (Section 5) is defined as the proportion of evaluated explanations receiving either accept or pass, grouping together outputs that provide factually sound pedagogical assistance without misguiding the learner, while isolating actively erroneous or misleading feedback under reject. In the human validation stage, an independent reviewer evaluated every model-generated explanation under this rubric, and a second reviewer resolved discrepancies to establish final aligned judgments. Across all three systems on Golden2, aligned human acceptability rates converged to approximately 95% with no statistically significant differences (χ2χ^2 test of independence, p>0.30p>0.30), in contrast to the uncalibrated LLM judge. E.4 Synthetic Distribution-Shift Probe We tested the selected Qwen3.5-0.8B model on 200 learner-like utterances independently authored and annotated by GPT-5.5. Concepts and correctness rates followed Golden2, while topics spanned 15 domains; deterministic checks enforced schema, span, and label constraints. Because the labels were teacher-generated, this is a diagnostic probe rather than a third golden benchmark. Partial-span matching requires at least 50% character overlap with the target span, whereas exact-span requires exact string equality of the text span. Metric Golden2 Synthetic L1 concept micro F1 92.58 93.50 Partial-span F1 94.53 93.75 Exact-span F1 87.61 62.92 Parse errors 1 / 2,977 0 / 200 Table 12: Performance of the same Qwen3.5-0.8B checkpoint under real and synthetic evaluation distributions. Scores are percentages. Span axes evaluate localization independently of concept classification; exact-span performance is diagnostic because span-boundary conventions were not calibrated across annotators. As reported in Table 12, concept and partial-span F1 remained stable and all synthetic outputs parsed, but exact-span F1 fell by 24.69 points. The synthetic-data generator used span-boundary rules that differed from those used for the human-annotated benchmarks. The drop is therefore consistent with an annotation-protocol mismatch.

Instant research discovery