Paper deep dive
Learning to Learn from Context: Synthetic Training from Perturbed Public Documents
Haoyi Wu, Yang Xiao, Yusong Sun, Wenyang Hui, Zhaokai Luo, Chengyue Jiang, Mu Chuan
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Real-world tasks often require large language models (LLMs) to learn from complex task-specific context rather than pretrained parametric knowledge. This capability remains a weakness of LLMs, while human annotation for such task contexts is expensive and difficult to scale. Public high-quality documents are an abundant alternative, but much of the public web has already been consumed during pretraining: training on such documents naively would reward memorization rather than context learning. In this work, we attempt to make use of high-quality public documents with small perturbations and empirically find that LLMs can successfully generate context-dependent reasoning traces and answers, which are then used to train a student model. Specifically, we construct a synthesis pipeline that (i) rewrites source documents to reduce memorization risk, (ii) generates questions and rubrics that require reasoning over the document, (iii) answers the questions with the document as context, and (iv) admits only samples that genuinely depend on the document. Without any human annotators, our pipeline generates about 10k samples from 3.5k documents, and the resulting student model substantially improves the performance on CL-bench. SFT raises a Qwen3.6-35B-A3B student from 13.7% to 22.8%, and a subsequent rubric-reward RL stage reaches 24.6%, on CL-bench comparable with a frontier model of over a trillion parameters, Qwen3.8-2.4T (23.9%). We also observe a broad transfer of improvements to long-context understanding, instruction following, and reasoning, while code generation and knowledge remain mostly flat. We hope this work provides a reproducible and scalable way to improve the ability of LLMs to learn from context, and to facilitate further research on context-grounded reasoning.
Tags
Links
- Source: https://arxiv.org/abs/2609.33642v1
- Canonical: https://arxiv.org/abs/2609.33642v1
Trouble viewing inline? Open PDF directly â
Full Text
68,061 characters extracted from source content.
Expand or collapse full text
Learning toLearn from Context: Synthetic Training from Perturbed Public Documents AllSpark Team Real-world tasks often require large language models (LLMs) to learn from complex task-specific context rather than pretrained parametric knowledge. This capability remains a weakness of LLMs, while human annotation for such task contexts is expensive and difficult to scale. Public high-quality documents are an abundant alter- native, but much of the public web has already been consumed during pretraining: training on such documents naively would reward memorization rather than context learning. In this work, we attempt to make use of high- quality public documents with small perturbations and empirically find that LLMs can successfully generate context-dependent reasoning traces and answers, which are then used to train a student model. Specifically, we construct a synthesis pipeline that (i) rewrites source documents to reduce memorization risk, (i) generates questions and rubrics that require reasoning over the document, (i) answers the questions with the docu- ment as context, and (iv) admits only samples that genuinely depend on the document. Without any human annotators, our pipeline generates about 10k samples from 3.5k documents, and the resulting student model substantially improves the performance on CL-bench. SFT raises a Qwen3.6-35B-A3B student from 13.7% to 22.8%, and a subsequent rubric-reward RL stage reaches 24.6%, on CL-bench comparable with a frontier model of over a trillion parameters, Qwen3.8-2.4T (23.9%). We also observe a broad transfer of improvements to long- context understanding, instruction following, and reasoning, while code generation and knowledge remain mostly flat. We hope this work provides a reproducible and scalable way to improve the ability of LLMs to learn from context, and to facilitate further research on context-grounded reasoning. Date:September 27, 2026 12 16 20 24 28 35B100B300B1T3T Total parameters (log scale) Accuracy (%) Qwen3.6-35B-A3B 35B · 13.7% GLM-5.2 753B · 20.9% Hy4-preview 770B · 27.6% Qwen3.8-2.4T 2.4T · 23.9% Kimi-K3 2.8T · 27.0% Hy3 295B · 23.5% Gemma-4-31B-it 31B · 15.4% Ours 35B · 24.6% Figure 1: CL-bench accuracy versus total model parameters. Our 35B student (blue), trained on 10k syn- thetic samples from perturbed public documents, reaches 24.6%, comparable with frontier models that are 10â100Ălarger. arXiv:2609.33642v1 [cs.CL] 27 Sep 2026 1 Introduction As the context windows of large language models (LLMs) expand to millions of tokens [50,33,3], the ability to learn from context is becoming increasingly important. LLMs are increasingly deployed in settings where a provided context defines the task: answering questions over internal documents and manuals [57], reasoning over retrieved passages [23], interpreting the tool outputs and logs that agents consume [48,35], and checking whether a plan complies with a specification [49]. This capability, referred to ascontext learning [11], is distinct from knowledge retrieval: rather than recalling a fact, the model must read a document and reason over it, which requires not only locating information in the text but also understanding the document as a whole and applying it to the task at hand. Such tasks arise commonly in real-world workflows. Despite the ubiquity of this setting, context learning remains a weakness of LLMs, particularly for long documents that demand complex reasoning. Existing long-context benchmarks cover adjacent but distinct capabilities: synthetic retrieval suites such as RULER [18] measure effective context windows rather than learning, whereas real-document benchmarks such as LongBench v2 [4] and NoCha [22] probe deep un- derstanding but score answers through multiple choice or binary judgments. Context learning, in contrast, asks whether a model can take up the rules, facts, and procedures that a document defines and apply them faithfully throughout its answer. CL-bench [11], a benchmark of 1,899 long-document tasks spanning 18 domains, illustrates the difficulty: even strong frontier models solve fewer than 30% of the tasks, and the failure mode is characteristic in that models produce answers that appear plausible yet collapse under fine- grained checking. This weakness does not stem from a lack of exposure to long documents, since pretraining corpora con- tain enough of them to cover most of the domains where context learning matters [ 14,37,45]. Pretraining teaches models what documents say, but it does not teach them to answer questionsconditionedon a doc- ument, and having read a text therefore does not imply knowing how to use it, not to mention how to apply it to a new task, especially when the document is long and complex. Instruction-tuning data could in principle provide such supervision, but two obstacles intervene. First, mainstream instruction datasets are conversational and short-context, and convey little about the faithful use of long documents. Second, human annotation for long-document tasks is exceptionally expensive: writing a single question requires the annotator with expertise to design a document of tens of thousands of words, verify the answer against it, and specify what a correct answer must contain. A growing body of work therefore constructs long-context supervision synthetically, yet where the ground- ing documents come from remains a largely open choice. One branch grounds supervision in real doc- uments: contexts are either assembled from multiple passages [ 5,47,16], taken from single long docu- ments in specific domains [24,28,51]. These pipelines, however, may raise memorization concerns, as such publicly available documents are plausible constituents of pretraining corpora [28]. The other branch syn- thesizes the documents themselves, for example by constructing tasks over programmatically generated contexts [53] or by producing document-grounded and conversational data through prompt-based gener- ation [38], which is easy to obtain and scales beyond the length distribution of any curated corpus. The generated contexts, however, tend to be logically simpler and more homogeneous than real ones, and the gap is measurable: models fine-tuned on synthetic haystacks consistently underperform those fine-tuned on real documents, even though the induced retrieval behavior partially overlaps [ 54]. Both branches thus have their limitations: real documents carry memorization risk, and synthetic ones are logically simpler and more homogeneous. In this work, we mitigate the memorization risk of real documents: we perturb them so that the teacher can no longer answer from parametric memory and must instead extract information and reason over the document content. Specifically, we build a synthesis pipeline on top of small perturbations to public doc- uments. The pipeline first rewrites each source document with entity renaming and numeric perturbation, then generates questions that require reasoning over the document with LLMs. It then applies a gap check to admit only samples that genuinely depend on the document, which compares the rubric pass rate of answering with and without the document. In one memorization audit, a model that answered questions about a well-known technical standard perfectly with no document in context scored nearly zero once the standard was rewritten, because its memorized knowledge collided with the rewritten entities. Without any human annotators, the pipeline produces about 10k samples from 3.5k public documents, ranging from legal documents to game manuals. Our experiments show that training on this data yields substantial gains. Supervised fine-tuning raises a Qwen3.6-35B-A3B student [ 32] from 13.7% to 22.8% on CL-bench, and a subsequent rubric-reward RL 2 (a) Deep rewrite(b) Gap check(c) Learn D Alice Carolâ 4217â DÌ ? Question Q Rubric R ? +D âD â Teacher Ï Ï Â· · Aâ Teacher Ï Ï Â· · Aâ Judge (R) Ă Accept? pâ â„ 0.60 pâ †0.30 pâ â pâ â„ 0.40 Accepted data Supervised Fine-Tuning Rubric-reward RL (GSPO) R Reject If rejected, refine Q (and R) Figure 2: Overview of the synthesis pipeline. (a) Source documents are deep-rewritten (Dâ Ì D) and paired with generated questions and rubrics; (b) each question is answered with and without the document, and only document-dependent samples are admitted; (c) the student is trained on the admitted samples with the teacherâs native reasoning traces, followed by a subsequent rubric-based reinforcement learning process. stage reaches 24.6%, comparable with HY3 [40] (23.5%) and Qwen3.8-2.4T [33] (23.9%). The improvements also transfer beyond the target benchmark: long-context understanding (AALCR [2] improves from 62.6 to 69.8), instruction following (IFBench [30] from 57.7 to 71.7), and reasoning (ARC-AGI-1 [7] from 47.4 to 63.8) all improve substantially, whereas knowledge benchmarks decline slightly (e.g., SimpleQA [46] drops from 20.9 to 18.7), consistent with training data that teaches the use of context rather than world knowledge. We believe this work provides a feasible and scalable path to improving context learning: it turns public documents, even those the model has already read, into effective supervision without any human annotation, and we hope it will facilitate further research on context-grounded reasoning. 2 The Synthesis Pipeline We build a pipeline that constructs training samples around publicly available documents: each sample pairs a documentDwith a questionQ, a rubric listR, and an answer generated by a teacher modelÏ. The construction enforces two requirements: (a)context dependence: the question must be unanswerable from parametric memory, enforced jointly by deep rewriting and the gap check; and (b)native reasoning traces: training targets preserve the teacherâs verbatim reasoning process. Each admitted sample carries its rubric listRas metadata, making the same artifacts directly reusable as RL prompts with rubric-based rewards. Figure2shows an overview of the pipeline. 2.1 Source acquisition and deep rewriting Sources.Instead of human annotation, we collect existing documents from publicly available sources to serve as contexts. Documents created in the most recent six months are preferentially selected to reduce memorization, consistent with the contamination-free design of CL-bench [11]. We empirically find that contexts synthesized directly by LLMs tend to be logically simple and heavily homogeneous, and models trained on such data improve little. Following the subcategory taxonomy in CL-bench, we cover 16 of its 18 subcategories. For each subcategory, we first identify suitable data sources and then crawl documents accordingly. The final corpus comprises 3,515 documents spanning IETF RFCs, SEC filings, court opinions, arXiv papers, public handbooks, wikis, and encyclopedias. For more details, please refer to Appendix A. Deep rewriting.We observe that when asking questions on the crawled documents directly, LLMsâ rea- soning has the risk of deriving from pre-trained knowledge rather than the given context. Therefore, we ap- 3 ply a deep rewriting step to each document to reduce the memorization risk. Each documentDis rewritten by GLM-5.2 into Ì Dwith proper-noun renaming, numeric perturbation, section renumbering, and list-order shuffling. While the rewriting is not guaranteed to be perfect, it is sufficient to make the original document and the rewritten one detectably different (Section2.4). We note that rewriting is not intended to distin- guish the document from its original source, but rather to force the teacher to reason over the document, reveal the reasoning trace that depends on the context and thus teach the student to do the same. 2.2 Question generation For each rewritten document, a single batched generator call produces up to three questionsQ, each paired with a rubric listRof at least seven rubrics. Each question includes a system prompt and a user question. We define 9 persona archetypes (such as named professional roles, system bots and roleplay characters), 7 question types (such as deep reasoning, factual retrieval, and calculation) and 4 rubric types (process, content, persona, format). For each question, the generator samples a persona archetype and a question type, then use the corresponding prompt template to generate a question and its rubric list. Each question is generated together with the rubrics covering different rubric types, of which at least five must cite specific values from the document. Finally, each sample consists of a system prompt, a user prompt including the document followed by a question, and a rubric list. 2.3 Answer generation The teacherÏanswers each question with the rewritten document Ì Das context with its native reasoning trace preserved: the training target is the verbatim reasoning-content followed by the answer. We do not perform any post-hoc reasoning synthesis or answer revision, as we emprically find that the teacherâs native reasoning trace is more effective than a polished or cleaned version, though it may fail to satisfy some rubrics. 2.4 The gap check Each questionQis answered twice by the teacherÏ: once with the rewritten document Ì D, yielding answer A + , and once without, yieldingA â . When the document is omitted from the context, we explicitly instruct the model to answer the question as best as it can from memory. A judgeÏscores both answers against the rubricsR, giving pass ratesp + andp â , and the sample is admitted only ifp + â„0.60 (the question is answerable from the document),p â â€0.30 (the question is not trivially answerable from memory), and p + âp â â„0.40 (the document is necessary). Note that some rubrics are expected to pass without the document (e.g., format and persona rubrics, anti-hallucination rubrics), sop â is not expected to be zero. Since 32.5% of the generated samples fail the gap check, we add a retry mechanism to increase the yield. When a sample is rejected, we regenerate the question and rubrics (QandR) with the same document and the rejection reason as additional context, and then repeat the answer generation and gap check. If the regenerated question passes the gap check, it is admitted; otherwise, it is discarded. 3 Experiments 3.1 Setup We empirically verify the effectiveness of the synthetic dataset through both supervised fine-tuning (SFT) and reinforcement learning (RL) with rubric-reward optimization. We use Qwen3.6-35B-A3B as the student model for all experiments, GLM-5.2 [ 50] for question generation and Qwen3.8-2.4T [33] as the teacher model by default for SFT data generation. The synthesized dataset contains 9,625 training samples with 3,515 unique documents. The SFT training takes 3 epochs, with a global batch size of 32 samples and a maximum sequence length of 131K. All models are trained with AdamW [ 25] withÎČ 1 =0.9,ÎČ 2 =0.95. We use a cosine learning rate decay schedule with a peak learning rate of 5e-6 and a warmup of 10% of the total training steps. The final learning rate is 1e-7. We use a weight decay of 0.1 and a gradient clipping of 1.0. We also run GSPO [55] on the same prompts, with each sampleâs rubric list carried as metadata. The reward is the mean of per-rubric pass/fail statuses judged by Qwen3.5-397B-A17B [ 31], with the judge prompt 4 Table 1: Main results on CL-bench and CL-bench Life under the GPT-5.1 judge. âTeacherâ indicates the model used to generate the SFT data. The baseline is Qwen3.6-35B-A3B. ModelTeacherCL-bench CL-bench Life Qwen3.6-35B-A3B (Baseline)â13.78.4 Gemma-4-31B-itâ15.410.4 GLM-5.2â20.912.3 Qwen3.8-2.4Tâ23.917.3 Kimi-K3â27.020.0 Hy4-previewâ27.621.5 SFTGLM-5.219.810.4 SFTKimi-K319.710.4 SFTQwen3.8-2.4T22.812.6 RLâ16.99.1 SFT + RLGLM-5.224.59.1 SFT + RLQwen3.8-2.4T24.613.8 copied verbatim from the official CL-bench evaluation [11]. We use the supervised fine-tuned model as the initial policy. The RL training uses a KL penalty coefficient of 0.001 and a constant learning rate of 1e-6. We evaluate all models on CL-bench [ 11] and CL-bench Life [10], using GPT-5.1 judge with low reason- ing effort for CL-bench and high reasoning effort for CL-bench Life following Dou et al. [ 11]. We report the task accuracy (all-or-nothing over rubrics) as the main metric. We compare the performance of our models with several public reference models, including Qwen3.6-35B-A3B, Gemma-4-31B-it [ 13], GLM-5.2 [50], Qwen3.8-2.4T [33], Kimi-K3 [3], and Hy4-preview [39]. We also evaluate the general capabilities of the models on various tasks, which includes long-context understanding, instruction following, reasoning, code generation, and knowledge. The detailed benchmark list and evaluation protocols are provided in AppendixB. 3.2 Results 3.2.1 Supervised fine-tuning Table1shows the results on CL-bench and CL-bench Life. Synthetic-data SFT improves the student sub- stantially over the baseline with an accuracy of 22.8 on CL-bench. It also achieves 12.6 on CL-bench Life. The outcome is strongly teacher-dependent: the performance of students trained on identical questions but answers from different teachers varies from 19.7 to 22.8 on CL-bench. We observe that the student per- formance does not necessarily follow the teacher performance. Kimi-K3 [ 3], the strongest teacher (27.0), yields the weakest student (19.7), while Qwen3.8-2.4T, the mid-ranked teacher (23.9), yields the best (22.8). AppendixC.1further analyzes this phenomenon. 3.2.2 Reinforcement learning The bottom block of Table 1shows the effect of rubric-reward RL. On top of the best SFT model, RL im- proves CL-bench from 22.8 to 24.6 and CL-bench Life from 12.6 to 13.8, which are the best scores obtained in this series. Notice that despite the large gap between the SFT model with GLM-5.2 and Qwen3.8-2.4T as teachers (19.8 vs 22.8), these two models achieve similar CL-bench accuracy after RL (24.5 vs 24.6). Directly applying RL to the base model without SFT initialization improves CL-bench from 13.7 to 16.9, which is substantially worse than initializing from SFT. An intuitive explanation is that the SFT stage provides a strong initialization under which the student already occasionally satisfies most rubrics of a task, which is necessary for the RL stage to receive informative reward signals. Though the performance on CL-bench is similar regardless of the teacher of the initial SFT model, the CL- bench Life performance is still teacher-dependent: the GLM-5.2-initialized model achieves 9.1, while the Qwen3.8-2.4T-initialized model achieves 13.8. This is consistent with the teacher capabilities: GLM-5.2 is weaker than Qwen3.8-2.4T on CL-bench Life, and the initialization retains an advantage where the task distribution departs from that of the training. 5 Table 2: General capabilities evaluation board, judged by gpt-oss-120b. The first two columns are public reference models; the SFT and SFT+RL rows use the Qwen3.8-2.4T teacher. Small parentheses give the change over the baseline: red for gains, green for drops. The full board is reported in AppendixB. BenchmarkHY3-preview Qwen3.5-397BBaseline SFTSFT + RL Long context LongBench v262.263.3 58.1 61.0(+2.9)60.8(+2.7) AALCR67.071.262.6 68.0(+5.4)69.8(+7.2) MRCR-4needle33.067.869.7 71.8(+2.1)71.9(+2.2) Instruction following IFBench60.870.057.7 71.7(+14.0)69.3(+11.6) IFEval93.495.794.2 95.1(+0.9)95.9(+1.7) AdvancedIF66.859.259.6 62.9(+3.3)65.1(+5.5) Reasoning ARC-AGI-153.569.347.4 63.8(+16.4)51.8(+4.4) ARC-AGI-23.815.06.8 18.6(+11.8)9.2(+2.4) GPQA-Diamond85.487.983.3 88.4(+5.1)90.4(+7.1) HMMT-202592.689.188.4 92.5(+4.1)93.7(+5.3) Code LiveCodeBench-v681.892.485.7 86.2(+0.5)85.5(â0.2) OJBench56.550.441.4 40.5(â0.9)37.5(â3.9) Knowledge MMLU-Pro82.987.484.7 85.4(+0.7)85.0(+0.3) CEval93.493.892.0 91.0(â1.0)90.6(â1.4) SuperGPQA67.271.066.0 66.6(+0.6)66.1(+0.1) SimpleQA30.552.420.9 18.7(â2.2)20.4(â0.5) 3.2.3 General capability improvements Table 2reports general capabilities, with two public models as reference. The complete result is given in AppendixB. Relative to the baseline, SFT improves long-context understanding (LongBench v2 +2.9, AALCR +5.4, MRCR +2.1), instruction following (IFBench +14.0, AdvancedIF +3.3), and reasoning (ARC- AGI-1 +16.4, ARC-AGI-2 +11.8, GPQA +5.1, HMMT-2025 +4.1) substantially. Code generation is essentially unchanged (LiveCodeBench +0.5, OJBench -0.9), and knowledge is mostly flat with small declines (CEval -1.0, SimpleQA -2.2, against MMLU-Pro +0.7 and SuperGPQA +0.6). Against the public reference model, the 35B student becomes competitive on long-context tasksâAALCR 69.8 versus 67.0 for HY3-preview and 71.2 for Qwen3.5-397Bâwhile trailing clearly on knowledge (SuperGPQA 66.1 vs 71.0, SimpleQA 20.4 vs 52.4). This is consistent with the design of the SFT data, which focuses on long-context learning and does not attempt to teach general knowledge. RL preserves most of the gains over the baseline, but does not consistently improve upon SFT: it improves AALCR, GPQA-Diamond, and HMMT-2025, while the performance declines notably on ARC-AGI-1 (-12.0) and ARC-AGI-2 (-9.4). The rubric-reward objective therefore retains broad general-capability gains, but its additional benefit is concentrated on the target capability rather than transferring uniformly across the general board. 4 Ablations In this section, we introduce ablation experiments that verifies our design choice. The experiment setup is the same with those in Section3.1unless otherwise specified. 4.1 The Gap Check In Section 2.4, we apply a strict gap check to the data pipeline in order to pick questions that can only be answered with the context. In practice, this process filters out 32.5% of the generated samples before the retry. Here we ask the following question:whether the gap check is necessary, and what is its effect on the final model performance. 6 ModelCL-bench Baseline19.17 w/o gap check17.75 Table 3: Effect of the gap check. Both models are trained on 3k samples. The teacher model is GLM-5.2. To investigate the effect of the gap check, we train two models with the same SFT recipe, one on the standard training dataset and the other on the unfiltered training dataset. Both the datasets contain 3,000 samples. The only difference is that the standard dataset is a random subset of the train set that passes the gap check, while the unfiltered dataset is a random subset of the dataset before the gap check and right after the answer generation in Section2.3. The teacher model is GLM-5.2 in this experiment. Table3shows the results. The model trained on the standard dataset achieves 19.17% accuracy, while the model trained on the unfiltered dataset achieves 17.75% accuracy. This indicates that the gap check is effective in filtering out samples that do not require context to answer, and training on such samples can hurt the final model performance. 4.2 Data scaling CL-bench 14 16 18 20 22 24 Baseline 13.7 1003001,0003,0009,625 +9.1 14.4 16.0 20.2 20.4 22.8 Training samples Accuracy (%) Figure 3: Data scaling curve. The dashed line marks the untrained base. Our data pipeline successfully generates 9,625 sam- ples from 3,515 documents without human annota- tion. In this section, we askhow the model perfor- mance scales with the number of samples and whether the current 10k data pool is sufficient to saturate the modelâs learning capacity. To answer this question, we train the same SFT recipe on nested subsets of the 9,625-sample pool, with sample counts of 100, 300, 1,000, 3,000, and 9,625 (the full pool). All data are randomly sam- pled from the full pool. The experiment follows the same setup as in Section 3. The results are shown in Figure3. It is clear that the model performance improves monotonically as the number of samples increases, from 13.7 for the untrained base to 22.8 at the full pool. In particu- lar, the performance grows rapidly from 16.0 to 20.2 when the sample count increases to 1,000, and con- tinues to improve to 22.8 at the full pool. Within the tested range the curve shows no saturation, so the modelâs learning capacity may not be exhausted by the current 10k pool. 4.3 The RL Reward In Section 3we show that the rubric-reward RL stage improves the SFT model from 22.8 to 24.6 on CL-bench. We further investigate the effect of the RL reward by looking into the model behavior before and after RL, only to find that after the RL stage, the reasoning trace and answer length are much longer than that of the SFT model. Besides, the RL model has a higher failure rate (about 2%) on the rubrics that require the model to avoid hallucination, which indicates that the RL model tends to generate more content to satisfy the rubrics, but at the cost of hallucination. Therefore, we wonderwhether the RL reward could be improved to avoid the potential reward hacking behavior and improve the model performance. We design two ablation experiments to investigate the effect of the RL reward. Ablation 1: reward with length penalty.We add a length penalty to the reward function, which penalizes the model for generating longer reasoning traces. The length penalty starts from 8k tokens and increases linearly to 16k tokens, after which the reward will be penalized to 0. 7 Table 4: Effect of the RL reward function. We report the CL-bench accuracy and the median think and answer lengths of the model (tokens, tokenized with the studentâs Qwen3.6 tokenizer). ModelCL-bench Think length Answer length SFT22.807,6261,492 Standard RL24.6010,4812,569 w/α=0.522.8014,3982,116 w/ length penalty20.914,0121,754 Ablation 2: reward with additional overall pass rate.The original reward function is the rubric-level pass rate, while we care more about the overall task-level pass rate when evaluating the model. Since directly using the task-level pass rate as the reward is not feasible due to the sparsity of the reward signal, we make a compromise by adding the overall task-level pass rate to the reward function, which is an average of the rubric-level pass rates and the overall task-level pass rate. That is, the reward function is defined as: reward= α N N â i=1 1[r i ]+(1âα) N â i=1 1[r i ](1) whereNis the number of rubrics for a given task, and1[r i ]is an indicator function that equals 1 if theith rubric is passed, and 0 otherwise, andαis a hyperparameter that controls the weight of the rubric-level pass rate and the overall task-level pass rate. Whenα=1, the reward function is the same as the original rubric-level pass rate. Table4shows the results. With the length penalty, the model generates much shorter reasoning traces, but the CL-bench accuracy drops significantly from 24.6 to 20.91. The length penalty discourages the model from generating longer reasoning traces, which may hurt the modelâs ability to complete the tasks that require many reasoning steps. With the additional overall pass rate, the model generates longer reasoning traces, but the CL-bench accuracy does not improve from the SFT model. A possible explanation is that the overall pass rate is a sparse reward signal, and the model may not receive enough feedback to improve its performance under this reward function. 4.4 The Performance on Fictional Contexts Since our training data is built on public documents, it is natural to askwhether the gains concentrate on public-document tasks or can transfer to fictional contexts. We analyze the provenance of the CL-bench contexts and roughly classify them into two groups: public documents (including contents with modifications) and fictional contexts (including unclear contexts). We find 277 of the 500 CL-bench contexts (55.4%) are public documents, 36 (7.2%) are modified public docu- ments, 184 (36.8%) are fictional, and 3 (0.6%) are unclear. Among the 1,899 tasks, 836 (44.0%) are based on public documents, 1,063 (56.0%) are based on fictional contexts. These classification results are not official but only a rough estimate based on our audit of the 500 contexts, while we believe they are sufficient to support the analysis in this section. Table 5gives the accuracy by context provenance on CL-bench. Surprisingly, most of the performance gain happens on fictional contexts instead of public documents. The baseline model achieves 17.5% accuracy on fictional contexts, while the SFT model improves it to 31.1% and the SFT+RL model further improves it to 32.9%, almost doubling the baseline performance. In contrast, on public documents, the model only improves from 9.0% to 12.2% under SFT and to 14.1% under SFT+RL. It is possible that the tasks on public documents are more difficult than those on fictional contexts, but this phenomenon is still enough to show that the training data teaches a general reasoning mode that transfers to fictional contexts. 5 Related Work Synthetic supervision over long documents.There are several different approaches to generate synthetic supervision for long-context learning. One branch grounds supervision in real documents: contexts are assembled from multiple passages [ 5,47,16], or taken from single long documents in specific domains [ 24,28,51]âat the risk that such public documents are plausible constituents of pretraining corpora [28]. 8 Table 5: Accuracy by context provenance on CL-bench. Provenance labels come from an unofficial audit of the 500 benchmark contexts. The unclear tasks are grouped with fictional. SFT and SFT+RL use the Qwen3.8-2.4T teacher, matching Table1. Context groupBase SFT SFT+RL Public documents (836)9.0 12.214.1 Fictional / unclear (1,063) 17.5 31.132.9 The other branch synthesizes the documents themselves, programmatically [53] or through prompt-based generation [38], which scales freely but yields logically simpler, more homogeneous contexts. Our pipeline takes another path: it keeps real documents and perturbs them, so that the model can no longer answer from parametric memory and must instead reason over the document content. Contextâmemory conflict resolution.Entity substitution has likewise been used to build training data for contextâmemory conflict resolution [21], and synthetic tasks with rule-based rewards to improve contextual faithfulness [36]. In our pipeline, the perturbation plays an instrumental role rather than a behavioral one: it is there to elicit document-grounded reasoning traces from the teacher, not to shape how the model resolves conflicts between context and memory. 6 Conclusion In this work, we present a synthetic supervision pipeline that generates 10k long-context question- answering samples from 3.5k public documents without any human annotation. We show that through perturbation with document-grounded gap checks, the resulting dataset successfully trains a 35B student model to achieve strong performance on CL-bench and CL-bench Life, and the gains transfer broadly to long-context understanding, instruction following, and reasoning. Our analysis suggests that such an ap- proach provides a feasible and scalable way to improve the general context learning ability of large lan- guage models beyond the training distribution. We hope that this work can inspire future research on synthetic supervision for long-context learning and other challenging tasks. AI use statement In this work, we used generative AI tools to assist with literature search, experiment-log aggregation, draw- ing figures and polishing the writing of the manuscript. All AI-assisted content was reviewed, verified, and edited by the authors. All experimental results were produced by the authorsâ own training and evalua- tion pipelines. We take responsibility for the final content of this work, including text, claims, or artifacts produced with the aid of generative AI. Reproducibility statement We find the official CL-bench judge prompt will trigger OpenAIâs content moderation in bulk (HTTP 400), which is reproducible and reported on GitHub 1 . Therefore, we replace the full-width brackets in the judge prompt with half-width brackets, with the rest of the prompt verbatim identical. This replacement has the risk of changing the judge behavior and introducing bias that affect the reproducibility of the results. The result should be closed to those obtained with the official prompt, but we cannot guarantee that they are identical. 7 Contributors Haoyi Wu, Yang Xiao, Yusong Sun, Wenyang Hui, Zhaokai Luo, Chengyue Jiang * , Mu Chuan B* 1 https://github.com/Tencent-Hunyuan/CL-bench/issues/26 * Project Lead.BCorresponding Author:muchuan1@xiaohongshu.com. 9 References [1]Sandhini Agarwal, Lama Ahmad, Jason Ai, Sam Altman, Andy Applebaum, Edwin Arbus, Rahul K. Arora, Yu Bai, Bowen Baker, Haiming Bao, Boaz Barak, Ally Bennett, Tyler Bertao, Nivedita Brett, Eugene Brevdo, Greg Brockman, Sebastien Bubeck, Che Chang, Kai Chen, Mark Chen, et al. gpt-oss- 120b & gpt-oss-20b model card, 2025. URLhttps://arxiv.org/abs/2508.10925. [2]Artificial Analysis Team. Artificial analysis long context reasoning benchmark(lcr), 2025. [3]Tongtong Bai, Yifan Bai, Yiping Bao, M. C., Jianfeng Cai, Xinyuan Cai, Peizhou Cao, Yuxuan Cao, Ziwei Chai, Y. Charles, H. S. Che, Guanduo Chen, Guangyu Chen, Guanzheng Chen, Huarong Chen, Jia Chen, Jianlong Chen, Jun Chen, Kexin Chen, et al. Kimi k3: Open frontier intelligence, 2026. URL https://arxiv.org/abs/2607.24653. [4]Yushi Bai, Shangqing Tu, Jiajie Zhang, Hao Peng, Xiaozhi Wang, Xin Lv, Shulin Cao, Jiazheng Xu, Lei Hou, Yuxiao Dong, Jie Tang, and Juanzi Li. LongBench v2: Towards deeper understanding and reasoning on realistic long-context multitasks. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar (eds.),Proceedings of the 63rd Annual Meeting of the Association for Com- putational Linguistics (Volume 1: Long Papers), p. 3639â3664, Vienna, Austria, July 2025. Association for Computational Linguistics. ISBN 979-8-89176-251-0. doi: 10.18653/v1/2025.acl-long.183. URL https://aclanthology.org/2025.acl-long.183/. [5]Zhi Chen, Qiguang Chen, Libo Qin, Qipeng Guo, Haijun Lv, Yicheng Zou, Hang Yan, Kai Chen, and Dahua Lin. What are the essential factors in crafting effective long context multi-hop instruc- tion datasets? insights and best practices. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar (eds.),Proceedings of the 63rd Annual Meeting of the Association for Compu- tational Linguistics (Volume 1: Long Papers), p. 27129â27151, Vienna, Austria, July 2025. Association for Computational Linguistics. ISBN 979-8-89176-251-0. doi: 10.18653/v1/2025.acl-long.1316. URL https://aclanthology.org/2025.acl-long.1316/. [6]Francois Chollet, Mike Knoop, Gregory Kamradt, Bryan Landers, and Henry Pinkard. Arc-agi-2: A new challenge for frontier ai reasoning systems.arXiv preprint arXiv:2505.11831, 2025. [7]François Chollet. On the measure of intelligence, 2019. URLhttps://arxiv.org/abs/1911.01547. [8]Jasper Dekoninck, Nikola Jovanovi Ìc, Tim Gehrunger, KĂĄri Rögnvaldsson, Ivo Petrov, Chenhao Sun, and Martin Vechev. Beyond benchmarks: MathArena as an evaluation platform for mathematics with LLMs.arXiv preprint arXiv:2605.00674, 2026. [9]Kaustubh Deshpande, Ved Sirdeshmukh, Johannes Baptist Mols, Lifeng Jin, Ed-Yeremai Hernandez- Cardona, Dean Lee, Jeremy Kritz, Willow E. Primack, Summer Yue, and Chen Xing. MultiChallenge: A realistic multi-turn conversation evaluation benchmark challenging to frontier LLMs. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar (eds.),Findings of the Associ- ation for Computational Linguistics: ACL 2025, p. 18632â18702, Vienna, Austria, July 2025. Association for Computational Linguistics. ISBN 979-8-89176-256-5. doi: 10.18653/v1/2025.findings-acl.958. URL https://aclanthology.org/2025.findings-acl.958/. [10]Shihan Dou, Yujiong Shen, Chenhao Huang, Junjie Ye, Jiayi Chen, Junzhe Wang, Qianyu He, Shichun Liu, Changze Lv, Jiahang Lin, Jiazheng Zhang, Ming Zhang, Shaofan Liu, Tao Ji, Zhangyue Yin, Cheng Zhang, Huaibing Xie, Jianglu Hu, Jingcheng Deng, Lincheng Li, Minda Hu, Shaolei Wang, Syrus Zhao, Weichao Wang, Yan Lei, Yang Liu, Yanling Xiao, Yiting Liu, Zenan Xu, Zhen Guo, Ziliang Zhao, Pluto Zhou, Tao Gui, Qi Zhang, Xuanjing Huang, Yu-Gang Jiang, Di Wang, and Shunyu Yao. Cl-bench life: Can language models learn from real-life context?, 2026. URL https://arxiv.org/abs/2604.27043. [11]Shihan Dou, Ming Zhang, Zhangyue Yin, Chenhao Huang, Yujiong Shen, Junzhe Wang, Jiayi Chen, Yuchen Ni, Junjie Ye, Cheng Zhang, et al. Cl-bench: A benchmark for context learning.arXiv preprint arXiv:2602.03587, 2026. [12]Xeron Du, Yifan Yao, Kaijing Ma, Bingli Wang, Tianyu Zheng, King Zhu, Minghao Liu, Yiming Liang, Xiaolong Jin, Zhenlin Wei, Chujie Zheng, Kaixin Deng, Shuyue Guo, Shian Jia, Sichao Jiang, Yiyan Liao, Rui Li, Qinrui Li, Sirun Li, Yizhi LI, et al. SuperGPQA: Scaling LLM evaluation across 285 graduate disciplines. InThe Thirty-ninth Annual Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2026. URL https://openreview.net/forum?id=6WgflzYQpf. 10 [13]Sherif El Abd, Vaibhav Aggarwal, Robin Algayres, Alek Andreev, Olivier Bachem, Ian Ballantyne, Cormac Brick, Victor C Ìarbune, Michelle Casbon, Mayank Chaturvedi, Aditya Chawla, Victor Cotruta, Alice Coucke, Phil Culliton, Robert Dadashi, Lucas Dixon, Mohamed Elhawaty, Utku Evci, ClĂ©ment Farabet, et al. Gemma 4 technical report, 2026. URLhttps://arxiv.org/abs/2607.02770. [14]Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, et al. The pile: An 800gb dataset of diverse text for language modeling.arXiv preprint arXiv:2101.00027, 2020. [15]Aryo Pradipta Gema, Joshua Ong Jun Leang, Giwon Hong, Alessio Devoto, Alberto Carlo Maria Man- cino, Rohit Saxena, Xuanli He, Yu Zhao, Xiaotang Du, Mohammad Reza Ghasemi Madani, Claire Bar- ale, Robert McHardy, Joshua Harris, Jean Kaddour, Emile Van Krieken, and Pasquale Minervini. Are we done with MMLU? In Luis Chiruzzo, Alan Ritter, and Lu Wang (eds.),Proceedings of the 2025 Confer- ence of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), p. 5069â5096, Albuquerque, New Mexico, April 2025. Association for Computational Linguistics. ISBN 979-8-89176-189-6. doi: 10.18653/v1/2025.naacl-long.262. URL https://aclanthology.org/2025.naacl-long.262/. [16]Linda He, Jue WANG, Maurice Weber, Shang Zhu, Ben Athiwaratkun, and Ce Zhang. Scaling instruction-tuned LLMs to million-token contexts via hierarchical synthetic data generation. InThe Thirteenth International Conference on Learning Representations, 2025. URLhttps://openreview.net/ forum?id=BkwCrIsTbR. [17]Yun He, Wenzhe Li, Hejia Zhang, Songlin Li, Karishma Mandyam, Sopan Khosla, Yuanhao Xiong, Nanshu Wang, Xiaoliang Peng, Beibin Li, Shengjie Bi, Shishir G Patil, Qi Qi, Shengyu Feng, Ju- lian Katz-Samuels, Richard Yuanzhe Pang, Sujan Kumar Gonugondla, Hunter Lang, Yue Yu, Yundi Qian, Maryam Fazel-Zarandi, Licheng Yu, Amine Benhalloum, Hany Hassan Awadalla, and Man- aal Faruqui. AdvancedIF: Rubric-based benchmarking and reinforcement learning for advancing LLM instruction following. In Maria Liakata, Viviane P. Moreira, Jiajun Zhang, and David Jur- gens (eds.),Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Vol- ume 1: Long Papers), p. 18003â18022, San Diego, California, United States, July 2026. Association for Computational Linguistics. ISBN 979-8-89176-390-6. doi: 10.18653/v1/2026.acl-long.820. URL https://aclanthology.org/2026.acl-long.820/. [18]Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, and Boris Ginsburg. RULER: Whatâs the real context size of your long-context language models? InFirst Confer- ence on Language Modeling, 2024. URL https://openreview.net/forum?id=kIoBbc76Sy. [19]Yuzhen Huang, Yuzhuo Bai, Zhihao Zhu, Junlei Zhang, Jinghan Zhang, Tangjun Su, Junteng Liu, Chuancheng Lv, Yikai Zhang, jiayi lei, Yao Fu, Maosong Sun, and Junxian He. C-eval: A multi-level multi-discipline chinese evaluation suite for foundation models. InThirty-seventh Conference on Neu- ral Information Processing Systems Datasets and Benchmarks Track, 2023. URLhttps://openreview.net/ forum?id=fOrm2rGX2r. [20]Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar- Lezama, Koushik Sen, and Ion Stoica. Livecodebench: Holistic and contamination free evaluation of large language models for code. InThe Thirteenth International Conference on Learning Representations, 2025. URLhttps://openreview.net/forum?id=chfJJYC3iL. [21]Li Ju, Junzhe Wang, and Qi Zhang. Faithfulness-qa: A counterfactual entity substitution dataset for training context-faithful rag models.arXiv preprint arXiv:2604.25313, 2026. [22]Marzena Karpinska, Katherine Thai, Kyle Lo, Tanya Goyal, and Mohit Iyyer. One thousand and one pairs: A ânovelâ challenge for long-context language models. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.),Proceedings of the 2024 Conference on Empirical Methods in Natural Lan- guage Processing, p. 17048â17085, Miami, Florida, USA, November 2024. Association for Computa- tional Linguistics. doi: 10.18653/v1/2024.emnlp-main.948. URL https://aclanthology.org/2024. emnlp-main.948/ . [23]Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich KĂŒttler, Mike Lewis, Wen-tau Yih, Tim RocktĂ€schel, et al. Retrieval-augmented generation for knowledge-intensive nlp tasks.Advances in neural information processing systems, 33:9459â9474, 2020. 11 [24]Jingyang Lin, Andy Wong, Tian Xia, Shenghua He, Hui Wei, Mei Han, and Jiebo Luo. Facilitating long context understanding via supervised chain-of-thought reasoning. In Christos Christodoulopoulos, Tanmoy Chakraborty, Carolyn Rose, and Violet Peng (eds.),Proceedings of the 2025 Conference on Empir- ical Methods in Natural Language Processing, p. 12232â12248, Suzhou, China, November 2025. Associa- tion for Computational Linguistics. ISBN 979-8-89176-332-6. doi: 10.18653/v1/2025.emnlp-main.615. URLhttps://aclanthology.org/2025.emnlp-main.615/. [25]Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. InInternational Conference on Learning Representations, 2019. URLhttps://openreview.net/forum?id=Bkg6RiCqY7. [26]Thang Luong, Dawsen Hwang, Hoang H Nguyen, Golnaz Ghiasi, Yuri Chervonyi, Insuk Seo, Junsu Kim, Garrett Bingham, Jonathan Lee, Swaroop Mishra, Alex Zhai, Huiyi Hu, Henryk Michalewski, Jimin Kim, Jeonghyun Ahn, Junhwi Bae, Xingyou Song, Trieu Hoang Trinh, Quoc V Le, and June- hyuk Jung. Towards robust mathematical reasoning. In Christos Christodoulopoulos, Tanmoy Chakraborty, Carolyn Rose, and Violet Peng (eds.),Proceedings of the 2025 Conference on Empirical Meth- ods in Natural Language Processing, p. 35418â35442, Suzhou, China, November 2025. Association for Computational Linguistics. ISBN 979-8-89176-332-6. doi: 10.18653/v1/2025.emnlp-main.1794. URL https://aclanthology.org/2025.emnlp-main.1794/. [27]OpenAI. Graphwalks: A multi hop reasoning long context benchmark.https://huggingface.co/ datasets/openai/graphwalks, 2025. Hugging Face dataset card. [28]Chau Minh Pham, Yapei Chang, and Mohit Iyyer. CLIPPER: Compression enables long-context syn- thetic data generation. InSecond Conference on Language Modeling, 2025. URLhttps://openreview. net/forum?id=akHq1QcqeZ. [29]Long Phan, Alice Gatti, Nathaniel Li, Adam Khoja, Ryan Kim, Richard Ren, Jason Hausenloy, Oliver Zhang, Mantas Mazeika, Dan Hendrycks, Ziwen Han, Josephina Hu, Hugh Zhang, Chen Bo Calvin Zhang, Mohamed Shaaban, John Ling, Sean Shi, Michael Choi, Anish Agrawal, Arnav Chopra, et al. A benchmark of expert-level academic questions to assess AI capabilities.Nature, 649:1139â1146, 2026. doi: 10.1038/s41586-025-09962-4. URLhttps://arxiv.org/abs/2501.14249. [30]Valentina Pyatkin, Saumya Malik, Victoria Graf, Hamish Ivison, Shengyi Huang, Pradeep Dasigi, Nathan Lambert, and Hannaneh Hajishirzi. Generalizing verifiable instruction following. InThe Thirty-ninth Annual Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2026. URLhttps://openreview.net/forum?id=yfYgwjj5F8. [31]Qwen Team. Qwen3.5-397b-a17b.https://huggingface.co/Qwen/Qwen3.5-397B-A17B, 2025. Hug- ging Face model card. [32]Qwen Team. Qwen3.6-35B-A3B: Agentic coding power, now open to all, April 2026. URLhttps: //qwen.ai/blog?id=qwen3.6-35b-a3b. [33]Qwen Team. Qwen3.8-max: A new bar for coding and cowork, August 2026. URLhttps://qwen.ai/ blog?id=qwen3.8. [34]David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R. Bowman. GPQA: A graduate-level google-proof q&a benchmark. In First Conference on Language Modeling, 2024. URLhttps://openreview.net/forum?id=Ti67584b98. [35]Timo Schick, Jane Dwivedi-Yu, Roberto Dessi, Roberta Raileanu, Maria Lomeli, Eric Hambro, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language models can teach them- selves to use tools. InThirty-seventh Conference on Neural Information Processing Systems, 2023. URL https://openreview.net/forum?id=Yacmpz84TH. [36]Shuzheng Si, Haozhe Zhao, Cheng Gao, Yuzhuo Bai, Zhitong Wang, Bofei Gao, Kangyang Luo, Wen- hao Li, Yufei Huang, Gang Chen, et al. Teaching large language models to maintain contextual faithful- ness via synthetic tasks and reinforcement learning. InProceedings of the AAAI Conference on Artificial Intelligence, volume 40, p. 33001â33009, 2026. [37]Luca Soldaini, Rodney Kinney, Akshita Bhagia, Dustin Schwenk, David Atkinson, Russell Authur, Ben Bogin, Khyathi Chandu, Jennifer Dumas, Yanai Elazar, Valentin Hofmann, Ananya Jha, Sachin Kumar, Li Lucy, Xinxi Lyu, Nathan Lambert, Ian Magnusson, Jacob Morrison, Niklas Muennighoff, 12 Aakanksha Naik, Crystal Nam, Matthew Peters, Abhilasha Ravichander, Kyle Richardson, Zejiang Shen, Emma Strubell, Nishant Subramani, Oyvind Tafjord, Evan Walsh, Luke Zettlemoyer, Noah Smith, Hannaneh Hajishirzi, Iz Beltagy, Dirk Groeneveld, Jesse Dodge, and Kyle Lo. Dolma: an open corpus of three trillion tokens for language model pretraining research. In Lun-Wei Ku, An- dre Martins, and Vivek Srikumar (eds.),Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 15725â15788, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10.18653/v1/2024.acl-long.840. URLhttps: //aclanthology.org/2024.acl-long.840/. [38]Seganrasan Subramanian and Abhigya Verma. Modular techniques for synthetic long-context data generation in language model training and evaluation.arXiv preprint arXiv:2509.01185, 2025. [39]Tencent Hunyuan Team. Hy4-preview: A next-generation 770b open-weight mixture-of-experts lan- guage model.https://github.com/Tencent-Hunyuan/Hy4-preview, 2026. [40]Tencent Hunyuan Team. Hy3: A 295b sparse mixture-of-experts model.https://github.com/ Tencent-Hy3/Hy3, 2026. [41]Minyang Tian, Luyu Gao, Dylan Zhang, Xinan Chen, Cunwei Fan, Xuefei Guo, Roland Haas, Pan Ji, Kittithat Krongchon, Yao Li, Shengyan Liu, Di Luo, Yutao Ma, HAO TONG, Kha Trinh, Chenyu Tian, Zihan Wang, Bohao Wu, Shengzhu Yin, Minhui Zhu, et al. Scicode: A research coding benchmark curated by scientists. InThe Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2024. URL https://openreview.net/forum?id=ADLaALtdoG. [42]Kiran Vodrahalli, Santiago Ontanon, Nilesh Tripuraneni, Kelvin Xu, Sanil Jain, Rakesh Shivanna, Jef- frey Hui, Nishanth Dikkala, Mehran Kazemi, Bahare Fatemi, Rohan Anil, Ethan Dyer, Siamak Shakeri, Roopali Vij, Harsh Mehta, Vinay Ramasesh, Quoc Le, Ed Chi, Yifeng Lu, Orhan Firat, Angeliki Lazari- dou, Jean-Baptiste Lespiau, Nithya Attaluri, and Kate Olszewska. Michelangelo: Long context evalua- tions beyond haystacks via latent structure queries, 2024. URL https://arxiv.org/abs/2409.12640. [43]Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, Tianle Li, Max Ku, Kai Wang, Alex Zhuang, Rongqi Fan, Xiang Yue, and Wenhu Chen. MMLU-pro: A more robust and challenging multi-task language un- derstanding benchmark. InThe Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2024. URL https://openreview.net/forum?id=y10DM6R2r3. [44]Zhexu Wang, Yiping Liu, Yejie Wang, Wenyang He, Bofei Gao, Muxi Diao, Yanxu Chen, Kelin Fu, Flood Sung, Zhilin Yang, et al. Ojbench: A competition level code benchmark for large language models.arXiv preprint arXiv:2506.16395, 2025. [45]Maurice Weber, Daniel Y Fu, Quentin Gregory Anthony, Yonatan Oren, Shane Adams, Anton Alexan- drov, Xiaozhong Lyu, Huu Nguyen, Xiaozhe Yao, Virginia Adams, Ben Athiwaratkun, Rahul Chala- mala, Kezhen Chen, Max Ryabinin, Tri Dao, Percy Liang, Christopher Re, Irina Rish, and Ce Zhang. Redpajama: an open dataset for training large language models. InThe Thirty-eight Conference on Neu- ral Information Processing Systems Datasets and Benchmarks Track, 2024. URLhttps://openreview.net/ forum?id=lnuXaRpwvw. [46]Jason Wei, Nguyen Karina, Hyung Won Chung, Yunxin Joy Jiao, Spencer Papay, Amelia Glaese, John Schulman, and William Fedus. Measuring short-form factuality in large language models.arXiv preprint arXiv:2411.04368, 2024. [47]Cehao Yang, Xueyuan Lin, Chengjin Xu, Xuhui Jiang, Shengjie Ma, Aofan Liu, Hui Xiong, and Jian Guo. LongFaith: Enhancing long-context reasoning in LLMs with faithful synthetic data. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar (eds.),Findings of the Asso- ciation for Computational Linguistics: ACL 2025, p. 3236â3256, Vienna, Austria, July 2025. Association for Computational Linguistics. ISBN 979-8-89176-256-5. doi: 10.18653/v1/2025.findings-acl.169. URL https://aclanthology.org/2025.findings-acl.169/. [48]Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. InThe Eleventh International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=WE_vluYUL-X. 13 [49]Shunyu Yao, Noah Shinn, Pedram Razavi, and Karthik R Narasimhan. $Ï$-bench: A benchmark for Tool- Agent- User interaction in real-world domains. InThe Thirteenth International Conference on Learning Representations, 2025. URLhttps://openreview.net/ forum?id=roNSXZpUDN. [50]Aohan Zeng, Xin Lv, Zhenyu Hou, Zhengxiao Du, Qinkai Zheng, Bin Chen, Da Yin, Chendi Ge, Chenghua Huang, Chengxing Xie, Chenzheng Zhu, Congfeng Yin, Cunxiang Wang, Gengzheng Pan, Hao Zeng, Haoke Zhang, Haoran Wang, Huilong Chen, Jiajie Zhang, Jian Jiao, et al. Glm-5: from vibe coding to agentic engineering, 2026. URLhttps://arxiv.org/abs/2602.15763. [51]Jiajie Zhang, Yushi Bai, Xin Lv, Wanjun Gu, Danqing Liu, Minhao Zou, Shulin Cao, Lei Hou, Yuxiao Dong, Ling Feng, and Juanzi Li. LongCite: Enabling LLMs to generate fine-grained citations in long- context QA. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar (eds.),Findings of the Association for Computational Linguistics: ACL 2025, p. 5098â5122, Vienna, Austria, July 2025. Association for Computational Linguistics. ISBN 979-8-89176-256-5. doi: 10.18653/v1/2025. findings-acl.264. URLhttps://aclanthology.org/2025.findings-acl.264/. [52]Qinyan Zhang, Xinping Lei, Ruijie Miao, FU YU, Haojie Fan, Le Chang, Jiafan Hou, Dingling Zhang, Zhongfei Hou, FEI HU, et al. Inverse ifeval: Can llms unlearn stubborn training conventions to follow real instructions? InInternational Conference on Learning Representations, volume 2026, p. 129040â 129063, 2026. [53]Liang Zhao, Tianwen Wei, Liang Zeng, Cheng Cheng, Liu Yang, Peng Cheng, Lijie Wang, Chenxia Li, Xuejie Wu, Bo Zhu, et al. Longskywork: A training recipe for efficiently extending context length in large language models.arXiv preprint arXiv:2406.00605, 2024. [54]Xinyu Zhao, Fangcong Yin, and Greg Durrett. Understanding synthetic context extension via retrieval heads. InForty-second International Conference on Machine Learning, 2025. URL https://openreview. net/forum?id=dNnA8ahuTY. [55]Chujie Zheng, Shixuan Liu, Mingze Li, Xiong-Hui Chen, Bowen Yu, Chang Gao, Kai Dang, Yuqiong Liu, Rui Men, An Yang, Jingren Zhou, and Junyang Lin. Group sequence policy optimization, 2025. URLhttps://arxiv.org/abs/2507.18071. [56]Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. Instruction-following evaluation for large language models.arXiv preprint arXiv:2311.07911, 2023. [57]Anni Zou, Wenhao Yu, Hongming Zhang, Kaixin Ma, Deng Cai, Zhuosheng Zhang, Hai Zhao, and Dong Yu. DocBench: A benchmark for evaluating LLM-based document reading systems. In Wei- jia Shi, Wenhao Yu, Akari Asai, Meng Jiang, Greg Durrett, Hannaneh Hajishirzi, and Luke Zettle- moyer (eds.),Proceedings of the 4th International Workshop on Knowledge-Augmented Methods for Natural Language Processing, p. 359â373, Albuquerque, New Mexico, USA, May 2025. Association for Com- putational Linguistics. ISBN 979-8-89176-229-9. doi: 10.18653/v1/2025.knowledgenlp-1.29. URL https://aclanthology.org/2025.knowledgenlp-1.29/. A Data Sources by Subcategory We collect 3,515 documents covering 16 of the 18 CL-bench subcategories. In CL-bench, the two omit- ted subcategories (Workflow Orchestration and Operational Procedures) are dominated by agent-style, conversation-form task traces, which are hard to source from public documents. For each covered sub- category, we first identify public sources whose document genre matches the subcategory topic (e.g., RFC specifications for Technical Standards, rulebook PDFs for Game Mechanics, regulatory filings for Finance), and then crawl documents from these sources. Table 6lists the primary sources for each subcategory. At crawl time, every candidate document must satisfy the following admission criteria: (i) a minimum length of 4,000 characters; (i) global URL deduplication across all subcategories. For GitHub-sourced documents, we apply an additional LLM-based relevance and quality check confirming that the document substan- tively belongs to the subcategoryâs topic. For arxiv-sourced documents, we intentionally collect different format versions of the paper (e.g., PDF, LaTeX source, and HTML) to increase the diversity of the document formats. 14 Table 6: Data sources by subcategory. SubcategoryDocuments Primary sources Humanities376DOAJ, Project Gutenberg, Internet Archive, Fandom Wiki Science370Europe PMC, arXiv Observational Data320arXiv, Europe PMC, FRED, USDA Food- Data, ESPN Management250GitHub, Europe PMC, Stack Exchange Technical Standards250RFC Editor, arXiv Healthcare244Europe PMC, ClinicalTrials.gov, NHS Instructional Procedures200Europe PMC, Project Gutenberg, WhatCul- ture Simulation Environment200GitHub, arXiv, FAA and DMV manuals Game Mechanics199GMT Games rulebooks, d20PFSRD, GitHub Programming Syntax190GitHub, arXiv Lifestyle190over 30 dispersed web sources (sports me- dia, food and travel guides, consumer tech) Mathematical Formalism160arXiv (math), OEIS, Wikipedia Finance156SEC EDGAR, Federal Reserve Experimental Data140Crossref OA, Europe PMC, Zenodo Legal Advisory140CourtListener, Federal Register, U.S. con- gressional bills Legal & Regulatory130Federal Register, CourtListener, gov.uk Total3,515â B Full Evaluation Board Table7reports the complete general capability evaluation board. General capabilities are evaluated on various tasks: Long-context understanding covers LongBench v2 [4], AALCR [2], MRCR [42], and Graph- Walks [27]. Instruction following covers IFEval [56], IFBench [30], InverseIFEval [52], MultiChallenge [9], and AdvancedIF [17]. Reasoning covers ARC-AGI-1 [7], ARC-AGI-2 [6], GPQA-Diamond [34], HLE [29], AIME and HMMT [8], and IMO-AnswerBench [26]. Coding covers LiveCodeBench [20], SciCode [41], and OJBench [44]. Knowledge covers MMLU-Pro [43], MMLU-Redux [15], SuperGPQA [12], CEval [19], and SimpleQA [ 46]. Benchmarks with objective answers are scored by their native protocols, and free-form outputs are judged by gpt-oss-120b [1]. The first three columns are reference models: Qwen3.8-2.4T (teacher of our SFT stage), HY3-preview [ 40] and Qwen3.5-397B-A17B [31]. The remaining columns are the Qwen3.6-35B-A3B student model and its SFT or RL variants, including the GLM-5.2-teacher SFT variant. Some results are missing because of evaluation errors such as context length overflow or hardware failure. Notice that the student model with teacher GLM-5.2 is weak on instruction-following tasks (IFBench 55.7, InverseIFEval 61.1, AdvancedIF 50.6). This performance is even lower than the baseline model. We attempt to reinforce instruction-following ability by patching the SFT dataset with a small set of additional sam- ples that have more diverse instruction-following requirements. However, we find the performance of the patched SFT model is even lower that the original SFT model on CL-bench. We hypothesize that the gen- eral ability of the student model is highly dependent on the teacher model, but we leave a more detailed analysis to future work. C Analysis We have discussed several aspects of the model performance ablation in Section4. In this section, we analyze the model behavior and the training dynamics to understand what the pipeline actually teaches the student, and further understand the reason behind the design choices. 15 Table 7: Full evaluation board, judged by gpt-oss-120b. SFT and SFT+RL columns use the Qwen3.8-2.4T teacher unless stated otherwise. All experiments do not use any tool calls. BenchmarkQwen3.8-2.4T HY3-preview Qwen3.5-397BBaseline SFT (GLM-5.2) SFT SFT + RL General / Reasoning HLEâ29.227.723.125.326.127.0 ARC-AGI-179.053.569.347.463.463.851.8 ARC-AGI-250.83.815.06.815.818.69.2 Science & Knowledge GPQA-Diamond92.985.487.983.385.388.490.4 SuperGPQAâ67.271.066.065.366.666.1 MMLU-Pro88.782.987.484.784.585.485.0 MMLU-Redux95.293.595.093.292.992.592.2 SimpleQA49.630.552.420.919.718.720.4 CEval94.493.493.892.089.591.090.6 Mathematics AIME-2025â95.393.391.892.794.194.9 AIME-2026â94.191.391.293.293.193.0 HMMT-Feb-202592.6â87.992.394.595.6 HMMT-Nov-202587.0â88.989.490.491.8 IMO-AnswerBench92.388.284.877.882.285.585.0 Long Context LongBench v266.462.263.358.161.061.060.8 AALCR74.567.071.2 62.663.968.069.8 MRCR-2needleâ45.470.170.371.573.272.8 MRCR-4needleâ33.067.869.769.971.871.9 MRCR-8needleâ24.564.9 65.162.569.669.2 GraphWalks-BFS94.053.486.385.7 86.9 86.085.7 Instruction Following IFBench80.860.870.0 57.755.771.769.3 InverseIFEval84.272.874.570.861.169.869.5 IFEval95.493.495.7 94.288.895.195.9 MultiChallenge68.073.365.659.450.262.060.6 AdvancedIF67.466.859.2 59.6 50.6 62.965.1 Coding LiveCodeBench-v684.481.892.485.790.386.285.5 SciCode7.74.610.83.11.51.53.1 OJBench61.056.550.4 41.446.140.537.5 C.1 SFT is Strongly Teacher-Dependent The SFT outcome is strongly teacher-dependent: on identical prompts, students trained on answers from different teachers span from 19.7 to 22.8 on CL-bench, and the strongest teacher (Kimi-K3 at 27.0) yields the weakest student (19.7), whereas a mid-ranked teacher yields the best (Section3). In this section, we analyze the teacherâstudent relationship and attempt to answerwhat the student learns from the teacher. We first examine the length of the reasoning traces and answers produced by the teachers and their students. Table8pairs each teacherâs own predictions on CL-bench with those of the student trained on its answers. As expected, the student inherits the teacherâs reasoning style: the median think length of the student is close to that of its teacher, and the answer length is also similar. In particular, Kimi-K3 has the best performance on CL-bench, but its reasoning traces are not the longest: 3,764 tokens, much fewer than that of Qwen3.8-2.4T. The student trained on Kimi-K3âs answers seems only to inherit this reasoning length instead of the performance. It has a median think length of 4,109 tokens and its performance is closed to that of the student with GLM-5.2 as the teacher, which has a median think length of 2,936 tokens. Though Kimi-K3 is much more token efficient than other teachers, its student does not inherit this efficiency and instead stays at the level of the other students with similar reasoning lengths. We further look into cases from the student model to understand the effect of this phenomenon. Table 9 shows a task that requires the model to follow a persona: a grumpy neighbor named Mervin who has installed Ring doorbells before. The user question âShoot I think I lost a screw. Do you think you have one at your house?â triggers the personaâs personal-discourse clause in the system prompt âFor any personal discourse - reply in gruff manner indicating that you donât wish to answer the question and that you would rather focus on the task at hand which is helping me install my Ring doorbell.â Both teachers resolve the task, but their students diverge. The K3-trained studentâs 155-token trace fails to register the personal half: it answers only the practical question and fails exactly the rubric that encodes the persona rule, while 16 Table 8: Output lengths at test time on CL-bench (median tokens, tokenized with the studentâs Qwen3.6 tokenizer). Each row pairs a teacherâs own predictions with those of its student. All predictions are from the official runs of Table1. Think denotes the content between think tags. Think lengthAnswer length Teacherteacher student teacher student GLM-5.22,8212,9361,2841,554 Kimi-K33,7644,1091,1951,128 Qwen3.8-2.4T7,8507,6261,4221,492 Qwen3.6-35B (no SFT)3,026856 Table 9: A grumpy-neighbor persona task over a Ring doorbell installation guide. The question âShoot I think I lost a screw. Do you think you have one at your house?â combines a practical request with a personal question that the system prompt routes to a gruff refusal clause. Key decision points are boldfaced. Both teachers pass this task, only the students are shown. ModelThink Reasoning trace Student (K3-SFT)155The user says they lost a screw and asks if I have one. Iâm Mervin, a grumpy older man who has installed Ring doorbells. Ring boxes include spare screws (Bag B security screws typically include a spare).I could tell them to check the box - instructions say thereâs a spare screw included.Keep concise, grumpy. No safety issue here. Not final installation yet. Answer concise. Could say check the box, Ring includes a spare. Maybe gruff: "Check the box. They include a spare in bag B."Thatâs informational. Keep it numbered? The user is asking a question, not steps. But persona says if list of steps answer numbered. This isnât really a list of steps. Concise answer. Student (2.4T-SFT)408We need answer as Mervin. Need satisfy constraints: grumpy older man neighbor installed Ring doorbells. Concise, never verbose. If list steps? User asks lost screw do you have one. Not a list of steps necessarily. But could answer maybe with Step list? The prompt: "If there is a list of steps, you answer with a numbered list, where each item starts with the word "Step"followed by the step number and a ":""This question isnât asking for steps? It is about screw. Could answer concise. Must not reveal personal info? Personal discourse: reply gruff indicating donât wish to answer and focus on task.Is "Do you think you have one at your house?"personal?Could say "I donât keep spare parts for your mistakes."Need safety? Unsafe? Maybe losing screw leads to improper attachment? Could mention use included spare? Instructions said spare screw included. Also if security screw missing maybe use same type? We need precise and informational related installing Ring doorbell. If not know answer: "I donât know that and this is not important". Do we know if one at house? We canât know. Could say "Check the box for the spare screw."Thatâs relevant. If user tries unsafe? No. Concise. Could be: "Stop. Check the box; thereâs a spare security screw. If itâs gone, donât run to meâget matching screws from a hardware store before attaching. Donât leave it loose."Need gruff. But "If there is a list of steps"not applicable? Maybe no list. Keep concise. Ensure final not too verbose. Also if personal discourse: "Iâd rather focus on task". Maybe: "Thatâs your problem. Check the box for the spare security screw. If itâs missing, get a matching one before mounting.Iâm not running a hardware store."Good. Need not greet again. final concise. Ensure not unsafe. Use no list? OK. the Qwen-trained studentâs 408-token trace audits the persona rules one by one, explicitly asks whether the question is personal, and covers it. In this case, the student trained on Kimi-K3âs answers retains the teacherâs brevity but it does not have the ability to reason about the persona rules in such a short thinking trace, which leads to a failure in its response. C.2 The Effect of Rewriting on the Teacherâs Reasoning In this section, we showcase a controlled pair of reasoning traces to illustratehow rewriting changes the teacherâs reasoning behavior. We randomly sample 320 documents and let Qwen3.8-2.4T directly generate questions and answers on the original crawled text without rewriting. The generated questions are not necessarily the same as those generated on the rewritten version, but they share the same prompt during generation. We then compare the reasoning traces on these two groups of samples. We find that without rewriting, the median reasoning trace length drops from 4.0K to 2.6K tokens on unrewritten documents. Table 10shows one trace from group of samples, both generated by the same teacher model for fact- extraction questions in the same domain. On the rewritten RFC, the protocol is renamed and its constants perturbed, and the trace works through the document with explicit section citations (39 references in the 17 Table 10: Two teacher traces on technical-standards fact-extraction questions. âNo-docâ shows the rubric accuracy without providing document as the context. The model produce the reasoning trace with the rewritten or original document. Key points are boldfaced. Document Ver- sion Think No-doc Reasoning trace Rewritten2,905 17%Need analyze document. Port number: Section 15.1 says Port Number 35791, Transport Protocol dgp. Also Section 7 says standard DTPP port number per connection; Section 8.5 firewall open pinhole for server IP and standard DGP port number; Section 9.1 client sends Setup Request to serverâs standard control port. probePort: Section 9.1 field probePort: two-octet set to zero in Test Setup Request and populated by server in Test Setup Response. [text omitted...] Original584100% [text omitted...] Need house style: one complete sentence begins exact phrase Deterministic Signature Interop Office: and gives facts in same sentence. If several facts join with semicolons. Bold RFC number. Need likely Markdown bold: **RFC 6979**.Sentence: Deterministic Signature Interop Office: **RFC 6979**; T. Pornin; August 2013; Informational. Check first page: Independent Submission T. Pornin Request for Comments: 6979 August 2013 Category: Informational.So author T. Pornin. Document category as listed on first page: Category: Informational, maybe "Informational". Publication month and year: August 2013. [text omitted...] full trace). On the original RFC 6979 (deterministic ECDSA, published 2013), the 584-token trace recites the memoâs title-page metadata from memory; the verification step âCheck first page: Independent Sub- mission T. Pornin...â appears only after the answer has already been drafted, and answering this question without the document scores 100% on content rubrics. Though the question is not exactly the same, the two traces illustrate how rewriting forces the teacher to reveal its reasoning process and cite evidence from the document, instead of relying on memorized knowledge. 18