Paper deep dive
No Test Cases, No Problem: Distillation-Driven Code Generation for Scientific Workflows
Siddeshwar Raghavan, Tanwi Mallick
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 98%
Last extracted: 6/21/2026, 6:16:11 AM
Summary
MOSAIC is a training-free, multi-agent framework designed for scientific code generation where traditional I/O test cases are unavailable. It utilizes a student-teacher knowledge distillation approach to achieve semantic grounding through domain-specific rationales. The framework features a Consolidated Context Window (CCW) to mitigate hallucinations in chained subproblems and employs specialized agents (Self-Reflection, Rationale, Coding, and Debugger) to decouple semantic and syntactic grounding. Experiments on the SciCode benchmark demonstrate that MOSAIC significantly improves accuracy and numerical precision across scientific domains compared to standard LLM baselines.
Entities (10)
Relation Signals (5)
MOSAIC → contains → Self-Reflection Agent
confidence 100% · MOSAIC consists of four main agents Self-Reflection, Rationale, Coding, and Debugging.
MOSAIC → evaluatedon → SciCode
confidence 100% · We evaluate extensively on SciCode Tian et al. (2024) benchmark
Self-Reflection Agent → partof → Teacher Module
confidence 100% · The Self-Reflection Agent is the key component of our Teacher Module.
Rationale Agent → partof → Student Module
confidence 100% · The Rationale Agent is the first of three components within the Student Module.
MOSAIC → uses → Consolidated Context Window
confidence 100% · Our proposed Consolidated Context Window (CCW) maintains consistent reasoning across chained subproblems
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Existing multi-agent Large Language Model (LLM) frameworks for code generation typically use execution feedback and improve iteratively using Input/Output (I/O) test cases. However, this does not work for scientific workflows, where I/O test cases do not exist, and generating them requires solving the very problem at hand. To address this, we introduce MOSAIC, a training-free multi-agent framework for scientific code generation without I/O supervision. Instead of execution feedback, MOSAIC employs a student-teacher knowledge distillation framework that grounds generation through domain-specific examples and structured problem decomposition. To further mitigate hallucinations across chained subproblems, we introduce a Consolidated Context Window (CCW) for maintaining consistent reasoning across agents. Experiments on the SciCode benchmark show that MOSAIC improves accuracy, executability, and numerical precision over existing approaches while relying on lightweight models.
Tags
Links
- Source: https://arxiv.org/abs/2604.23106v1
- Canonical: https://arxiv.org/abs/2604.23106v1
Trouble viewing inline? Open PDF directly →
Full Text
41,418 characters extracted from source content.
Expand or collapse full text
No Test Cases, No Problem: Distillation-Driven Code Generation for Scientific Workflows Siddeshwar Raghavan Purdue University Electrical and Computer Engineering West Lafayette, USA raghav12@purdue.edu &Tanwi Mallick Argonne National Laboratory Mathematics and Computer Sciences Lemont, USA tmallick@anl.gov Abstract Existing multi-agent Large Language Model (LLM) frameworks for code generation typically use execution feedback and improve iteratively using Input/Output (I/O) test cases. However, this does not work for scientific workflows, where I/O test cases do not exist, and generating them requires solving the very problem at hand. To address this, we introduce MOSAIC, a training-free multi-agent framework for scientific code generation without I/O supervision. Instead of execution feedback, MOSAIC employs a student-teacher knowledge distillation framework that grounds generation through domain-specific examples and structured problem decomposition. To further mitigate hallucinations across chained subproblems, we introduce a Consolidated Context Window (CCW) for maintaining consistent reasoning across agents. Experiments on the SciCode benchmark show that MOSAIC improves accuracy, executability, and numerical precision over existing approaches while relying on lightweight models. No Test Cases, No Problem: Distillation-Driven Code Generation for Scientific Workflows Siddeshwar Raghavan Purdue University Electrical and Computer Engineering West Lafayette, USA raghav12@purdue.edu Tanwi Mallick Argonne National Laboratory Mathematics and Computer Sciences Lemont, USA tmallick@anl.gov 1 Introduction Scientific coding plays a very crucial role in the process of modern empirical discovery. Across physics, biology, and materials science, computational problems demand numerical precision, reproducibility, and deep domain reasoning. These problems decompose naturally into chains of interdependent subproblems, and errors accumulate fast. As context grows, LLMs struggle to retain critical information and become prone to hallucinations and inconsistencies Zhang et al. (2025). Collectively, these requirements exceed the capabilities of standalone LLMs and simple agent setups. Existing multi-agent code generation frameworks try to handle coding complexity through planning, execution, and iterative debugging. However, they have one important hidden dependency, that is, Input/Output (I/O) test cases. From one-shot synthesis Chen et al. (2021) and chain-of-thought prompting Wei et al. (2022) to more advanced multi-agent pipelines Huang et al. (2024); Islam et al. (2024, 2025); Mallick et al. (2024), the standard approach improves the generated code by refining it with I/O samples. This type of structure works well in competitive programming settings, where test cases are already available, but it is not suitable for scientific discovery. Competitive coding (I/O test cases available) Problem: Write a function to find the sum of all even numbers in a list. Test cases: ⬇ assert sum_even([1, 2, 3, 4]) == 6 assert sum_even([1, 3, 5]) == 0 Scientific coding (no I/O test cases) Problem: Write a Haldane model Hamiltonian on a hexagonal lattice. Function header: ⬇ def hamiltonian(kx, ky, a, t1, t2, phi, m): Background info: Defines what a Haldane model is. Figure 1: Problem Structure Differences: General and Competitive coding benchmarks provide explicit I/O pairs for validation during code generation. Scientific coding benchmarks typically provide only a function signature and background information. The verification cycle. In scientific workflows, validation test cases are not available in advance as shown in Figure 1. Unlike competitive programming benchmarks, where expected outputs are fixed, scientific problems are controlled by domain-specific tolerances and numerical limits that vary from problem to problem. More importantly, constructing a valid I/O sample for these physical simulations or biological models is not straightforward, since the underlying algorithm is what we are aiming to generate. This leads to a fundamental deadlock: the code cannot be verified without knowing the answer, but if the answer is already known, then the problem is effectively solved. Prior frameworks such as MapCoder Islam et al. (2024), CodeSIM Islam et al. (2025), and AgentCoder Huang et al. (2024) designed for competitive programming are fundamentally unsuitable for scientific settings, as their entire verification loop depends on I/O test cases. Grounding without ground truth: To address this issue, we propose MOSAIC, a fully autonomous and training-free multi-agent framework that helps grounding of scientific code at the architectural level, instead of entirely depending on execution-based feedback. Rather than verifying code against I/O test cases, MOSAIC decouples semantic grounding by generating domain-specific rationales through a student–teacher knowledge distillation framework. This process is kept separate from syntactic grounding, which is managed by a dedicated Debugger Agent that focuses on resolving syntax and import errors. Our proposed Consolidated Context Window (CCW) maintains consistent reasoning across chained subproblems without expanding the context window. We evaluate extensively on SciCode Tian et al. (2024) benchmark, MOSAIC consistently outperforms all baselines across LLM backbones and scientific domains. Our main contributions are: • We identify the verification cycle as the fundamental obstacle to scientific code generation and propose architectural grounding as a principled solution. • We introduce MOSAIC, a training-free, LLM-agnostic multi-agent framework that operates entirely without I/O test cases, decoupling semantic and syntactic grounding. • We introduce the Consolidated Context Window (CCW), which preserves reasoning coherence across interdependent subproblems without growing the context window. • Extensive experiments and ablations on SciCode show the promise of our proposed approach achieving up to 24% accuracy gains across the five scientific domains. 2 Related Work LLMs for Science & Code Generation: LLMs are increasingly adopted for scientific discovery Wang et al. (2023a); Microsoft Research AI4Science and Microsoft Azure Quantum (2023); Zhang et al. (2024) and code generation Chen et al. (2021); Nijkamp et al. (2023); Ben Allal et al. (2023); Li et al. (2023). While techniques like Chain-of-Thought Wei et al. (2022), Self-Consistency Wang et al. (2023b), and Graph of Thoughts Besta et al. (2023) improve multi-step reasoning, and methods like Reflexion Shinn et al. (2023) mimic the experimental trial-and-error cycle, these methods predominantly rely on execution feedback. General coding benchmarks (e.g., HumanEval Chen et al. (2021), APPS Hendrycks et al. (2021), MBPP Austin et al. (2021)) and repository-level tasks (SWE-bench Jimenez et al. (2024)) provide validation test cases to guide algorithm improvement. This dependency limits their applicability in scientific domains where I/O targets cannot be trivially generated. In contrast, the SciCode benchmark Tian et al. (2024) evaluates executable scientific programs without relying on I/O tests, exposing a gap in current methods. Agentic LLM Frameworks. Recent research explores collaborative multi-agent pipelines, where specialized agents interact to tackle complex programming tasks. Frameworks like MapCoder Islam et al. (2024) and CodeSIM Islam et al. (2025) employ planning, coding, and debugging agents to iteratively verify inputs and outputs. Some systems also integrate external APIs and mathematical toolkits Wu et al. (2023b); Schick et al. (2023); Wu et al. (2023a). However, while effective for standard programming tasks, these systems fail to generalize to scientific problem-solving where reasoning must be coupled with domain knowledge and where validation I/O is unavailable to correct semantic logic. Motivated by this gap, we propose a modular, training-free multi-agent framework driven by knowledge distillation, separating semantic grounding from syntactic grounding to operate without an I/O oracle. 3 Method In this section, we present our framework, MOSAIC, detailing the various agents it orchestrates as well as the underlying design choices that allow it to operate without execution feedback. 3.1 MOSAIC Framework We design MOSAIC as a modular, multi-agent framework for solving complex scientific coding problems. It operates by breaking problems into smaller steps and coordinating specialized agents to produce correct and executable code. The framework is LLM-agnostic and adapts to various fields by integrating domain-specific knowledge and memory into each agent, without fine-tuning. MOSAIC consists of four main agents Self-Reflection, Rationale, Coding, and Debugging. A Bucketing Module first assigns each problem to the appropriate domain. While all domains share the same agent architecture, each maintains its own dedicated memory to prevent cross-domain interference. Figure 2: MOSAIC: our distillation-driven multi-agent framework for scientific code generation without I/O supervision. The Teacher Module derives domain-specific guidance from a small validation set and passes it to the Student Module via few-shot prompting. The Consolidated Context Window (CCW) maintains focus on the current subproblem by retaining only prior function signatures and summaries. Inspired by the concept of Knowledge Distillation (KD) Hinton et al. (2015), MOSAIC is structured as a teacher-student system, as illustrated in Figure 2. Within the Teacher Module, we use ground-truth scientific code from a small non-overlapping subset (≤ 5%) of the validation data to create detailed domain specific rationales that break each problem into a sequence of solution steps. These rationale steps, along with the corresponding ground-truth code, enable the teacher to create domain-specific guidance templates. These templates then help the Student Module learn to solve problems within a given scientific domain. Because I/O test cases are absent, this distillation process acts as the primary mechanism for semantic grounding. In the following sections, we explore the core agents that form the foundation of our architecture. 3.1.1 Self-Reflection Agent The Self-Reflection Agent is the key component of our Teacher Module. To construct a strong domain guidance template, this agent receives the ground-truth rationale and learns to evaluate its own reasoning steps for each domain. It identifies potential mistakes or omissions and refines its logic iteratively before arriving at the final pseudocode. By verbalizing its thought process and critically analyzing its reasoning path, the Self-Reflection Agent establishes the semantic baseline for the framework, enhancing output reliability and correcting logical flaws without executing code. 3.1.2 Rationale Agent The Rationale Agent is the first of three components within the Student Module. It uses the template pseudocode generated by the Self-Reflection Agent as few-shot examples to process scientific problems from the test set. It then produces a clear, step-by-step reasoning plan similar to the structured guidance offered by the Teacher Module. Scientific problems typically involve a sequence of dependent subproblems that must be addressed in a specific order. To address the risk of hallucination as the context window grows Zhang et al. (2025); Banerjee et al. (2024), we implement a Consolidated Context Window (CCW) within the Rationale Agent. As illustrated in Figure 2, to mitigate hallucination, the CCW contains only previously implemented function signatures and brief one-sentence summaries, rather than the full code and reasoning history. Thus, the CCW helps the agent remain focused on the current task while maintaining enough historical awareness to determine the logical next steps in the reasoning process. 3.1.3 Coding Agent The Coding Agent uses the detailed plan provided by the Rationale Agent to generate the corresponding code block in Python, maintaining awareness of both the subproblem and the broader problem context with the help of the CCW. While MOSAIC is designed for I/O-free environments, the Coding Agent remains flexible, if a prompt does include I/O test cases (as seen in general-purpose coding datasets Austin et al. (2021); Chen et al. (2021); Hendrycks et al. (2021)), it can successfully incorporate them to improve quality of generated code. 3.1.4 Debugger Agent The final core agent in our MOSAIC framework is the Debugger Agent, which executes the generated code and performs up to k rounds of error correction in collaboration with the Coding Agent. Because scientific coding benchmarks such as SciCode Tian et al. (2024) lack I/O test cases to verify algorithmic logic, the Debugger Agent is strictly constrained to syntactic grounding. This iterative process strictly resolves syntax and import errors. By ensuring code executability, the Debugger Agent guarantees that the semantically grounded logic generated by the previous agents can be successfully evaluated. 4 Experiments 4.1 Datasets and Comparison Approaches We evaluate MOSAIC on the challenging SciCode dataset Tian et al. (2024), which covers five domains (Physics, Chemistry, Biology, Mathematics, and Materials Science) and comprises 65 main problems split into 283 subproblems. Each subproblem provides a prompt, background context, a function signature for code generation (as shown in Figure 3, Appendix Figure 8) without access to the gold standard ground truth code. The dataset provides a validation set of 15 main problems and 50 subproblems with ground truth code that is disjoint with the test set. We compare MOSAIC against four baseline methods: Direct, Chain of Thought (CoT), Self Planning, and Analogical. Because current state-of-the-art LLM coding frameworks rely on sample test cases to generate code and do not support integrating multiple subproblems into a unified workflow, they cannot be evaluated directly on SciCode. The test suite provided by SciCode Tian et al. (2024) evaluates the correctness of a problem if all the subproblems are executable and match the ground truth target value. To demonstrate MOSAIC’s versatility, we also evaluate it alongside leading multi-agent coding frameworks on the general-purpose MBPP dataset Austin et al. (2021) (1,000 problems) and the HumanEval dataset Chen et al. (2021) (164 problems), as well as on the APPS benchmark Hendrycks et al. (2021), which comprises over 5,000 problems spanning introductory, interview, and competition-level challenges. On the general purpose coding datasets, we compare MOSAIC against MapCoder Islam et al. (2024) and CodeSIM Islam et al. (2025) where it achieves comparable performance. 4.2 Implementation Details We build on the open-source PyTorch implementation of SciCode Tian et al. (2024) for our experiments. Within MOSAIC, we employ LangGraph LangChain (2024) to orchestrate agent communication and ensure reproducibility. For each problem domain, we have a dedicated memory for the agentic framework to ensure encountering only the domain knowledge and prevent cross-domain interference. Each agent (Self-Reflection, Rationale, Coding, and Debugging) is guided by tailored prompts that constrain it to its specific role and yield the outputs needed to arrive at the final solution (Included in the Appendix A). In MOSAIC’s teacher module, designed for knowledge distillation via few-shot prompting, we sample twenty problems at random from the APPS training set (seed 1993) Hendrycks et al. (2021), use the ten MBPP problems provided for few-shot examples Austin et al. (2021), and include five problems from the HumanEval dataset Chen et al. (2021) taken out of the entire dataset. We evaluate our MOSAIC framework, other baselines and benchmarks using Open AI GPT-4o, Claude Sonnet 4 and Gemini 2.5 Flash. In our ablation studies 6 we explore other open source LLM backbones. 4.3 Evaluation Metrics In this paper, we adopt the SciCode evaluation protocol Tian et al. (2024) for the scientific dataset, counting solved sub-problems and main problems. We also report metrics that quantify how closely our outputs align with the reference solutions in SciCode (Figure 5) in our Ablation section 6, even for cases that don’t fully succeed, since precision and accuracy are critical in scientific problem solving. For MBPP Austin et al. (2021), HumanEval Chen et al. (2021), and APPS Hendrycks et al. (2021), we report performance as the percentage of test cases passed. 5 Results and Discussion LLM Backbone Methods Total Physics Chemistry Biology Material Science Mathematics GPT-4o SciCode Baseline 7/65 94/283 3/30 48/145 1/7 13/42 0/7 5/25 2/11 24/50 1/10 4/24 Analogical 1/65 32/283 1/30 18/145 0/7 2/42 0/7 3/25 0/11 6/50 0/10 3/24 CoT 2/65 38/283 1/30 21/145 0/7 2/42 0/7 3/25 1/11 8/50 0/10 4/24 LATS 4/65 49/283 2/30 34/145 0/7 2/42 0/7 3/25 2/11 8/50 0/10 2/24 MOSAIC (ours) 12/65 113/283 4/30 56/145 2/7 14/42 0/7 7/25 3/11 26/50 3/10 10/24 Claude Sonnet 4 SciCode Baseline 9/65 109/283 4/30 71/145 1/7 13/42 1/7 8/25 2/11 8/50 1/10 8/24 MOSAIC (ours) 13/65 118/283 4/30 77/145 2/7 17/42 1/7 9/25 3/11 8/50 3/10 8/24 Gemini 2.5 flash SciCode Baseline 7/65 112/283 5/30 67/145 1/7 6/42 1/7 5/25 2/11 6/50 1/10 1/24 MOSAIC (ours) 11/65 117/283 5/30 88/145 2/7 14/42 1/7 11/25 2/11 9/50 1/10 12/24 Table 1: Performance comparison between baselines and MOSAIC on scientific datasets with different LLM backbones. Best results are highlighted. The SciCode benchmark consists of 65 main problems comprising a total of 283 subproblems spanning physics, chemistry, biology, materials science, and mathematics. A problem is considered solved only when all of its subproblems pass the corresponding test suites. Figure 3: Structure of problems and subproblems in the SciCode dataset. Each main problem is composed of multiple subproblems, all of which must be solved correctly for the main problem to be considered successfully solved. Figure 4: Error statistics on SciCode benchmarks. The figure distinguishes syntactic errors (failed execution) from semantic errors (output–target mismatches). Semantic errors are shown in gray, while other colors represent different categories of syntactic errors. MOSAIC substantially reduces both the overall error rate and the relative proportion of syntactic errors compared to the baseline. Method HumanEval (30 val / 134 test) MBPP (10 val / 500 test) APPS Direct 89.63 48.80 12.70 CoT 87.20 54.92 11.30 Self-Planning 89.63 49.64 14.70 Analogical 90.85 49.81 12.00 MapCoder 90.26 77.92 22.37 CodeSIM 93.60 80.49 22.56 MOSAIC (ours) 92.53 84.90 24.71 Table 2: Performance comparison of different code generation methods on HumanEval, MBPP, and APPS benchmarks. Reported values represent accuracy scores (%). The Best results for each benchmark are highlighted in blue, while the second best results are highlighted in orange. MOSAIC achieves competitive performance with CodeSIM, ranking first on MBPP and APPS and second on HumanEval Table 1 reports performance across five scientific domains and three LLM backbones. MOSAIC consistently outperforms all baselines under every backbone. With GPT-4o, it solves 12/65 main problems and 113/283 subproblems, with strongest gains in Physics (56/145) and Material Science (26/50). Claude Sonnet 4 yields the best overall result (13/65, 118/283), with clear improvements in Physics (77/145) and Chemistry (17/42). Gemini 2.5 Flash achieves 11/65 and 117/283, with the largest gains in Physics (88/145) and Mathematics (12/24). Across all settings, MOSAIC’s gains stem from its multi-agent orchestration: the Rationale Agent proposes a structured plan guided by the Teacher Module through domain-specific few-shot examples, while the CCW reduces hallucination by exposing only prior function signatures and one-line summaries to each subsequent agent. Performance in Biology is consistently the weakest across all backbones. We observed incorrect ordering of steps and oversimplified algorithmic logic errors that are very different from numerical precision failures in physics or mathematics. We identified three factors that contribute to this lower performance. (1) LLMs struggle to transfer concepts across structurally related prompts Xu et al. (2024) (2) biological knowledge in closed-source models is difficult to verify and (3) Biology is the least represented domain in the SciCode validation set. Which leaves fewer ground-truth samples for creating guidance templates to use for few-shot prompting. For problems with more than 10 subproblems, even the CCW is insufficient to maintain full context the model does not consistently reuse prior function headers or carry forward intermediate results. LLM backbone insights Different backbones excel in different roles, Claude Sonnet 4 performs better on code generation tasks, while Gemini provides stronger reasoning and GPT-4o does moderately well on both. This points to the potential of heterogeneous agent configurations, which we leave for future work. We also found that capitalizing critical instructions in prompts (e.g., DO NOT) reliably kept agents within intended bounds, consistent with tokenization differences between text and TEXT. Error analysis. Figure 4 shows the distribution of syntactic and semantic errors across benchmarks. The baseline produces nearly half its programs with syntactic errors, rendering them inexecutable. MOSAIC shifts this distribution, the overall error count drops, and the remaining errors are predominantly semantic (AssertionError), meaning the code runs but produces an incorrect value. This shift is deliberate. MOSAIC’s Debugger Agent is constrained to syntactic grounding only, so semantic errors surface as a natural consequence of ensuring executability first. Once execution is guaranteed, algorithmic refinement becomes easier. Figure 5 further shows that MOSAIC outputs have substantially smaller deviations from target values, reflecting improved numerical precision across domains. General coding benchmarks. Despite being designed for I/O-free scientific settings, MOSAIC also performs competitively on standard benchmarks (Table 2), ranking first on MBPP (84.90%) and APPS (24.71%) and second on HumanEval (92.53%), behind CodeSIM. This confirms that architectural grounding does not compromise general coding capability. 6 Ablation Studies Method Total Phys prob Chem Biology Mat Sci Math Baseline 7/65, 94/283 3/30, 48/145 1/7, 13/42 0/7, 5/25 2/11, 24/50 1/10, 4/24 Baseline + Rationale + Coding and Debug Agent 9/65, 97/283 3/30, 47/145 2/7, 15/42 0/7, 6/25 3/11, 25/50 1/10, 4/24 Baseline + Few-shot prompting + Rationale + CCW(all prev. code) + Coding and Debug Agent 4/65, 57/283 1/30, 32/145 1/7, 6/42 0/7, 6/25 1/11, 9/50 1/10, 4/24 Baseline + Self Reflection (stepwise) + Few-shot prompting + Rationale + CCW(prev. headers) + Coding and Debug Agent 6/ 65, 81/283 2/30, 48/145 1/7, 8/42 0/7, 6/25 2/11, 14/50 0/10, 5/24 MOSAIC(ours) = Baseline + Self Reflection (whole) + Few-shot prompting + Rationale + CCW(prev. headers) + Coding and Debug Agent 12/ 65, 113/283 4/30, 56/145 2/7, 14/42 0/7, 7/25 3/11, 26/50 3/10, 10/24 Table 3: Performance comparison of MOSAIC with incremental addition of specialized agents (Rationale, Coding, Debugger) and mechanisms (Consolidated Context Window (CCW), Self-Reflection, and Few-shot prompting). The benchmark baseline Tian et al. (2024) is included for reference. The results highlight how each component contributes to overall performance, with the full MOSAIC framework yielding the most significant improvements. Method Type LLM backbone Parameters Problems solved Main Sub Problems MOSAIC(ours) Open Source Mistral 7B 0/65 24/283 Open Source Gemma 3 27B 1/65 39/283 Open Source Llama 4 16×17B 2/65 41/283 Open Source DeepSeek R1 32B 4/65 84/283 Closed Source Gemini 2.5 Flash NA 11/65 117/283 Closed Source Claude Sonnet 4 NA 13/65 118/283 Closed Source GPT-4o NA 12/65 113/283 Table 4: Performance comparison of Open and Closed Source models on SciCode benchmark dataset. Open-source models solve at most 4/65 main problems and 84/283 subproblems (DeepSeek R1), while smaller ones like Mistral fail to solve any main problems. Closed-source models perform substantially better, with Claude Sonnet 4 achieving 13/65 main and 118/283 subproblems solved. On average, closed-source backbones solve nearly 3× more main problems and 2× more subproblems, highlighting the current performance gap. Figure 5: Precision differences between target and generated outputs. Compared to the baseline, MOSAIC produces a larger proportion of executable code, which introduces slightly more detectable errors. However, MOSAIC outputs exhibit substantially smaller deviations from the target values, indicating improved numerical precision. 6.1 Component Analysis Table 3 reports results as agents and mechanisms are added incrementally to the SciCode baseline (GPT-4o). Adding a Rationale Agent with iterative Coding and Debugger Agents improves performance from 7/65 to 9/65 main problems (+4% solve rate), primarily by reducing syntactic errors. Adding few-shot prompting alongside an unrestricted CCW (retaining all prior code) surprisingly drops performance to 4/65, as the expanded context caused the model to replicate irrelevant code fragments and produce logically inconsistent outputs. Restricting the CCW to function signatures and one-line summaries recovers most of this loss but still lacks a mechanism to generalize reasoning across problems. Introducing a Self-Reflection Agent that processes rationales step-by-step gives 6/65 still below baseline because fragmenting the rationale loses overall problem context. Applying self-reflection over the entire rationale preserves this context and yields best performance with 12/65 main problems solved and 113/283 subproblems solved, an 8.5% gain over baseline. Two key insights emerge. First, components added in isolation can hurt performance and careful orchestration mattered more than component count. Second, each configuration solves a somewhat different subset of problems; because overlap is minimal, combining all agents broadens coverage and gives the largest overall gain. 6.2 Open- vs. Closed-Source Models Table 4 compares closed-source and open-source backbones. Among open-source models, DeepSeek R1 (32B) performs best (4/65, 84/283), but does not surpass the SciCode baseline with GPT-4o. Mistral 7B solves no main problems. Closed-source models substantially outperform open-source counterparts on average solving 3×3× more main problems and 2×2× more subproblems which we attribute to larger and more diverse training data, proprietary fine-tuning, and stronger alignment. Domain-specific fine-tuning of open-source models on curated scientific data is a promising direction to close this gap. 6.3 Numerical Precision Figure 5 reports the distribution of output deviations from target values (binned from <10−10<10^-10 to >101>10^1). MOSAIC consistently reduces large deviations, producing outputs that are both more precise and better aligned with the underlying problem structure. In scientific and engineering contexts, small numerical errors accumulate and can substantially affect downstream tasks. The improvement is a direct consequence of the student-teacher structure, which ensures domain specific guidance is provided through structured templates to help the Rationale Agent refine it’s plan before code generation. 7 Conclusion Scientific code generation is inherently more challenging than general-purpose coding, as there are no I/O test cases for validation, and generating such test cases itself requires solving the underlying problem. Instead of relying entirely on execution-based feedback for verifying algorithmic correctness, our solution MOSAIC adopts a student–teacher distillation framework to guide code generation through domain-specific reasoning. Extensive experiments and ablation studies on the SciCode benchmark demonstrate the effectiveness and robustness of our approach, achieving up to 24% higher accuracy even with lightweight models. Overall, our results highlight the practical utility of MOSAIC and provide a strong foundation for future research in scientific code generation. References Austin et al. (2021) J. Austin, A. Odena, M. I. Nye, M. Bosma, H. Michalewski, D. Dohan, E. Jiang, C. Cai, M. Terry, Q. V. Le, and C. Sutton. 2021. Program synthesis with large language models. CoRR, abs/2108.07732. Banerjee et al. (2024) S. Banerjee, A. Agarwal, and S. Singla. 2024. Llms will always hallucinate, and we need to live with this. arXiv preprint arXiv:2409.05746. Ben Allal et al. (2023) L. Ben Allal, R. Li, D. Kocetkov, C. Mou, C. Akiki, C. M. Ferrandis, N. Muennighoff, M. Mishra, A. Gu, M. Dey, and 1 others. 2023. Santacoder: Don’t reach for the stars! arXiv preprint arXiv:2301.03988. Besta et al. (2023) M. Besta and 1 others. 2023. Graph of thoughts: Solving elaborate problems with large language models. arXiv preprint arXiv:2308.09687. Chen et al. (2021) M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. O. Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman, and 1 others. 2021. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374. Hendrycks et al. (2021) D. Hendrycks, S. Basart, S. Kadavath, M. Mazeika, A. Arora, E. Guo, C. Burns, S. Puranik, H. He, D. Song, and J. Steinhardt. 2021. Measuring coding challenge competence with apps. In NeurIPS. Hinton et al. (2015) G. Hinton, O. Vinyals, and J. Dean. 2015. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531. Huang et al. (2024) D. Huang, J. M. Zhang, M. Luck, Q. Bu, Y. Qing, and H. Cui. 2024. Agentcoder: Multi-agent-based code generation with iterative testing and optimisation. arXiv preprint arXiv:2312.13010. Islam et al. (2024) M. A. Islam, M. E. Ali, and M. R. Parvez. 2024. Mapcoder: Multi-agent code generation for competitive problem solving. arXiv preprint arXiv:2405.11403. Islam et al. (2025) M. A. Islam, M. E. Ali, and M. R. Parvez. 2025. Codesim: Multi-agent code generation and problem solving through simulation-driven planning and debugging. arXiv preprint arXiv:2502.05664. Jimenez et al. (2024) C. Jimenez and 1 others. 2024. Swe-bench: Can language models resolve real-world github issues? In Proc. ICLR. LangChain (2024) LangChain. 2024. Langgraph: Cyclical graphs for agent runtimes. Blog post. Li et al. (2023) R. Li, L. Ben Allal, Y. Zi, N. Muennighoff, D. Kocetkov, C. Mou, M. Marone, C. Akiki, J. Li, J. Chim, and 1 others. 2023. Starcoder: May the source be with you! arXiv preprint arXiv:2305.06161. Mallick et al. (2024) T. Mallick, O. Yildiz, D. Lenz, and T. Peterka. 2024. Chatvis: Automating scientific visualization with a large language model. In Proc. SC24-W: Workshops of the International Conference for High Performance Computing, Networking, Storage and Analysis, pages 49–55. IEEE. Microsoft Research AI4Science and Microsoft Azure Quantum (2023) Microsoft Research AI4Science and Microsoft Azure Quantum. 2023. The impact of large language models on scientific discovery: a preliminary study using gpt-4. arXiv preprint arXiv:2311.07361. Nijkamp et al. (2023) E. Nijkamp, B. Pang, H. Hayashi, L. Tu, H. Wang, Y. Zhou, S. Savarese, and C. Xiong. 2023. Codegen: An open large language model for code with multi-turn program synthesis. arXiv preprint arXiv:2203.13474. Schick et al. (2023) Timo Schick, Jane Dwivedi-Yu, Roberto Dessì, Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. 2023. Toolformer: Language models can teach themselves to use tools. In Advances in Neural Information Processing Systems. Shinn et al. (2023) N. Shinn, F. Cassano, A. Gopinath, K. R. Narasimhan, and S. Yao. 2023. Reflexion: Language agents with verbal reinforcement learning. In Advances in Neural Information Processing Systems. Tian et al. (2024) M. Tian, L. Gao, S. D. Zhang, X. Chen, C. Fan, X. Guo, R. Haas, P. Ji, K. Krongchon, Y. Li, S. Liu, D. Luo, Y. Ma, H. Tong, K. Trinh, C. Tian, Z. Wang, B. Wu, Y. Xiong, and 11 others. 2024. Scicode: A research coding benchmark curated by scientists. arXiv preprint arXiv:2407.13168. Wang et al. (2023a) H. Wang, T. Fu, Y. Du, W. Gao, K. Huang, Z. Liu, P. Chandak, S. Liu, P. Van Katwyk, A. Deac, A. Anandkumar, K. J. Bergen, C. P. Gomes, S. Ho, P. Kohli, J. Lasenby, J. Leskovec, T.-Y. Liu, A. K. Manrai, and 11 others. 2023a. Scientific discovery in the age of artificial intelligence. Nature, 620:47–60. Wang et al. (2023b) X. Wang, J. Wei, D. Schuurmans, Q. Le, E. Chi, S. Narang, A. Chowdhery, and D. Zhou. 2023b. Self-consistency improves chain of thought reasoning in language models. arXiv preprint arXiv:2203.11171. Wei et al. (2022) J. Wei, X. Wang, D. Schuurmans, M. Bosma, F. Xia, E. Chi, Q. V. Le, D. Zhou, and 1 others. 2022. Chain-of-thought prompting elicits reasoning in large language models. In Advances in Neural Information Processing Systems, volume 35, pages 24824–24837. Wu et al. (2023a) Q. Wu and 1 others. 2023a. Autogen: Enabling next-gen llm applications via multi-agent conversation. arXiv preprint arXiv:2308.08155. Wu et al. (2023b) Y. Wu and 1 others. 2023b. Mathchat: Converse to tackle challenging math problems with llm agents. arXiv preprint arXiv:2306.01337. Xu et al. (2024) Y. Xu, W. Li, P. Vaezipoor, S. Sanner, and E. B. Khalil. 2024. Llms and the abstraction and reasoning corpus: Successes, failures, and the importance of object-based representations. arXiv preprint arXiv:2305.18354. Zhang et al. (2024) Y. Zhang and 1 others. 2024. A comprehensive survey of scientific large language models and their applications in scientific discovery. arXiv preprint. Zhang et al. (2025) Z. Zhang, C. Wang, Y. Wang, E. Shi, Y. Ma, W. Zhong, J. Chen, M. Mao, and Z. Zheng. 2025. Llm hallucinations in practical code generation: Phenomena, mechanism, and mitigation. Proc. ACM Softw. Eng., 2(ISSTA):1–23. Appendix A Appendix In the Appendix, we illustrate the overall format of the SciCode benchmark input prompt (Figure 8), present the code generated by the SciCode baseline and MOSAIC side by side (Figure 7), and provide the benchmark’s test-suite snippet and error traceback (Figure 6). We also provide the complete set of prompt templates used in MOSAIC (Figures 9, 13). We also discuss some key limitations of our work. A.1 Limitations MOSAIC has two main limitations. First, it relies on the domain labels provided by SciCode rather than inferring them automatically. When we tested LLM-based domain bucketing using domain-specific keywords, the model frequently assigned problems to the wrong domain, leading to a performance drop of around 10 to 12%. Second, since MOSAIC is training-free and LLM-agnostic, its performance is bounded by the capabilities of the underlying backbone. As shown in Main paper Table 4, open-source models lag significantly behind closed-source ones on SciCode. Domain-specific fine-tuning of open-source models and heterogeneous agent configurations, where different models specialize in different roles, are promising directions for future work. A.2 Code Outputs Each problem in SciCode may contain several subproblems, and a problem is considered complete only when all subproblems are solved without errors. Every subproblem includes a step description, a set of inputs, an expected output type, and a function header that the agent must follow precisely. A central challenge of SciCode is the absence of sample inputs and outputs for checking intermediate correctness; the agent has access only to domain-level validation examples that differ from the problem in the input prompt (Figure 8). The SciCode test suite calls ket(2, [1,1]) (Figure 6), intended to construct the tensor product state |1⟩⊗|1⟩ 1 1 , whose expected output is the four-dimensional column vector [0,0,0,1]T[0,0,0,1]^T. The baseline does not distinguish between the case where dim is an integer and args is a list, treating the list as a NumPy fancy indexing operation and producing a vector of shape (2,1)(2,1) instead of (4,1)(4,1). NumPy cannot broadcast these shapes for element-wise comparison, causing a runtime error. This is an algorithmic error rather than a syntactic one. MOSAIC instead constructs each basis vector separately and computes their Kronecker product, producing the correct shape (4,1)(4,1) that satisfies the test without error. ⬇ from parse.parse import process_hdf5_to_tuple targets = process_hdf5_to_tuple(’11.1’, 3) target = targets[0] assert np.allclose(ket(2, 0), target) target = targets[1] assert np.allclose(ket(2, [1,1]), target) target = targets[2] assert np.allclose(ket([2,3], [0,1]), target) ⬇ Traceback (most recent call last): File "main.py", line 42, in <module> assert np.allclose(ket(2, [1,1]), target) ... ValueError: operands could not be broadcast together with shapes (2,1) (4,1) (a) Test suite snippet used to evaluate correctness. (b) Python traceback when running the SciCode baseline. Figure 6: SciCode evaluation: test suite call and the resulting baseline error. ⬇ def ket(dim, args): ’Input: dim: int or list, dimension of the ket args: int or list, the i-th basis vector Output: out: dim dimensional array of float, the matrix representation of the ket ’ if isinstance(dim, int): out = np.zeros(dim) out[args] = 1.0 else: basis_vectors = [] for d, j in zip(dim, args): vec = np.zeros(d) vec[j] = 1.0 basis_vectors.append(vec) out = basis_vectors[0] for vec in basis_vectors[1:]: out = np.kron(out, vec) return out.reshape(-1, 1) ⬇ def ket(dim, args): ’Input: dim: int or list, dimension of the ket args: int or list, the i-th basis vector Output: out: dim dimensional array of float, the matrix representation of the ket ’ if isinstance(dim, int) and isinstance(args, int): out = np.zeros(dim) out[args] = 1.0 return out.reshape(-1, 1) elif isinstance(dim, int) and isinstance( args, (list, tuple)): basis_vectors = [] for j in args: vec = np.zeros(dim) vec[j] = 1.0 basis_vectors.append(vec) out = basis_vectors[0] for vec in basis_vectors[1:]: out = np.kron(out, vec) return out.reshape(-1, 1) else: basis_vectors = [] for d, j in zip(dim, args): vec = np.zeros(d) vec[j] = 1.0 basis_vectors.append(vec) out = basis_vectors[0] for vec in basis_vectors[1:]: out = np.kron(out, vec) return out.reshape(-1, 1) (a) SciCode baseline Tian et al. (2024): fails to handle the case where dim is int and args is a list. (b) MOSAIC (ours): correctly computes the Kronecker product for all input combinations. Figure 7: Side-by-side comparison of the SciCode baseline and MOSAIC outputs for the ket function. Input prompt Main problem: Consider sending a bipartite maximally entangled state where both parties are encoded by m-rail encoding through m uses of the generalized amplitude damping channel γ1,N1A_ _1,N_1 to receiver 1 and m uses of another generalized amplitude damping channel γ2,N2A_ _2,N_2 to receiver 2. Each of the two receivers measures whether the m qubits are in the one-particle sector, i.e., whether there are m−1m-1 zeros and one excitation. If so, they keep the state. Otherwise, they discard the state. They then perform the hashing protocol on the post-selected state. Calculate the rate of entanglement that can be generated per channel use in this set-up. Step description: Given j and d, write a function that returns a standard basis vector |j⟩ j in d-dimensional space. If d is given as an int and j is given as a list [j1,j2,…,jn][j_1,j_2,…,j_n], return the tensor product |j1⟩|j2⟩⋯|jn⟩ j_1 j_2 ·s j_n of d-dimensional basis vectors. If d is also a list [d1,d2,…,dn][d_1,d_2,…,d_n], return the tensor product of d1,d2,…,dnd_1,d_2,…,d_n-dimensional basis vectors. Inputs: • rails: int, number of rails • γ1 _1: float, damping parameter of the first channel • N1N_1: float, thermal parameter of the first channel • γ2 _2: float, damping parameter of the second channel • N2N_2: float, thermal parameter of the second channel Output: • float, the achievable rate of our protocol Function header: def ket(dim): ’Input: dim: int or list, dimension of the ket args: int or list, the i-th basis vector Output: out: dim dimensional array of float ’ Figure 8: Input prompt containing the main problem, step description, and function header. A.3 Prompt Templates Figure 9: Prompt template for the Self-Reflection Agent, which analyzes ground-truth code from the validation set to identify domain-specific patterns and prepares gold-standard pseudocode as few-shot examples for the Rationale Agent. Figure 10: Prompt template for the Rationale Agent. Guided by few-shot examples from the Self-Reflection Agent, it converts the subproblem into a detailed rationale for the Coding Agent. Figure 11: Prompt template for the Coding Agent, which converts each pseudocode step into executable code. Figure 12: Prompt template for the Debugger Agent, which identifies and corrects syntactic errors to ensure executability of the generated code. Figure 13: A sample of code generated by MOSAIC, alongside the test suite evaluation code used to verify correctness.