Paper deep dive
MetaInfer: A Knowledge Only LLM Inference Engine Generator SKILL Toolbox
Zhenwen Miao, Honglin Wang, Mingheng Mi
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 94%
Last extracted: 7/15/2026, 8:13:11 AM
Summary
The paper introduces MetaInfer, an LLM-driven multi-agent system that automatically generates compact, customized inference frameworks from runtime constraints without relying on existing source code. By utilizing a Contract Knowledge Base (CKB) and a zero-reference generation approach, it avoids the complexity of general-purpose frameworks like vLLM and SGLang. The system incorporates a knowledge-base evolution loop to handle new models and hardware platforms, enabling the continuous consolidation of domain knowledge into reusable contracts. Evaluations demonstrate its ability to produce runnable, optimized inference engines tailored to specific constraints.
Entities (15)
Relation Signals (12)
MetaInfer → uses → Contract Knowledge Base (CKB)
confidence 97% · The system first parses these constraints and checks whether the contract knowledge base (CKB) covers the target model and platform.
MetaInfer → employs → Zero-reference generation
confidence 96% · MetaInfer therefore shifts the operative knowledge source from source-code implementations to contract knowledge... zero-reference denotes an operational constraint during generation
Implementation Agent → collaborateswith → Specification-review Agent
confidence 95% · These three roles are deliberately separated: the implementer does not declare success, the reviewer does not replace testing, and the verifier does not participate in implementation.
Specification-review Agent → collaborateswith → Verification Agent
confidence 95% · These three roles are deliberately separated: the implementer does not declare success, the reviewer does not replace testing, and the verifier does not participate in implementation.
MetaInfer → employs → Knowledge-base evolution
confidence 95% · If information is missing, the system does not force generation. Instead, it treats the missing items as knowledge to be learned. Knowledge-base evolution is performed by independent agents.
MetaInfer → testedon → Qwen3.6-27B
confidence 95% · using Qwen3.6-27B and the Z200 platform as target scenarios with insufficient knowledge coverage
VibeServe → dependson →
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:As LLM technology advances, the space of model families, compute hardware, quantization schemes, parallelization strategies, and specialized optimization kernels continues to expand, sharply increasing the code complexity and maintenance cost of general-purpose inference frameworks. Conventional software engineering uses multiple layers of abstraction to support diverse application scenarios, but these abstractions also increase system complexity and may introduce additional performance overhead. This paper presents metainfer, an 'LLM-as-Compiler' approach in which users specify only the runtime constraints of an inference program. An LLM-driven multi-agent collaboration system, coupled with a contract knowledge base, then automatically generates a compact customized inference framework that satisfies these constraints. We evaluate metainfer from three perspectives: the effect of source-code reference, the runtime behavior and performance profile of engines generated under the zero-reference constraint on CKB-covered targets, and knowledge-base evolution for new model and platform scenarios. The results show that metainfer organizes generation constraints, validation feedback, and knowledge consolidation into a continuous closed loop, enabling runnable customized inference solutions to be generated from explicit knowledge. The code is publicly available at this https URL.
Tags
Links
- Source: https://arxiv.org/abs/2607.12875v1
- Canonical: https://arxiv.org/abs/2607.12875v1
PDF not stored locally. Use the link above to view on the source site.
Full Text
80,317 characters extracted from source content.
Expand or collapse full text
MetaInfer: A Knowledge Only LLM Inference Engine Generator SKILL Toolbox Zhenwen Miao, Honglin Wang, Mingheng Mi Abstract As LLM technology advances, the space of model families, compute hardware, quantization schemes, parallelization strategies, and specialized optimization kernels continues to expand, sharply increasing the code complexity and maintenance cost of general-purpose inference frameworks. Conventional software engineering uses multiple layers of abstraction to support diverse application scenarios, but these abstractions also increase system complexity and may introduce additional performance overhead. This paper presents MetaInfer, an “LLM-as-Compiler” approach in which users specify only the runtime constraints of an inference program. An LLM-driven multi-agent collaboration system, coupled with a contract knowledge base, then automatically generates a compact customized inference framework that satisfies these constraints. We evaluate MetaInfer from three perspectives: the effect of source-code reference, the runtime behavior and performance profile of engines generated under the zero-reference constraint on CKB-covered targets, and knowledge-base evolution for new model and platform scenarios. The results show that MetaInfer organizes generation constraints, validation feedback, and knowledge consolidation into a continuous closed loop, enabling runnable customized inference solutions to be generated from explicit knowledge. The code is publicly available at https://github.com/MetaInfer/MetaInfer. 1 Introduction Large language model (LLM) inference services have become a core component of AI infrastructure. For high-throughput and low-latency serving, Orca [20] introduced iteration-level scheduling for generative models, vLLM [8] improved KV cache management through PagedAttention and continuous batching, and SGLang [21] organized efficient execution for structured language model programs. These systems constitute important baselines for current research and engineering practice in inference serving. However, the pursuit of generality in general-purpose frameworks also creates structural problems. To support diverse model architectures and hardware combinations, a framework must expose a large number of configuration options that cover many usage scenarios, whereas the large framework collapses into only one specific execution path for any concrete inference task. Since only a small subset of code paths is active at runtime, a natural question arises: why not generate a compact, customized inference framework for the specific runtime environment? Such customized systems could remove redundant performance overhead caused by multiple abstraction layers, apply optimizations tailored to workload characteristics, and substantially reduce the adaptation cost for heterogeneous hardware platforms, including edge computing and embodied-intelligence settings. One natural route to such customized framework generation is to use code-generation agents, so that users can describe requirements in natural language and flexibly construct compact inference engines. Self-Refine uses model feedback for iterative rewriting [9], Reflexion improves agent decisions through verbal feedback [15], and SWE-agent improves software-engineering task execution from the perspective of an Agent–Computer Interface [19]. Survey work also identifies such systems as an important direction for LLM autonomous agents [17]. In fact, using agents to optimize LLM inference efficiency has made notable progress in several areas, especially in GPU kernel writing and optimization, where KernelBench [12], EvoEngineer [5], and KernelFoundry [18] demonstrate the effectiveness of agent-driven evolutionary search. However, most of these studies focus on closed operator-level optimization problems, and exploration of framework-level generation and optimization for inference engines remains limited. A few recent efforts move in this direction, but they still do not achieve the goal of building a customized inference engine from scratch. VibeServe [7] is the first attempt to build a complete LLM inference service system with agents, but its pipeline explicitly depends on the source code of vLLM and SGLang as references, so the generated systems remain structurally constrained by existing implementations. Glia [6] applies agent-based optimization to existing codebases rather than constructing a new engine from scratch. ForgeTrain [11] uses an Agent loop for LLM pretraining framework generation, but its current public repository mainly provides the generated training engine and project description rather than a verifiable paper or a complete harness implementation. Generating directly from existing framework source code faces three obstacles. First, the multiple abstraction layers and conditional branches in general-purpose source code can easily leak into the customized framework, weakening the intended simplification. Second, explicit dependence on source-code reference confines the search space to established implementation paradigms and suppresses architectural innovation. Third, many frontier advances are first released as technical reports, release notes, or configuration files rather than mature reference implementations. Qwen3.6-27B is one such case: its official technical report describes a shift from the dense-attention baseline of Qwen3-8B to a hybrid attention architecture, introducing GatedDeltaNet (GDN) states, MRoPE, and RMSNorm variants as implementation constraints [13]. If generation depends only on existing framework source code, such changes are difficult to translate directly into compact execution paths for new platforms. Therefore, true on-demand generation should move away from concrete source-code reference and instead allow agents to construct systems from distilled domain knowledge, such as computational semantics, optimization principles, and parallelization strategies. This shift can remove inherited redundancy and create a direct path for new results that do not yet have reference implementations. Based on these observations, this paper first uses comparative experiments to analyze how source-code reference affects the process by which agents build customized inference frameworks, quantifying its role in reducing search cost and error-localization cost. We then evaluate the real runtime behavior of engines generated under the zero-reference constraint on CKB-covered targets on the K100 platform and compare them with vLLM under matched workloads and graph-capture states. Finally, using Qwen3.6-27B and the Z200 platform as target scenarios with insufficient knowledge coverage, we construct a knowledge-base evolution loop that proceeds from controlled exploration and knowledge consolidation to revalidation after source-code reference is disabled. This loop tests whether the system can convert information about new models and platforms into reusable contract knowledge for subsequent generation. This work offers a new path for accelerating the transfer of research advances into production environments and for efficient adaptation across multiple platforms. 2 Related Work The central problem in LLM inference-serving systems is end-to-end optimization across dynamic request batching, KV cache management, memory and bandwidth constraints, and the phase differences between prefill and decode. Orca [20] applies iteration-level scheduling to generative model serving, and vLLM [8] improves KV cache management through PagedAttention. DeepSpeed-Inference [2] and FlexGen [14] improve LLM inference from the perspectives of parallel execution, memory management, and offloading. Sarathi-Serve [1] and DistServe [22] further optimize the throughput–latency trade-off around the different characteristics of prefill and decode, while SpecInfer [10] accelerates generation through speculative verification. These systems show that the performance of an inference framework comes from the joint design of scheduling, caching, operators, parallelism, and hardware constraints, rather than from the implementation of a single model forward function alone. VibeServe [7], proposed by the University of Washington, is a representative work on using LLMs to generate customized inference frameworks. It introduces a multi-agent loop in which the outer loop plans system-design search, while the inner loop implements candidates and verifies correctness and performance. This design can automatically generate serving systems for specific scenarios end to end. Under standard deployment settings, it is comparable to a highly optimized vLLM system, and under six classes of non-standard scenarios, including non-standard model architectures, prior workload knowledge, and hardware-specific optimization, it significantly outperforms general-purpose systems. However, VibeServe includes the source code of vLLM and SGLang as part of its reference knowledge base. It therefore does not realize inference-framework generation from natural-language requirements and knowledge alone. This use of source-code reference weakens the agent system’s ability to handle entirely new models. Glia [6], proposed by MIT, follows a human-expert-style research workflow. Through collaboration between a Researcher agent and a Supervisor agent, it iteratively runs experiments, analyzes bottlenecks, and generates and optimizes algorithms through white-box reasoning and hypothesis-driven exploration. Glia focuses on local strategy optimization on top of existing inference frameworks such as vLLM, including request routing, batch scheduling, and autoscaling. It can obtain optimization effects in two hours that are comparable to two weeks of work by human experts, and the generated designs are interpretable and adaptable to workload changes. However, its scope is limited to tuning parameters and strategies in existing systems. It does not reduce the complexity of existing inference frameworks or address the problem of generating compact, specialized inference frameworks for different hardware platforms. In addition, the ForgeTrain project released by OpenBMB [11] applies an Agent loop to LLM pretraining framework generation. Its official repository reports that the public version already includes training engines for MiniCPM4-0.5B and MiniCPM4-8B, and achieves an approximately 10% MFU improvement over the Megatron-LM [16] baseline on the H100 platform. However, the project primarily targets training rather than inference. More importantly, the current public repository marks harness scaffolding as coming soon. It is therefore more appropriate to cite ForgeTrain as a public software project than as a peer-reviewed paper supporting the problem of framework-level inference-engine generation. Compared with the above work, MetaInfer has three distinguishing features: • It generates compact inference frameworks from scratch using a knowledge base composed of natural-language descriptions and code-example fragments. • It has a self-learning capability: through knowledge-base evolution, it can update the knowledge base and generate inference frameworks for new models. • Its code is fully open source and is presented in the standard Agent SKILL format. 3 Method 3.1 MetaInfer inference-framework generation workflow As discussed in the previous section, prior work has shown that LLM agents can generate inference-serving systems for specific scenarios. MetaInfer therefore treats code generation as a baseline capability rather than the central question. Given model, hardware, and deployment constraints, the system must organize the relevant knowledge, generate code, review the implementation, and complete independent verification. This workflow provides an executable basis for zero-reference generation and knowledge-base evolution. The input to MetaInfer is a set of deployment-oriented constraints, including the target model, weight path, parallelization mode, hardware platform, runtime mode, and performance targets. The system first parses these constraints and checks whether the contract knowledge base (CKB) covers the target model and platform. The CKB stores reusable contract files, supporting technical documents, and accumulated verification and optimization experience. If coverage is sufficient, the system enters the regular generation workflow. If key information is missing, such as the model structure, weight mapping, parallelization strategy, or platform backend, it switches to knowledge-base evolution. This entry mechanism makes MetaInfer a generation system that selects its execution path according to knowledge coverage, rather than a one-shot code generator. Figure 1 presents the overall architecture of MetaInfer. The system organizes three loops around the CKB. Under normal conditions, agents perform zero-reference generation from the CKB and are evaluated by test contracts. When knowledge coverage is insufficient, the system enters knowledge-base evolution and consolidates information about new models or platforms back into the CKB. When generation and verification produce effective fixes or performance experience, the system feeds this experience back as reusable knowledge. Figure 1: Overall architecture of MetaInfer. The system organizes three loops around the CKB. The zero-reference generation loop generates and verifies a single-path inference engine from user constraints; the knowledge-base evolution loop supplements knowledge about new models or platforms when coverage is insufficient; and the experience-feedback loop writes reusable verification and optimization experience back into the CKB. In the regular generation workflow, MetaInfer converts user constraints into interface contracts and test contracts, then uses multi-agent collaboration to build the inference framework. A test contract specifies the required behavior and acceptance criteria for a generation stage; fixed test scripts execute the contract, and the resulting test gate determines whether the workflow advances. The implementation agent generates code from the CKB and the current stage objective. The specification-review agent checks whether the code conforms to the required interfaces, data flow, and runtime constraints. The verification agent runs the fixed test scripts and judges success from execution results. These three roles are deliberately separated: the implementer does not declare success, the reviewer does not replace testing, and the verifier does not participate in implementation. This division of labor reduces the risk of a self-confirming loop between implementation, explanation, and verification. It also discourages hard-coded outputs and false passes, while preserving a traceable, reviewable, and reversible engineering workflow for complex system generation. The generation process is staged, but the stages are not manual development tasks. They define the internal construction order for the agents. The system proceeds from low-level execution capability to the model execution path, weight loading, scheduling logic, and end-to-end engine integration. After each stage, the corresponding test contracts determine whether the current artifact meets the expected behavior. If a test fails, the system feeds the failure evidence back into the implementation workflow. If the test passes, the system advances to the next stage. The final artifact is a single-path inference engine for the target constraints, rather than a runtime path selected from a general-purpose framework through many configuration options. The acceptance criteria in MetaInfer address both correctness and authenticity. The system checks module-level unit tests and end-to-end outputs, and also verifies whether the generated code comes from the current working directory, whether it loads real weights, whether execution follows the target parallelization path, and whether hard-coded outputs or external-framework bypasses are present. At the end-to-end stage, the system also collects evidence for decoding consistency, runtime performance, and resource usage. A generated artifact is accepted only when all test contracts are satisfied. Section 3.1 thus defines the basic execution loop of MetaInfer: constraint input, coverage detection, code generation, contract review, independent verification, and failure feedback. This loop establishes the system’s ability to generate a runnable inference framework. The rest of this section focuses on two mechanisms that distinguish MetaInfer from this baseline workflow. First, generation does not directly consult source code from open-source inference frameworks such as vLLM or SGLang, but reconstructs the target framework from the CKB. Second, when the CKB does not cover a new model or platform sufficiently, MetaInfer supplements the missing knowledge through knowledge-base evolution and then reruns generation and verification after consolidation. The following subsections introduce these two mechanisms. 3.2 Zero-reference generation from the contract knowledge base In this workflow, the key issue is not only whether agents can implement code, but also which knowledge source drives generation. Work such as VibeServe [7] has shown that including the source code of mature systems such as vLLM and SGLang as reference knowledge can support automatic generation and verification of inference frameworks. However, source-code reference also couples generation to existing framework structures. Agents can inherit module boundaries, control flow, and abstraction levels from reference implementations, which weakens the simplicity expected from a customized framework and limits adaptation to models or platforms without mature open-source implementations. MetaInfer therefore shifts the operative knowledge source from source-code implementations to contract knowledge. During generation, agents reconstruct the target framework from abstracted interfaces, data flow, and execution constraints, rather than opening and modifying existing inference-framework source code. In this paper, zero-reference denotes an operational constraint during generation, not a claim about the pretraining data of the underlying LLM. The training corpora of modern LLMs may contain code from vLLM, SGLang, and other open-source projects, so we do not claim that the model has never been exposed to these implementations. The zero-reference constraint in MetaInfer means that, during a concrete inference-framework generation run, the agent does not actively retrieve, open, or copy the source code of open-source inference frameworks. It may use model configurations, hardware information, GPU programming principles, and contracts already consolidated in the CKB. It may also call public high-performance low-level operators or operator wrappers such as FlashAttention [3, 4]. However, it does not directly reuse model implementations, scheduling logic, weight-loading paths, or KV cache management code from existing frameworks. The zero-reference constraint therefore does not deny the latent experience of LLMs. Instead, it prevents generation from degenerating into translation or assembly of reference implementations. To support this constraint, MetaInfer organizes key inference-system knowledge as contracts. A contract does not store complete framework source code. Instead, it describes the interfaces, tensor shapes, data types, device placement, state transitions, and error boundaries that a component must satisfy on the target execution path. For agents, these contracts serve as design boundaries for the executable system. They specify what must hold, not which existing code should be copied. For review and verification, contracts also provide checkable criteria: the specification-review agent can judge whether an implementation deviates from the design intent, and the verification agent can confirm through test contracts whether these constraints hold during real execution. Figure 2 shows the zero-reference generation branch. Its forward path is driven jointly by target constraints and contract knowledge, producing the target engine through task decomposition, code generation, specification review, and test acceptance. Its backward path feeds review failures, test failures, and effective fixes back to the implementation workflow or the CKB, thereby avoiding ad hoc repair based on external framework source code. Figure 2: Zero-reference generation branch. Without actively consulting the source code of open-source inference frameworks, the forward path uses contract knowledge to drive code generation, specification review, and test acceptance. The backward path feeds failure reports, repair experience, and knowledge gaps back into the implementation workflow or the CKB. This design distinguishes MetaInfer from methods driven by source-code reference. Generation based on source-code reference is closer to cutting or rewriting a target path from a mature system, whereas contract-knowledge-driven generation abstracts transferable design principles from mature systems and recombines them under the target constraints. The former reduces implementation difficulty. The latter decouples generation from any specific framework implementation, making it easier to form a compact single-path inference engine and leaving room for knowledge-base evolution. When the CKB already covers the target model and platform, MetaInfer can complete generation directly under the zero-reference constraint. When coverage is insufficient, the system enters knowledge-base evolution, converts missing information into new contract knowledge, and then returns to the zero-reference generation path. 3.3 Knowledge-base evolution for new models and platforms Zero-reference generation depends on one premise: the CKB must contain enough model, platform, and execution constraints to support the target scenario. The rapid evolution of LLM architectures and hardware platforms makes this premise difficult to maintain statically. New model structures may change attention mechanisms, normalization methods, weight-naming rules, or expert-routing logic. New hardware platforms may introduce different communication primitives, operator backends, and memory constraints. If the CKB can only be maintained manually, the system will quickly degrade into a generation tool for a small number of known configurations. MetaInfer therefore introduces coverage detection before regular generation and starts knowledge-base evolution when coverage is insufficient. Coverage detection determines whether the current CKB can support a zero-reference generation run. The system checks whether the target model specification is explicit, whether execution contracts exist for key modules, whether weight mapping and parallel partitioning rules are available, and whether the target platform has constraints for communication and operator backends. If these conditions are met, MetaInfer enters the zero-reference generation path described above. If information is missing, the system does not force generation. Instead, it treats the missing items as knowledge to be learned. Knowledge-base evolution is performed by independent agents. The exploration agent first collects external information about the target model or platform, such as model configurations, technical reports, hardware documents, and public descriptions. When necessary, the system also permits open-source implementations to be consulted in a controlled stage. The purpose is not to make final generation depend on source code, but to extract architectural differences, interface rules, and implementation constraints that have not yet been expressed in the CKB. The implementation agent then attempts to generate the target inference framework from the newly added information, and the verification agent uses test contracts to judge whether the generated artifact is usable. If generation and verification succeed, the knowledge-consolidation agent writes the effective information from the current round back into the CKB, converting one-time context into reusable contract knowledge. Figure 3 shows the knowledge-base evolution branch. Its forward path starts from a coverage gap and obtains a usable implementation through external exploration, candidate generation, and verification. Its backward path uses verification results to decide whether to consolidate knowledge, revalidate after source-code reference is disabled, or send the failure cause to the failure-analysis agent for further incremental exploration. Figure 3: Knowledge-base evolution branch. When knowledge coverage is insufficient, the system permits controlled exploration of external information and obtains a usable implementation through candidate generation and verification. A target model or platform is considered absorbed by the CKB only after effective information has been consolidated and revalidation passes with source-code reference disabled. The key criterion for knowledge-base evolution is not whether the system can generate after consulting an open-source implementation, but whether it can generate again after knowledge consolidation with source-code reference disabled. Open-source-assisted success is therefore only an intermediate state. The system must disable source-code reference and rerun generation and verification from the updated CKB. Only when revalidation passes without source-code reference does MetaInfer consider the model or platform absorbed by the CKB. If revalidation fails, the failure-analysis agent records the missing knowledge or incorrect assumptions and redirects the system to incremental exploration and consolidation. Through this mechanism, the unit of extension in MetaInfer is no longer a copied framework implementation, but a set of reusable knowledge constraints. Differences introduced by new models and platforms are distilled into model specifications, execution contracts, platform constraints, or failure modes, and are then invoked as ordinary knowledge in subsequent generation tasks. This allows MetaInfer to expand the range of supported models and hardware while preserving the zero-reference constraint. 4 Experiments 4.1 Experimental setup and evaluation metrics The experiments evaluate whether the generation loop of MetaInfer works in practice, specifically whether the system can connect the CKB, specification review, verification feedback, and knowledge-base evolution during inference-engine generation. We address three progressive questions. First, how much does source-code reference help agents build inference engines? Second, when the target is already covered by the CKB, can an engine generated under the zero-reference constraint run stably on a real platform and exhibit the performance and resource characteristics of a customized single-path inference engine? Third, when the target model exceeds current knowledge coverage, can knowledge-base evolution distill new information into reusable knowledge constraints through multiple rounds of exploration, verification, and consolidation? To answer these questions, we divide the experiments into three groups. The first group is a controlled comparison of construction processes with source-code reference enabled and disabled. The target task is to generate a Qwen3-8B inference engine, and the goal is to quantify how source-code reference affects generation difficulty. The experiment records the construction process under the two settings for source-code reference. The evaluation metrics include build time, number of participating agents, number of tool calls, scale of read/write and edit operations, token consumption, and whether the source-access constraint is violated. This group focuses on the generation process itself and explains why knowledge extraction is necessary. The second group profiles the runtime behavior and performance of an engine generated under the zero-reference constraint on the K100 platform. It also uses Qwen3-8B as the target model and evaluates runtime behavior, resource usage, and relative baseline performance on a real DCU platform when the target is already covered by the CKB. The experiment uses TP=4, BF16 precision, and greedy decoding. The main performance workload is a 3×33× 3 matrix: prompt lengths are 11, 256, and 1021 tokens, and generation lengths are 256, 1024, and 2048 tokens. The evaluation metrics include time to first token (TTTFTT_TTFT), per-token decoding latency (TTPOTT_TPOT), end-to-end throughput, decode throughput, and per-GPU memory usage. In addition, chunked prefill and multi-batch experiments examine the memory–latency trade-off and batch-throughput scalability, respectively. The third group is a Qwen3.6-27B knowledge-base evolution experiment, which evaluates system extensibility when knowledge coverage is insufficient. Compared with Qwen3-8B, Qwen3.6-27B introduces mixed attention structures, GatedDeltaNet states, MRoPE, RMSNorm variants, and new weight-naming rules, forming a target scenario not sufficiently covered by the CKB. The experiment records the multi-round evolution process from controlled exploration, knowledge consolidation, candidate generation, and component verification to revalidation after source-code reference is disabled. The evaluation metrics include the test pass rate in each round, whether generation succeeds under the CKB-only condition, whether greedy decode tokens match the HF greedy-output trace, bug types and severities, and whether the final PASS_VERIFIED state is reached. After correctness verification, we further report the performance profile of the generated engine. This overview states only the experimental questions, target models, and evaluation metrics. The corresponding subsections describe the specific hardware platforms, backend environments, and platform-related constraints. Together, they form a progressive evidence chain: generation difficulty, runtime and performance profiling of artifacts generated under the zero-reference constraint on CKB-covered targets, and extension through knowledge-base evolution. To keep performance experiments comparable, we fix the model, parallelism, weight precision, and decoding strategy on each platform, and run warmup before formal measurements for each configuration. For framework comparisons, we prioritize fair analysis under the same CUDA Graph state. Metrics that cannot be stably obtained in the platform environment, or that require heuristic estimation, are reported only as auxiliary information and are not used as the basis for the main conclusions. The following subsections report the corresponding results in order. 4.2 Effect of source-code reference on the generation process We first examine how source-code reference affects the process by which agents build an inference engine. The target construction task is a customized Qwen3-8B inference engine with TP=4, running in the K100 DCU environment with four K500SM_AI cards and 65520 MiB memory per card. The experiment compares the same full construction task under the with_ref and no_ref conditions. Both groups cover Phase 1–11, and neither group produces blacklist violations. This experiment focuses on construction-process complexity rather than the runtime performance of the final engine. The analysis therefore emphasizes process metrics such as time, agent-call scale, tool usage, and token consumption. Table 1 reports the main results. In this construction run, the with-reference condition completed the full workflow in approximately 14.6 h, whereas the no-reference condition required approximately 20.5 h, or 1.40 times longer. Under the no-reference condition, the number of participating agents increased from 31 to 51, assistant messages increased from 1,227 to 2,472, and total tool calls increased from 2,054 to 3,291. The difference in token consumption was even more pronounced: total token usage increased from 3.04M to 6.93M, approximately 2.28 times the with-reference condition. Table 1: Comparison of the full construction process with and without source-code reference. Metric with_ref no_ref Build time ∼ 14.6 h ∼ 20.5 h Total agents 31 51 Assistant messages 1,227 2,472 Total tool calls 2,054 3,291 Edit calls 98 404 Bash calls 664 1,456 Total token usage 3.04M 6.93M Blacklist violations 0 0 The tool-call structure shows the same pattern. Under the no-reference condition, Read calls increased from 464 to 921, Edit calls from 98 to 404, and Bash calls from 664 to 1,456. This indicates that, without source-code reference, agents need more retrieval, local modification, and runtime verification to compensate for implementation uncertainty. By contrast, the with-reference condition still required the full planning, implementation, review, and verification process, but existing implementations helped agents determine module boundaries, interface conventions, and key execution paths more quickly, reducing repeated trial and error. The scale of code changes followed the same trend. Under the no-reference condition, the net increase was 8,195 lines, higher than the 6,429 lines in the with-reference condition. Edit deletions and additions reached 3,598 and 4,269 lines, respectively, far above the 194 deletions and 500 additions in the with-reference condition. This shows that no-reference generation did not simply write longer code in one pass. Instead, it involved more structural rewriting and repair during implementation. The main role of source-code reference was therefore not to contribute final code directly, but to reduce the agents’ search cost in interface selection, state management, and error localization. Notably, the recorded usage rate of source-code reference in the with-reference condition was 3/10, rather than continuous dependence on source code. This shows that even partial source-code reference can substantially affect the construction path. Together with the higher time, tool-call, and token costs under the no-reference condition, this result indicates that source-code reference reduces the initial difficulty of inference-engine generation. At the same time, the no-reference construction still covered Phase 1–11, showing that knowledge-driven generation is feasible, although it requires higher exploration and repair costs. This result directly motivates the following experiments: if transferable knowledge from reference implementations can be consolidated into the CKB, it may reduce the difficulty of zero-reference generation after source-code reference is disabled. 4.3 Zero-reference generation results on the K100 platform After confirming that source-code reference substantially reduces construction cost, we evaluate the performance difference between the customized inference engine generated with source-code reference disabled and the current state-of-the-art general-purpose inference framework, vLLM, under the same workloads and runtime configurations. The K100 experiment uses Qwen3-8B as the target model and follows the hardware configuration of the first experiment, with four K500SM_AI cards and 65520 MiB memory per card. We run the TP=4 inference engine generated by MetaInfer and compare it with vLLM under the same CUDA Graph state. All main experiments use Chinese natural-language prompts truncated by the tokenizer to the target token counts, avoiding interference from empty tokens or padding in prefill computation. The main performance experiment uses a 3×33× 3 matrix with prompt length p∈11,256,1021p∈\11,256,1021\ and generation length g∈256,1024,2048g∈\256,1024,2048\. Table 2 reports the full performance profile of the MetaInfer-generated engine on K100, including end-to-end throughput, TTPOTT_TPOT, and TTTFTT_TTFT. Without CUDA Graph, the generated engine maintained an end-to-end throughput of approximately 24–25 tok/s and a TTPOTT_TPOT of approximately 40 ms. With CUDA Graph enabled, throughput increased to approximately 50–54 tok/s, and TTPOTT_TPOT decreased to approximately 18–19 ms. TTTFTT_TTFT mainly varied with prompt length, from approximately 0.1 s for short prompts to approximately 0.34–0.37 s for 1021-token prompts. Table 2: Full 3×33× 3 performance profile of the MetaInfer engine generated under the zero-reference constraint on the K100 DCU platform (Qwen3-8B, TP=4). Values are means ± standard deviations over three runs. p g Eager tok/s Eager TPOT (ms) Eager TTFT (ms) Graph tok/s Graph TPOT (ms) Graph TTFT (ms) 11 256 24.5 ± 0.2 40.5 ± 0.3 106.6 ± 4.5 53.1 ± 0.2 18.4 ± 0.1 120.0 ± 5.1 11 1024 24.6 ± 0.1 40.5 ± 0.1 104.5 ± 1.2 54.4 ± 0.0 18.3 ± 0.0 120.6 ± 12.0 11 2048 24.7 ± 0.1 40.4 ± 0.1 88.0 ± 27.6 54.3 ± 0.1 18.3 ± 0.0 118.8 ± 6.2 256 256 24.4 ± 0.0 40.5 ± 0.0 135.1 ± 3.0 52.7 ± 0.2 18.4 ± 0.0 138.3 ± 3.9 256 1024 24.7 ± 0.1 40.4 ± 0.1 136.2 ± 4.6 53.9 ± 0.1 18.4 ± 0.0 145.7 ± 7.1 256 2048 24.7 ± 0.1 40.4 ± 0.1 132.8 ± 2.6 54.2 ± 0.2 18.4 ± 0.1 143.4 ± 6.3 1021 256 23.8 ± 0.0 40.6 ± 0.1 340.0 ± 3.2 49.8 ± 0.1 18.7 ± 0.0 362.5 ± 6.4 1021 1024 24.6 ± 0.1 40.3 ± 0.1 344.5 ± 5.9 53.0 ± 0.1 18.5 ± 0.0 361.8 ± 6.8 1021 2048 24.7 ± 0.1 40.3 ± 0.1 339.9 ± 4.6 53.4 ± 0.1 18.5 ± 0.0 365.1 ± 4.3 These results show that the inference engine generated under the zero-reference constraint handled real inference workloads on the K100 DCU platform and obtained its main benefit in the decode phase. CUDA Graph consistently improved TTPOTT_TPOT, reducing it from approximately 40 ms to approximately 18–19 ms across all configurations. By contrast, TTTFTT_TTFT did not decrease after graph capture was enabled, indicating that this optimization mainly reduced scheduling overhead during per-token decoding rather than the prefill computation itself. Table 3: Full 3×33× 3 performance profile of vLLM on the K100 DCU platform (Qwen3-8B, TP=4). Values are means ± standard deviations over three runs. p g Eager tok/s Eager TPOT (ms) Eager TTFT (ms) Graph tok/s Graph TPOT (ms) Graph TTFT (ms) 11 256 16.2 ± 0.1 52.6 ± 0.5 N/A† 43.4 ± 0.0 19.7 ± 0.0 N/A† 11 1024 16.1 ± 0.1 52.8 ± 0.3 N/A† 43.3 ± 0.0 19.7 ± 0.0 N/A† 11 2048 16.1 ± 0.1 52.7 ± 0.5 N/A† 43.2 ± 0.0 19.7 ± 0.0 N/A† 256 256 16.4 ± 0.0 52.0 ± 0.1 N/A† 43.1 ± 0.0 19.8 ± 0.0 N/A† 256 1024 16.1 ± 0.1 52.7 ± 0.4 N/A† 43.1 ± 0.0 19.7 ± 0.0 N/A† 256 2048 16.2 ± 0.1 52.5 ± 0.4 N/A† 43.2 ± 0.0 19.7 ± 0.0 N/A† 1021 256 16.1 ± 0.1 53.0 ± 0.4 N/A† 43.1 ± 0.0 19.8 ± 0.0 N/A† 1021 1024 16.3 ± 0.1 52.2 ± 0.3 N/A† 43.1 ± 0.1 19.7 ± 0.0 N/A† 1021 2048 24.4 ± 0.3∗ 34.9 ± 0.4∗ N/A† 64.1 ± 0.0∗ 13.3 ± 0.0∗ N/A† ∗This cell is an anomalous outlier for vLLM under the long-prompt and long-generation setting, and is not used as evidence for a stable trend in the fair comparison below. †In this ROCm environment, vLLM could not stably provide the first-token timestamp, so TTFT is not reported in the main text. Table 3 reports the full vLLM results under the same workloads. Compared with MetaInfer, vLLM achieved a stable throughput of approximately 16 tok/s in Eager mode and approximately 43 tok/s after CUDA Graph was enabled. Using p=11,g=256p=11,g=256 as a representative configuration, MetaInfer was approximately 51% and 22% faster under the two graph-capture states, respectively. The corresponding TTPOTT_TPOT decreased from 52.6 ms in Eager mode to 19.7 ms in CUDA Graph mode, indicating that graph capture also substantially reduced vLLM’s decode scheduling overhead. vLLM’s TTTFTT_TTFT could not be obtained stably from metrics.first_token_time in this ROCm environment. The wall-time heuristic in the original script did not satisfy the definition of first-token latency, so we mark TTFT as N/A in the main text and do not use it for comparison. Memory usage showed a more pronounced system-level difference. Under the same model and TP=4 configuration, MetaInfer used approximately 5.3 GB per GPU, whereas vLLM pre-allocated approximately 56 GB per GPU with gpu_memory_utilization=0.85. This difference does not mean that vLLM has larger model weights. It mainly reflects the large KV cache space reserved by vLLM for general-purpose serving scenarios. By contrast, the single-path inference engine generated by MetaInfer retains only the runtime structure needed by the target workload, so its memory usage is closer to the sum of model weights and actual runtime overhead. This result supports the claim that customized generation can reduce redundancy from general-purpose frameworks. As supplementary validation, the chunked prefill experiment showed that chunking can reduce peak memory usage but increases first-token latency, and the multi-batch experiment showed that the generated engine scaled nearly linearly to B=8B=8 in short-prompt scenarios. These two experiments mainly check scheduling and memory-management boundaries. Full results are reported in Appendix Tables 11 and 12; they are not expanded as main evidence in the body text. Overall, the K100 experiment provides the runtime and performance profile of an artifact generated under the zero-reference constraint on a CKB-covered target. The generated engine completed Qwen3-8B inference on a real DCU platform and showed decode-throughput and memory-usage characteristics consistent with a customized single-path design. The next subsection examines whether MetaInfer can extend through knowledge-base evolution when the target model structure and platform constraints exceed existing coverage. 4.4 Knowledge-base evolution results on the Z200 platform The K100 experiment reports the runtime behavior and performance profile of an artifact generated under the zero-reference constraint on a CKB-covered model. To test whether the knowledge-base evolution loop can handle insufficiently covered targets, we select Qwen3.6-27B and run an end-to-end evolution experiment in the Z200 DCU environment, using four Z200SM_80 cards with 16368 MB memory per card. Compared with Qwen3-8B, Qwen3.6-27B [13] introduces mixed attention structures, GatedDeltaNet states, MRoPE, RMSNorm variants, and new weight-naming rules. We use SWITCH to indicate whether agents are allowed to access external source-code reference during knowledge-base evolution. SWITCH=ON denotes controlled source-code-assisted exploration, whereas SWITCH=OFF denotes source-code reference disabled, so that regeneration depends only on the consolidated CKB and test contracts. The key criterion in this experiment is therefore not whether the system can generate a candidate implementation when source-code reference is enabled, but whether it can regenerate, verify, and run the inference engine under SWITCH=OFF after knowledge consolidation. Table 4 reports the main evolution trajectory of Qwen3.6-27B on Z200. The evo-001 round identified differences between the target model and existing knowledge, and consolidated new components, weight rules, and risk areas into the CKB. The evo-002 round exposed OOM, TIMEOUT, configuration-parsing, and architecture-identification issues. The evo-003 round completed all non-hardware component tests after hardware-blocking cases were excluded. After switching to SWITCH=OFF, evo-004 passed 21/21 component tests and achieved greedy match. Finally, after the candidate engine had been generated, evo-005 compared its output token by token with an HF greedy-output trace; all 24 greedy decode token IDs matched exactly, reaching PASS_VERIFIED. We define PASS_VERIFIED as the final acceptance state of knowledge-base evolution: after source-code reference is disabled, the system passes the preset component tests and independent correctness verification, and remains token-wise consistent with the HF output trace under greedy decoding. The HF output trace is used only as a post-generation correctness criterion, not as source-code reference or as input to candidate generation. Table 4: Knowledge-base evolution trajectory of Qwen3.6-27B on the Z200 platform. Round Source switch Objective Verification result Knowledge or repair output evo-001 ON Architecture exploration and knowledge consolidation PASS Consolidated 7 knowledge items; identified differences across 8 dimensions and 7 risk areas evo-002 ON First candidate verification 21/31 PASS Exposed 9 OOM issues, 1 TIMEOUT, and 2 config/arch bugs evo-003 ON Component retest after repair 19/19 PASS All non-hardware-blocking tests passed, confirming that source-code reference could be disabled evo-004 OFF KB-only full generation 21/21 PASS Repaired KV cache, scheduler, and test-encoding issues; TP=4 decode aligned with the HF greedy output evo-005 OFF Independent correctness verification PASS_VERIFIED Repaired 3 numerical bugs; all 24 decode token IDs matched the HF greedy-output trace exactly This trajectory shows that the main role of knowledge-base evolution is not simply to add model-parameter descriptions, but to convert failure cases into reusable constraints. Across the process, nine categories of issues were recorded and repaired (the full matrix is reported in Appendix Table 13), including 4 CRITICAL, 2 HIGH, 2 MEDIUM, and 1 LOW issue. Model-implementation errors concentrated on L2 normalization in GatedDeltaNet, gate timing in RMSNormGated, and repeated scaling in the prefill path. Scheduler errors concentrated on KV cache length maintenance and decode sequence advancement. Script-level errors came from configuration nesting, architecture-name recognition, and test encoding. These issues correspond to model specifications, state contracts, weight mapping, and verification experience, showing that the evolved CKB covers a set of generation constraints rather than a single debugging log. After reaching PASS_VERIFIED, we measured the runtime behavior of the evolved engine on Z200. Table 5 reports the full 3×33× 3 performance matrix for Qwen3.6-27B. The table retains five metrics, TTFT, TPOT, end-to-end throughput, decode throughput, and memory usage, to observe prefill cost, decode stability, and the 16 GB per-GPU memory boundary jointly. Table 5: Full 3×33× 3 performance matrix of Qwen3.6-27B on the Z200 platform (TP=4, Eager mode). Values are means ± standard deviations over three runs. p g TTFT (ms) TPOT (ms) tok/s (e2e) tok/s (decode) VRAM (MB/GPU) 11 256 526.5 ± 20.2 174.8 ± 1.9 5.7 ± 0.1 5.7 ± 0.1 16368 11 1024 531.9 ± 7.9 175.7 ± 0.6 5.7 ± 0.0 5.7 ± 0.0 16368 11 2048 524.8 ± 16.5 174.3 ± 0.5 3.9 ± 1.0∗ 5.7 ± 0.0 16368 256 256 5114.5 ± 163.3 170.6 ± 0.8 5.2 ± 0.0 5.9 ± 0.0 16368 256 1024 5238.2 ± 143.7 174.4 ± 1.9 5.5 ± 0.1 5.7 ± 0.1 16368 256 2048 5236.1 ± 90.9 175.5 ± 0.1 5.6 ± 0.0 5.7 ± 0.0 16368 1021 256 19957.5 ± 488.1 166.4 ± 0.4 4.0 ± 0.0 6.0 ± 0.0 16368 1021 1024 20292.0 ± 130.5 173.7 ± 0.6 5.1 ± 0.0 5.8 ± 0.0 16368 1021 2048 19972.2 ± 481.8 174.7 ± 1.0 5.4 ± 0.0 5.7 ± 0.0 16368 ∗One run in this configuration may have hit EOS early, causing a larger standard deviation in end-to-end throughput. Decode throughput was not affected. The results show that the decode phase of the evolved engine was relatively stable. Across all configurations, TPOT remained approximately 166–176 ms, decode throughput remained 5.7–6.0 tok/s, and standard deviations were small. This indicates that the model implementation, cache state, and scheduling path consolidated during evolution can sustain continuous generation stably. More importantly, Z200 provides an adaptation scenario distinct from K100, with limited backend capability and insufficient prior knowledge coverage. In this environment, MetaInfer must handle a new model structure, a tight memory budget, and unavailable platform operators at the same time. TTFT was highly sensitive to prompt length: it was approximately 0.53 s for short prompts, approximately 5.2 s for 256-token prompts, and approximately 20 s for 1021-token prompts. The main reason is that although flash_attn [3, 4] was installed on the gfx906 architecture of Z200, it could not be used on this backend. The full-attention layers therefore fell back to the PyTorch SDPA math backend, making prefill the main bottleneck in long-prompt scenarios. The purpose of this dataset is therefore not to show single-point performance optimality on Z200, but to show that MetaInfer can generate a verifiable and runnable inference workflow on an unfamiliar and constrained hardware backend, while exposing the platform bottleneck as a target for subsequent operator adaptation and tuning. The memory results further define the hardware boundary of this experiment. Per-GPU memory usage was close to 16368 MB in all main configurations, indicating that Qwen3.6-27B on four Z200 cards was already near the 16 GB per-card capacity limit. Under a BF16 weight estimate, a 27B-parameter model occupies approximately 54 GB, or approximately 13.5 GB per GPU after TP=4, leaving the remaining space for the KV cache, GatedDeltaNet states, and runtime buffers. The Z200 experiment therefore deliberately retains a more constrained adaptation environment than K100, testing whether MetaInfer can complete the loop from knowledge completion to end-to-end execution under non-ideal hardware conditions, rather than reporting throughput only on a platform with more abundant performance resources. As supplementary validation, the multi-batch experiment on Z200 showed that throughput scaled to B=4B=4, whereas B=8B=8 triggered OOM because of the 16 GB per-GPU memory limit. The chunked prefill experiment showed that TTFT differed little across chunk sizes and was mainly dominated by prefill computation under the SDPA math fallback. Full supplementary data are reported in Appendix Tables 14 and 15; the main text retains only the principal results above. Overall, the Z200 experiment extends the conclusions of the first two experiment groups from knowledge-covered targets to insufficiently covered targets. The result shows that the extension target of MetaInfer is not a one-off framework implementation, but a set of transferable model structures, state-management rules, and platform constraints. After controlled exploration, consolidation, and revalidation with source-code reference disabled, this knowledge can re-enter the zero-reference generation path. The three experiment groups therefore answer the three questions posed in this paper: source-code reference reduces construction cost; artifacts generated under the zero-reference constraint on knowledge-covered targets can run on real platforms and form measurable performance profiles; and knowledge-base evolution can expand the system’s coverage. 5 Discussion 5.1 Implications of zero-reference generation Zero-reference generation in this paper is not only a stricter experimental setting. It changes the source of knowledge used during inference-framework generation. Generation based on source-code reference can reduce implementation difficulty, but it may also carry the module boundaries, control flow, and abstraction levels of the reference framework into the generated artifact, leaving the customized engine constrained by existing engineering structures. By contrast, the zero-reference constraint requires agents to reconstruct the target execution path from contract knowledge, model specifications, platform constraints, and test feedback. It therefore shifts the basis of generation from concrete code implementations to transferable system knowledge. This shift makes the generated artifact closer to a single-path inference engine synthesized for the target constraints, rather than an execution branch cut out of a general-purpose framework. The deeper implication of zero-reference generation is that implicit engineering experience must be made explicit. Model-structure differences, weight-mapping rules, KV cache state maintenance, tensor-parallel partitioning, platform communication interfaces, and failure modes must all be written into the CKB before agents can invoke, review, and verify them after source-code reference is disabled. Zero-reference generation is therefore not an isolated capability, but a prerequisite for knowledge-base evolution. Only when knowledge exists in a checkable and updatable form can the system identify gaps, supplement knowledge for new models or platforms, and return to the generation path without source-code reference after consolidation. 5.2 From project to generation Skill At the system level, the long-term goal of MetaInfer is not to maintain an ever-expanding general-purpose inference framework, but to maintain procedural knowledge that can generate inference frameworks. Traditional frameworks are usually centered on a long-lived codebase, which accumulates configuration branches as models, hardware, and optimization strategies expand. MetaInfer is instead centered on the CKB, generation SOPs, test contracts, verification evidence, and experience entries, and generates a single-path inference engine for the target constraints when needed. In other words, the maintenance focus shifts from a continuously expanding engineering implementation to generation specifications that can be invoked, verified, and evolved. This form resembles a compiler: user constraints and the CKB describe what should be generated, generation SOPs and test contracts specify how generation and acceptance should proceed, and the final engine is an executable artifact for a specific model and platform. The current form of MetaInfer already shows the early shape of Skill-based distribution. It encodes the execution workflow, role division, knowledge-base organization, and test gates as reusable rules, and can drive agents to generate target engines. Strictly speaking, however, it is not yet a fully packaged, plug-and-play Skill for arbitrary environments. Future work needs to reduce user-side configuration cost, standardize model and platform description files, stabilize knowledge-base version management, and package generation, verification, performance tuning, and knowledge feedback into a unified invocation interface. Along this direction, MetaInfer can gradually evolve from a generation project into a generation Skill for inference infrastructure. 5.3 Limitations and risks of the evolution strategy Knowledge-base evolution expands the applicability of MetaInfer, but it does not remove all human and environmental dependencies. First, the success rate of evolution depends on the quality of external information and the search coverage of the Explorer. If the target model or hardware platform lacks public documents, configuration descriptions, or verifiable reference behavior, the system may struggle to determine whether a failure comes from a knowledge gap, an implementation error, or a platform limitation. Second, knowledge consolidation carries a risk of over-specialization. A successful evolution run may preserve only local experience about the current model, without abstracting reusable general rules for the same model family or similar platforms. This risk must be mitigated through repeated validation across more model variants and hardware scenarios. In addition, correctness evolution is not equivalent to performance optimization. Knowledge-base evolution first requires the generated artifact to pass interface checks, weight loading, decoding consistency, and end-to-end execution verification. However, an engine that runs correctly does not necessarily achieve optimal throughput, minimal latency, or best VRAM efficiency. In scenarios with persistent failures or long-standing performance bottlenecks that cannot be localized, the system still requires human expert intervention or an independent performance-tuning loop. MetaInfer is therefore currently better viewed as a closed loop for verifiable generation and knowledge expansion, rather than a fully unattended production-grade optimization system. 5.4 Performance evolution for automatic operator adaptation The key to performance improvement is not to reintroduce a profiler, but to strengthen the existing profiler-driven evolution loop. The current system already uses profiler traces, benchmarks, VRAM monitoring, and hardware-utilization evidence during end-to-end acceptance and performance optimization to locate bottlenecks, and it feeds some effective optimization experience back into the CKB. However, the mapping from performance symptoms to operator selection, implementation repair, platform-constraint updates, and knowledge consolidation is still not systematic enough. Major bottlenecks on different hardware may come from prefill attention, decode-stage communication, KV cache management, memory allocation, kernel-launch overhead, or platform-backend fallback. These phenomena are also long-standing concerns in existing inference-serving and attention-optimization work [8, 1, 22, 3], and should be standardized further into reusable bottleneck types and optimization rules. A natural next step is therefore to build an automatic operator adaptation and tuning mechanism. Based on profiler results, the system could propose candidate operator implementations or invocation strategies, screen effective versions through benchmarks and correctness tests, and then write successful experience back into the operator-knowledge and platform-constraint layers of the CKB. This path aligns with recent work on LLM-driven GPU kernel generation and evolutionary optimization [12, 5, 18]. Under this mechanism, modules such as attention, communication, normalization, activation functions, KV cache updates, and memory management would no longer be only static contracts. Instead, their implementations could be selected dynamically according to target hardware and workload characteristics. If this loop runs stably, MetaInfer would move from generating runnable specialized inference engines toward generating inference systems that continuously self-optimize from performance feedback. 6 Conclusion and Outlook This paper presents MetaInfer, a multi-agent system for generating customized LLM inference engines. The system abstracts model structures, execution interfaces, tensor shapes, state transitions, parallelization strategies, and platform constraints into contract knowledge. Through isolated collaboration among implementation, specification review, and independent verification roles, it generates single-path inference engines for target constraints. Compared with directly consulting the source code of general-purpose frameworks and then cutting or rewriting target paths, MetaInfer organizes generation through explicit knowledge, test contracts, and verification feedback. The experimental results support the feasibility and extensibility of this design. The controlled comparison between source-code reference enabled and disabled shows that source-code reference can reduce agent construction cost, but is not a necessary condition for generating a usable inference engine. The Qwen3-8B experiment on the K100 platform provides the real runtime and performance profile of an artifact generated under the zero-reference constraint on a CKB-covered target, showing decode-throughput and memory-usage characteristics consistent with a customized single-path design. The Qwen3.6-27B experiment on the Z200 platform further shows that, when the target model and platform exceed existing coverage, the knowledge-base evolution mechanism can convert new information into reusable contract knowledge through controlled exploration, knowledge consolidation, and revalidation after source-code reference is disabled. Looking ahead, MetaInfer will continue to develop in two directions. The first is to move from its current project form toward a more complete generation Skill, packaging the CKB, generation SOPs, test contracts, performance tuning, and knowledge feedback into a more stable invocation interface. The second is to improve the profiler-driven automatic operator adaptation and tuning mechanism, so that the system can select or generate more suitable operator implementations from performance feedback under different hardware and workload conditions. Along these directions, MetaInfer is expected to evolve from a verifiable specialized inference-engine generation system into an inference infrastructure generator that can continuously learn and self-optimize. References [1] A. Agrawal, N. Kedia, A. Panwar, J. Mohan, N. Kwatra, B. S. Gulavani, A. Tumanov, and R. Ramjee (2024) Taming throughput-latency tradeoff in LLM inference with Sarathi-Serve. arXiv. Note: 10.48550/arXiv.2403.02310 External Links: Document, Link Cited by: §2, §5.4. [2] R. Y. Aminabadi, S. Rajbhandari, A. A. Awan, C. Li, D. Li, E. Zheng, O. Ruwase, S. Smith, M. Zhang, J. Rasley, and Y. He (2022) DeepSpeed-Inference: enabling efficient inference of transformer models at unprecedented scale. In SC22: International Conference for High Performance Computing, Networking, Storage and Analysis, p. 1–15. External Links: Document, Link Cited by: §2. [3] T. Dao, D. Y. Fu, S. Ermon, A. Rudra, and C. Ré (2022) FlashAttention: fast and memory-efficient exact attention with IO-Awareness. arXiv. Note: 10.48550/arXiv.2205.14135 External Links: Document, Link Cited by: §3.2, §4.4, §5.4. [4] T. Dao (2023) FlashAttention-2: faster attention with better parallelism and work partitioning. arXiv. Note: 10.48550/arXiv.2307.08691 External Links: Document, Link Cited by: §3.2, §4.4. [5] P. Guo, C. Zhu, S. Chen, F. Liu, X. Lin, Z. Lu, and Q. Zhang (2025) EvoEngineer: mastering automated CUDA kernel code evolution with large language models. arXiv. Note: 10.48550/arXiv.2510.03760 External Links: Document, Link Cited by: §1, §5.4. [6] P. Hamadanian, P. Karimi, A. Nasr-Esfahany, K. Noorbakhsh, J. Chandler, A. ParandehGheibi, M. Alizadeh, and H. Balakrishnan (2026) Glia: a human-inspired AI for automated systems design and optimization. In Proceedings of the ACM Conference on AI and Agentic Systems, p. 61–84. External Links: Document, Link Cited by: §1, §2. [7] K. Kamahori, S. Li, S. Peter, and B. Kasikci (2026) VibeServe: can AI agents build bespoke LLM serving systems?. arXiv. Note: 10.48550/arXiv.2605.06068 External Links: Document, Link Cited by: §1, §2, §3.2. [8] W. Kwon, Z. Li, S. Zhuang, Y. Sheng, L. Zheng, C. H. Yu, J. Gonzalez, H. Zhang, and I. Stoica (2023) Efficient memory management for large language model serving with PagedAttention. In Proceedings of the 29th Symposium on Operating Systems Principles, p. 611–626. External Links: Document, Link Cited by: §1, §2, §5.4. [9] A. Madaan, N. Tandon, P. Gupta, S. Hallinan, L. Gao, S. Wiegreffe, U. Alon, N. Dziri, S. Prabhumoye, Y. Yang, S. Gupta, B. P. Majumder, K. Hermann, S. Welleck, A. Yazdanbakhsh, and P. Clark (2023) Self-Refine: iterative refinement with self-feedback. arXiv. Note: 10.48550/arXiv.2303.17651 External Links: Document, Link Cited by: §1. [10] X. Miao, G. Oliaro, Z. Zhang, X. Cheng, Z. Wang, Z. Zhang, R. Y. Y. Wong, A. Zhu, L. Yang, X. Shi, C. Shi, Z. Chen, D. Arfeen, R. Abhyankar, and Z. Jia (2024) SpecInfer: accelerating large language model serving with tree-based speculative inference and verification. In Proceedings of the 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 3, p. 932–949. External Links: Document, Link Cited by: §2. [11] OpenBMB (2026) ForgeTrain: an LLM pretraining framework built end-to-end by an autonomous agent loop. Note: Software, version 0.1.0; accessed 2026-07-10 External Links: Link Cited by: §1, §2. [12] A. Ouyang, S. Guo, S. Arora, A. L. Zhang, W. Hu, C. Ré, and A. Mirhoseini (2025) KernelBench: can LLMs write efficient GPU kernels?. arXiv. Note: 10.48550/arXiv.2502.10517 External Links: Document, Link Cited by: §1, §5.4. [13] Qwen Team (2026-04) Qwen3.6 Technical Report. Note: Technical reportOfficial technical report for Qwen3.6-27B External Links: Link Cited by: §1, §4.4. [14] Y. Sheng, L. Zheng, B. Yuan, Z. Li, M. Ryabinin, D. Y. Fu, Z. Xie, B. Chen, C. Barrett, J. E. Gonzalez, P. Liang, C. Ré, I. Stoica, and C. Zhang (2023) FlexGen: high-throughput generative inference of large language models with a single GPU. arXiv. Note: 10.48550/arXiv.2303.06865 External Links: Document, Link Cited by: §2. [15] N. Shinn, F. Cassano, E. Berman, A. Gopinath, K. Narasimhan, and S. Yao (2023) Reflexion: language agents with verbal reinforcement learning. arXiv. Note: 10.48550/arXiv.2303.11366 External Links: Document, Link Cited by: §1. [16] M. Shoeybi, M. Patwary, R. Puri, P. LeGresley, J. Casper, and B. Catanzaro (2019) Megatron-LM: training multi-billion parameter language models using model parallelism. arXiv. Note: 10.48550/arXiv.1909.08053 External Links: Document, Link Cited by: §2. [17] L. Wang, C. Ma, X. Feng, Z. Zhang, H. Yang, J. Zhang, Z. Chen, J. Tang, X. Chen, Y. Lin, W. X. Zhao, Z. Wei, and J. Wen (2024) A survey on large language model based autonomous agents. Frontiers of Computer Science 18 (6), p. 186345. External Links: Document, Link Cited by: §1. [18] N. Wiedemann, Q. Leboutet, M. Paulitsch, D. Wofk, and B. Ummenhofer (2026) KernelFoundry: hardware-aware evolutionary GPU kernel optimization. arXiv. Note: 10.48550/arXiv.2603.12440 External Links: Document, Link Cited by: §1, §5.4. [19] J. Yang, C. E. Jimenez, A. Wettig, K. Lieret, S. Yao, K. Narasimhan, and O. Press (2024) SWE-agent: agent-computer interfaces enable automated software engineering. arXiv. Note: 10.48550/arXiv.2405.15793 External Links: Document, Link Cited by: §1. [20] G. Yu, J. S. Jeong, G. Kim, S. Kim, and B. Chun (2022) Orca: a distributed serving system for Transformer-Based generative models. In 16th USENIX Symposium on Operating Systems Design and Implementation (OSDI 22), p. 521–538. External Links: Link Cited by: §1, §2. [21] L. Zheng, L. Yin, Z. Xie, C. Sun, J. Huang, C. Yu, S. Cao, C. Kozyrakis, I. Stoica, J. Gonzalez, C. Barrett, and Y. Sheng (2024) SGLang: efficient execution of structured language model programs. In Advances in Neural Information Processing Systems 37, p. 62557–62583. External Links: Document, Link Cited by: §1. [22] Y. Zhong, S. Liu, J. Chen, J. Hu, Y. Zhu, X. Liu, X. Jin, and H. Zhang (2024) DistServe: disaggregating prefill and decoding for goodput-optimized large language model serving. arXiv. Note: 10.48550/arXiv.2401.09670 External Links: Document, Link Cited by: §2, §5.4. Appendix A Implementation Details of the MetaInfer Method This appendix provides engineering details that are not expanded in the main Method section, including the generation workflow, agent role division, and CKB organization. The main text keeps the methodological narrative concise, whereas this appendix explains how MetaInfer turns these designs into an executable engineering workflow. A.1 Generation workflow and staged gates The generation workflow of MetaInfer is routed by the main agent. The system first reads user constraints and environment configurations, parses the target model configuration, and determines whether the CKB covers the target model, parallelization mode, and platform backend. If coverage is sufficient, the system enters the regular generation path. If coverage is insufficient, it enters knowledge-base evolution. The regular generation path is further divided into stages, each with an explicit implementation objective and test gate. The staged organization is not intended to require manual step-by-step development. Instead, it allows agents to generate and verify complex systems within smaller interface boundaries. Table 6: Main stages in the MetaInfer generation workflow. Stage Role Basic execution layer Generates or assembles numerical primitives, kernel wrappers, and tensor-parallel communication interfaces, providing low-level capabilities for upper-level model execution. Model component layer Builds components required by the model forward path, including tensor-parallel linear layers, embeddings, attention/KV cache, MLP, and decoder-layer modules. Framework integration layer Completes weight loading, scheduler, sampler, request-state management, and LLMEngine integration, turning generated code into an executable inference loop. End-to-end acceptance layer Uses greedy alignment, benchmarks, path checks, resource-usage evidence, and no-compile checks to confirm that the generated artifact is usable. Performance optimization layer After correctness passes, performs local optimization based on profiler and benchmark results, and feeds effective experience back into the CKB. Acceptance uses multi-level test contracts, which specify the required behavior and acceptance criteria. Fixed test scripts execute these contracts, and their results form the test gates for stage progression. L0 checks whether import paths and runtime objects come from the current generated directory. L1 runs the fixed test scripts for the current stage. L2 performs cross-stage regression to prevent later modifications from breaking earlier capabilities. L3 collects end-to-end performance, memory, and profiler evidence. Correctness errors, hard-coded outputs, external-framework bypasses, or failures to load real weights all cause the generated artifact to be rejected. A.2 Multi-agent role division MetaInfer separates implementation, review, verification, and knowledge management across different agents to reduce the risk of self-confirmation caused by a single agent writing, reviewing, and testing its own output. Roles are invoked through independent processes. By default, they do not share internal memory from previous rounds and collaborate only through strategy files, test reports, and CKB artifacts. Table 7: Main agent roles in MetaInfer. Role Responsibility Main agent Reads user constraints, performs coverage detection, selects the regular generation or knowledge-base evolution path, and schedules sub-agents. Implementation agent Generates code according to the CKB and stage objective, and submits implementation results without claiming that tests have passed. Specification-review agent Checks interfaces, data flow, shape, dtype, device, and implementation boundaries according to contracts, without replacing testing. Verification agent Runs fixed test scripts against the test contracts and performs end-to-end acceptance; it is the only role allowed to declare PASS or FAIL. Exploration agent Collects model configurations, papers, technical reports, hardware documents, and controlled source-code information when knowledge coverage is insufficient. Knowledge-consolidation agent Organizes verified information into model specifications, contracts, platform constraints, debugging experience, or failure modes. Failure-analysis agent Analyzes root causes when generation or revalidation fails, and provides structured feedback for the next round of exploration or repair. Experiment-summary agent Writes experience back into the CKB after performance optimization or bug repair produces stable gains. This role division forms an adversarial loop of implementation, review, and verification. Implementation failures return to the implementation agent, specification-review failures require contract realignment, and verification failures produce reproducible error evidence. In knowledge-base evolution, the exploration and consolidation roles further ensure that external information does not remain only in a one-time context, but is distilled into knowledge that can be reused by subsequent zero-reference generation. A.3 Contract knowledge base organization The CKB of MetaInfer is not a collection of source code, but a generation-oriented knowledge organization. Its core consists of contract files and supporting technical documents. The former define interfaces and execution constraints that must be satisfied, whereas the latter supplement model, operator, parallelization, and debugging knowledge. Fixed test scripts serve as the harness, executing test contracts as acceptance checks. Table 8: Main components of the MetaInfer CKB. Component Content and role Contract layer Specifies interface and data-flow constraints for modules such as kernels, communication, tensor-parallel layers, attention/KV cache, MLP, weight loading, framework shell, and engine integration. Framework-design layer Describes framework-level designs such as scheduler, KV cache, model executor, sampler, memory pool, and request lifecycle. Model-knowledge layer Stores structural parameters, weight naming, special modules, and architectural differences for specific models or model families. Parallelism and operator layer Records tensor parallelism, communication primitives, kernel wrappers, and platform-backend knowledge. Experience and optimization layer Stores debugging experience, failure modes, performance optimizations, and reusable conclusions formed during knowledge-base evolution. Test-contract layer Consists of fixed scripts used to verify whether each generation stage and end-to-end inference path meets expectations. After a new model or platform is absorbed by knowledge-base evolution, its differences are not stored as complete source code. Instead, they are decomposed into reusable knowledge across the layers above. Subsequent generation tasks can then invoke this knowledge under the zero-reference constraint without reopening the original open-source inference-framework implementations. Appendix B Historical Platform Results This section reports platform data that were verified as usable in the initial with-reference experiments. Because Section 4.3 reports the latest zero-reference benchmark on the K100 platform, this section does not repeat the main K100 3×33× 3 performance table. It retains only A800 and Apple M5 results as cross-platform reference points. B.1 A800 platform results Table 9 reports the full 3×33× 3 benchmark results for Qwen3-8B on the A800 platform. This experiment comes from a with-reference construction version and shows the executability and relative performance of early MetaInfer outputs on an NVIDIA platform. It is not used as main evidence for zero-reference generation in this paper. Table 9: Full benchmark data on the A800 platform (Qwen3-8B, TP=4, BF16). p g MetaInfer (tok/s) vLLM Eager (tok/s) Gain 11 256 64.8 42.5 +52.5% 11 1024 66.1 43.3 +52.7% 11 2048 65.9 43.1 +52.9% 256 256 65.2 42.8 +52.3% 256 1024 66.0 43.2 +52.8% 256 2048 65.6 43.1 +52.2% 1021 256 64.9 42.2 +53.8% 1021 1024 65.3 42.6 +53.3% 1021 2048 65.6 43.0 +52.6% B.2 Apple M5 platform results Table 10 reports the full 3×33× 3 benchmark results for Qwen3-8B on the Apple M5 Pro platform. This experiment uses TP=1 and mlx_lm as the platform-native baseline, supplementing executability evidence for early MetaInfer outputs on a non-CUDA/DCU platform. Table 10: Full benchmark data on the Apple M5 Pro platform (Qwen3-8B, TP=1, BF16). p g MetaInfer (tok/s) mlx_lm (tok/s) Ratio 11 256 17.6 17.2 1.02 11 1024 17.4 17.0 1.02 11 2048 17.3 17.1 1.01 256 256 17.5 17.0 1.03 256 1024 17.4 17.0 1.02 256 2048 17.3 16.9 1.02 1021 256 17.2 18.0 0.96 1021 1024 17.1 17.8 0.96 1021 2048 17.4 17.0 1.02 Appendix C Supplementary Experimental Data This section reports experimental data that are not expanded in Section 4. The main text retains only construction cost, single-sequence performance, and knowledge-base evolution results that directly support the main conclusions. This section reports chunked prefill, multi-batch results, and the knowledge-base evolution bug matrix to clarify the boundaries of scheduling, memory usage, and failure repair. C.1 K100 platform supplementary experiments Table 11 reports the chunked prefill results for Qwen3-8B on the K100 platform. This experiment fixes p=1021p=1021 and g=256g=256, and compares the effects of different chunk sizes on first-token latency, decode latency, throughput, and peak memory usage on the MetaInfer engine with CUDA Graph disabled. Table 11: Supplementary chunked prefill experiment on the K100 platform (Qwen3-8B, TP=4, p=1021p=1021, g=256g=256). Chunk Chunks TTFT (ms) TPOT (ms) Prefill (ms) Decode (ms) Throughput (tok/s) Peak VRAM (MB) None 1 216.3 ± 6.3 51.1 ± 0.9 212.5 ± 6.7 13020 ± 228 19.3 5269 128 8 571.7 ± 8.8 51.2 ± 0.2 568.5 ± 7.7 13060 ± 70 18.7 4673 256 4 375.2 ± 10.0 51.1 ± 0.2 371.7 ± 10.3 13031 ± 56 19.1 4764 512 2 263.5 ± 16.3 51.2 ± 0.2 259.8 ± 16.1 13046 ± 65 19.3 4933 Table 12 reports the multi-batch results for Qwen3-8B on the K100 platform. This experiment fixes each sequence to p=256p=256 and g=256g=256, and measures the effects of batch scaling on throughput, first-token latency, and peak memory usage in short-prompt scenarios. Table 12: Supplementary multi-batch experiment on the K100 platform (Qwen3-8B, TP=4, p=256p=256, g=256g=256). Batch TTFTavg (ms) Throughput (tok/s) Speedup Peak VRAM (MB) KV Blocks KV Util. 1 122.2 ± 9.0 19.4 1.00× 4764 2/160 1.2% 2 158.8 ± 12.3 38.7 1.99× 4933 2/160 1.2% 4 317.0 ± 27.1 76.3 3.93× 5271 2/160 1.2% 8 599.6 ± 23.4 147.8 7.61× 5956 2/160 1.2% C.2 Z200 platform supplementary experiments Table 13 reports the bug matrix recorded during knowledge-base evolution for Qwen3.6-27B. This matrix corresponds to the multi-round evolution process in Table 4 in the main text and clarifies the types of issues repaired before PASS_VERIFIED. Table 13: Bug matrix during Qwen3.6-27B knowledge-base evolution on Z200. ID Round Module Severity Symptom and root cause CONFIG-001 evo-002 scripts MEDIUM Nested text_config was not handled, causing missing configuration parsing. ENGINE-001 evo-002 scripts MEDIUM The Qwen3_5ForConditionalGeneration architecture name was not recognized, causing model routing failure. BUG-004-1 evo-004 model CRITICAL KV cache state maintenance was incorrect, causing repeated loops in decode output. BUG-004-2 evo-004 scheduler CRITICAL The scheduler did not correctly advance the generated sequence, causing generate to idle. BUG-004-3 evo-004 scripts LOW Unicode characters in the test script did not match, affecting automated verification. BUG-KVLEN transition scheduler CRITICAL kv_len initialization and update were incorrect, causing output to degrade into invalid tokens. BUG-005-1 evo-005 model CRITICAL GatedDeltaNet L2 normalization used mean rather than sum, causing numerical explosion. BUG-005-2 evo-005 model HIGH The RMSNormGated gate should be applied after normalization; the wrong order caused abnormal DeltaNet output. BUG-005-3 evo-005 model HIGH The prefill path applied Q scaling repeatedly, making attention scores too small and causing degradation. Table 14 reports the supplementary multi-batch results for Qwen3.6-27B on the Z200 platform. This experiment fixes p=256p=256 and g=256g=256 and tests batch scalability under the 16 GB per-GPU memory constraint. B=8B=8 triggered OOM because of insufficient memory and is therefore not included in the table. Table 14: Supplementary multi-batch experiment on the Z200 platform (Qwen3.6-27B, TP=4, p=256p=256, g=256g=256). Batch Throughput (tok/s) Decode Step (ms) Prefill Step (ms) KV Blocks 1 5.4 ± 0.0 165.4 ± 1.7 1486.1 ± 8.8 2/152 2 10.2 ± 0.1 174.9 ± 1.7 1644.0 ± 73.7 4/148 4 18.3 ± 0.1 191.8 ± 1.5 2363.1 ± 46.5 8/148 Table 15 reports the chunked prefill results for Qwen3.6-27B on the Z200 platform. This experiment fixes p=1021p=1021 and g=256g=256, and measures the effects of chunk size on TTFT, total throughput, and peak memory usage. Table 15: Supplementary chunked prefill experiment on the Z200 platform (Qwen3.6-27B, TP=4, p=1021p=1021, g=256g=256). Configuration Chunks TTFT (s) Prefill (s) Decode (s) Throughput (tok/s) VRAM (GiB) full 1 12.04 12.04 42.90 4.49 14.5 chunk=128 8 12.22 12.22 43.15 4.45 13.6 chunk=256 4 11.70 11.70 42.80 4.52 13.8 chunk=512 2 11.89 11.89 42.80 4.51 14.3