Paper deep dive
CangjieBench: Benchmarking LLMs on a Low-Resource General-Purpose Programming Language
Junhang Cheng, Fang Liu, Jia Li, Chengru Wu, Nanxiang Jiang, Li Zhang
Intelligence
Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 96%
Last extracted: 3/22/2026, 5:09:53 AM
Summary
CangjieBench is a contamination-free benchmark designed to evaluate Large Language Models (LLMs) on Cangjie, a low-resource, general-purpose programming language. The study introduces 248 manually translated samples from HumanEval and ClassEval, covering Text-to-Code and Code-to-Code tasks. Experiments across four paradigmsâDirect Generation, Syntax-Constrained Generation, RAG, and Agent-based methodsâreveal that while direct generation fails due to lack of syntactic knowledge, syntax-constrained prompting provides the best balance of accuracy and cost, while agentic frameworks achieve state-of-the-art performance at higher token costs.
Entities (6)
Relation Signals (4)
CangjieBench â contains â HumanEval
confidence 100% ¡ The benchmark comprises 248 high-quality samples manually translated from HumanEval and ClassEval
CangjieBench â contains â ClassEval
confidence 100% ¡ The benchmark comprises 248 high-quality samples manually translated from HumanEval and ClassEval
CangjieBench â evaluates â Cangjie
confidence 100% ¡ we introduce CangjieBench, a contamination-free benchmark for Cangjie
Syntax-Constrained Generation â improvesperformanceof â LLM
confidence 90% ¡ Syntax-Constrained Generation offers the best trade-off between accuracy and computational cost.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Large Language Models excel in high-resource programming languages but struggle with low-resource ones. Existing research related to low-resource programming languages primarily focuses on Domain-Specific Languages (DSLs), leaving general-purpose languages that suffer from data scarcity underexplored. To address this gap, we introduce CangjieBench, a contamination-free benchmark for Cangjie, a representative low-resource general-purpose language. The benchmark comprises 248 high-quality samples manually translated from HumanEval and ClassEval, covering both Text-to-Code and Code-to-Code tasks. We conduct a systematic evaluation of diverse LLMs under four settings: Direct Generation, Syntax-Constrained Generation, Retrieval-Augmented Generation (RAG), and Agent. Experiments reveal that Direct Generation performs poorly, whereas Syntax-Constrained Generation offers the best trade-off between accuracy and computational cost. Agent achieve state-of-the-art accuracy but incur high token consumption. Furthermore, we observe that Code-to-Code translation often underperforms Text-to-Code generation, suggesting a negative transfer phenomenon where models overfit to the source language patterns. We hope that our work will offer valuable insights into LLM generalization to unseen and low-resource programming languages. Our code and data are available at this https URL.
Tags
Links
- Source: https://arxiv.org/abs/2603.14501v1
- Canonical: https://arxiv.org/abs/2603.14501v1
Trouble viewing inline? Open PDF directly â
Full Text
73,844 characters extracted from source content.
Expand or collapse full text
CANGJIEBENCH: Benchmarking LLMs on a Low-Resource General-Purpose Programming Language Junhang Cheng 1 , Fang Liu 1 * , Jia Li 2 , Chengru Wu 1 , Nanxiang Jiang 1 , Li Zhang 1 1 Beihang University, 2 Wuhan University chengjunhang7@gmail.com, fangliu@buaa.edu.cn Abstract Large Language Models excel in high-resource programming languages but struggle with low- resource ones. Existing research related to low- resource programming languages primarily fo- cuses on Domain-Specific Languages (DSLs), leaving general-purpose languages that suf- fer from data scarcity underexplored. To ad- dress this gap, we introduce CANGJIEBENCH, a contamination-free benchmark for Cangjie, a representative low-resource general-purpose language. The benchmark comprises 248 high- quality samples manually translated from Hu- manEval and ClassEval, covering both Text- to-Code and Code-to-Code tasks. We conduct a systematic evaluation of diverse LLMs un- der four settings: Direct Generation, Syntax- Constrained Generation, Retrieval-Augmented Generation (RAG), and Agent. Experiments reveal that Direct Generation performs poorly, whereas Syntax-Constrained Generation offers the best trade-off between accuracy and com- putational cost. Agent achieve state-of-the- art accuracy but incur high token consump- tion. Furthermore, we observe that Code-to- Code translation often underperforms Text- to-Code generation, suggesting a negative transfer phenomenon where models overfit to the source language patterns. We hope that our work will offer valuable insights into LLM generalization to unseen and low- resource programming languages. Our code and data are available athttps://github. com/cjhCoder7/CangjieBench. 1 Introduction Large Language Models (LLMs) have revolution- ized software engineering, particularly in code gen- eration and translation tasks (Zhao et al., 2025; He et al., 2025; Zhang et al., 2025; Xue et al., 2024). Models such as GPT-5 (OpenAI, 2025b) and Deepseek-V3 (Liu et al., 2024) have demon- * Corresponding author. strated remarkable proficiency in mainstream pro- gramming languages like Python (Chen, 2021) and C++ (Chervyakov et al., 2025). However, the con- vergence of AI and Edge-Cloud collaboration de- mands next-generation languages capable of unify- ing diverse computing scenariosâfrom embedded devices to cloud serversâwhile maintaining high performance. [Importance of Cangjie] Cangjie 1 , a modern programming language developed by Huawei, has emerged to meet these challenges. Designed as the cornerstone of the HarmonyOS ecosystem 2 , Cangjie features native support for all-scenario application development, integrating modern lan- guage characteristics with advanced compilation optimization and runtime performance (Huawei, 2025). As HarmonyOS expands to govern a vast ecosystem of smart devices, Cangjie is poised to become a critical language in the global software market, particularly within China. [Motivation 1] Despite its growing industrial significance, Cangjie remains a low-resource lan- guage. Unlike Python or C++, which have billions of tokens available in open-source repositories 3 , Cangjie lacks a large-scale public corpus, making it challenging for current LLMs to generate valid Cangjie code. This scarcity presents a unique op- portunity to probe the boundaries of LLM general- ization (Joel et al., 2024). [Motivation 2] Existing low-resource program- ming language studies largely focus on Domain- Specific Languages (DSLs) such as Verilog or So- lidity (Liu et al., 2023; Peng et al., 2025). These languages are tightly coupled with specific domains (e.g., hardware or blockchain), making it hard to distinguish whether a model fails due to a lack of syntactic knowledge or domain expertise. In con- trast, Cangjie is a general-purpose language that 1 https://cangjie-lang.cn/en 2 https://w.harmonyos.com/en/ 3 https://github.com/ 1 arXiv:2603.14501v1 [cs.SE] 15 Mar 2026 BenchmarkLanguagesTask FocusManualLang. DomainLang. Popularity HumanEval (2021)PythonGenerationâ â âGeneral ClassEval (2023)PythonGenerationâ â âGeneral ClassEval-T (2025)Java, C++Translationâ â âGeneral BIRD (2023)SQLTranslationâ â âDatabase CRust-Bench (2025)RustTranslationâ â âGeneral RTest (2025)RGenerationĂ Ă ĂData TeXpert (2025)LaTeXGenerationâ â âTypesetting NL2Bash (2024)BashGenerationâ â âScript BenchSol (2024)SolidityGenerationâ â âSmart Contract SolEval (2025)SolidityGenerationĂ Ă ĂSmart Contract MultiPL-E-Lua (2023)LuaGenerationĂ Ă ĂScript MultiPL-E-Racket (2023)RacketGenerationĂ Ă ĂEducation VerilogEval (2023)VerilogGenerationâ â âHardware KernelBench (2025)CUDAGenerationâ â âHardware Clever (2025)LeanGenerationâ â âProof VHDL-Eval (2024)VHDLGenerationâ â âHardware CANGJIEBENCHCangjieGeneration + Translationâ â âGeneral Table 1: Overview of Code Benchmarks. The Popularity estimates are derived from the November 2025 data of the TIOBE Programming Community Index. possesses algorithmic universality. Since Cangjie allows for expressing logic and algorithms with- out requiring domain knowledge, it can assess the modelâs ability to generalize to general-purpose languages in low-resource settings. [Motivation 3] Beyond generation, converting code from languages like Python to new ones like Cangjie is a key industrial need. However, most existing translation benchmarks focus on pairs of high-resource languages such as Java-Python (Ah- mad et al., 2023). Translating from a high-resource language to a low-resource one is far more challeng- ing. It requires the LLM to understand the logic in the source language and accurately map it to the syntax of a strictly constrained target language. [Novelty of CangjieBench] To bridge this gap and facilitate research in this direction, we intro- duce CANGJIEBENCH, the first comprehensive benchmark designed for Cangjie. Due to the data scarcity, scraping sufficient high-quality code from open-source repositories is currently infeasible. Consequently, we adopt a translation-based strat- egy, manually translating the widely adopted Hu- manEval (Chen, 2021) and ClassEval (Du et al., 2023) datasets from Python to Cangjie. This ap- proach brings three advantages:âśZero Contam- ination: Since the benchmark is constructed via manual translation rather than crawling existing internet data, it avoids the data leakage issues, en- suring a rigorous test of the modelâs generalization capabilities rather than memorization. By selecting tasks where LLMs already demonstrate strong al- gorithmic reasoning in Python, we deliberately con- trol the logic variable to isolate syntax adaptability as the sole evaluation target.âˇMulti-Difficulty Coverage: By combining HumanEval and ClassE- val, CANGJIEBENCH covers a diverse spectrum of difficulty. It spans from function-level algorithmic tasks to complex class-level object-oriented pro- gramming scenarios.â¸Dual-Task Utility: The translation process naturally yields a high-quality parallel corpus between Python and Cangjie. This enables CANGJIEBENCH to support not only Text- to-Code generation tasks but also Code-to-Code translation tasks. [Exploration of Methods] Beyond establish- ing the dataset, we aim to answer a key research question: which paradigm most effectively enables LLMs to generalize to an unseen programming language without parameter updates? To this end, we conduct extensive experiments across four paradigms:âśDirect Generation: Evaluating the capability of SOTA LLMs in zero-shot settings. âˇSyntax-Constrained Generation: Augment- ing prompts with expert-curated concise grammar rules to help LLM better learn Cangjie program- ming language.â¸RAG: Leveraging external knowledge basesâincluding official documenta- tion and crawled Cangjie code snippets to provide in-context few-shot guidance.âšAgent: Utilizing 2 CLI-based agents to simulate iterative development, self-learning, and self-correction processes. In summary, our main contributions are listed as follows: â˘We introduce CANGJIEBENCH 4 , the first high- quality benchmark for Cangjie. By manually translating existing datasets, we ensure zero con- tamination and cover diverse difficulties, ranging from simple functions to complex classes. ⢠We propose a novel research perspective by treat- ing Cangjie as a low-resource general-purpose language. Unlike DSLs, this allows us to strictly evaluate an LLMâs ability to learn new syntax without the interference of domain knowledge. â˘We establish two tasks: Text-to-Code genera- tion and Code-to-Code translation. These tasks simulate real-world scenarios, such as migrating projects to the HarmonyOS ecosystem, providing a practical testbed for the community. â˘We conduct a comprehensive evaluation using four strategies, ranging from syntax-constrained method to RAG and Agents. Our experiments provide solid baselines and offer effective in- sights into how LLMs can master new program- ming languages with limited data. 2 Related Work Low-Resource Language Benchmarks. Though LLMs have demonstrated exceptional performance in high-resource languages (Jimenez et al., 2023; Jain et al., 2024), they failed significantly when applied to low-resource programming languages due to the scarcity of data. Early efforts such as MultiPL-E (Cassano et al., 2023) extended the Hu- manEval (Chen, 2021) and MBPP (Austin et al., 2021) benchmarks to 18 diverse languages. How- ever, as LLMs evolve, these established bench- marks have become less discriminative. Recent works have introduced specialized benchmarks such as NL2Bash (Vo et al., 2024) for shell script- ing, SolEval (Peng et al., 2025) for Solidity smart contracts, VerilogEval (Liu et al., 2023) for hard- ware description, and others (detailed in Table 1). However, using them to evaluate the generalization capabilities of LLMs presents two specific chal- lenges: (1) Entanglement with Domain Knowl- edge: Benchmarks based on DSLs (e.g., Verilog, Solidity) mix evaluation with domain-specific con- straints (e.g., hardware logic). (2) Data Leakage 4 The data and code are available athttps://github.com/ cjhCoder7/CangjieBench Risks: Languages traditionally labeled as âlow- resourceâ (e.g., Lua, R; see Table 1) still exist in massive pre-training corpora due to their long his- tory. In contrast, Cangjie is a nascent language released in July 2025, eliminating pre-training leak- age and providing a rigorous testbed. Methods of Code Generation. The landscape of code generation has evolved from code models like Codex (Chen, 2021) to state-of-the-art founda- tion models such as GPT-5 (OpenAI, 2025b), and Qwen-3 (Yang et al., 2025a). However, LLMs still struggle with complex problems. Consequently, the research community has shifted towards enhancing generation through advanced methods, primarily categorized into three paradigms: Prompt Engi- neering (Puerto et al., 2024; Wang et al., 2024), Retrieval-Augmented Generation (Liu et al., 2025; Yang et al., 2025b), and Agentic Frameworks (Ope- nAI, 2025a; Anthropic, 2025). However, previous studies on low-resource code typically limit their evaluation to direct zero-shot or few-shot prompt- ing. To the best of our knowledge, this work com- prehensively evaluates these paradigms, systemati- cally exploring which methodology is better. 3CANGJIEBENCH As illustrated in Figure 1, the CANGJIEBENCH framework comprises three principal components: Dataset Construction, Evaluation Task, and Evaluation Framework. 3.1 Dataset Construction High-quality open-source Cangjie code is scarce, making the traditional crawling approach infeasible. Thus, we adopt a translation-based strategy using HumanEval (Chen, 2021) and ClassEval (Du et al., 2023). These datasets represent function-level and class-level Python programming tasks, respectively, meaning the code is self-contained in a single file without external dependencies. Unlike repository- level benchmarks (Yu et al., 2024; Jimenez et al., 2023) that involve complex multi-file environments, this standalone format minimizes translation errors and ensures the correctness of the ground truth. To rigorously translate these Python benchmarks to Cangjie, we adhere to a set of strict construction principles, detailed below. â˘Type Adaptation: Basic types are mapped to Cangjie equivalents such asintâInt64, floatâFloat64,strâString,boolâBool. And data structures to CangjieâsCollectionli- 3 Method Cangjie Code NL Prompt Task 1: Text-to-Code Generation Task 2: Code-to-Code Translation Method Cangjie Code Python Code Cangjie Code Cangjie Runtime Cangjie Libraries Test Frameworks Test Result HumanEvalClassEval Solution & Prompt Translation Test Case Translation Interactive Verification Type Adaptation Naming Convention Algorithm Utilization Prompt Transformation Dependency Management PrinciplesPipelines Dataset Construction Evaluation TaskEvaluation Framework Figure 1: Overview of CANGJIEBENCH framework. brary (ArrayList, HashMap). â˘Naming Convention: We retain the original snake_case naming to maintain fidelity and mini- mize ambiguity during translation. ⢠Algorithm Utilization: Cangjie solutions strictly replicate the algorithmic flow of the original Python solutions. â˘Prompt Transformation: All code-related con- tent within prompts (function signatures, doc- strings, few-shot examples) is manually trans- lated to Cangjie syntax. â˘Dependency Management: Tasks dependent on third-party libraries lacking Cangjie equivalents (e.g.,sqlite3,PIL) are excluded (see Table 3), though we manually implemented lightweight ones like hashlib.md5. The construction of CANGJIEBENCH was exe- cuted entirely by the first author, an early adopter who has studied Cangjie since its official release. This process spanned 1.5 months and involved three steps: (1) Manual translation of solutions and prompts; (2) Construction of Cangjie test cases; and (3) Iterative verification to eliminate transla- tion bias. Subsequently, two other authors, who also possess substantial experience with Cangjie, independently reviewed and verified all translated problems, solutions, and test cases to ensure cor- rectness and consistency. The final dataset con- sists of 248 high-quality samples, comprising 164 problems from HumanEval and 84 problems from ClassEval. 3.2 Evaluation Task We design two tasks in CANGJIEBENCH, cover- ing different granularities: function-level code for the HumanEval subset and class-level code for the ClassEval subset. The first is the classic Text-to- Code generation task, which evaluates the ability of LLMs to synthesize syntactically valid Cangjie code from natural language instructions. In addi- tion to this, we introduce a Code-to-Code transla- tion task, which aims to evaluate the modelâs code translation capacity from a high-resource program- ming language to a low-resource programming lan- guage. To evaluate correctness, we follow the testing logic of the original benchmarks. For HumanEval, a sample is considered correct if the generated func- tion passes the test cases. For ClassEval, the criteria are stricter: a sample is correct only if all methods within the class pass their tests. Additionally, we prepare a main test case to verify the overall func- tionality of the generated class. 3.3 Evaluation Framework To ensure an automated, secure, and reproducible evaluation process, we build a dedicated evaluation sandbox encapsulated via Docker 5 . It integrates the complete Cangjie runtime, standard libraries, and testing frameworks to form a lightweight, iso- lated execution environment. Beyond executing the test suites for CANGJIEBENCH, the sandbox also supports the running of any Cangjie code. 4 Methods It is worth noting that we exclude fine-tuning ap- proaches in this study. The primary motivation be- hind CANGJIEBENCH is not to engineer a special- ized model via massive data, but rather to probe the 5 https://github.com/cjhCoder7/CangjieBench/ tree/main/CangjieBench 4 Direct Generation Syntax-Constrained Generation Grammar Rules LLM + Question Cangjie Code Agent Usage Guides API Docs Iterative Query Agent LLM RAG Code Repos Usage Guides API Docs Query Transform Search Figure 2: Overview of our evaluated methods. generalization boundaries of existing foundation models. We aim to investigate how well current LLMs can transfer their vast programming knowl- edge from high-resource languages to a syntac- tically distinct, unseen general-purpose language without parameter updates. In practice, when a new language emerges, developers typically lack the compute, data, or time to immediately fine-tune an LLM, making ICL and agentic approaches the most practical options. As Figure 2 shows, we fo- cus on four mainstream paradigms for code-related tasks: Direct Generation, Syntax-Constrained Gen- eration, Retrieval-Augmented Generation (RAG), and Agent-based approaches. Direct Generation. In this setting, the model is provided solely with the natural language problem description (for Text-to-Code) or the Python source code (for Code-to-Code), wrapped in a prompt tem- plate (see Appendix A.8). The model relies entirely on its pre-trained weights to infer the syntax and semantics of Cangjie. Syntax-Constrained Generation. Cangjie is likely absent from the pre-training data of most models, so they frequently rely on heuristics from other high-resource languages, leading to invalid syntax. To mitigate this, we define a set of simpli- fied yet comprehensive Cangjie grammar rules to guide LLMs. These rules, injected into the prompt, cover essential syntactic structures, type defini- tions, and standard library interfaces (see Appendix A.4.1). This method evaluates whether providing concise grammar rules is sufficient to guide the modelâs generalization via in-context learning. RAG. We implement two strategies, i.e., RAG (Docs) and RAG (Code). The first targets official usage guides and API documentation. Instead of using the raw problem description directly, it em- ploys a query transformation approach: the model first generates precise keywords to retrieve the most relevant documentation segments, ensuring the re- trieved context closely aligns with the problem. The second strategy leverages a curated repository of crawled Cangjie code snippets to supply few- shot examples for models. We use lexical matching (via BM25) in RAG. More details are provided in Appendix A.4.2. Agent. Unlike RAG where retrieval is prede- termined by the system, this approach employs a CLI-based agent to autonomously consult official Cangjie usage guides and API references. This setup simulates a realistic development process where a program beginner proactively looks up syn- tax details or library functions as needed. By grant- ing the model the agency to decide what to look up and when, this method evaluates the modelâs ability to plan and execute a research-driven workflow to bridge its knowledge gap. 5 Experiments In this section, we evaluate the performance of cur- rent state-of-the-art LLMs using our four proposed methods on CANGJIEBENCH. 5.1 Experimental Setup Evaluated LLMs. We evaluated six open-source and closed-source LLMs on CANGJIEBENCH. To ensure we assess state-of-the-art performance, we specifically selected open-source models exceed- ing 200B parameters and widely adopted closed- source model. This includes: DeepSeek-V3 (Liu et al., 2024), ERNIE-4.5 (Team, 2025), Kimi-K2 (Team et al., 2025), Qwen3, Qwen3-Coder (Yang et al., 2025a), and GPT-5 (OpenAI, 2025b). Spe- cific details are provided in Appendix A.5.1. Evaluated Agents. We pair each agent with its rec- ommended backbone: (1) Codex CLI with GPT-5; (2) Qwen Code CLI with Qwen3-Coder; and (3) iFlow CLI with Qwen3-Coder and Kimi-K2. 5 MethodModel HumanEvalClassEvalAvg. Pass@1CompilePass@1CompilePass@1Compile Direct DeepSeek-V33.03.03.03.01.21.21.21.22.12.12.12.1 ERNIE-4.54.34.94.34.90.01.20.01.22.23.12.23.1 Kimi-K223.823.823.823.87.18.38.38.315.516.116.116.1 Qwen3 4.34.34.34.31.22.41.22.42.83.42.83.4 Qwen3-Coder4.37.94.37.91.21.21.21.22.84.62.84.6 GPT-57.38.57.38.51.23.61.23.64.36.14.36.1 Syntax- Constrained DeepSeek-V347.644.547.644.516.76.017.96.032.225.332.825.3 ERNIE-4.539.035.440.935.42.46.03.66.020.720.722.320.7 Kimi-K262.256.162.256.122.615.523.815.542.435.843.035.8 Qwen3 57.347.657.347.622.614.322.614.340.031.040.031.0 Qwen3-Coder47.651.851.851.822.611.923.813.135.131.937.832.5 GPT-567.145.167.145.140.531.040.531.053.838.153.838.1 RAG (Code) DeepSeek-V316.515.916.515.93.64.83.64.810.110.410.110.4 ERNIE-4.515.918.915.918.96.03.66.03.611.011.311.011.3 Kimi-K233.531.734.231.713.19.513.19.523.320.623.720.6 Qwen313.47.913.47.93.67.13.67.18.57.58.57.5 Qwen3-Coder24.418.324.418.37.17.17.17.115.812.715.812.7 GPT-549.447.049.447.013.113.113.113.131.330.131.330.1 RAG (Docs) DeepSeek-V333.522.633.522.69.59.59.59.521.516.121.516.1 ERNIE-4.513.412.813.412.83.63.63.63.68.58.28.58.2 Kimi-K234.834.136.634.111.916.711.916.723.425.424.325.4 Qwen3 12.812.212.812.21.24.81.24.87.08.57.08.5 Qwen3-Coder22.618.922.618.96.09.56.09.514.314.214.314.2 GPT-537.232.337.232.315.516.715.516.726.424.526.424.5 Agent Kimi-K2 (iFlow CLI)44.551.844.551.826.227.426.228.635.439.635.440.2 Qwen3-Coder (iFlow CLI) 32.333.532.333.59.516.710.717.920.925.121.525.7 Qwen3-Coder (Qwen Code CLI) 30.527.430.527.48.38.38.38.319.417.919.417.9 GPT-5 (Codex CLI) 87.287.887.287.867.965.567.965.577.676.777.676.7 Table 2: Main results on CANGJIEBENCH benchmarks.Light graycolumns indicate Text-to-Code task, while dark graycolumns represent Code-to-Code task. The best result for each method is highlighted in red, and the overall best result across all methods isunderlined. Metrics. We employ three primary metrics to com- prehensively evaluate both the correctness of the generated results and the computational cost of dif- ferent methods: Pass@1 (Chen, 2021), Compile Rate and Token Usage. Appendix A.5.3 provides more detailed definitions. 5.2 Results on Text-to-Code Task Based on the results on the CANGJIEBENCH text- to-code task in Table 2, we can observe several significant trends: Challenges in Cangjie Code. The results from the Direct Generation confirm the extreme diffi- culty of generating Cangjie code without assistance. Specifically, most models, including DeepSeek- V3 and Qwen3, achieve negligible pass@1 scores (<5%), and even GPT-5 only reaches 4.3% on av- erage. Crucially, the Compile Rates are nearly identical to these low pass@1 scores. This implies that the modelsâ poor performance is not due to logical flaws in algorithm, but rather a fundamental deficiency in syntactic knowledge. The vast ma- jority of generated samples simply fail to compile, confirming that Cangjie is effectively absent from the pre-training corpora. The only outlier is Kimi- K2, which achieves a moderate pass@1 (15.5%), suggesting potential differences in its training data distribution. Efficacy of Syntax-Constraints. A significant performance breakthrough is observed when ap- plying the Syntax-Constrained method. By sim- ply injecting expert-curated grammar rules into the prompt, GPT-5âs average pass@1 surges from 4.3% to 53.8% and other models also saw a relatively significant increase under the same method. This result highlights a critical insight: SOTA LLMs already possess the abstract algorithmic logic re- quired to solve these problems, which is transferred from high-resource languages; their primary bottle- neck is merely the lack of surface-level syntactic knowledge. RAG vs. Syntax-Constraints. While RAG (Code) and RAG (Docs) improve performance over the direct baseline, they generally underperform compared to the Syntax-Constrained method. For example, GPT-5 with RAG (Code) achieves 31.3%, significantly lower than the 53.8% achieved with syntax constraints. This suggests that for a com- pletely unseen language, explicit and structured grammatical guidance is more effective than re- trieving noisy few-shot examples or fragmented 6 Better Performance & Fewer Token Ideal Case Syntax-Constrained + GPT-5 Token: 100k Figure 3: Performance-Cost trade-off on Text-to-Code task. documentation, which the model may struggle to generalize from in a zero-shot manner. Specifi- cally, RAG (Code) fails because LLMs struggle to generalize complex grammar from isolated code snippets, while RAG (Docs) fails primarily due to poor query generation, as LLMs lacking Cangjie knowledge produce low-quality search keywords that cause retriever failure at the source. Agent Capabilities and Limitations.The Agent paradigm yields the state-of-the-art result, with the GPT-5 driven Codex CLI achieving an impressive 77.6% pass@1. Unlike RAG, where retrieval is passive and predetermined, the Agent autonomously consults documentation to bridge specific knowledge gaps, simulating a human de- veloperâs âresearch-then-codeâ workflow. How- ever, it demands high-level planning and reasoning capabilities. Agents driven by Qwen3-Coder and Kimi-K2 show limited improvement compared to their Syntax-Constrained methods. This indicates that open-resource models struggle to effectively decide what to look up or how to integrate the re- trieved syntax into their generation logic. 5.3 Results on Code-to-Code Task The Code-to-Code task evaluates the modelsâ abil- ity to translate Python solutions into Cangjie. Our analysis focuses on the following aspects: Performance Comparison. Similar to the Text- to-Code task, the Direct translation performance is poor across most models due to the lack of Cangjie knowledge. The ranking of models remains con- sistent, with GPT-5 generally leading the perfor- mance in advanced settings (Syntax-Constrained and Agent), while open-resource models struggle to generate compilable code. Syntax as the Primary Barrier. Comparing the two tasks, we observe that Code-to-Code trans- lation does not offer a significant advantage over Text-to-Code generation. In low-resource settings, the primary challenge is not deriving the algorith- mic logic (which is provided in the Python source) but rather generating syntactically valid code. The low Compile Rates across both tasks indicate that regardless of the input modality (Natural Language or Python Code), the models struggle equally to generate correct Cangjie Code. The Interference of Source Language. Closer examination reveals an important phenomenon: having access to the source code can sometimes reduce performance. Notably, under the Syntax- Constrained method, GPT-5 achieves a pass@1 of 53.8% on the Text-to-Code task but significantly drops to 38.1% on the Code-to-Code task. We hypothesize that this degradation is caused by the modelâs tendency to overfit to the Python source patterns. When translating, the model is prone to imitating Pythonâs dynamic typing and idioms line-by-line, different from Cangjieâs static syn- tax. Conversely, Text-to-Code allows the model to generate the target structure directly, avoiding the interference of the source languageâs syntax. 5.4 Token Usage We analyze the token consumption across different methods, as illustrated in Figure 3 and Figure 4. Performance-Cost Efficiency. Figure 3 illus- trates the trade-off between model performance (Pass@1) and cost (Token Usage). The top-left corner represents the ideal region, defined as the area achieving maximum accuracy with minimal token consumption. We observe that the Syntax- Constrained method clusters closest to this optimal region. Compared to the Direct method, it yields 7 Direct Syntax- Constrained RAG (Code) RAG (Docs) Agent 10 2 10 3 10 4 10 5 Token Usage (Log Scale) 37.6% 62.4% 1,313 79.4% 3,556 66.6% 33.4% 2,214 84.2% 5,171 99.1% 504,972 Input (Prompt)Output (Completion) Figure 4: Analysis of token consumption for GPT-5 on Text-to-Code task. We report the total token usage (log scale) split into Input and Output percentages. substantial performance gains with only a slight increase in token usage, demonstrating the high- est cost-effectiveness. In contrast, while the Agent method (specifically Codex CLI) achieves superior performance, its cost control falls short. The itera- tive process of constantly querying documentation and analyzing feedback results in excessive token consumption. Token Composition. Figure 4 details the ra- tio of input (prompt) to output (completion) to- kens. Methods like Syntax-Constrained and RAG maintain a relatively balanced distribution, with input percentages ranging from 66.6% to 84.2%. This indicates that these methods efficiently utilize contextâsuch as grammar rules or retrieved snip- petsâto guide generation without overwhelming the context window. In contrast, the Agent method exhibits an extreme skew, where input tokens ac- count for 99.1% of the total usage. This extreme ratio indicates low generation efficiency, as the vast majority of computational resources are spent on reading context rather than producing valid code. This is because the agent must repeatedly read ex- tensive documentation and interaction history dur- ing its iterative workflow. While this massive input improves accuracy, the resulting high cost and la- tency make the Agent less suitable for applications. 6 Discussion The Future Direction of Low-Resource Code Generation. Traditionally, adapting models to a new language involves expensive pre-training or fine-tuning with massive datasets. However, as new programming languages constantly emerge, the training data inevitably lags behind. First, the Syntax-Constrained method demonstrates that high-quality context can substitute for weight up- dates. Since LLMs already understand algorithmic logic from high-resource languages, future work should focus on developing automated methods to extract and inject minimal yet effective gram- mar rules, which could allow models to master any new language instantly. Notably, our Pass@10 experiments (Appendix A.7) show that the Syntax- Constrained method with an expanded sampling budget achieves dramatic improvements, suggest- ing models can correct syntax errors when allowed to explore the syntax space. Second, we observe a clear trade-off between performance and cost. While Agents achieve the best results, their high token consumption makes them impractical for ap- plications. A promising direction is to use Syntax- Constrained method for simple tasks and dynami- cally switch to an Agent approach only when the code fails to compile or requires complex library usage. The Future Direction of Cross-Lingual Trans- fer. A surprising finding is that source code (Code- to-Code) can sometimes hinder performance, as LLMs tend to overfit to the source languageâs syn- tax. Future research should explore semantics- aligned translation. Instead of translating code line- by-line, models should perhaps first summarize the source code into a intermediate representation be- fore generating the target code. This would help strip away the âsyntax noiseâ of the source lan- guage, allowing the model to focus on correctly implementing the logic in the target languageâs structure. 7 Conclusion In this work, we introduced CANGJIEBENCH, a contamination-free benchmark for Cangjie derived from HumanEval and ClassEval. Unlike prior DSL- focused benchmarks, it targets general-purpose syn- tax and supports both code generation and transla- tion tasks. We systematically evaluated four strate- gies, finding that Syntax-Constrained Generation offers the best balance between cost and accuracy, while Agent-based methods achieve the state-of- the-art. Surprisingly, our experiments reveal that Code-to-Code translation often suffers from neg- ative transfer compared to Text-to-Code genera- tion. CANGJIEBENCH aims to drive future research 8 in efficient adaptation and generalization for low- resource languages. Limitations The official version of Cangjie was released on July 1, 2025. LLMs used in our experiments have training data cutoff dates before this time. However, as Cangjie becomes more popular, future versions of LLMs will inevitably include Cangjie code in their training sets. Therefore, the low-resource status of our benchmark may change in the future. Despite this, Cangjie currently serves as an ideal window to rigorously test how models generalize to low-resource languages. We acknowledge that HarmonyOS also uses ArkTS for application development. However, ArkTS is a superset of TypeScript. Since current LLMs are already very good at TypeScript, ArkTS does not present a major challenge for them. In contrast, Cangjie is a natively developed general- purpose language. Its syntax is distinct from other mainstream languages. We chose Cangjie to strictly test the modelâs ability to learn a new syntax. Additionally, our current benchmark mainly fo- cuses on standalone code snippets. However, real- world Cangjie development is often more complex. It usually involves multiple files, external depen- dencies, and interactions across a whole project. This creates a gap between our current tests and actual coding scenarios. In preliminary cross-file experiments on two multi-file Cangjie reposito- ries (Markdown4cj and Httpclient4cj), all evalu- ated models achieved near-zero success rates, as they could neither correctly infer syntax rules nor adhere to cross-file API contracts. Future research should consider Cangjie code generation and trans- lation at the repository level to better reflect these real-world challenges. Finally, we could not test all available LLMs on the market due to the high cost of APIs. Instead, we selected a representative set of high-performance models from both open-source and closed-source categories. Future research could expand this eval- uation to a wider range of models. Ethical Considerations The construction of CANGJIEBENCH strictly ad- heres to the licensing terms of all source materials. The core evaluation dataset is derived from Hu- manEval and ClassEval. Both source datasets are distributed under the MIT License, which permits modification, distribution, and private use. Conse- quently, our manually translated derivative work complies with these legal frameworks. For the RAG experiments, we utilize two types of external knowledge: official documentation and public code repositories. The official Cangjie doc- umentation is used in accordance with the C-BY- 4.0 license, allowing us to incorporate it into our retrieval corpus with appropriate attribution. Re- garding the public Cangjie repositories used for code retrieval, we do not redistribute the raw code data directly. Instead, we release only the repos- itory metadata. This ensures that access to the data remains subject to the original repositoriesâ availability and licensing terms, thereby avoiding copyright infringement. References Wasi Ahmad, Md Golam Rahman Tushar, Saikat Chakraborty, and Kai-Wei Chang. 2023. Avatar: A parallel corpus for java-python program translation. InFindingsoftheAssociationforComputational Linguistics:ACL2023, pages 2268â2281. Anthropic. 2025. Claude code. URLhttps://w. claude.com/product/claude-code. Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, and 1 others. 2021. Program synthesis with large language models.arXivpreprintarXiv:2108.07732. Federico Cassano, John Gouwar, Daniel Nguyen, Syd- ney Nguyen, Luna Phipps-Costin, Donald Pinckney, Ming-Ho Yee, Yangtian Zi, Carolyn Jane Anderson, Molly Q Feldman, and 1 others. 2023. Multipl-e: A scalable and polyglot approach to benchmarking neu- ral code generation.IEEETransactionsonSoftware Engineering, 49(7):3675â3691. Mark Chen. 2021. Evaluating large language models trained on code.arXivpreprintarXiv:2107.03374. Artem Chervyakov, Alexander Kharitonov, Pavel Zadorozhny, Adamenko Pavel, Rodion Levichev, Dmitrii Vorobev, Dmitrii Salikhov, Aidar Valeev, Alena Pestova, Maria Dziuba, and 1 others. 2025. Mera code:A unified framework for evaluat- ing code generation across tasks.arXivpreprint arXiv:2507.12284. Etienne Daspe, Mathis Durand, Julien Hatin, and Salma Bradai. 2024. Benchmarking large language mod- els for ethereum smart contract development. In 20246thConferenceonBlockchainResearch& ApplicationsforInnovativeNetworksandServices (BRAINS), pages 1â4. IEEE. 9 Xueying Du, Mingwei Liu, Kaixin Wang, Hanlin Wang, Junwei Liu, Yixuan Chen, Jiayi Feng, Chaofeng Sha, Xin Peng, and Yiling Lou. 2023.Classe- val: A manually-crafted benchmark for evaluating llms on class-level code generation.arXivpreprint arXiv:2308.01861. Minghua He, Yue Chen, Fangkai Yang, Pu Zhao, Wen- jie Yin, Yu Kang, Qingwei Lin, Saravan Rajmohan, and Dongmei Zhang. 2025. Execoder: Empowering large language models with executability represen- tation for code translation. InProceedingsofthe 2025ConferenceonEmpiricalMethodsinNatural LanguageProcessing, pages 7110â7136. Huawei. 2025.Cangjie language documentation (v1.0.0).URLhttps://cangjie-lang.cn/en/ docs?url=%2F1.0.0%2F. Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar- Lezama, Koushik Sen, and Ion Stoica. 2024. Live- codebench: Holistic and contamination free eval- uation of large language models for code.arXiv preprintarXiv:2403.07974. Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. 2023. Swe-bench: Can language mod- els resolve real-world github issues?arXivpreprint arXiv:2310.06770. Sathvik Joel, Jie Wu, and Fatemeh Fard. 2024. A survey on llm-based code generation for low-resource and domain-specific programming languages.ACMTransactionsonSoftware EngineeringandMethodology. Sahil Kale and Vijaykant Nadadur. 2025. Texpert: A multi-level benchmark for evaluating latex code gen- eration by llms.arXivpreprintarXiv:2506.16990. Anirudh Khatry, Robert Zhang, Jia Pan, Ziteng Wang, Qiaochu Chen, Greg Durrett, and Isil Dil- lig. 2025. Crust-bench: A comprehensive bench- mark for c-to-safe-rust transpilation.arXivpreprint arXiv:2504.15254. Jinyang Li, Binyuan Hui, Ge Qu, Jiaxi Yang, Binhua Li, Bowen Li, Bailin Wang, Bowen Qin, Ruiying Geng, Nan Huo, and 1 others. 2023. Can llm already serve as a database interface? a big bench for large-scale database grounded text-to-sqls.AdvancesinNeural InformationProcessingSystems, 36:42330â42357. Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, and 1 others. 2024. Deepseek-v3 technical report.arXivpreprint arXiv:2412.19437. Mingjie Liu, Nathaniel Pinckney, Brucek Khailany, and Haoxing Ren. 2023.Verilogeval: Evaluat- ing large language models for verilog code gener- ation. In2023IEEE/ACMInternationalConference onComputerAidedDesign(ICCAD), pages 1â8. IEEE. Yang Liu, Li Zhang, Fang Liu, Zhuohang Wang, Donglin Wei, Zhishuo Yang, Kechi Zhang, Jia Li, and Lin Shi. 2025. Reposcope: Leveraging call chain- aware multi-view context for repository-level code generation.arXivpreprintarXiv:2507.14791. OpenAI. 2025a. Introducing codex. URLhttps:// openai.com/index/introducing-codex/. OpenAI. 2025b. Introducing gpt-5. URLhttps:// openai.com/index/introducing-gpt-5/. Anne Ouyang, Simon Guo, Simran Arora, Alex L Zhang, William Hu, Christopher RĂŠ, and Azalia Mirhoseini. 2025. Kernelbench: Can llms write effi- cient gpu kernels?arXivpreprintarXiv:2502.10517. Zhiyuan Peng, Xin Yin, Rui Qian, Peiqin Lin, Yongkang Liu, Hao Zhang, Chenhao Ying, and Yuan Luo. 2025. Soleval: Benchmarking large language models for repository-level solidity code generation.arXiv preprintarXiv:2502.18793. Haritz Puerto, Martin Tutek, Somak Aditya, Xiaodan Zhu, and Iryna Gurevych. 2024. Code prompting elicits conditional reasoning abilities in text+ code llms. InProceedingsofthe2024Conferenceon EmpiricalMethodsinNaturalLanguageProcessing, pages 11234â11258. Baidu ERNIE Team. 2025. Ernie 4.5 technical report. Kimi Team, Yifan Bai, Yiping Bao, Guanduo Chen, Jiahao Chen, Ningxin Chen, Ruijue Chen, Yanru Chen, Yuankun Chen, Yutian Chen, and 1 others. 2025. Kimi k2: Open agentic intelligence.arXiv preprintarXiv:2507.20534. Amitayush Thakur, Jasper Lee, George Tsoukalas, Meghana Sistla, Matthew Zhao, Stefan Zetzsche, Greg Durrett, Yisong Yue, and Swarat Chaud- huri. 2025.Clever: A curated benchmark for formally verified code generation.arXivpreprint arXiv:2505.13938. Prashanth Vijayaraghavan, Luyao Shi, Stefano Am- brogio, Charles Mackin, Apoorva Nitsure, David Beymer, and Ehsan Degan. 2024. Vhdl-eval: A framework for evaluating large language models in vhdl code generation. In2024IEEELLMAided DesignWorkshop(LAD), pages 1â6. IEEE. Ngoc Phuoc An Vo, Brent Paulovicks, and Vadim Sheinin. 2024. Tackling execution-based evaluation for nl2bash.CoRR. Hanbin Wang, Zhenghao Liu, Shuo Wang, Ganqu Cui, Ning Ding, Zhiyuan Liu, and Ge Yu. 2024. Inter- venor: Prompting the coding ability of large lan- guage models with the interactive chain of repair. InFindingsoftheAssociationforComputational Linguistics:ACL2024, pages 2081â2107. Min Xue, Artur Andrzejak, and Marla Leuther. 2024.An interpretable error correction method for enhancing code-to-code translation.In 10 TheTwelfthInternationalConferenceonLearning Representations. Pengyu Xue, Linhao Wu, Zhen Yang, Chengyi Wang, Xiang Li, Yuxiang Zhang, Jia Li, Ruikai Jin, Yifei Pei, Zhaoyan Shen, and 1 others. 2025. Classeval-t: Evaluating large language models in class-level code translation.ProceedingsoftheACMonSoftware Engineering, 2(ISSTA):1421â1444. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, and 1 others. 2025a. Qwen3 technical report.arXivpreprint arXiv:2505.09388. Zezhou Yang, Sirong Chen, Cuiyun Gao, Zhenhao Li, Xing Hu, Kui Liu, and Xin Xia. 2025b. An empir- ical study of retrieval-augmented code generation: Challenges and opportunities.ACMTransactionson SoftwareEngineeringandMethodology. Hao Yu, Bo Shen, Dezhi Ran, Jiaxin Zhang, Qi Zhang, Yuchi Ma, Guangtai Liang, Ying Li, Qianxiang Wang, and Tao Xie. 2024. Codereval: A benchmark of prag- matic code generation with generative pre-trained models.InProceedingsofthe46thIEEE/ACM InternationalConferenceonSoftwareEngineering, pages 1â12. Longhui Zhang, Bin Wang, Jiahao Wang, Xiaofeng Zhao, Min Zhang, Hao Yang, Meishan Zhang, Yu Li, Jing Li, and Jun Yu. 2025. Function-to-style guid- ance of llms for code translation. InForty-second InternationalConferenceonMachineLearning. Qianhui Zhao, Li Zhang, Fang Liu, Junhang Cheng, Chengru Wu, Junchen Ai, Qiaoyuanhe Meng, Lichen Zhang, Xiaoli Lian, Shubin Song, and 1 others. 2025. Towards realistic project-level code generation via multi-agent collaboration and semantic architecture modeling.arXivpreprintarXiv:2511.03404. Zixiao Zhao and Fatemeh Fard. 2025. Do current language models support code intelligence for r programming language?ACMTransactionson SoftwareEngineeringandMethodology, 34(8):1â 39. A Appendix A.1 Cangjie Programming Language Figure 8 illustrates the syntactic distinctions be- tween Cangjie and other programming languages through concise code snippets. A.2CANGJIEBENCH Statics The final CANGJIEBENCH dataset comprises 248 high-quality samples. For the HumanEval source, we successfully migrated all 164 original problems, as they primarily focus on fundamental algorithmic logic and standard data structures that map directly to the Cangjie standard library. For the ClassEval source, we retained 84 out of the original 100 problems. We excluded 16 tasks that heavily relied on specific Python third- party ecosystems which currently lack equivalents in Cangjie. These exclusions ensure that the bench- mark remains fair and does not penalize models for the lack of language maturity. Table 3 provides a detailed list of the excluded tasks and the specific reasons for their removal. A.3 Example of Problem Figure 10 and Figure 9 illustrate sample problems from the HumanEval and ClassEval subsets of CANGJIEBENCH, respectively. A.4 Details of Methods This section provides specific implementation de- tails for the methods used in our experiments. A.4.1 Syntax-Constrained Generation To guide the LLMs in generating valid Cangjie code, we injected a concise set of grammar rules into the input prompt. We defined 20 specific categories of constraints to cover the essential as- pects of the Cangjie: Program Structure, Vari- ables, Numeric Types, Strings and Characters, Boolean Type, Arrays and Dynamic Arrays, Hash Maps, Hash Sets, Sorting, Tuples, Con- ditional Statements, Loop Statements, Option Type, Match Expressions, Lambda Expressions, Input and Output, Functions, Classes, Inter- faces, Keywords. This grammar introduction com- prises 2,146 tokens based on the GPT-5 tokenizer. These rules serve as a quick reference for the model to understand the distinct syntax of Cangjie via in- context learning. A.4.2 RAG Corpus Preparation We constructed two dis- tinct corpora to support our RAG experiments: â˘Official Documentation Corpus: Derived from official GitCode repository 6 , this corpus consists of the Cangjie Developer Manual and Standard Library API documentation. To better improve retrieval, we pre-processed the data by remov- ing HTML tags. We also divided the text into segments based on headers. â˘Cangjie Codebase Corpus: We established a high-quality code repository by crawling open- source projects from GitCode 7 . We employed 6 https://gitcode.com/Cangjie/CangjieCorpus 7 https://gitcode.com/ 11 Probelem IDProblem NameExclusion Reason ClassEval_14 BookManagementDBDependency on sqlite3 for database operations ClassEval_25 CookiesUtilDependency on json for data serialization ClassEval_28 DatabaseProcessorDependency on sqlite3 for database operations ClassEval_34 DocFileHandlerDependency on docx for Word document processing ClassEval_38 ExcelProcessorDependency on openpyxl for Excel spreadsheet manipulation ClassEval_44 HtmlUtilDependency on bs4 for parsing and gensim for NLP processing ClassEval_45 ImageProcessorDependency on PIL for image processing ClassEval_50 JSONProcessorDependency on json for data serialization ClassEval_52 LemmatizationDependency on nltk for NLP processing ClassEval_60 MovieTicketDBDependency on sqlite3 for database operations ClassEval_69 PDFHandlerDependency on PyPDF2 for PDF document manipulation ClassEval_83 StudentDatabaseProcessorDependency on sqlite3 for database operations ClassEval_84 TextFileProcessorDependency on json for data serialization ClassEval_92 UserLoginDBDependency on sqlite3 for database operations ClassEval_98 XMLProcessorDependency on xml for XML data parsing ClassEval_99 ZipFileProcessorDependency on zipfile for ZIP archive management Table 3: List of excluded ClassEval tasks and exclusion reasons. the keyword âCangjieâ for initial search. To ensure corpus quality, we applied a strict filter- ing criterion, retaining only repositories where Cangjie constitutes more than 80% of the code- base. The collected files were parsed using ab- stract syntax trees (AST) to extract standalone functions and class definitions. Data collection was concluded prior to November 10. The final codebase corpus consists of 30,341 functions and 31,483 classes. Retrieval StrategyWe tailored our retrieval strat- egy to the distinct nature of each corpus. For both strategies described below, we retrieve the topk = 3most relevant candidates to augment the generation context without introducing excessive noise. â˘Documentation Retrieval via Query Transfor- mation. Directly using problem descriptions for retrieval is often ineffective due to the semantic gap between the problem statement and technical documentation. To address this, we employ a Query Transformation method. An LLM acts as an intermediate agent, analyzing the input prob- lem to extract and generate technical keywords (e.g., specific library names, API methods, or pro- gramming concepts). These refined keywords are then used to query the Official Documentation corpus. â˘Code Retrieval via Direct Matching. For the Codebase corpus, the goal is to find syntactically or functionally similar implementations. Since the input prompt often contains code snippets or function signatures, we perform direct matching against the crawled code snippets. Retrieval Algorithm We employ the BM25 al- gorithm for similarity calculation: ⢠BM25: We utilize the Okapi BM25 algorithm, a probabilistic information retrieval model that ranks documents based on the term frequency- inverse document frequency (TF-IDF) schema. We utilized the standard implementation 8 with default parameters. As mentioned, the top 3 re- sults with the highest BM25 scores are selected as the retrieved context. A.4.3 Agent We implemented a CLI-based agent that interacts with the official documentation. To ensure the safety and integrity of the local environment, we use Git 9 for version control. The agent is granted read access to the documen- tation but might attempt invalid write operations. To prevent this, we track the documentation di- rectory with Git. After every interaction turn, the system automatically checks for file changes. If any modification is detected, the system executes a rollback command to revert the environment to its original state. This ensures a consistent and isolated environment for each inference step. 8 https://github.com/dorianbrown/rank_bm25 9 https://git-scm.com/ 12 A.5 Details of Experiments A.5.1 Details of Models To ensure that our experiments can explore the per- formance boundaries of current LLMs, we selected a set of high-performing SOTA models. These selections include both open-source and closed- source models, representing the most advanced LLMs developed by their respective organizations to date. For the inference process, we utilized the open- source models through the SiliconFlow platform 10 . And the closed-source models were accessed directly via their official APIs. For the generation parameters, we set themax_tokensto 8192, while keeping all other hyperparameters at modelâs de- fault values. This limit was chosen to guarantee that the models have capacity to complete their generation, as the token length of the code solu- tions for every problem in our dataset is well below 8192. The detailed specifications and sources for all models used in this study are listed in Table 4. A.5.2 Details of Agents Table 5 provides an overview of the CLI-based agent frameworks employed in our study, along with their corresponding open-source repositories. A.5.3 Metric Definitions In this section, we provide the formal definitions and motivations for the evaluation metrics used in our experiments. Pass@1 Following the standard evaluation met- ric established by HumanEval (Chen, 2021), we use Pass@1 to measure the functional correctness of the generated code. A solution is considered cor- rect if and only if it passes all the unit tests provided in the problem description. Formally, for a given problemP, letSbe the generated solution. The score for this problem is defined as: Score(P) = ( 1 if Eval(S, T ests P ) 0 otherwise (1) The final Pass@1 score is the average accuracy across the entire dataset. This metric serves as the primary indicator of the modelâs ability to solve programming tasks in the Cangjie programming language. 10 https://w.siliconflow.com/ Compile RateSince Cangjie is a statically typed, compiled language with strict syntax rules, the abil- ity to generate syntactically valid code is a prereq- uisite for functional correctness. Large Language Models, due to a lack of pre-training data, often hal- lucinate syntax from other high-resource languages, e.g., Python or Java. To quantify this, we define Compile Rate as the proportion of generated solutions that success- fully pass the Cangjie compiler without raising any syntax errors: Compile_Rate = 1 N N X i=1 I(Compiles(S i ))(2) whereNis the total number of problems andI(¡) is the indicator function. A high compile rate indi- cates that the model has successfully grasped the grammatical structure of the unseen language, even if the logic is incorrect. Token Usage While Agent and RAG workflows typically yield higher performance, they incur sig- nificantly higher computational costs due to multi- turn interactions and context retrieval. To evaluate the trade-off between performance and efficiency, we track Token Usage. We record the total number of tokens pro- cessedâincluding both input prompts and output completionsâacross the entire problem-solving session. This metric serves as a proxy for the in- ference latency and financial cost associated with each method. We adopt distinct calculation ways based on model accessibility: â˘Open-Source Models: For models with acces- sible weights (e.g., DeepSeek-V3, Qwen3), we utilize the official tokenizers provided via the Hugging Face transformers library. â˘Closed-Source Models: For closed-source mod- els (e.g., GPT-5), we log the exact token usage (prompt_tokensandcompletion_tokens) di- rectly from the API response metadata for every call. A.6 Full Results This section presents the comprehensive experi- mental data. Table 6 and Table 7 detail the de- tailed performance metrics for the Text-to-Code and Code-to-Code tasks, respectively. Table 8 pro- vides the baseline performance of the evaluated 13 Open Source Models Model NameModel SizeHuggingFace URL DeepSeek-V3671B https://huggingface.co/deepseek-ai/DeepSeek-V3-0324 ERNIE-4.5300B https://huggingface.co/baidu/ERNIE-4.5-300B-A47B-PT Kimi-K21T https://huggingface.co/moonshotai/Kimi-K2-Instruct-0905 Qwen3235B https://huggingface.co/Qwen/Qwen3-235B-A22B-Instruct-2507 Qwen3-Coder480B https://huggingface.co/Qwen/Qwen3-Coder-480B-A35B-Instruct Closed Source Models Model NameModel SizeBlog URL GPT-5 https://openai.com/index/introducing-gpt-5 Table 4: Overview of LLMs used in our experiments. Agent NameBackbone ModelsVersionGitHub URL Codex CLIGPT-50.63.0 https://github.com/openai/codex Qwen Code CLIQwen3-Coder0.2.3 https://github.com/QwenLM/qwen-code iFlow CLIQwen3-Coder, Kimi-K20.4.8 https://github.com/iflow-ai/iflow-cli Table 5: Source codes and frameworks for the evaluated Agents. models on the original Python datasets using the Direct generation method. Figure 5 illustrates the performance-cost trade- off for the Code-to-Code task. Figure 6 breaks down the input and output token distribution for Qwen-Coder, while Figure 7 compares the gener- ation efficiency across methods using the Pass@1 per 1k input tokens metric. Finally, the specific statistics about token consumption for each method are listed in Table 9 for the Text-to-Code task and Table 10 for the Code-to-Code task. A.7 Pass@10 Results To provide a fairer comparison with the Agent methodâs substantially larger token budget, we conducted Pass@10 experiments for the Syntax- Constrained method on the Text-to-Code task. As shown in Table 11, all models exhibit dramatic improvements under Pass@10, with several even surpassing the Agentâs Pass@1 results. This sug- gests that the low Pass@1 rates are primarily due to modelsâ difficulty in perfectly aligning all strict syntax constraints in a single generation; when al- lowed to explore the syntax space with a modestly expanded sampling budget, models can success- fully self-correct and produce compilable code. A.8 Prompts In this section, we present all the prompts utilized for CANGJIEBENCH, as illustrated in Figures 11, 12, 13, 14, 15, 16, 17, 18, 19 and 20. 14 MethodModel HumanEvalClassEval PassCompilePassCompileFuncMain %#%#%#%#%#%# Direct DeepSeek-V33.053.051.211.211.551.21 ERNIE-4.54.374.370.000.000.000.00 Kimi-K223.83923.8397.168.378.5297.16 Qwen34.374.371.211.211.241.21 Qwen3-Coder4.374.371.211.211.241.21 GPT-57.3127.3121.211.211.861.21 Syntax- Constrained DeepSeek-V347.67847.67816.71417.91515.85416.714 ERNIE-4.539.06440.9672.423.635.3182.42 Kimi-K2 62.210262.210222.61923.82025.48722.619 Qwen357.39457.39422.61922.61922.27622.619 Qwen3-Coder47.67851.88522.61923.82022.57722.619 GPT-567.111067.111040.53440.53435.412140.534 RAG (Code) DeepSeek-V316.52716.5273.633.634.7163.63 ERNIE-4.515.92615.9266.056.055.3186.05 Kimi-K2 33.55534.25613.11113.11114.95113.111 Qwen313.42213.4223.633.633.8133.63 Qwen3-Coder24.44024.4407.167.167.3257.16 GPT-5 49.48149.48113.11113.11113.54613.111 RAG (Docs) DeepSeek-V3 33.55533.5559.589.588.5299.58 ERNIE-4.513.42213.4223.633.633.5123.63 Kimi-K2 34.85736.66011.91011.91012.34211.910 Qwen3 12.82112.8211.211.210.621.21 Qwen3-Coder22.63722.6376.056.055.6196.05 GPT-5 37.26137.26115.51315.51313.74715.513 Agent Kimi-K2 (iFlow CLI)44.57344.57326.22226.22225.48726.222 Qwen3-Coder (iFlow CLI) 32.35332.3539.5810.7912.0419.58 Qwen3-Coder (Qwen Code CLI) 30.55030.5508.378.379.7338.37 GPT-5 (Codex CLI) 87.214387.214367.95767.95769.923967.957 Table 6: Full results on CANGJIEBENCH Text-to-Code task. The symbol % denotes the pass rate (pass@1 or success rate), and # denotes the count of successful instances. Func and Main refer to function-level and main- function-level evaluations on ClassEval, respectively. Best results per method are red, overall best areunderlined. Better Performance & Fewer Token Ideal Case Direct + Kimi-K2 Token: 100k Figure 5: Performance-Cost trade-off on the Code-to-Code task. 15 MethodModel HumanEvalClassEval PassCompilePassCompileFuncMain %#%#%#%#%#%# Direct DeepSeek-V3 3.053.051.211.211.241.21 ERNIE-4.54.984.981.211.211.241.21 Kimi-K223.83923.8398.378.378.8308.37 Qwen34.374.372.422.422.382.42 Qwen3-Coder7.9137.9131.211.211.241.21 GPT-58.5148.5143.633.634.1143.63 Syntax- Constrained DeepSeek-V344.57344.5736.056.055.3186.05 ERNIE-4.535.45835.4586.056.055.8206.05 Kimi-K256.19256.19215.51315.51313.24515.513 Qwen347.67847.67814.31214.31211.74014.312 Qwen3-Coder51.88551.88511.91013.11112.04111.910 GPT-545.17445.17431.02631.02627.59431.026 RAG (Code) DeepSeek-V315.92615.9264.844.844.4154.84 ERNIE-4.518.93118.9313.633.633.5123.63 Kimi-K231.75231.7529.589.5811.7409.58 Qwen37.9137.9137.167.167.6267.16 Qwen3-Coder18.33018.3307.167.167.6267.16 GPT-547.07747.07713.11113.11113.24513.111 RAG (Docs) DeepSeek-V322.63722.6379.589.589.1319.58 ERNIE-4.5 12.82112.8213.633.633.2113.63 Kimi-K2 34.15634.15616.71416.71415.85416.714 Qwen312.22012.2204.844.843.8134.84 Qwen3-Coder 18.93118.9319.589.589.4329.58 GPT-532.35332.35316.71416.71416.75416.714 Agent Kimi-K2 (iFlow CLI)51.88551.88527.42328.62427.59428.624 Qwen3-Coder (iFlow CLI)33.55533.55516.71417.91518.76416.714 Qwen3-Coder (Qwen Code CLI)27.44527.4458.378.378.8308.37 GPT-5 (Codex CLI)87.814487.814465.55565.55565.222365.555 Table 7: Full results on CANGJIEBENCH Code-to-Code task. The symbol % denotes the pass rate (pass@1 or success rate), and # denotes the count of successful instances. Func and Main refer to function-level and main- function-level evaluations on ClassEval, respectively. Best results per method are red, overall best areunderlined. Direct Syntax- Constrained RAG (Code) RAG (Docs) Agent (iFlow) Agent (Qwen Code CLI) 10 2 10 3 10 4 10 5 Token Usage (Log Scale) 46.1% 53.9% 1,097 82.6% 3,464 72.8% 27.2% 2,058 87.8% 4,662 97.6% 36,141 99.2% 218,845 Input (Prompt)Output (Completion) (a) Text-to-Code Task Direct Syntax- Constrained RAG (Code) RAG (Docs) Agent (iFlow) Agent (Qwen Code CLI) 10 2 10 3 10 4 10 5 Token Usage (Log Scale) 51.5% 48.5% 1,022 82.0% 3,514 74.8% 25.2% 2,367 88.2% 4,871 97.5% 35,394 99.3% 244,363 Input (Prompt)Output (Completion) (b) Code-to-Code Task Figure 6: Token usage analysis for Qwen-Coder on (a) Text-to-Code and (b) Code-to-Code tasks. We report the total token usage (log scale) split into Input and Output percentages. 16 Direct Syntax- Constrained RAG (Code) RAG (Docs) Agent (iFlow CLI) 0 5 10 15 20 25 30 35 Pass@1 per 1k input tokens 31.2 15.1 15.8 5.9 1.4 Kimi-K2 Direct Syntax- Constrained RAG (Code) RAG (Docs) Agent (iFlow CLI) 0 2 4 6 8 10 12 Pass@1 per 1k input tokens 5.5 12.3 10.5 3.5 0.6 Qwen3-Coder Direct Syntax- Constrained RAG (Code) RAG (Docs) Agent (Codex CLI) 0 5 10 15 20 Pass@1 per 1k input tokens 8.7 19.1 21.2 6.1 0.2 GPT-5 (a) Text-to-Code Task Direct Syntax- Constrained RAG (Code) RAG (Docs) Agent (iFlow CLI) 0 5 10 15 20 25 30 35 Pass@1 per 1k input tokens 31.2 12.7 11.8 5.9 1.8 Kimi-K2 Direct Syntax- Constrained RAG (Code) RAG (Docs) Agent (iFlow CLI) 0 2 4 6 8 10 12 Pass@1 per 1k input tokens 8.7 11.1 7.2 3.3 0.7 Qwen3-Coder Direct Syntax- Constrained RAG (Code) RAG (Docs) Agent (Codex CLI) 0.0 2.5 5.0 7.5 10.0 12.5 15.0 17.5 Pass@1 per 1k input tokens 11.9 13.4 17.2 5.2 0.2 GPT-5 (b) Code-to-Code Task Figure 7: Efficiency analysis measured by Pass@1 per 1k input tokens. We compare Kimi-K2, Qwen3-Coder and GPT-5 of different methods on (a) Text-to-Code and (b) Code-to-Code tasks. While Agent methods achieve high accuracy, their efficiency score is significantly lower due to massive token consumption. In contrast, Syntax- Constrained and RAG (Code) methods often demonstrate higher efficiency. Model HumanEval (Python)ClassEval (Python) PassPassFuncMain %#%#%#%# DeepSeek-V391.515032.12771.724666.756 ERNIE-4.5 93.915431.02670.024063.153 Kimi-K295.715735.73074.325572.661 Qwen393.915441.73579.027177.465 Qwen3-Coder 93.915444.13778.727082.169 GPT-596.315841.73576.726377.465 Table 8: Full results on the original Python versions of HumanEval and ClassEval benchmarks. All models utilize the Direct generation method. The symbol % denotes the pass rate (pass@1 or success rate), and # denotes the count of successful instances. Best results are highlighted in red. As shown in the table, the performance on Python is significantly higher compared to the Cangjie results, highlighting the substantial challenges posed by low-resource languages like Cangjie for LLMs. 17 MethodModel HumanEval (Avg. Tokens)ClassEval (Avg. Tokens) InputOutputTotalInputOutputTotal Direct DeepSeek-V3212.9121.5334.4795.8517.61,313.5 ERNIE-4.5251.4240.1491.4943.8993.11,936.9 Kimi-K2209.5128.1337.6782.9620.91,403.8 Qwen3214.4150.5 364.8797.7845.41,643.1 Qwen3-Coder214.4163.7378.0797.71,019.11,816.8 GPT-5209.7271.5481.3778.41,367.52,145.9 Syntax- Constrained DeepSeek-V32,550.9135.42,686.33,133.8590.73,724.5 ERNIE-4.52,940.4310.63,251.03,632.8844.84,477.6 Kimi-K22,520.5142.32,662.83,093.9543.93,637.8 Qwen32,568.4201.7 2,770.13,151.7714.53,866.2 Qwen3-Coder2,568.4206.1 2,774.53,151.71,002.14,153.8 GPT-52,539.7255.62,795.33,108.41,208.94,317.3 RAG (Code) DeepSeek-V3907.3116.61,023.92,118.7522.02,640.6 ERNIE-4.51,047.7229.91,277.62,462.11,035.53,497.6 Kimi-K2896.5127.41,023.92,062.2657.62,719.9 Qwen3915.0132.7 1,047.62,081.7953.53,035.2 Qwen3-Coder915.0155.1 1,070.12,081.7965.23,046.9 GPT-5896.8230.51,127.32,055.51,246.93,302.4 RAG (Docs) DeepSeek-V34,404.6243.94,648.55,377.3716.36,093.7 ERNIE-4.54,737.0360.45,097.45,753.91,142.96,896.8 Kimi-K23,483.3169.53,652.84,485.9565.05,051.0 Qwen33,575.6184.63,760.24,591.7866.65,458.3 Qwen3-Coder3,608.2192.53,800.74,578.9945.75,524.6 GPT-53,903.3287.74,191.14,806.91,344.56,151.5 Agent Kimi-K2 (iFlow CLI)23,981.0242.724,223.626,306.41,154.327,460.7 Qwen3-Coder (iFlow CLI)35,752.9365.036,117.834,831.21,334.536,165.8 Qwen3-Coder (Qwen Code CLI)223,526.01,312.8224,838.8210,736.42,115.1212,851.5 GPT-5 (Codex CLI)388,428.03,316.6 391,744.7612,463.85,737.2618,201.0 Table 9: Full token usage statistics on CANGJIEBENCH Text-to-Code task. We report the average Input (prompt), Output (completion), and Total token counts per problem. 18 MethodModel HumanEval (Avg. Tokens)ClassEval (Avg. Tokens) InputOutputTotalInputOutputTotal Direct DeepSeek-V3231.9125.3357.2814.8534.71,349.5 ERNIE-4.5277.4251.1528.5969.8920.41,890.2 Kimi-K2229.5151.4380.9802.9690.01,492.9 Qwen3234.4137.4 371.7817.7718.61,536.3 Qwen3-Coder234.4151.1385.5817.7840.81,658.4 GPT-5229.7272.3502.0798.41,178.81,977.2 Syntax- Constrained DeepSeek-V32,570.9144.42,715.33,153.8513.33,667.2 ERNIE-4.52,966.4258.73,225.13,658.8783.74,442.4 Kimi-K22,540.5136.52,677.03,113.9560.03,673.9 Qwen32,589.4164.6 2,754.03,172.7693.63,866.2 Qwen3-Coder2,589.4257.62,847.03,172.71,008.44,181.0 GPT-52,560.7267.52,828.33,129.41,210.34,339.7 RAG (Code) DeepSeek-V31,115.4120.41,235.82,468.1546.23,014.3 ERNIE-4.51,293.0246.41,539.42,892.1984.33,876.4 Kimi-K21,102.0121.81,223.82,399.2581.62,980.9 Qwen31,123.3133.81,257.22,416.8946.63,363.4 Qwen3-Coder1,123.3159.6 1,282.92,416.81,035.53,452.3 GPT-51,102.2244.81,347.02,390.31,155.73,546.0 RAG (Docs) DeepSeek-V34,606.3240.44,846.75,587.5723.06,310.5 ERNIE-4.54,979.5406.15,385.66,134.71,136.27,270.9 Kimi-K23,727.4158.33,885.84,824.9574.95,399.8 Qwen33,811.3176.83,988.14,971.9723.55,695.5 Qwen3-Coder3,712.4182.2 3,894.64,876.9972.35,849.2 GPT-54,126.8304.14,430.95,206.61,321.76,528.3 Agent Kimi-K2 (iFlow CLI)22,209.9283.3 22,493.222,549.41,126.923,676.3 Qwen3-Coder (iFlow CLI)31,008.1455.731,463.838,036.41,288.339,324.8 Qwen3-Coder (Qwen Code CLI)252,853.41,312.9254,166.3232,396.02,165.1234,561.1 GPT-5 (Codex CLI)318,735.13,227.4321,962.5514,826.55,926.7520,753.2 Table 10: Full token usage statistics on CANGJIEBENCH Code-to-Code task. We report the average Input (prompt), Output (completion), and Total token counts per problem. Model HumanEvalClassEvalAvg. Pass@1Pass@10Pass@1Pass@10Pass@1Pass@10 DeepSeek-V347.673.816.735.732.254.8 ERNIE-4.539.077.42.419.120.748.3 Kimi-K262.290.922.650.042.470.5 Qwen357.378.722.634.540.056.6 Qwen3-Coder47.668.922.632.135.150.5 GPT-567.197.640.566.753.882.2 Table 11: Pass@10 results for the Syntax-Constrained method on the Text-to-Code task. All models show substantial gains, indicating that syntax errors can be effectively corrected with an expanded sampling budget. 19 Figure 8: Comparative implementation of thesumPositivefunction in Cangjie versus 11 other programming languages. 20 QuestionId:ClassEval_Cangjie-05 Prompt: import std.collection.HashMap import std.collection.ArrayList import std.unicode.UnicodeStringExtension class AutomaticGuitarSimulator /* This class is an automatic guitar simulator that can interpret and play based on the input guitar sheet music. */ let play_text: String public init(text: String) /* Initialize the score to be played :param text:String, score to be played */ this.play_text = text public func interpret(display!: Bool = false): ArrayList<HashMap<String, String>> /* Interpret the music score to be played :param display:Bool, representing whether to print the interpreted score :return: list of HashMap, The HashMap includes two fields, Chord and Tune, which are letters and numbers, respectively. If the input is empty or contains only whitespace, an empty list is returned. >>> context = AutomaticGuitarSimulator("C53231323 Em43231323 F43231323 G63231323") >>> play_list = context.interpret(display: false) ['Chord': 'C', 'Tune': '53231323', 'Chord': 'Em', 'Tune': '43231323', 'Chord': 'F', 'Tune': '43231323', 'Chord': 'G', 'Tune': '63231323'] */ public func display(key: String, value: String): String /* Print out chord and play tune with following format: Normal Guitar Playing -- Chord: %s, Play Tune: %s :param key:String, chord :param value:String, play tune :return: String >>> context = AutomaticGuitarSimulator("C53231323 Em43231323 F43231323 G63231323") >>> context.display("C", "53231323") Normal Guitar Playing -- Chord: C, Play Tune: 53231323 */ Example of CANGJIEBENCH from ClassEval Figure 9: Example of CANGJIEBENCH from ClassEval Example of CANGJIEBENCH from HumanEval QuestionId:HumanEval_Cangjie-01 Prompt: import std.collection.ArrayList func separate_paren_groups(paren_string: String): ArrayList<String> /* Input to this function is a string containing multiple groups of nested parentheses. Your goal is to separate those group into separate strings and return the list of those. Separate groups are balanced (each open brace is properly closed) and not nested within each other Ignore any spaces in the input string. >>> separate_paren_groups('( ) (( )) (( )( ))') ['()', '(())', '(()())'] */ Figure 10: Example of CANGJIEBENCH from ClassEval 21 Prompt for Direct Generation on Text-to-Code Task You are an expert assistant for the Cangjie programming language. Write the complete code and wrap it in ```cangjie```. Do not give any explanation. --- cangjie_question Figure 11: Prompt for Direct Generation on Text-to-Code Task Prompt for Direct Generation on Code-to-Code Task You are an expert assistant for the Cangjie programming language. Write the complete code and wrap it in ```cangjie```. Do not give any explanation. --- This is the corresponding Python implementation code. python_code This is the Cangjie question. cangjie_question Figure 12: Prompt for Direct Generation on Code-to-Code Task Prompt for Syntax-Constrained Generation on Text-to-Code Task You are an expert assistant for the Cangjie programming language. Please answer questions based on the **syntax rules** provided by the user. Please adhere to the following principles: 1. **Prioritize explicit rules**: If the question is directly addressed in the provided syntax rules, answer strictly according to those rules. 2. **Reasonable inference (only when necessary)**: If the question is not explicitly covered but can be answered through **conservative and logically consistent analogy** based on other programming languages' syntax structures, naming conventions, or semantic patterns, you may attempt to answer. 3. **Keep answers concise and accurate**: Avoid redundant explanations. 4. **Import Statements**: This question code may not import the necessary packages required to resolve the problem. If necessary, import the required modules at the top of the program. Please read the specific syntax rules provided by the user before handling subsequent questions. And write the complete code and wrap it in ```cangjie```. --- ## Syntax Rules Below are the general syntax rules of the Cangjie language: syntax_rule --- ## Question cangjie_question Figure 13: Prompt for Syntax-Constrained Generation on Text-to-Code Task 22 Prompt for Syntax-Constrained Generation on Code-to-Code Task You are an expert assistant for the Cangjie programming language. Please answer questions based on the **syntax rules** provided by the user. Please adhere to the following principles: 1. **Prioritize explicit rules**: If the question is directly addressed in the provided syntax rules, answer strictly according to those rules. 2. **Reasonable inference (only when necessary)**: If the question is not explicitly covered but can be answered through **conservative and logically consistent analogy** based on other programming languages' syntax structures, naming conventions, or semantic patterns, you may attempt to answer. 3. **Keep answers concise and accurate**: Avoid redundant explanations. 4. **Import Statements**: This question code may not import the necessary packages required to resolve the problem. If necessary, import the required modules at the top of the program. Please read the specific syntax rules provided by the user before handling subsequent questions. And write the complete code and wrap it in ```cangjie```. --- ## Syntax Rules Below are the general syntax rules of the Cangjie language: syntax_rule --- ## Question This is the corresponding Python implementation code. python_code This is the Cangjie question. cangjie_question Figure 14: Prompt for Syntax-Constrained Generation on Code-to-Code Task Prompt for RAG (Code) on Text-to-Code Task You are an expert assistant for the Cangjie programming language. Write the complete code and wrap it in ```cangjie```. Do not give any explanation. --- ## Relative Cangjie Code relative_cangjie_code ## Question cangjie_question Figure 15: Prompt for RAG (Code) on Text-to-Code Task 23 Prompt for RAG (Code) on Code-to-Code Task You are an expert assistant for the Cangjie programming language. Write the complete code and wrap it in ```cangjie```. Do not give any explanation. --- ## Relative Cangjie Code relative_cangjie_code ## Question: This is the corresponding Python implementation code. python_code This is the Cangjie question. cangjie_question Figure 16: Prompt for RAG (Code) on Code-to-Code Task Prompt for RAG (Docs) on Text-to-Code Task You are an expert assistant for the Cangjie programming language. Write the complete code and wrap it in ```cangjie```. Do not give any explanation. --- ## Relative Document relative_doc ## Question cangjie_question Figure 17: Prompt for RAG (Docs) on Text-to-Code Task 24 Prompt for RAG (Docs) on Code-to-Code Task You are an expert assistant for the Cangjie programming language. Write the complete code and wrap it in ```cangjie```. Do not give any explanation. --- ## Relative Document relative_doc ## Question: This is the corresponding Python implementation code. python_code This is the Cangjie question. cangjie_question Figure 18: Prompt for RAG (Docs) on Code-to-Code Task Prompt for Agent on Text-to-Code Task You must use Cangjie to finish the code. Even though you are not familiar with Cangjie, you can use `ls`, `cat`, `grep` or other command in `CangjieCorpus/` directory to learn it. Write the complete code directly and wrap it in ```cangjie```. Note: 1. Use commands to learn Cangjie in `CangjieCorpus/` directory first. 2. Only output the solution codeă 3. Do not write test code. 4. Do not write into a file. --- cangjie_question Figure 19: Prompt for Agent on Text-to-Code Task 25 Prompt for Agent on Code-to-Code Task You must use Cangjie to finish the code. Even though you are not familiar with Cangjie, you can use `ls`, `cat`, `grep` or other command in `CangjieCorpus/` directory to learn it. Write the complete code directly and wrap it in ```cangjie```. Note: 1. Use commands to learn Cangjie in `CangjieCorpus/` directory first. 2. Only output the solution codeă 3. Do not write test code. 4. Do not write into a file. --- This is the corresponding Python implementation code. python_code This is the Cangjie question. cangjie_question Figure 20: Prompt for Agent on Code-to-Code Task 26