Paper deep dive

Beyond Text Matching: Towards Reference-Free Evaluation for Human-Oriented Binary Reverse Engineering

Xiuwei Shang, Li Hu, Xiao Jiang, Jieke Shi, Junda He, Zhou Yang, Shaoyin Cheng, Guoqiang Chen, Weiming Zhang, David Lo

Year: 2026Venue: arXiv preprintArea: cs.SEType: PreprintEmbeddings: 94

Intelligence

Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 94%

Last extracted: 8/10/2026, 4:00:14 AM

Summary

This paper introduces BinJudgeBench, the first expert-annotated, reference-free evaluation benchmark for Human-Oriented Binary Reverse Engineering (HOBRE). It systematically investigates the LLM-as-a-Judge paradigm for evaluating function name recovery, binary code summarization, and decompilation optimization. The study finds that LLM-as-a-Judge correlates better with human judgment (63.20%) than traditional metrics (35.04%). To address the variability in optimal judge configurations, the authors propose BinJudge, a lightweight routing mechanism that adaptively selects the best LLM configuration per sample, improving correlation by 4.5%-24.7% while reducing API costs.

Entities (10)

BinJudgeBench · dataset · 99%BinJudge · system · 98%HOBRE · concept · 98%LLM-as-a-judge · method · 97%Binary Code Summarization · task · 95%Decompilation Optimization · task · 95%Function Name Recovery · task · 95%IDA Pro · tool · 92%BERTScore · metric · 90%BLEU · metric · 90%

Relation Signals (8)

BinJudge → improves → correlation_with_human_experts

confidence 95% · BinJudge improves correlation with human experts by 4.5%-24.7%

Function Name Recovery → ispartof → HOBRE

confidence 95% · HOBRE... covering three representative tasks: function name recovery...

Decompilation Optimization → ispartof → HOBRE

confidence 95% · HOBRE... covering three representative tasks: ... and decompilation optimization.

Binary Code Summarization → ispartof → HOBRE

confidence 95% · HOBRE... covering three representative tasks: ... binary code summarization...

BinJudge → reduces → API_cost

confidence 95% · BinJudge... reduces API cost to 0.06×-0.84× of that of static best configurations

BinJudgeBench → supportsevaluationof → HOBRE

confidence 95% · We introduce BinJudgeBench, the first expert-annotated, reference-free evaluation benchmark based on multi-dimensional human judgment... for HOBRE

LLM-as-a-judge → outperforms → BERTScore

confidence 90% · LLM-as-a-Judge achieves an average correlation of 63.20% with human judgment, outperforming traditional automated metrics at 35.04%

LLM-as-a-judge → outperforms → BLEU

confidence 90% · LLM-as-a-Judge achieves an average correlation of 63.20% with human judgment, outperforming traditional automated metrics at 35.04%

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:Human-Oriented Binary Reverse Engineering (HOBRE) aims to transform decompiled pseudocode into a more human-friendly representation, thereby reducing the cognitive burden of reverse analysis and improving efficiency. However, reliably evaluating HOBRE outputs remains a fundamental challenge: human evaluation is costly, time-consuming, and difficult to scale, while existing automated metrics either require executable test cases and runtime environments that are often unavailable for real-world binaries, or rely on high-quality source code references that are typically inaccessible and fail to capture semantically equivalent but lexically diverse outputs. Although LLM-as-a-Judge paradigm is naturally well-suited to HOBRE evaluation, its effectiveness remains underexplored. This paper presents the first systematic investigation of the LLM-as-a-Judge paradigm for HOBRE across three representative tasks: function name recovery, binary code summarization, and decompilation optimization. We introduce BinJudgeBench, the first expert-annotated, reference-free evaluation benchmark based on multi-dimensional human judgment, where LLM-as-a-Judge achieves an average correlation of 63.20\% with human judgment, outperforming traditional automated metrics at 35.04\%. By analyzing judge configurations across backbone LLMs, prompting strategies, and decoding temperatures, we find that no ``one-size-fits-all'' configuration exists, as the optimal setup varies across tasks and individual samples. To address this, we propose BinJudge, which employs a lightweight routing mechanism to adaptively select the optimal judge configuration for each task and sample. BinJudge improves correlation with human experts by 4.5\%-24.7\% and reduces API cost to 0.06$\times$-0.84$\times$ of that of static best configurations, providing a scalable, cost-effective, and high-fidelity automated evaluation scheme for HOBRE.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

92,957 characters extracted from source content.

Expand or collapse full text

Beyond Text Matching: Towards Reference-Free Evaluation for Human-Oriented Binary Reverse Engineering Xiuwei Shang ∗ shangxw@mail.ustc.edu.cn University of Science and Technology of China Hefei, China Li Hu pdxbshx@mail.ustc.edu.cn University of Science and Technology of China Hefei, China Xiao Jiang jx_xiao@mail.ustc.edu.cn University of Science and Technology of China Hefei, China Jieke Shi † jiekeshi@smu.edu.sg Singapore Management University Singapore Junda He jundahe.2022@phdcs.smu.edu.sg Singapore Management University Singapore Zhou Yang zy25@ualberta.ca University of Alberta & Alberta Machine Intelligence Institute Edmonton, Canada Shaoyin Cheng †‡ sycheng@ustc.edu.cn University of Science and Technology of China Hefei, China Guoqiang Chen ch3nye@mail.ustc.edu.cn University of Science and Technology of China Hefei, China Weiming Zhang ‡ zhangwm@ustc.edu.cn University of Science and Technology of China Hefei, China David Lo davidlo@smu.edu.sg Singapore Management University Singapore Abstract Human-Oriented Binary Reverse Engineering (HOBRE) aims to transform decompiled pseudocode into a more human-friendly representation, thereby reducing the cognitive burden of reverse analysis and improving efficiency. However, reliably evaluating HOBRE outputs remains a fundamental challenge. On one hand, human evaluation is costly, time-consuming, as well as difficult to scale. On the other hand, existing automated metrics either rely on executable test cases and runtime environments, which are of- ten unavailable for real-world binaries, or depend on high-quality source code references that are typically inaccessible and fail to cap- ture semantically equivalent but lexically diverse outputs. Although the emerging LLM-as-a-Judge paradigm is naturally well-suited to HOBRE evaluation, its effectiveness has not yet been fully studied. This paper presents the first systematic investigation of the LLM- as-a-Judge paradigm for HOBRE, covering three representative tasks: function name recovery, binary code summarization, and decompilation optimization. We introduce BinJudgeBench, the first expert-annotated, reference-free evaluation benchmark based ∗ Xiuwei Shang is also affiliated with Singapore Management University, Singapore. † Jieke Shi and Shaoyin Cheng are the corresponding authors. ‡ Also affiliated with Anhui Province Key Laboratory of Digital Security, Hefei, China. This work is licensed under a Creative Commons Attribution 4.0 International License. ASE ’26, Munich, Germany © 2026 Copyright held by the owner/author(s). ACM ISBN 979-8-4007-2882-2/2026/10 https://doi.org/10.1145/3832783.3834339 on multi-dimensional human judgment. Our empirical study re- veals that LLM-as-a-Judge achieves an average correlation of 63.20% with human judgment, significantly outperforming traditional auto- mated metrics at 35.04%. By analyzing the impact of various judge configurations, including backbone LLMs, prompting strategies, and decoding temperatures, on both correlation and cost, we find that no “one-size-fits-all” configuration exists, as the optimal setup varies across tasks and individual samples. To address this, we pro- pose BinJudge, which employs a lightweight routing mechanism to adaptively select the optimal judge configuration for each spe- cific task and sample. BinJudge improves correlation with human experts by 4.5%-24.7% and reduces API cost to 0.06×-0.84×of that of static best configurations, providing a scalable, cost-effective, and high-fidelity automated evaluation scheme for HOBRE. CCS Concepts • General and reference→Metrics; Measurement;• Software and its engineering→Software reverse engineering;• Com- puting methodologies→ Artificial intelligence. Keywords Binary Reverse Engineering, Evaluation, LLM-as-a-Judge ACM Reference Format: Xiuwei Shang, Li Hu, Xiao Jiang, Jieke Shi, Junda He, Zhou Yang, Shaoyin Cheng, Guoqiang Chen, Weiming Zhang, and David Lo. 2026. Beyond Text Matching: Towards Reference-Free Evaluation for Human-Oriented Binary Reverse Engineering. In Proceedings of the 41st IEEE/ACM International Conference on Automated Software Engineering (ASE ’26), October 12–16, arXiv:2608.07038v1 [cs.SE] 7 Aug 2026 ASE ’26, October 12–16, 2026, Munich, GermanyXiuwei Shang et al. 2026, Munich, Germany. ACM, New York, NY, USA, 13 pages. https://doi. org/10.1145/3832783.3834339 1 Introduction Binary reverse engineering is the cornerstone of security-critical scenarios such as malware analysis [36], software vulnerability de- tection [31], and patch analysis [61], where reverse analysts often lack access to source code and must confront obscure machine code directly. Modern decompilers (e.g., IDA Pro’s Hex-Rays [20]) have become the industry standard, lifting low-level assembly into C- style pseudocode to lower the barrier to analysis. Nevertheless, due to aggressive compiler optimizations and symbol table stripping, the resulting pseudocode often lacks essential semantic information, such as meaningful function names and developer comments, and suffers from distorted code structures [28,72], making reverse engi- neering still a cognitively demanding and labor-intensive task [47]. Driven by the advancements in neural machine translation and large language models, Human-Oriented Binary Reverse Engi- neering (HOBRE) [9,51] has emerged. Researchers leverage deep learning models to post-process decompiled pseudocode, aiming to elevate it into human-readable content that resembles source code and bridge the semantic gap between binary and high-level source code. Such efforts include recovering meaningful function names [23], generating natural language summaries [60], and re- constructing pseudocode to improve readability [53,54], thereby significantly reducing the cognitive burden on analysts and em- powering downstream expert analysis tasks. Limitations of Existing Metrics. However, effective evaluation of these HOBRE tasks remains significant challenges. First, while human evaluation is regarded as the “gold standard” for verify- ing model performance, it has obvious limitations in large-scale assessment. The high barrier to entry in binary reverse engineering makes human auditing not only expensive and time-consuming but also prone to consistency issues, as analysts suffer from fa- tigue and diminished attention when processing massive datasets. Second, several decompilation optimization studies, exemplified by LLM4Decompile [54], employ test-based metrics such as re- executability rate [6,59] to verify functional correctness. These metrics, however, require comprehensive unit tests and execution environments, limiting their application to single-function or single- file “toy programs”. In practical reverse engineering scenarios, com- plex binary projects often lack corresponding unit test cases, and their environmental dependencies are extremely difficult to sim- ulate, making dynamic verification infeasible. Furthermore, such metrics are unable to evaluate non-functional aspects like human readability [58], nor can they be applied to tasks like function name recovery or binary code summarization. To achieve scalable evaluation for HOBRE tasks, several tradi- tional automated evaluation metrics have been considered. One type is match-based metrics such as BLEU [39] and CodeBLEU [43], which assess surface-level overlap between generated content and references (i.e., ground truth). Another type is embedding-based metrics, such as BERTScore [68] and CodeBERTScore [70], which first encode both generated and reference texts into embeddings and then measure their similarity. However, these metrics also possess inherent limitations. First, they rely on high-quality and accessible reference texts. In HOBRE tasks, references are typi- cally derived from explicit information in the source code (e.g., function names and developer comments). Yet, in the most press- ing real-world scenarios beyond experimental environments, such as malware analysis and vulnerability discovery in closed-source firmware, source code is often unavailable, rendering evaluation methods based on source-derived references ineffective. Even in open-source software, only a small fraction of functions contain usable developer comments, which vary significantly in quality and style, making them unreliable as stable evaluation ground truth. Most importantly, after compilation optimization and stripping, the source code exhibits a fundamental semantic mismatch with its corresponding binary code; thus, forcibly using source code content as ground truth for text matching lacks justification. Furthermore, these metrics focus more on lexical similarity than semantic simi- larity, whereas correct results often exist in various forms that are semantically equivalent but phrased differently. Potential of LLM-as-a-Judge in HOBRE Evaluation.Recent ad- vancements in LLMs have fostered the rise of the LLM-as-a-Judge [27] paradigm, making it a scalable and cost-effective alternative to human expert evaluation. By guiding LLMs to perform automated audits of generated content based on predefined criteria, this para- digm has demonstrated high human alignment in tasks like code generation [65] and program repair [71], showcasing its significant promise within the software engineering community [19]. Similarly, several key attributes make LLMs particularly suitable for evaluating the HOBRE tasks. First, existing research indicates that LLMs exhibit a certain level of binary program comprehen- sion [48,49] and human-like reasoning abilities [17]. Second, LLMs are typically trained through reinforcement learning from human feedback (RLHF) [24], aligning closely with human judgment pref- erences [58] and naturally fitting the objectives like "readability" and "usefulness" of HOBRE. Most importantly, unlike metrics like BLEU that mechanically compare text similarity, LLMs excel at han- dling "semantically equivalent but diversely expressed" scenarios, while leveraging internal reasoning to directly verify generated artifacts against the code’s inherent logic, obviating the depen- dency on explicit reference text, which is critical for "in-the-wild" binary reverse engineering where original source code is inacces- sible. Nevertheless, the evaluation reliability of LLM-as-a-Judge in the specialized domain of binary reverse engineering, and the degree of its consistency with human experts, has not yet been systematically investigated. Benchmark and Empirical Study.To this end, this paper pro- poses an evaluation perspective that goes beyond text matching, systematically investigating for the first time the applicability and capability boundaries of LLM-as-a-Judge in HOBRE tasks, covering three of the most representative tasks: function name recovery, binary code summarization, and decompilation optimization. To enable a reliable empirical study, we construct BinJudgeBench, which is carefully annotated by three human experts in reverse engi- neering. BinJudgeBench records human experts’ judgments across multiple quality dimensions, providing a curated benchmark for reference-free evaluation. Based on this, we systematically compare LLM-as-a-Judge with traditional automated metrics in terms of their correlation with human expert judgment, and further analyze how different judge configurations, including backbone LLMs, prompt- ing strategies, and decoding temperatures, affect both correlation Beyond Text Matching: Towards Reference-Free Evaluation for Human-Oriented Binary Reverse EngineeringASE ’26, October 12–16, 2026, Munich, Germany voidparse_long_options(intargc, /* omitted */, void(*usage_func) (int), ...) if(argc == 2&& (c = getopt_long (argc, argv, "+", long_options, NULL)) != -1) switch(c) case'h': (*usage_func) (EXIT_SUCCESS); break; case'v': va_list authors; va_start (authors, usage_func); version_etc_va (stdout, command_name, package, version, authors); exit(0); default: break; SoftwareForwardEngineering BinaryReverseEngineering Human-Oriented BinaryReverse Engineering voidsub_401B20(__int64 a1 , /* omitted */, void(__fastcall *a6)(_QWORD), ...) intv6; // ebx intv11; // eax gcc_va_list va; if( (_DWORD)a1 == 2) v11 = sub_404DF0(a1, a2, "+", &off_405A40, 0L); if( v11 != -1) if( v11 == 104) a6(0L); else if ( v11 == 118) va_start(va, a6); sub_403C70(stdout, a3, a4, a5, va); exit(0); Evaluation ABC D # Test-basedMetrics # Source Code# Decompiled Pseudocode # Optimized Pseudocode ##Parses a single long command-line option voidhandle_options(intargc, /* omitted */, void(*help_func)(void*)) va_list args; if(argc == 2) intoption = parse_options(argc, argv, "+", &option_config, 0); if(option != -1) if(option == 'h') help_func(NULL); else if (option == 'v') va_start(args, help_func); vfprintf(stdout, format, args); exit(0); ##This function parses command-line – h/-v options to print help or version information and exit. # Match-based/Embedding-basedMetrics # HumanEvaluation Binary Files Stripped Binary Task2: Binary Code Summarization Task1: FunctionNameRecovery Task3: Decompilation Optimization Compilation Strip Assembly Code PseudoCode Decompilation Disassembly Paradigm Limitations TestCases LLMs-powered DLModel-driven 1.High Expertise Barrier 2.Labor-intensive & Costly&Fatigue 3.Poor Scalability 1.StrictUnitTest& Environment Dependency 2.Limited to Isolated“Toy Programs” 3.Incapable of Non-functional Assessment 1.HeavyHigh-quality Reference Dependency 2.LimitedtoSurface-level Lexical Overlap 3.Source-Binary Semantic Mismatch Figure 1: Background of Human-Oriented Binary Reverse Engineering (HOBRE) and cost. Our empirical results reveal that LLM-as-a-Judge sub- stantially outperforms traditional metrics in approximating human judgment, achieving an average correlation of 63.20% compared to 35.04% for traditional metrics. Moreover, the optimal judge config- uration varies across tasks and individual samples, indicating that no “one-size-fits-all” configuration exists. Proposed Approach.Motivated by the empirical findings, we fur- ther propose BinJudge, a lightweight routing mechanism designed for task- and sample-aware adaptive configuration selection, which employs a distribution-aligned training strategy, explicitly optimiz- ing both evaluation correlation and cost. By dynamically selecting the optimal judge configuration for each sample, BinJudge im- proves the correlation with human experts by 4.5%-24.7% while re- ducing API cost to only 0.06×-0.84×of that of static configurations, effectively narrowing the gap between the static best configurations and oracle upper bound. This provides a scalable, cost-effective, and high-fidelity automated evaluation solution for the HOBRE tasks. Contributions. Our major contributions are as follows: •BinJudgeBench Benchmark: We release the first high-quality, expert-annotated dataset specifically designed for reference- free HOBRE evaluation. It contains 1,233 meticulously audited samples, providing a reliable ground-truth based on multi- dimensional human judgment. •Systematic Empirical Study: We conduct the first compre- hensive study to explore the effectiveness of LLM-as-a-judge for HOBRE evaluation, analyzing its correlation with human experts across three core tasks. •BinJudge Router: We propose a lightweight adaptive rout- ing mechanism that can dynamically select the optimal judge configuration for each individual sample, thereby improving correlation while reducing costs. 2 Background and Related Works 2.1 Human-Oriented Binary Reverse Engineering (HOBRE) As illustrated in Figure 1, traditional binary reverse engineering pri- marily focuses on “recovering program behavior from binaries,” and its outputs are often intermediate results oriented toward machine semantics, including disassembled instruction sequences, control- flow graphs, data-flow relations, and C-like pseudocode produced by decompilation [52]. Although modern decompilers have made substantial progress in recovering logical equivalence, these outputs remain at a low semantic level. While they can also support down- stream tasks such as vulnerability detection and malware analysis, a cognitive gap still exists for human analysts: on the one hand, compiler optimizations can significantly alter the forms of control and data structures [62,63]; on the other hand, symbol stripping removes critical semantic cues like function names, variable names, and type information, making the analysis process heavily reliant on expert experience and extensive manual reasoning. To bridge the aforementioned gap, HOBRE shifts the perspec- tive toward a goal that is closer to “human readability and us- ability” [9,51]: taking decompiled pseudocode as the core input, it performs semantic completion and expression optimization to produce a more human-oriented representation that is closer to source-level expression. Therefore, in contrast to classical “Software Forward Engineering” (source code→compilation→binary), HOBRE can be regarded as a post-processing paradigm that, building upon “Tra- ditional Binary Reverse Engineering” (binary→decompilation→ decompiled pseudocode), takes an additional step toward a more human-friendly high-level representation. 2.2 Target Tasks To comprehensively cover the research scope of HOBRE, this paper focuses on the following three core tasks, which are representa- tive and complementary across semantic dimensions, spanning identifier-level function name recovery, natural-language–level summarization, and program-level decompilation optimization. To- gether, they cover two major task paradigms, i.e., Code2NL and Code2Code, and encompass three typical output forms, i.e., short- form text, long-form text, and structured code. 2.2.1 Function Name Recovery (FNR). The goal of this task is to rename anonymous functions in stripped binaries with descriptive names (e.g.,sub_401B20→ handle_optionsin Figure 1), which provide direct clues to the program’s intent. Early research mainly relied on heuristic rules or similarity-based retrieval [18,33]. In re- cent years, deep learning approaches (e.g., NFRE [14], SymLM [23], ASE ’26, October 12–16, 2026, Munich, GermanyXiuwei Shang et al. and Llasm [45]) have become dominant, achieving notable progress by learning mappings between assembly or pseudocode features and source-level symbols. The evaluation challenge of this task lies in the inherently highly diverse and style dependence of function naming, as well as the prevalence of domain-specific abbreviations and acronyms, both of which make hard string-matching metrics inadequate for fair assessment. 2.2.2 Binary Code Summarization (BCS). This task aims to gen- erate concise natural-language descriptions that explain the func- tionality of a given function. With the advance of multi-modal pretraining, models such as BinT5 [4] and HexT5 [60] attempt to align binary features with natural-language descriptions within a unified representation space. Evaluation also remains challenging because summarization is a typical open-ended generation problem in which a single function can be described in many semantically equivalent ways, causing traditional n-gram–based matching met- rics to over-penalize reasonable rewriting. In addition, these metrics cannot properly distinguish the importance of informative key- words from non-informative tokens like stop words. Furthermore, in real-world reverse engineering scenarios, it is often difficult to obtain high-quality and reliable reference texts. 2.2.3 Decompilation Optimization (DO). Compared to the first two tasks, this task emphasizes the structural quality and readability of the code itself. It aims to post-edit decompiler (e.g., IDA Pro) outputs through transformations including redundancy elimina- tion, variable and type enhancement, and statement restructuring, making the result closer to source code conventions and style. Re- cent work, including LLM4Decompile [54], SK 2 Decompile [53], and ReCopilot [8], demonstrates the potential of LLMs for such Code2Code transformations, improving the readability of decom- piled artifacts. Although the Re-executability Rate is commonly used for evaluation, it only reflects functional correctness and can- not capture non-functional properties such as readability. Moreover, real-world binaries often lack unit tests and have hard-to-reproduce environments, which makes dynamic test-based metrics ineffective in in-the-wild settings, while match-based metrics still struggle with semantically equivalent but differently expressed outputs. 2.3 LLM-as-a-Judge in SE The widespread adoption of LLMs in software engineering has led to the generation of a large volume of automated software artifacts, and the difficulty of evaluating such massive outputs has in turn given rise to the LLM-as-a-Judge paradigm [19,71]. This paradigm extends the role of LLMs from mere artifact generation to artifact quality assessment, and it has been proven to achieve high agreement with human experts across a variety of scenarios, including requirements engineering [3,32], code generation [55, 58,65,75], and software maintenance [22,26,67], thus becoming a scalable and cost-effective alternative to manual review, and, by virtue of its ability to operate without reference texts and to perform deep semantic understanding, it offers inherent advantages over traditional automated metrics. As discussed earlier, this paradigm is naturally suited for HOBRE evaluation. However, there is still a lack of systematic empirical studies, benchmark datasets, and task-specific judging models for the HOBRE setting, and this critical gap motivates our research. 3 BinJudgeBench Benchmark To systematically evaluate the effectiveness of LLM-as-a-Judge on the HOBRE task and its alignment with human expert judgment, we construct BinJudgeBench, the first HOBRE-oriented evaluation benchmark, meticulously annotated by reverse engineering experts. 3.1 Datasets Collection and Preprocessing To ensure both realism and diversity, we first select 51 represen- tative projects from the GNU [15] repository, which are widely adopted in binary reverse engineering research [46,48,73]. We then compile these source projects with GCC 8.2.0 compiler while preserving DWARF debugging information, producing 24 distinct binary variants that span 6 target architectures (ARM_32,ARM_64, X86,X64,MIPS_32, andMIPS_64) and 4 optimization levels (-O0, -O1,-O2, and-O3). For each compiled binary, we retain both the original version with debugging information and its corresponding stripped version, in which symbols and debug sections are removed. Subsequently, we employ IDA Pro [20] to decompile both the original and stripped binaries, and extract function boundaries, names, and corresponding pseudocode at the function level. While the stripped binaries replace function names with placeholder ad- dresses, the function boundaries remain consistent, and we leverage this to align and merge content extracted from the two versions. To further obtain information from the source-code side, we use the srcML [10] tool to parse the source code into XML format and extract developer-written comments located above function definitions and declarations. Based on the established function boundaries and name mappings, we then align the source function body, source function name, and developer comments with the stripped pseudocode. Through this process, each function sample is organized into a unified structured representation, including the stripped pseudocode, original (symbol-preserving) pseudocode, source function body, source function name, and available developer comments, etc. In total, we obtain 346,596 function-level samples. 3.2 Response Generation To evaluate the discriminative capability of LLM-as-a-Judge across diverse HOBRE outputs with varying quality and styles, we select 8 representative models for each task to generate corresponding responses for every sample. These models fall into three categories: •DL-based Binary-specific Models: For each task, we select two models, including SymLM [23] and HexT5 [60] for function name recovery, BinT5 [4] and CP-BCS [66] for binary code sum- marization, and LLM4Decompile [54] and SK 2 Decompile [53] for decompilation optimization. These models are specifically designed or trained for their corresponding tasks. • General-Purpose LLMs: This category covers the typical closed- source model GPT-4o [2], the ultra-large-scale open-source model DeepSeek-V3.2 (685B) [30], as well as code-oriented lightweight models DeepSeek-Coder-V2-Lite-Instruct (16B) [74], Qwen3-Coder- 30B-A3B-Instruct [64], and CodeLlama-34B-Instruct [44]. •Binary-Domain LLMs: Specifically ReCopilot [8], which is built upon general-purpose LLMs and undergo continued pre-training and instruction-tuning on binary-specific corpora to better adapt to binary reverse engineering tasks. Beyond Text Matching: Towards Reference-Free Evaluation for Human-Oriented Binary Reverse EngineeringASE ’26, October 12–16, 2026, Munich, Germany Table 1: Human Annotation Guideline ([green] [yellow] [blue] represent task-specific information for Function Name Recovery, Binary Code Summarization, and Decompilation Optimization tasks, respectively.) For each instance, annotators are provided with the stripped decompiled pseudocode푃 푓 and the candidate output푂 푓 generated by the model for the [Function Name Recovery] [Binary Code Summarization] [Decompilation Optimization] task (i.e., [Function Name] [Summary] [Optimized Pseudocode]). The pseudocode’s project origin, corresponding source code 푆 푓 , and compilation settings are provided for reference, but the model’s identity remains anonymous. The annotator needs to assign a single overall score (1–5 Likert scale) for each instance, which jointly reflects the following three dimensions: Evaluation Dimensions: 1. Semantic Correctness: To what extent does the output 푂 푓 faithfully reflect the semantics and functional intent of the given pseudocode 푃 푓 ? 2. Reverse Analyst Utility: To what extent does the output푂 푓 reduce the cognitive load for the reverse engineer? Is it closer to a higher-level representation of the source code푆 푓 ? 3. [Naming Distinctiveness & Naturalness: Is the output푂 푓 specific and natural, effectively distinguishing between similar functions, and adhering to common engineering naming conventions?] [Information Coverage & Density: Does the output 푂 푓 cover the key operations while avoiding lengthy, line-by-line translation and providing high-level abstraction?] [Idiomization & Faithfulness: Does the style and structure of the output푂 푓 conform to human programming habits while avoiding the addition of overly speculative semantics?] Scoring: Rate outputs on a scale of 1 to 5: ·5 (Excellent): Highly accurate and faithful, significantly reducing the reverse analysts’ cognitive burden, [name is specific and natural, clearly distinguishing similar functions.] [summary covers key operations with high information density, avoiding verbose translation.] [optimized pseudocode is highly idiomatic and free of any groundless speculation.] ·4 (Good): Mostly correct and effectively reduces cognitive burden, with only minor, non-misleading deviations, [name is correct but slightly generic, following basic naming conventions.] [summary covers nearly all key operations but is slightly wordy.] [optimized pseudocode follows most habits with very few groundless guesses.] · 3 (Acceptable): Partially correct but of limited value; helpful to the analyst but contains some deviations that could be misleading, [name is basically correct but too broad, partially violating naming conventions.][summary covers basic functions but is quite verbose.][optimized pseudocode follows basic habits but contains some unsupported speculation.] · 2 (Poor): Contains obvious semantic errors and offers minimal help, with significant misleading information, [name is weakly related and violates naming conventions.][summary severely misses operations and contains meaningless descriptions.][optimized pseudocode is generally non-idiomatic and increases auditing burden due to over-speculation.] ·1 (Bad): Fundamentally incorrect, fails to support analysis, and severely misleads the analyst, [name is entirely unrelated or uses random/meaningless characters.][summary is irrelevant, consisting entirely of verbose and meaningless descriptions.][optimized pseudocode is completely non-idiomatic and consists entirely of model hallucinations.] For each function sample, we input the stripped pseudocode into the above models and generate task-specific outputs, including function names, natural language summaries, or optimized pseu- docode. Regarding prompt templates, models in categories (1) and (3) follow their default settings, while models in category (2) adopt the settings from [48]. It is worth emphasizing that this stage aims not to pursue optimal generation quality, but rather to cover diverse model sources, output qualities, and expressive styles. Ultimately, this process yields a total of 2,772,768 (346,596×8) candidate re- sponses for each task. 3.3 Manual Annotation 3.3.1 Sampling Strategy. Given the prohibitively high cost of manual expert annotation, we design a rigorous sampling strat- egy to ensure both statistical significance and diversity of Bin- JudgeBench. First, targeting a 95% confidence level with a 5% con- fidence interval, we randomly sample 385 samples from the pool of 2,772,768 candidate responses. We then introduce an additional coverage constraint requiring the samples to span diverse com- binations of "generative model—target architecture—optimization level" (totaling 8×6×4=192 triplets). Since the initial random samples already cover 166 such combinations, we perform targeted supplementary sampling for an additional 26 samples, ultimately forming a set of 411 samples per task for annotation. 3.3.2Multi-dimensional Evaluation Framework. We employ a five-point multi-dimensional Likert scale to assess the overall qual- ity of candidate outputs, incorporating a two-tier evaluation frame- work composed of shared and task-specific dimensions. Specifically, (i) shared dimensions are applicable to all three HOBRE tasks, aiming to ensure Semantic Faithfulness (i.e., correctly reflecting the functionality of the pseudocode) and Analyst-centered Utility (i.e., aligning with source-code-level representations to reduce cog- nitive burden); and (i) task-specific dimensions, i.e., Naming Distinctiveness & Naturalness for function names, Information Cov- erage & Density for summaries, and Idiomization & Faithfulness for optimized pseudocode, are designed to capture the unique qual- ity characteristics of different output forms. Detailed definitions of each dimension and the corresponding scoring criteria are provided in the human annotation guideline shown in Table 1. Overall, this framework comprehensively describes the quality implications of the HOBRE task. 3.3.3Human Annotation and Quality Assurance. The three authors of this paper, each with over three years of experience in binary reverse engineering, served as annotation experts. Prior to the formal annotation, the three experts jointly annotate 10 sam- ples for each task and discuss the scoring criteria based on the annotation guideline in Table 1 to calibrate their judgments and ensure a consistent understanding. Subsequently, the remaining samples are annotated independently, with the entire process span- ning approximately two weeks per expert. Upon completion, we measure the inter-annotator agreement among the three experts using ordinal Krippendorff’s Alpha. The coefficients for the three tasks reach 0.7996, 0.7077, and 0.6619, respectively, indicating a satisfactory level of agreement and confirming the reliability of the initial annotations. To further improve annotation quality and mitigate subjective bias, we convene review meetings for samples with substantial disagreements (i.e., where the range between the maximum and minimum scores among the three experts is≥2, e.g., 3, 4, 5) to re-evaluate and establish final scores. Across the three tasks, 73, 105, and 76 samples undergo this review process, respectively. For samples with minor disagreements (i.e., a score range < 2), we directly use the mode of the three scores as the final score (e.g., 4, 4, 5→4). The resulting expert scores are treated as the "gold standard" in BinJudgeBench and are used to measure the consistency of subsequent automated evaluation methods with human judgment. 3.4 Dataset Analysis Figure 2 illustrates the distribution of expert annotation scores in BinJudgeBench. The results indicate that BinJudgeBench covers the full score spectrum from low quality (score 1) to high quality (score 5), demonstrating a well-defined gradient and discriminabil- ity. Among the three tasks, function name recovery receives the low- est average score (2.42), with the largest proportion of low-scoring ASE ’26, October 12–16, 2026, Munich, GermanyXiuwei Shang et al. 050100150200250300350400 Number of Samples FNR BCS DO Task 151 94 41 86 73 161 69 112 122 57 80 70 48 52 17 Avg: 2.43 Avg: 2.81 Avg: 2.66 1 (Bad)2 (Poor)3 (Acceptable)4 (Good)5 (Excellent) Figure 2: Distribution of human expert-annotated scores across three HOBRE tasks in BinJudgeBench. samples; binary code summarization exhibits the most balanced distribution and the highest average score (2.81); while decompi- lation optimization obtains an average score of 2.66 and exhibits a clear scarcity of high-scoring samples. Overall, BinJudgeBench provides reasonable quality coverage and sample diversity, serving as a reliable benchmark for subsequent evaluation. 4 Empirical Study Our empirical study investigates the following research questions to explore the effectiveness of LLM-as-a-Judge on the HOBRE task: •RQ1: Can LLM-as-a-Judge better approximate human ex- pert judgment in HOBRE tasks? We compare the perfor- mance of LLM-as-a-Judge against 12 traditional match-based and embedding-based automated metrics, verifying its higher correlation with human expert judgment. •RQ2: How do different configurations of LLM-as-a-Judge affect evaluation correlation and cost? We analyze three primary factors: (1) diverse backbone LLMs; (2) prompting strate- gies, i.e., Zero-Shot Learning (ZSL), Few-Shot Learning (FSL), and Chain-of-Thought (CoT); and (3) decoding strategies, specifically varying temperature. We explore the correlation with human judgment and API cost overhead under each configuration. •RQ3: Does the optimal configuration of LLM-as-a-Judge vary across tasks or samples? We investigate whether a "one- size-fits-all" configuration exists, i.e., whether the optimal config- uration, which ensures high correlation while minimizing cost, varies across different tasks or sample characteristics. 4.1 Experimental Setup 4.1.1 Environments. Our experimental environment operates on an Ubuntu 22.04 server, equipped with two 28-core Intel Xeon 6330 CPUs, 512GB RAM, 64TB storage, and 8*NVIDIA RTX A6000 GPUs, each with 48GB of VRAM. These GPUs run NVIDIA driver version 525.116.03 along with CUDA version 12.0. All LLMs used in this study are accessed via API calls through the OpenRouter [37] platform, while all other experiments are deployed and trained locally on the above experimental environment. 4.1.2Baseline Metrics. We select the following 2 categories, totaling 11 traditional automated evaluation metrics, as baselines: Match-based Metrics. These metrics assess quality by measur- ing the surface-level overlap between generated text and references. BLEU [39] and Rouge-L [29] capture lexical similarity based on푛- gram overlaps and the longest common subsequence, respectively. Table 2: The Detailed Information of LLMs Employed in the LLM-as-a-Judge Evaluation. ModelSizeModel ID Input 1 Tokens Output 1 Tokens Publisher Usage 2 Proprietary LLMs GPT-4o [2]-GPT-4o-2024-11-20$2.50/M $10/MOpenAI • Claude-3.5 [5]-Claude-3.5-Sonnet$6/M$30/M Anthropic ◦ Gemini-2.5 [11]-Gemini-2.5-Flash$0.30/M $2.50/MGoogle ◦ Open-Source General LLMs DeepSeek-V3.2 [30] 685B DeepSeek-V3.2$0.26/M $0.38/M DeepSeek • Mistral-Large-3 [35] 675B Mistral-Large-3-675B-Instruct-2512 $0.50/M $1.50/M Mistral AI ◦ Llama-3.3 [16]70B Llama-3.3-70B-Instruct$0.10/M $0.32/M Meta AI ◦ Phi-4 [1]14B Phi-4$0.065/M $0.14/M Microsoft ◦ Open-Source Code LLMs Qwen3-Coder [64]30B Qwen3-Coder-30B-A3B-Instruct$0.07/M $0.27/MQwen • Codestral [34]-Codestral-2508$0.30/M $0.90/M Mistral AI ◦ 1 Token prices (per million tokens) are sourced from OpenRouter [37] as of March 22, 2026. 2 "•" indicates that the model was used in Response Generation (§3.2), "◦" indicates it was not. METEOR [7] improves upon basic푛-gram matching by consid- ering stemmed forms, synonym matches, and word order, while ChrF++ [41] further assesses similarity using character-level푛- gram precision and recall. CrystalBLEU [13] filters out the most frequently occurring푛-grams before computing BLEU to better capture substantive differences between generated content and reference. CodeBLEU [43] enhances traditional푛-gram matching by incorporating structural features like abstract syntax trees and data-flow dependencies. RUBY [56] evaluates similarity by jointly considering lexical, syntactic, and semantic representations of code. F1-score is also adopted in some function name recovery works [14], which compute token-level F1-score by ignoring non-alphabetical characters and being case-, order-, and duplication-insensitive at the token level, and we follow their practice. Embedding-based Metrics. These metrics map the generated text and the reference text into high-dimensional semantic spaces, measuring their semantic proximity via vector similarity. Sentence- BERT [42] employs the cosine similarity of sentence-level embed- dings to assess overall semantic similarity. MoverScore [69] repre- sents text as sequences of token embeddings and measures semantic divergence using Earth Mover’s Distance. BERTScore [68] computes token-level similarity using contextual representations from a pre- trained BERT model, while CodeBERTScore [70] adapts BERTScore specifically for code-related tasks to enhance its representation capabilities for programming languages. 4.1.3 Meta-Evaluation. To measure the consistency between various automated evaluation metrics (including both traditional baselines and the LLM-as-a-Judge paradigm) and human expert judgment, we employ three statistical correlation coefficients widely used in prior research [12,38,57,75], i.e., Kendall’s휏coefficient, Spearman’s푟 푠 , and Pearson’s푟 푝 , to quantify the association be- tween automated scores and the expert-annotated "gold standard" in BinJudgeBench. Specifically, Kendall’s휏coefficient [25] mea- sures ordinal associations between sample pairs, Spearman’s푟 푠 [50] assesses the monotonic correlation between their ranking orders, and Pearson’s푟 푝 [40] measures the linear correlation between them. Considering that expert ratings are ordinal discrete values (on a 1–5 scale), we prioritize Kendall’s휏and Spearman’s푟 푠 as the primary reported metrics, while treating Pearson’s푟 푝 as a supplementary reference. Taken together, these three coefficients provide a meta- evaluation of automated evaluation metrics. Beyond Text Matching: Towards Reference-Free Evaluation for Human-Oriented Binary Reverse EngineeringASE ’26, October 12–16, 2026, Munich, Germany Table 3: Correlation of traditional metrics and LLM-as-a-Judge with human expert judgment across HOBRE tasks in Bin- JudgeBench. The highest correlation is highlighted in bold and underlined, and the highest in traditional metrics is underlined. Metrics Function Name Recovery (FNR) Binary Code Summarization (BCS) Decompilation Optimization (DO) Avg. 휏 (%)푟 푠 (%)푟 푝 (%)휏 (%)푟 푠 (%)푟 푝 (%)휏 (%)푟 푠 (%)푟 푝 (%) Existing Match-based / Embedding-based Metrics F1-score48.5154.6560.15– BLEU31.5840.1850.6926.8935.4527.9125.8033.0133.5033.89 METEOR49.97 56.7957.3627.8936.4534.2524.6731.9532.2939.05 ROUGE-L48.2954.5159.5131.1240.0338.7328.6337.0539.4841.93 ChrF++47.5560.3765.2435.3246.8846.6429.2437.6340.6045.40 CrystalBLEU32.6042.1953.1219.8026.7725.6624.4931.3029.5533.06 CodeBLEU13.0814.498.6512.4916.2700.7416.6222.0223.2914.18 RUBY24.2826.8330.7427.8836.0231.7316.7821.7825.8726.88 SentenceBERT43.8657.0265.5441.6754.1756.6628.1036.5138.1946.86 MoverScore31.4340.9253.4928.4138.1438.0111.9215.9818.6830.78 BERTScore36.0347.1651.4528.6238.1836.9522.2129.0133.2135.87 CodeBERTScore38.2549.1551.3225.2633.6734.8228.1336.4040.1437.46 Average37.1245.3650.6127.7636.5533.8323.5530.5332.8235.04 LLM-as-a-Judge Paradigm Codestral-250855.2865.0260.8348.4458.4057.6340.3845.4548.2853.30 Phi-448.6456.5055.2356.8965.4065.7741.6847.2250.2054.17 Llama-3.3-70B60.9770.0464.4261.2768.8869.7848.1452.8252.4760.97 Qwen3-Coder-30B63.6472.8570.0165.6875.0974.1342.8247.8348.7862.32 Mistral-Large-359.2868.7166.9563.9672.7173.0253.9559.3259.8964.20 GPT-4o64.4473.8669.5765.9475.2675.4351.1757.5557.3865.62 Claude-3.5-Sonnet67.0475.6673.3367.4676.4476.9854.7061.1061.9868.30 Gemini-2.5-Flash69.0278.9275.9368.0977.4577.9052.7660.2860.9469.03 DeepSeek-V3.266.1775.0374.2867.3676.5876.6862.1169.7569.8870.87 Average61.6170.7367.8462.7971.8071.9249.7455.7056.6563.20 4.2 RQ1: Meta-Evaluation of LLM-as-a-Judge Evaluation Setup. To comprehensively evaluate the effectiveness of the LLM-as-a-Judge paradigm, we select 9 representative back- bone LLMs, with detailed information provided in Table 2. These models span three categories—Proprietary, Open-Source General, and Open-Source Code—and encompass a range of parameter scales. Notably, within each category, we included one model that is previ- ously used for response generation in §3.2 (i.e., GPT-4o, DeepSeek- V3.2, and Qwen3-Coder), aiming to investigate whether LLMs ex- hibit self-evaluation bias when judging HOBRE tasks. Subsequently, we adopt the human expert annotations in Bin- JudgeBench as the "gold standard" and compute Kendall’s휏, Spear- man’s푟 푠 , and Pearson’s푟 푝 correlation coefficients for both 11 tra- ditional automated metrics and the 9 backbone LLMs serving as judges. Due to space constraints, Table 3 reports only the config- uration (i.e., prompting strategy and decoding temperature) that achieves the highest average correlation for each LLM across the three tasks. The impact of different parameter configurations on evaluation consistency will be discussed in detail in RQ2 (§4.3). Superiority over Traditional Metrics. As illustrated in Table 3, the LLM-as-a-Judge paradigm achieves an average correlation of 63.20% across all three HOBRE tasks, substantially outperforming the average of traditional metrics (35.04%). Taking the FNR task as an example, the best-performing LLM, Gemini-2.5-Flash, reaches a Kendall’s휏of 69.02%, nearly 1.4 times that of the best traditional metric, METEOR (49.97%). This indicates that LLMs can transcend surface-level lexical matching to deeply comprehend functional semantics and expressive intent, yielding judgments that closely align with human experts. While certain match-based metrics show marginal utility in specific scenarios (e.g., ChrF++ achieves 60.37% 휏in FNR, surpassing the lower-tier LLM Phi-4 at 56.50%), they consistently fail to compete with high-parameter LLM evaluators. Among embedding-based metrics, SentenceBERT slightly outper- forms a few lightweight LLMs on FNR in terms of Pearson’s푟 푝 , yet remains below the median level of LLMs and exhibits even weaker performance on the BCS and DO tasks. Task-Specific Performance Variation. Considering task types, LLMs perform best on the BCS task (average Kendall’s휏of 62.68%), where the performance gap over traditional metrics is most pro- nounced (a 35.03% difference in휏). This can be attributed to the nature of code summarization as a typical open-ended generation task, which allows for numerous semantically equivalent expres- sions. In such cases, surface-level lexical overlap often decouples from actual semantic similarity, rendering푛-gram matching or static embedding-based metrics ineffective. In contrast, the perfor- mance gap narrows slightly in the DO task (average휏of 49.74% for LLMs, and 23.55% for traditional metrics). This suggests that the structured nature of Code2Code transformations allows traditional metrics to capture basic syntactic correctness. However, the critical advantage of LLMs on DO task lies in their ability to detect "super- ficially plausible but logically flawed" refactorings, which involves a deep semantic equivalence judgment that remains beyond the capability boundary of traditional metrics. ASE ’26, October 12–16, 2026, Munich, GermanyXiuwei Shang et al. Table 4: Correlation of various LLM-as-a-Judge configurations (including different LLMs, prompting strategies, and decoding temperatures) with human expert judgment in BinJudgeBench. Each cell reports the "Kendall’s 휏 coefficient (%) / API cost ($)". For each LLMs, the highest correlation within its 3× 3 sub-block is highlighted in bold and underlined. ModelsPrompt Function Name Recovery (FNR)Binary Code Summarization (BCS)Decompilation Optimization (DO) Average T=0.1T=0.5T=1.0T=0.1T=0.5T=1.0T=0.1T=0.5T=1.0 Codestral-2508 ZSL45.44 / 0.10046.61 / 0.10044.71 / 0.10053.15/ 0.10651.67 / 0.10647.56 / 0.10639.81 / 0.15241.63/ 0.15229.29 / 0.15244.43 / 0.119 FSL31.70 / 0.32230.28 / 0.32229.22 / 0.32249.35 / 0.23844.87 / 0.23843.95 / 0.23835.08 / 0.49432.32 / 0.49434.97 / 0.49436.86 / 0.351 CoT55.28 / 0.23551.12 / 0.23756.98/ 0.23748.44 / 0.21848.33 / 0.22046.04 / 0.22040.38 / 0.29438.11 / 0.29236.29 / 0.29346.77/ 0.250 Phi-4 ZSL47.29 / 0.02042.31 / 0.02037.68 / 0.02047.57 / 0.02248.14 / 0.02248.22 / 0.02230.47 / 0.03142.84/ 0.03025.13 / 0.03141.07 / 0.024 FSL36.69 / 0.06633.25 / 0.06648.64 / 0.06545.02 / 0.04842.85 / 0.04856.89/ 0.04836.67 / 0.10128.54 / 0.10141.68 / 0.10041.14 / 0.071 CoT42.80 / 0.04440.38 / 0.04549.66/ 0.04851.61 / 0.04250.33 / 0.04354.74 / 0.04535.56 / 0.05635.72 / 0.05732.18 / 0.05943.66/ 0.049 Llama-3.3-70B ZSL60.97 / 0.03261.20 / 0.03261.82/ 0.03261.27 / 0.03460.30 / 0.03462.36 / 0.03448.14/ 0.04845.96 / 0.04843.54 / 0.04756.17/ 0.038 FSL59.53 / 0.10158.33 / 0.10159.61 / 0.10262.79/ 0.07461.50 / 0.07461.53 / 0.07547.80 / 0.15546.17 / 0.15541.63 / 0.15555.43 / 0.110 CoT53.50 / 0.09449.74 / 0.09352.73 / 0.09458.08 / 0.08959.79 / 0.09057.01 / 0.09046.05 / 0.10945.89 / 0.11146.45 / 0.10852.14 / 0.097 Qwen3-Coder-30B ZSL64.00 / 0.02363.20 / 0.02362.40 / 0.02363.16 / 0.02563.07 / 0.02563.47 / 0.02538.06 / 0.03538.67 / 0.03539.35 / 0.03555.04 / 0.028 FSL63.64 / 0.07564.28/ 0.07562.73 / 0.07565.68 / 0.05664.33 / 0.05666.96/ 0.05642.82 / 0.11541.84 / 0.11540.28 / 0.11556.95/ 0.082 CoT56.25 / 0.07953.37 / 0.07849.71 / 0.08355.43 / 0.07554.00 / 0.07655.29 / 0.08143.68/ 0.10041.38 / 0.10138.83 / 0.10849.77 / 0.087 Mistral-Large-3 ZSL63.40 / 0.16764.09/ 0.16763.11 / 0.16758.25 / 0.17760.31 / 0.17758.12 / 0.17750.08 / 0.25451.95 / 0.25451.23 / 0.25457.84 / 0.199 FSL59.98 / 0.53759.28 / 0.53759.72 / 0.53764.23/ 0.39663.96 / 0.39663.98 / 0.39652.84 / 0.82453.95/ 0.82452.80 / 0.82458.97/ 0.586 CoT53.98 / 0.57053.77 / 0.57054.93 / 0.57061.02 / 0.48858.00 / 0.49459.40 / 0.49045.88 / 0.69649.53 / 0.69545.92 / 0.68353.60 / 0.584 GPT-4o ZSL65.53 / 0.79665.78 / 0.79665.08 / 0.79660.48 / 0.84560.70 / 0.84559.63 / 0.84547.51 / 1.18146.32 / 1.18146.04 / 1.18157.45 / 0.941 FSL64.44 / 2.53364.29 / 2.53363.33 / 2.53365.94/ 1.85664.82 / 1.85663.48 / 1.85651.17/ 3.87550.19 / 3.87547.96 / 3.87559.51/ 2.754 CoT65.95/ 3.07564.64 / 3.11261.76 / 3.26758.11 / 2.90961.90 / 3.01157.14 / 3.25650.88 / 4.28851.58 / 4.14850.64 / 4.28158.07 / 3.483 Claude-3.5-Sonnet ZSL70.31/ 2.33369.11 / 2.33366.93 / 2.33260.23 / 2.46559.44 / 2.46560.84 / 2.46457.83/ 3.52656.59 / 3.52553.55 / 3.52561.65/ 2.774 FSL67.04 / 7.60657.39 / 7.61655.21 / 7.63667.46/ 5.51664.41 / 5.54663.37 / 5.59654.70 / 11.6952.13 / 11.7048.74 / 11.7158.94 / 8.290 CoT61.17 / 6.09556.77 / 6.09155.73 / 6.07156.08 / 5.73057.08 / 5.72256.34 / 5.75950.75 / 7.30245.95 / 7.22042.07 / 7.05753.55 / 6.339 Gemini-2.5-Flash ZSL66.56 / 0.11066.61 / 0.11068.31 / 0.11066.36 / 0.11666.83 / 0.11666.40 / 0.11649.11 / 0.16750.15 / 0.16751.94 / 0.16761.36 / 0.131 FSL69.02/ 0.35568.20 / 0.35565.81 / 0.35568.09 / 0.26168.13/ 0.26164.90 / 0.26152.76/ 0.55051.88 / 0.55048.50 / 0.55061.92/ 0.389 CoT57.23 / 1.18156.12 / 1.25256.98 / 1.31360.77 / 1.43661.19 / 1.50061.60 / 1.54239.96 / 2.31438.76 / 2.36933.77 / 2.66351.82 / 1.730 DeepSeek-V3.2 ZSL63.96 / 0.08363.72 / 0.08361.61 / 0.08363.46 / 0.08863.52 / 0.08861.38 / 0.08853.51 / 0.12553.89 / 0.12551.72 / 0.12459.64 / 0.099 FSL66.17/ 0.26762.76 / 0.26762.82 / 0.26767.36 / 0.19768.18/ 0.19761.00 / 0.19762.11/ 0.41157.25 / 0.41157.17 / 0.41162.76/ 0.291 CoT57.90 / 0.16958.81 / 0.16455.76 / 0.16859.82 / 0.16463.38 / 0.16562.39 / 0.17053.49 / 0.21951.14 / 0.22150.97 / 0.22857.07 / 0.185 Average58.14/ 1.00356.50 / 1.00756.78 / 1.01559.23/ 0.87758.93 / 0.88458.67 / 0.89846.56/ 1.44945.94 / 1.44343.80 / 1.45353.84 / 1.114 Investigation of Self-Evaluation Bias. We further analyze whether GPT-4o, DeepSeek-V3.2, and Qwen3-Coder, which serve as gen- erators in our response generation stage, exhibit any bias when acting as evaluators. To investigate this, we partition the samples into푆 self (outputs generated by the LLM itself ) and푆 others (outputs generated by other LLMs) subsets, and calculate the correlation of the LLM’s judgment with human experts for each subset. The results show that휏 self ≈ 휏 others , with the difference not reaching statistical significance, indicating that in the HOBRE evaluation context, LLMs maintain an objective evaluation scale and do not exhibit any preference toward their own generation style. Investigation of Hard Samples and the "Fluency Trap". We de- fine a "hard sample" as one with substantial disagreement between human experts and LLMs, i.e.,|HumanScore− avg. LLMScore| ≥2. Across BinJudgeBench, 149 out of 1,233 samples (12.08%) fall into this category (70/18/61 in FNR/BCS/DO). Task-stratified manual analysis on 100 hard samples reveals three primary difficulty pat- terns: (1) Lack of semantic anchors (34%): pseudocode lacking resid- ual semantic clues (e.g., hard-coded strings or external API calls) forces LLM judges into speculative scoring. (2) Lack of contextual information (14%): isolated single-function snippets miss caller/- callee relations, key global variables, or struct fields required for reliable judgment. (3) Rewarding fluent-but-unsupported outputs (12%): LLMs tend to over-score syntactically fluent or conventional outputs even if semantically unsupported. To further quantify this "Fluency Trap", we treat samples withHumanScore ≤2 as "se- mantically unsupported," and from this set manually identify 100 seemingly fluent samples as a sub-pool. In this fluent sub-pool, the proportion of LLM judge scores >3 rises to 59.0% (compared to 46.7% in the overall unsupported pool), which mainly comes from the FNR task, which tends to give medium scores (3–4) to plausible-looking names; for BCS and DO, the bias is not obvious. Overall, while LLM judges exhibit a slight bias toward fluency, its impact remains within a controllable and acceptable margin. Answering RQ1: LLM-as-a-Judge achieves an average corre- lation of 63.20% with human judgement, substantially supe- rior traditional metrics at 35.04%, accurately capturing deep semantics while maintaining an objective scale, providing a high-fidelity, scalable automated evaluation solution. 4.3 RQ2: Impact of Judge Configuration Evaluation Setup. To investigate the impact of different configu- rations (including backbone LLMs, prompting strategies, and de- coding temperatures) on the correlation and overhead of LLM-as-a- Judge, we employ the 9 backbone LLMs listed in Table 2. Regarding prompting strategies, Zero-Shot Learning (ZSL) provides the LLMs with task descriptions, evaluation dimensions, scoring criteria, as well as the stripped decompiled pseudocode and candidate outputs, while Few-Shot Learning (FSL) builds upon this by providing one example for each score level (1–5). Chain-of-Thought (CoT) requires the LLMs to provide a step-by-step reasoning process before out- putting the final score. Decoding temperatures are set to 0.1, 0.5, and 1.0 following prior work [38]. Due to space constraints, we report only Kendall’s휏and API costs in Table 4, as Spearman’s푟 푠 and Pearson’s 푟 푝 exhibit trends consistent with 휏 . Beyond Text Matching: Towards Reference-Free Evaluation for Human-Oriented Binary Reverse EngineeringASE ’26, October 12–16, 2026, Munich, Germany Table 5: Comparison of Correlation and Cost Across Different Judge Configuration Selection Strategies (↓and×represent the correlation gap and cost ratio relative to the Oracle Configuration, respectively) (Due to space constraints, we ignore 푟 푝 ). ConfigurationFunction Name Recovery (FNR) Binary Code Summarization (BCS) Decompilation Optimization (DO) ModelsPrompt Temp. 휏 (%)푟 푠 (%)Cost ($) 휏 (%)푟 푠 (%)Cost ($)휏 (%)푟 푠 (%)Cost ($) Random Configuration54.08 (↓41.8%) 63.56 (↓33.3%) 1.080 (11.24×) 55.02 (↓44.1%) 64.76 (↓34.6%) 1.035 (15.30×) 39.67 (↓56.1%) 45.82 (↓50.4%) 1.389 (6.90×) Static Configuration Claude-3.5-SonnetZSL0.1 70.31 (↓24.4%) 79.26 (↓16.9%) 2.333 (24.27×) 60.23 (↓38.8%) 69.47 (↓29.9%) 2.465 (36.46×) 57.83 (↓36.0%) 64.88 (↓29.8%) 3.526 (17.51×) Gemini-2.5-FlashFSL0.5 68.20 (↓26.6%) 78.12 (↓18.1%) 0.355 (3.69×) 68.13 (↓30.7%) 77.66 (↓21.6%) 0.261 (3.86×) 51.88 (↓42.6%) 59.41 (↓35.7%) 0.550 (2.73×) DeepSeek-V3.2FSL0.166.17 (↓28.8%) 75.03 (↓21.3%) 0.267 (2.78×) 67.36 (↓31.5%) 76.58 (↓22.7%) 0.197 (2.91×) 62.11 (↓31.3%) 69.75 (↓24.5%) 0.411 (2.04×) Oracle Configuration92.9595.330.09698.3699.020.06890.4192.380.201 Impact of Prompting Strategy. The impact of the prompt strat- egy on correlation varies significantly across models of differ- ent scales. For ultra-large-scale LLMs (e.g., GPT-4o, Gemini-2.5- Flash, DeepSeek-V3.2, and Mistral-Large-3), FSL generally achieves the highest correlation. For instance, Gemini-2.5-Flash reaches a Kendall’s휏of 69.02% in the FNR task at T=0.1 under FSL, outper- forming ZSL (66.56%) and CoT (57.23%). This superiority stems from the ability of large LLMs to rapidly extract evaluation patterns from few-shot examples. However, the increased input tokens brought by FSL make its cost the highest, even exceeding that of CoT. Con- versely, for small- to medium-scale LLMs (e.g., Codestral-2508 and Phi-4), FSL leads to a notable decline in correlation. This may be attributed to capacity constraints, where LLMs become "anchored" to specific patterns within the provided examples, thereby hinder- ing their generalization capabilities. In contrast, CoT provides a substantial boost for these smaller LLMs, achieving peak average휏 values for Codestral-2508 (46.77%) and Phi-4 (43.66%). This suggests that explicit reasoning helps compensate for the weaker native rea- soning capabilities of smaller LLMs. Interestingly, CoT often leads to a reduction in correlation for ultra-large LLMs in most cases, potentially because excessive reasoning induces hallucinations or over-interpretation, thereby interfering with judgment. Impact of Decoding Temperature. Lower temperatures (0.1 or 0.5) generally yield higher correlation, particularly for ultra-large LLMs, as they help maintain a consistent evaluation scale in com- plex semantic judgments. For smaller LLMs, however, higher tem- peratures sometimes achieve optimal correlation under certain configurations. For example, Codestral-2508 and Phi-4 attain their highest correlation on the FNR task using CoT at T=1.0. This sug- gests that moderate randomness may help weaker LLMs escape local scoring traps and activate more diverse reasoning paths. Fur- thermore, increased temperature only results in slight cost growth under CoT due to higher variability in output length. Pareto Differences between LLMs. Overall, correlation is pos- itively associated with the LLM scale, accompanied by a marked increase in cost. However, within the ultra-large LLMs, substan- tial Pareto differences exist regarding evaluation efficiency and economy. For example, on the FNR task using ZSL and T=0.1 con- figurations, Gemini-2.5-Flash and DeepSeek-V3.2 demonstrate ex- tremely high cost-effectiveness. Gemini-2.5-Flash achieves a 61.36% average correlation at a cost of only $0.131, whereas Claude-3.5- Sonnet requires $2.774 (approximately 21 times higher) to reach a similar correlation of 61.65%. This finding indicates that cost and performance are not strictly linearly coupled in HOBRE evaluation. Table 6: Cross-Task Kendall’s휏Correlation Analysis of Cor- relation and Cost Rankings for Judge Configurations. Task-Task Meta-Metrics 휏 (%)푟 푠 (%) 푟 푝 (%)Cost ($) FNR - BCS61.3961.4960.5994.83 FNR - DO55.3856.2861.1895.98 BCS - DO52.5655.9559.2093.74 Answering RQ2: The configuration of backbone LLMs, prompting strategy, and decoding temperature exhibit sig- nificant and non-linear impacts on correlation and cost. The optimal prompting strategy (especially FSL and CoT) shows a performance inversion across LLMs’scale, while lower temper- atures generally benefit the correlation of large-scale LLMs. 4.4 RQ3: Optimal Configuration Variability Following the analysis of judge configurations in RQ2, this section further investigates whether a "one-size-fits-all" configuration ex- ists that can consistently maintain high correlation and low cost across different task types and input samples. Task-level Configuration Variability. As previously discussed, for each task, we construct 81 judge configurations by combining 9 backbone LLMsM, 3 prompting strategiesP, and 3 decoding temperaturesT . A single configuration for task 푖 is defined as: 퐶 푖 (푚,푝,푡) → (휏,푟 푠 ,푟 푝 , Cost) where푚 ∈M,푝 ∈ P,푡 ∈ T. We calculate the Kendall’s휏correla- tion of all configurations across the three tasks in terms of the four metrics, and the results are shown in Table 6. The rank correlations for the three performance metrics (휏,푟 푠 ,푟 푝 ) across tasks are only at a moderate level. For example, the Kendall’s 휏between BCS and DO is only 52.56%. This "rank flipping" phenom- enon proves that a configuration performing excellently in one task may fail to maintain its superiority in another, reflecting inherent disparities in the capability requirements of different HOBRE tasks. In contrast, the rank correlation for Cost across tasks is extremely high, indicating that costs are minimally affected by the task type. Therefore, when selecting configurations for different tasks, one can prioritize task characteristics to predict evaluation effectiveness, while treating cost as a relatively independent dimension. Sample-level Configuration Variability. To further verify whether the optimal configuration varies across samples, we compare the performance of three configuration selection strategies: ASE ’26, October 12–16, 2026, Munich, GermanyXiuwei Shang et al. Table 7: Correlation and Cost of BinJudge routing configuration compared to other configuration strategies and fine-tuned judge model (↑represents the correlation improvement and×represents the cost ratio of BinJudge relative to each baseline) (All results are averaged over 5-fold cross-validation in BinJudgeBench). ConfigurationFunction Name Recovery (FNR) Binary Code Summarization (BCS) Decompilation Optimization (DO) ModelsPrompt Temp. 휏 (%)푟 푠 (%)Cost ($)휏 (%)푟 푠 (%)Cost ($)휏 (%)푟 푠 (%)Cost ($) Random Configuration54.40 (↑35.1%) 63.55 (↑30.2%) 0.216 (0.14×) 55.44 (↑30.7%) 65.09 (↑25.6%) 0.207 (0.15×) 38.20 (↑65.5%) 43.76 (↑62.2%) 0.278 (0.25×) Static Configuration Claude-3.5-SonnetZSL0.1 70.33 (↑4.5%) 78.99 (↑4.8%) 0.467 (0.07×) 60.17 (↑20.4%) 69.16 (↑18.2%) 0.493 (0.06×) 57.42 (↑10.1%) 64.16 (↑10.6%) 0.705 (0.10×) Gemini-2.5-FlashFSL0.5 68.06 (↑8.0%) 77.79 (↑6.4%) 0.071 (0.44×) 68.10 (↑6.4%) 77.61 (↑5.3%) 0.052 (0.61×) 50.67 (↑24.7%) 57.88 (↑22.6%) 0.110 (0.63×) DeepSeek-V3.2FSL0.165.64 (↑12.0%) 74.41 (↑11.2%) 0.053 (0.58×) 66.62 (↑8.8%) 75.66 (↑8.1%) 0.039 (0.81×) 59.75 (↑5.8%) 67.39 (↑5.3%) 0.082 (0.84×) Fine-tuned Judge Model UniXcoder53.89 (↑36.4%) 68.05 (↑21.6%) -45.22 (↑60.2%) 58.51 (↑39.7%) -38.70 (↑63.3%) 49.76 (↑42.6%) - Qwen2.5-Coder-7B69.27 (↑6.1%) 77.75 (↑6.4%) -64.11 (↑13.0%) 73.47 (↑11.3%) -51.36 (↑23.1%) 56.56 (↑25.5%) - BinJudge Routing Configuration73.5182.740.03172.4581.750.03263.2170.970.069 Oracle Configuration92.7295.080.01998.4099.040.01490.2992.180.040 •Random Configuration: A configuration is randomly selected from the 81 groups for each sample. • Static Configuration: A single fixed configuration is used for all samples. We select the one with the highest overall correlation for each task as a representative. •Oracle Configuration: For each sample, we select the configu- ration that minimizes the squared error between its score and the human expert’s rating (if multiple configurations achieve the same error, the one with the lowest cost is chosen). This repre- sents the upper bound of performance under an “ideal scenario”. The results in Table 5 show that Random Configuration is signif- icantly worse than Static Configuration, indicating that improper selection severely degrades evaluation quality. Moreover, Static Configuration is far below Oracle Configuration, with a significant "Oracle Gap" existing across all three tasks. This quantitatively proves that using a single static configuration for all samples can- not reach the theoretical upper bound of evaluation performance, as any single configuration faces a clear "capability ceiling" when dealing with heterogeneous samples. Answering RQ3: There is no "one-size-fits-all" judge configu- ration. Different tasks have distinct configuration preferences, and even within the same task, the performance of a fixed con- figuration (max 70.31%휏) is far below the 92.95% theoretical upper bound of sample-level adaptive configurations. 5 BinJudge Router 5.1 Adaptive Configuration Routing Empirical results indicate that the optimal judge configuration dynamically varies with task types and sample characteristics, re- vealing a substantial Oracle Gap between the "single static best" configuration and the theoretical upper bound. This underscores the high heterogeneity of HOBRE evaluation tasks, where a one-size- fits-all configuration fails to accommodate all tasks and samples. Motivated by this, we introduce BinJudge, a lightweight routing mechanism designed for task- and sample-aware adaptive configu- ration selection. BinJudge takes the stripped pseudocode and the task type as input, and dynamically routes the optimal judge configuration for each sample. Specifically, it employs a pretrained UniXcoder-base encoder to encode the input pseudocode, while freezing all param- eters during training so that it serves as a fixed feature extractor. Building upon this, a learnable task embedding layer maps different task types into vector representations, which are then concatenated with the code embeddings to equip the model with cross-task aware- ness. After feature fusion, a stacked multi-layer perceptron is used to predict the preference distribution over all judge configurations. During training, the optimization objective of BinJudge is to maximize the sample-level utility function푈, aiming to prioritize high correlation while reducing cost. The utility function of config- uration 푐 on sample 푖 is defined as: 푈 푖,푐 =−(MSE 푖,푐 + 휆· Cost 푖,푐 ) =− (푆 푖,푐 −ℎ 푖 ) 2 + 휆· ln(1+ 10 3 · Cost 푖,푐 ) ln(1+ 10 3 · max 푘∈C 푣푎푙푖푑 Cost 푖,푘 +휖) where푆 푖,푐 denotes the score assigned by configuration푐to sample 푖, ℎ 푖 denotes the corresponding human expert score, and Cost 푖,푐 is the actual API expenditure. For the cost term, we apply logarithmic scaling for normalization, and set휆=0.1 to control the trade-off between correlation and cost. We adopt distribution alignment for training, where the utility푈 is first transformed into a target probability distribution푃 푡푎푟푔푒푡 us- ing a Softmax function with a temperature푇=0.5. The model’s out- put layer then yields a predicted distribution푄 휃 via a Log-Softmax operation. The training loss is defined as the KL divergence between the target and the predicted distribution: L= KL(푃 푡푎푟푔푒푡 ∥ 푄 휃 )= 1 푁 푁 ∑︁ 푖=1 ∑︁ 푐∈C 푃 푡푎푟푔푒푡 (푐|푖) log 푃 푡푎푟푔푒푡 (푐|푖) 푄 휃 (푐|푖) This training strategy enables BinJudge not only to learn the top- ranked configuration, but also to capture the relative utility ordering of all 81 configurations for a given sample, thereby substantially improving robustness under limited training data. During inference, for a new sample, BinJudge requires only a single forward pass to obtain the predicted probability distribution over all configurations, and selects the configuration with the high- est probability as the final judge configuration. Subsequently, we conduct experiments to answer the following research question: •RQ4: Can BinJudge routing effectively improve correlation and maintain low cost? We explore the performance of Bin- Judge in terms of correlation and cost compared to static optimal Beyond Text Matching: Towards Reference-Free Evaluation for Human-Oriented Binary Reverse EngineeringASE ’26, October 12–16, 2026, Munich, Germany configuration, random configuration, and oracle upper bound, as well as two supervised fine-tuning judge models. 5.2 RQ4: Effectiveness of Configuration Routing Considering that human expert annotations are limited and that obtaining additional annotations is costly, we evaluate the effec- tiveness of BinJudge using five-fold cross-validation on the Bin- JudgeBench dataset. Experimental results, as detailed in Table 7, demonstrate that BinJudge consistently outperforms all Static Con- figurations across the three HOBRE tasks. Specifically, our routing mechanism improves Kendall’s휏by ranges of 4.5%–12.0%, 6.4%– 20.4%, and 5.8%–24.7%, respectively, effectively narrowing the oracle gap. Beyond superior correlation with human experts, BinJudge exhibits remarkable economic efficiency. The average API expendi- ture across all tasks is significantly lower than that of the Random Configuration, representing only 0.14×, 0.15×, and 0.25×of its cost, respectively. When compared to the Static Configuration, Bin- Judge maintains a cost ratio of only 0.07–0.58×, 0.06–0.81×, and 0.10–0.84×. Even relative to the Oracle Configuration, its cost is only about 2–3 times higher. These results validate the feasibility and practicality of learning-based configuration routing for scalable and accurate HOBRE evaluation. Meanwhile, we fine-tuned UniXcoder and a lightweight LLM, Qwen2.5-Coder-7B [21], to directly predict scores using the same five-fold cross-validation. Both significantly underperformed Bin- Judge. This indicates that, under the same annotation budget, a fine-tuned judge generalizes poorly, whereas BinJudge leverages complementary prior knowledge from different LLMs and adapts to sample variations via lightweight routing. Answering RQ4: BinJudge effectively bridges the gap be- tween static and oracle configurations by adaptively select- ing judge configurations for individual samples, improv- ing Kendall’s휏by 4.5%–24.7% and reducing API costs to 0.06×–0.84× of static best configurations. 6 Discussion 6.1 Analysis of Evaluator Disagreements We analyze the causes of disagreements across three dimensions: LLM vs. Traditional Metrics. Discrepancies stem from two pri- mary causes: (1) low-quality or incomplete references reduce the reliability of reference-based metrics; (2) traditional metrics un- derestimate outputs that are semantically equivalent but lexically different from the reference. LLM vs. Human. As detailed in §4.2, our manual inspection reveals three main patterns leading to LLM-Human disagreements (i.e., |HumanScore− avg. LLMScore| ≥2): (1) lack of semantic anchors, (2) lack of contextual information, and (3) rewarding fluent-but- unsupported outputs. Human vs. Human. We define this disagreement as samples with a score range≥2 among the three annotators (253 out of 1233 samples; 73/105/75 for FNR/BCS/DO). Manual inspection shows this disagreement mainly occurs on medium-quality outputs due to intrinsic subjectivity, while annotators are generally consistent on clearly good or clearly poor outputs, with no clear pattern tied to specific input features. Compared with LLM-Human disagreement (149 out of 1233 samples; 70/18/61 for FNR/BCS/DO), the overlap is only 23 samples (12/6/5 for FNR/BCS/DO). This low overlap indicates that the two are largely distinct phenomena. We therefore do not claim LLM judges can fully replace humans, but that they are a superior option to traditional metrics. 6.2 Threats to Validity Internal Validity. The primary threat to internal validity relates to the inherent subjectivity of human evaluation used to construct the BinJudgeBench “gold standard”. Human assessments of “read- ability” and “usefulness” can vary significantly among individuals. To mitigate this threat, three experts from our author team (each with over three years of binary reverse engineering experience) annotated the dataset using a rigorous three-stage protocol: ini- tial calibration, independent scoring, and a consensus meeting for high-variance samples. To prevent potential author bias, we further invited an independent industrial expert to annotate a subset (50 samples per task); merging these independent annotations with our original three-expert scores yields evaluation trends that remain highly consistent with original overall trends. External Validity. Threats to external validity primarily concern the generalizability of our findings. To ensure a diverse and repre- sentative dataset, BinJudgeBench encompasses 51 real-world GNU projects compiled across 6 target architectures and 4 optimization levels. We also evaluated responses generated by 8 distinct models, ranging from binary-specific deep learning models to ultra-large- scale general-purpose LLMs. This benchmark collectively spans multiple major HOBRE tasks: function name recovery, binary code summarization, and decompilation optimization. Construct Validity. Threats to construct validity concern whether our experimental design and evaluation metrics accurately capture the phenomena under investigation. To measure the alignment be- tween automated approaches and human judgments, we adopted Kendall’s휏rank correlation coefficient, Spearman’s푟 푠 rank correla- tion coefficient, and Pearson’s푟 푝 correlation coefficient. Following established methodologies in recent software engineering litera- ture [67,71], the rank-based metrics (휏and푟 푠 ) are particularly well-suited for evaluating our ordinal Likert-scale data, while푟 푝 serves as a supplementary measure of linear correlation. 7 Conclusion and Future Work This paper presents the first systematic investigation into the LLM- as-a-Judge paradigm for Human-Oriented Binary Reverse Engi- neering (HOBRE) tasks. To overcome the inherent limitations of traditional match-based and test-based metrics, we introduced Bin- JudgeBench, a comprehensive, expert-annotated, reference-free evaluation benchmark. Our extensive empirical study revealed that LLM evaluators significantly outperform traditional metrics in ap- proximating human judgment, effectively capturing deep semantic equivalence rather than relying on superficial lexical overlaps. Fur- thermore, recognizing that no single judge configuration is optimal across all scenarios, we proposed BinJudge, a lightweight, adaptive routing mechanism. By dynamically selecting the most suitable LLM configuration for each individual sample, BinJudge substan- tially improves evaluation correlation while drastically reducing API costs. Overall, this work provides the binary reverse engineer- ing community with a scalable, high-fidelity, and cost-effective framework for automated artifact evaluation. ASE ’26, October 12–16, 2026, Munich, GermanyXiuwei Shang et al. Data Availability Statement To support the reproducibility and replicability of the research, we release BinJudgeBench and BinJudge at https://doi.org/10.5281/ zenodo.19247812. The released resources comprise all source code, datasets, and a subset of experimental results. Acknowledgments This research / project is supported by the National Key Research and Development Program of China under Grant 2025QY2840, and the Natural Science Foundation of China under Grant U20B2047, 62072421, 62002334, 62102386, and 62121002. This research / project is also supported by the National Research Foundation, under its In- vestigatorship Grant (NRF-NRFI08-2022-0002). Any opinions, find- ings and conclusions or recommendations expressed in this material are those of the author(s) and do not reflect the views of National Research Foundation, Singapore. References [1]Marah Abdin, Jyoti Aneja, Harkirat Behl, Sébastien Bubeck, Ronen Eldan, Suriya Gunasekar, Michael Harrison, Russell J Hewett, Mojan Javaheripi, Piero Kauff- mann, et al.2024. Phi-4 technical report. arXiv preprint arXiv:2412.08905 (2024). [2]Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Floren- cia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al.2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 (2023). [3] Toufique Ahmed, Premkumar Devanbu, Christoph Treude, and Michael Pradel. 2025. Can llms replace manual annotation of software engineering artifacts?. In 2025 IEEE/ACM 22nd International Conference on Mining Software Repositories (MSR). IEEE, 526–538. doi:10.1109/msr66628.2025.00086 [4]Ali Al-Kaswan, Toufique Ahmed, Maliheh Izadi, Anand Ashok Sawant, Premku- mar Devanbu, and Arie van Deursen. 2023. Extending source code pre-trained language models to summarise decompiled binaries. In 2023 IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER). IEEE, 260–271. doi:10.1109/saner56733.2023.00033 [5] Anthropic. 2024. Claude 3.5 Sonnet. https://w.anthropic.com/news/claude- 3-5-sonnet Accessed: 2026-03-22. [6]Jordi Armengol-Estapé, Jackson Woodruff, Alexander Brauckmann, José Wesley de Souza Magalhaes, and Michael FP O’Boyle. 2022. ExeBench: an ML-scale dataset of executable C functions. In Proceedings of the 6th ACM SIGPLAN Interna- tional Symposium on Machine Programming. 50–59. doi:10.1145/3520312.3534867 [7]Satanjeev Banerjee and Alon Lavie. 2005. METEOR: An automatic metric for MT evaluation with improved correlation with human judgments. In Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization. 65–72. https://aclanthology.org/W05-0909/ [8]Guoqiang Chen, Huiqi Sun, Daguang Liu, Zhiqi Wang, Qiang Wang, Bin Yin, Lu Liu, and Lingyun Ying. 2025. ReCopilot: Reverse Engineering Copilot in Binary Analysis. arXiv preprint arXiv:2505.16366 (2025). https://arxiv.org/html/2505. 16366v1 [9]Qibin Chen, Jeremy Lacomis, Edward J Schwartz, Claire Le Goues, Graham Neubig, and Bogdan Vasilescu. 2022. Augmenting decompiler output with learned variable names and types. In 31st USENIX Security Symposium (USENIX Security 22). 4327–4343. https://w.usenix.org/conference/usenixsecurity22/ presentation/chen-qibin [10]Michael L Collard, Michael John Decker, and Jonathan I Maletic. 2013. srcml: An infrastructure for the exploration, analysis, and manipulation of source code: A tool demonstration. In 2013 IEEE International conference on software maintenance. IEEE, 516–519. doi:10.1109/icsm.2013.85 [11]Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, et al.2025. Gemini 2.5: Pushing the frontier with advanced reasoning, multi- modality, long context, and next generation agentic capabilities. arXiv preprint arXiv:2507.06261 (2025). https://arxiv.org/abs/2507.06261 [12] Yihong Dong, Jiazheng Ding, Xue Jiang, Ge Li, Zhuo Li, and Zhi Jin. 2025. Code- score: Evaluating code generation by learning code execution. ACM Transactions on Software Engineering and Methodology 34, 3 (2025), 1–22. doi:10.1145/3695991 [13]Aryaz Eghbali and Michael Pradel. 2022. CrystalBLEU: precisely and efficiently measuring the similarity of code. In Proceedings of the 37th IEEE/ACM Interna- tional Conference on Automated Software Engineering. 1–12. doi:10.1145/3551349. 3556903 [14]Han Gao, Shaoyin Cheng, Yinxing Xue, and Weiming Zhang. 2021. A lightweight framework for function name reassignment based on large-scale stripped binaries. In Proceedings of the 30th ACM SIGSOFT International Symposium on Software Testing and Analysis. 607–619. doi:10.1145/3460319.3464804 [15]GNU Project. 2026. GNU FTP Site. http://ftp.gnu.org/gnu. Accessed: 2026-03-22. [16]Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al.2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783 (2024). https://arxiv.org/pdf/2407.21783v3 [17]Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al.2025. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948 (2025). https://arxiv.org/abs/2501.12948 [18]Jingxuan He, Pesho Ivanov, Petar Tsankov, Veselin Raychev, and Martin Vechev. 2018. Debin: Predicting debug information in stripped binaries. In Proceedings of the 2018 ACM SIGSAC Conference on Computer and Communications Security. 1667–1680. doi:10.1145/3243734.3243866 [19]Junda He, Jieke Shi, Terry Yue Zhuo, Christoph Treude, Jiamou Sun, Zhenchang Xing, Xiaoning Du, and David Lo. 2026. LLM-as-a-Judge for Software Engineering: Literature Review, Vision, and the Road Ahead. ACM Transactions on Software Engineering and Methodology (2026). doi:10.1145/3797276 [20] Hex-RaysSA. 2026. "IDA Pro". https://w.hex-rays.com/products/ida. [21]Binyuan Hui, Jian Yang, Zeyu Cui, Jiaxi Yang, Dayiheng Liu, Lei Zhang, Tianyu Liu, Jiajun Zhang, Bowen Yu, Keming Lu, et al.2024. Qwen2.5-coder technical report. arXiv preprint arXiv:2409.12186 (2024). [22]Hongchao Jiang, Yiming Chen, Yushi Cao, Hung-yi Lee, and Robby T Tan. 2026. Codejudgebench: Benchmarking llm-as-a-judge for coding tasks. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 19416–19448. doi:10.18653/v1/2026.acl-long.888 [23] Xin Jin, Kexin Pei, Jun Yeon Won, and Zhiqiang Lin. 2022. Symlm: Predicting function names in stripped binaries via context-sensitive execution-aware code embeddings. In Proceedings of the 2022 ACM SIGSAC Conference on Computer and Communications Security. 1631–1645. doi:10.1145/3548606.3560612 [24] Timo Kaufmann, Paul Weng, Viktor Bengs, and Eyke Hüllermeier. 2024. A survey of reinforcement learning from human feedback. Transactions on Machine Learning Research (2024). https://openreview.net/pdf ?id=f7OkIurx4b [25] Maurice G Kendall. 1938. A new measure of rank correlation. Biometrika 30, 1-2 (1938), 81–93. doi:10.2307/2332226 [26] Abhishek Kumar, Sonia Haiduc, Partha Pratim Das, and Partha Pratim Chakrabarti. 2024. LLMs as Evaluators: A Novel Approach to Evaluate Bug Report Summarization. arXiv preprint arXiv:2409.00630 (2024). https://arxiv.org/ abs/2409.00630 [27] Dawei Li, Bohan Jiang, Liangjie Huang, Alimohammad Beigi, Chengshuai Zhao, Zhen Tan, Amrita Bhattacharjee, Yuxuan Jiang, Canyu Chen, Tianhao Wu, et al. 2025. From generation to judgment: Opportunities and challenges of llm-as- a-judge. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing. 2757–2791. doi:10.18653/v1/2025.emnlp-main.138 [28] Gangyang Li, Xiuwei Shang, Shaoyin Cheng, Junqi Zhang, Li Hu, Xu Zhu, Weim- ing Zhang, and Nenghai Yu. 2025. PseudoFix: Refactoring Distorted Structures in Decompiled C Pseudocode. In 2025 40th IEEE/ACM International Conference on Automated Software Engineering (ASE). IEEE, 841–853. doi:10.1109/ase63991. 2025.00075 [29] Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out. 74–81. https://aclanthology.org/W04-1013/ [30]Aixin Liu, Aoxue Mei, Bangcai Lin, Bing Xue, Bingxuan Wang, Bingzheng Xu, Bochao Wu, Bowei Zhang, Chaofan Lin, Chen Dong, et al.2025. Deepseek- v3.2: Pushing the frontier of open large language models. arXiv preprint arXiv:2512.02556 (2025). https://arxiv.org/abs/2512.02556 [31]Haolang Lu, Hongrui Peng, Guoshun Nan, Jiaoyang Cui, Cheng Wang, Weifei Jin, Songtao Wang, Shengli Pan, and Xiaofeng Tao. 2025. Malsight: Exploring malicious source code and benign pseudocode for iterative binary malware summarization. IEEE Transactions on Information Forensics and Security (2025). doi:10.1109/tifs.2025.3583552 [32]Sebastian Lubos, Alexander Felfernig, Thi Ngoc Trang Tran, Damian Garber, Merfat El Mansi, Seda Polat Erdeniz, and Viet-Man Le. 2024. Leveraging llms for the quality assurance of software requirements. In 2024 IEEE 32nd International Requirements Engineering Conference (RE). IEEE, 389–397. doi:10.1109/re59067. 2024.00046 [33]Jiang Ming, Meng Pan, and Debin Gao. 2012. iBinHunt: Binary hunting with inter-procedural control flow. In International Conference on Information Security and Cryptology. Springer, 92–109. doi:10.1007/978-3-642-37682-5_8 [34] Mistral AI Team. 2025. Announcing Codestral 25.08 and the Complete Mistral Coding Stack for Enterprise. https://mistral.ai/news/codestral-25-08 Accessed: 2026-03-22. [35] Mistral AI Team. 2025. Introducing Mistral 3. https://mistral.ai/news/mistral-3 Accessed: 2026-03-22. [36]Thinh Nguyen Hung, Hai Nguyen Phuc, Khoa Tran Dinh, Nhan Le Tran Thanh, Nghia To Trong, Khoa Ngo Khanh, Duy Phan The, and Hau Pham Van. 2023. Binary Representation Embedding and Deep Learning For Binary Code Similarity Detection in Software Security Domain. In Proceedings of the 12th International Beyond Text Matching: Towards Reference-Free Evaluation for Human-Oriented Binary Reverse EngineeringASE ’26, October 12–16, 2026, Munich, Germany Symposium on Information and Communication Technology. 785–792. doi:10.1145/ 3628797.3628996 [37]OpenRouter. 2026. The Unified Interface For LLMs.https://openrouter.ai/ Accessed: 2026-03-22. [38]Wendkûuni C Ouédraogo, Yinghua Li, Xueqi Dang, Pawel Borsukiewicz, Xin Zhou, Anil Koyuncu, Jacques Klein, David Lo, and Tegawendé F Bissyandé. 2025. Human-Aligned Code Readability Assessment with Large Language Models. arXiv preprint arXiv:2510.16579 (2025). https://arxiv.org/abs/2510.16579 [39] Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics. 311–318. doi:10.3115/1073083.1073135 [40] Karl Pearson. 1896. VII. Mathematical contributions to the theory of evolu- tion.—I. Regression, heredity, and panmixia. Philosophical Transactions of the Royal Society of London. Series A, containing papers of a mathematical or physical character 187 (1896), 253–318. doi:10.1098/rsta.1896.0007 [41]Maja Popović. 2017. chrF++: words helping character n-grams. In Proceedings of the second conference on machine translation. 612–618. doi:10.18653/v1/w17-4770 [42]Nils Reimers and Iryna Gurevych. 2019. Sentence-bert: Sentence embeddings using siamese bert-networks. In Proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (EMNLP-IJCNLP). 3982–3992. doi:10.18653/v1/d19- 1410 [43]Shuo Ren, Daya Guo, Shuai Lu, Long Zhou, Shujie Liu, Duyu Tang, Neel Sundare- san, Ming Zhou, Ambrosio Blanco, and Shuai Ma. 2020. Codebleu: a method for automatic evaluation of code synthesis. arXiv preprint arXiv:2009.10297 (2020). https://arxiv.org/abs/2009.10297 [44] Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiao- qing Ellen Tan, Yossi Adi, Jingyu Liu, Romain Sauvestre, Tal Remez, et al.2023. Code llama: Open foundation models for code. arXiv preprint arXiv:2308.12950 (2023). https://arxiv.org/abs/2308.12950 [45] Zihan Sha, Hao Wang, Zeyu Gao, Hui Shu, Bolun Zhang, Ziqing Wang, and Chao Zhang. 2025. llasm: Naming functions in binaries by fusing encoder-only and decoder-only llms. ACM Transactions on Software Engineering and Methodology 34, 4 (2025), 1–22. doi:10.1145/3702988 [46]Xiuwei Shang, Guoqiang Chen, Shaoyin Cheng, Shikai Guo, Yanming Zhang, Weiming Zhang, and Nenghai Yu. 2025. Foc: Figure out the cryptographic func- tions in stripped binaries with llms. ACM Transactions on Software Engineering and Methodology 35, 1 (2025), 1–38. doi:10.1145/3731449 [47]Xiuwei Shang, Guoqiang Chen, Shaoyin Cheng, Benlong Wu, Li Hu, Gangyang Li, Weiming Zhang, and Nenghai Yu. 2025. BinMetric: A Comprehensive Binary Code Analysis Benchmark for Large Language Models. In Proceedings of the Thirty-Fourth International Joint Conference on Artificial Intelligence, IJCAI-25. 7715–7723. doi:10.24963/ijcai.2025/858 [48]Xiuwei Shang, Shaoyin Cheng, Guoqiang Chen, Yanming Zhang, Li Hu, Xiao Yu, Gangyang Li, Weiming Zhang, and Nenghai Yu. 2024. How Far Have We Gone in Binary Code Understanding Using Large Language Models. In 2024 IEEE International Conference on Software Maintenance and Evolution (ICSME). IEEE, 1–12. doi:10.1109/icsme58944.2024.00012 [49]Xiuwei Shang, Zhenkan Fu, Shaoyin Cheng, Guoqiang Chen, Gangyang Li, Li Hu, Weiming Zhang, and Nenghai Yu. 2026. An empirical study on the effectiveness of large language models for binary code understanding. Empirical Software Engineering 31, 1 (2026), 1–38. doi:10.1007/s10664-025-10748-5 [50] Charles Spearman. 1961. The proof and measurement of association between two things. (1961). doi:10.1037/11491-005 [51] Zian Su, Xiangzhe Xu, Ziyang Huang, Kaiyuan Zhang, and Xiangyu Zhang. 2024. Source code foundation models are transferable binary analysis knowledge bases. Advances in Neural Information Processing Systems 37 (2024), 112624–112655. doi:10.52202/079017-3578 [52]Iain Sutherland, George E Kalb, Andrew Blyth, and Gaius Mulley. 2006. An em- pirical examination of the reverse engineering process for binary files. Computers & Security 25, 3 (2006), 221–228. doi:10.1016/j.cose.2005.11.002 [53]Hanzhuo Tan, Weihao Li, Xiaolong Tian, Siyi Wang, Jiaming Liu, Jing Li, and Yuqun Zhang. 2025. SK2Decompile: LLM-based Two-Phase Binary Decompilation from Skeleton to Skin. arXiv preprint arXiv:2509.22114 (2025). https://arxiv.org/ abs/2509.22114 [54]Hanzhuo Tan, Qi Luo, Jing Li, and Yuqun Zhang. 2024. LLM4Decompile: De- compiling Binary Code with Large Language Models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing. 3473–3487. doi:10.18653/v1/2024.emnlp-main.203 [55]Weixi Tong and Tianyi Zhang. 2024. CodeJudge: Evaluating Code Generation with Large Language Models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing. 20032–20051. doi:10.18653/v1/2024. emnlp-main.1118 [56]Ngoc Tran, Hieu Tran, Son Nguyen, Hoan Nguyen, and Tien Nguyen. 2019. Does BLEU score work for code migration?. In 2019 IEEE/ACM 27th International Conference on Program Comprehension (ICPC). IEEE, 165–176. doi:10.1109/icpc. 2019.00034 [57]Ruiqi Wang, Jiyu Guo, Cuiyun Gao, Guodong Fan, Chun Yong Chong, and Xin Xia. 2025. Can llms replace human evaluators? an empirical study of llm-as-a- judge in software engineering. Proceedings of the ACM on Software Engineering 2, ISSTA (2025), 1955–1977. doi:10.1145/3728963 [58]Martin Weyssow, Aton Kamanda, Xin Zhou, and Houari Sahraoui. [n. d.]. CodeUl- traFeedback: An LLM-as-a-Judge Dataset for Aligning Large Language Models to Coding Preferences. ACM Transactions on Software Engineering and Methodology ([n. d.]). doi:10.1145/3736407 [59] Wai Kin Wong, Huaijin Wang, Zongjie Li, Zhibo Liu, Shuai Wang, Qiyi Tang, Sen Nie, and Shi Wu. 2023. Refining decompiled c code with large language models. arXiv preprint arXiv:2310.06530 (2023). https://arxiv.org/abs/2310.06530 [60]Jiaqi Xiong, Guoqiang Chen, Kejiang Chen, Han Gao, Shaoyin Cheng, and Weim- ing Zhang. 2023. Hext5: Unified pre-training for stripped binary code information inference. In 2023 38th IEEE/ACM International Conference on Automated Software Engineering (ASE). IEEE, 774–786. doi:10.1109/ase56229.2023.00099 [61]Zhengzi Xu, Bihuan Chen, Mahinthan Chandramohan, Yang Liu, and Fu Song. 2017. Spain: security patch analysis for binaries towards understanding the pain and pills. In 2017 IEEE/ACM 39th International Conference on Software Engineering (ICSE). IEEE, 462–472. doi:10.1109/icse.2017.49 [62]Zhihao Xu, Shikai Guo, Xiaochen Li, Zun Wang, and He Jiang. 2025. Simtam: Generation diversity test programs for fpga simulation tools testing via timing area mutation. ACM Transactions on Design Automation of Electronic Systems 30, 2 (2025), 1–25. doi:10.1145/3705730 [63]Zhihao Xu, Shikai Guo, Guilin Zhao, Peiyu Zou, Xiaochen Li, and He Jiang. 2025. A novel HDL code generator for effectively testing FPGA logic synthesis compilers. IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems (2025). doi:10.1109/tcad.2025.3565488 [64] An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al.2025. Qwen3 technical report. arXiv preprint arXiv:2505.09388 (2025). https://arxiv.org/abs/2505.09388 [65]Guang Yang, Yu Zhou, Xiang Chen, Wei Zheng, Xing Hu, Xin Zhou, David Lo, and Taolue Chen. 2025. Code-DiTing: Automatic Evaluation of Code Generation without References or Test Cases. In 2025 40th IEEE/ACM International Conference on Automated Software Engineering (ASE). IEEE, 154–165. doi:10.1109/ase63991. 2025.00021 [66]Tong Ye, Lingfei Wu, Tengfei Ma, Xuhong Zhang, Yangkai Du, Peiyu Liu, Shouling Ji, and Wenhai Wang. 2023. CP-BCS: Binary Code Summarization Guided by Control Flow Graph and Pseudo Code. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. 14740–14752. doi:10.18653/ v1/2023.emnlp-main.911 [67] Qunhong Zeng, Yuxia Zhang, Zexiong Ma, Bo Jiang, Ningyuan Sun, Klaas-Jan Stol, Xingyu Mou, and Hui Liu. 2025. Evaluating Generated Commit Messages with Large Language Models. arXiv preprint arXiv:2507.10906 (2025). https: //arxiv.org/abs/2507.10906 [68]Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. 2019. Bertscore: Evaluating text generation with bert. arXiv preprint arXiv:1904.09675 (2019). https://arxiv.org/abs/1904.09675 [69]Wei Zhao, Maxime Peyrard, Fei Liu, Yang Gao, Christian M Meyer, and Steffen Eger. 2019. MoverScore: Text generation evaluating with contextualized embed- dings and earth mover distance. In Proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (EMNLP-IJCNLP). 563–578. doi:10.18653/v1/d19-1053 [70]Shuyan Zhou, Uri Alon, Sumit Agarwal, and Graham Neubig. 2023. Code- BERTScore: Evaluating Code Generation with Pretrained Models of Code. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. 13921–13937. doi:10.18653/v1/2023.emnlp-main.859 [71]Xin Zhou, Kisub Kim, Ting Zhang, Martin Weyssow, Luís F Gomes, Guang Yang, Kui Liu, Xin Xia, and David Lo. 2025. SE-Jury: An LLM-as-Ensemble-Judge Metric for Narrowing the Gap with Human Evaluation in SE. In 2025 40th IEEE/ACM International Conference on Automated Software Engineering (ASE). IEEE, 2606– 2618. doi:10.1109/ase63991.2025.00214 [72]Zhiping Zhou, Xiaohong Li, Ruitao Feng, Yao Zhang, Yuekang Li, Wenbu Feng, Yunqian Wang, and Yuqing Li. 2025. FidelityGPT: Correcting Decompilation Dis- tortions with Retrieval Augmented Generation. arXiv preprint arXiv:2510.19615 (2025). https://arxiv.org/abs/2510.19615 [73]Kangchen Zhu, Zhiliang Tian, Shangwen Wang, Weiguo Chen, Zixuan Dong, Mingyue Leng, and Xiaoguang Mao. 2025. Misum: multi-modality heterogeneous code graph learning for multi-intent binary code summarization. Proceedings of the ACM on Software Engineering 2, FSE (2025), 1339–1362. doi:10.1145/3715780 [74]Qihao Zhu, Daya Guo, Zhihong Shao, Dejian Yang, Peiyi Wang, Runxin Xu, Y Wu, Yukun Li, Huazuo Gao, Shirong Ma, et al.2024. Deepseek-coder-v2: Breaking the barrier of closed-source models in code intelligence. arXiv preprint arXiv:2406.11931 (2024). https://arxiv.org/abs/2406.11931 [75]Terry Yue Zhuo. 2024. Ice-score: Instructing large language models to evaluate code. In Findings of the Association for Computational Linguistics: EACL 2024. 2232–2242. doi:10.18653/v1/2024.findings-eacl.148 Received 2026-03-26; accepted 2026-06-18

Instant research discovery