Paper deep dive
Style Wins, Substance Loses: A Diagnosis of LLM-as-Judge in Idea Generation
Fengxian Ji, Yuke Li, Jingpu Yang, Juanfan Wu, Fan Zhang, Zhexuan Cui, Yu Xie, Min Peng, Qianqian Xie, Xiuying Chen, Zhuohan Xie
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 92%
Last extracted: 8/5/2026, 4:16:15 AM
Summary
The paper introduces SciStyleBench, a framework to diagnose and mitigate stylistic bias in LLM-as-Judge systems for scientific idea generation. It proposes SciStyleStage for controlled style perturbation, SciStyleMetrics (SBI, SRR, AWR) for quantifying bias, and SciStyleExtractor to separate style from substance. Experiments show that while direct LLM judges are sensitive to style, SciStyleExtractor significantly improves robustness and substance recognition.
Entities (10)
Relation Signals (10)
SciStyleBench → containscomponent → SciStyleMetrics
confidence 95% · (ii) Second, SciStyleMetrics, a set of quantitative measures...
SciStyleBench → containscomponent → SciStyleStage
confidence 95% · SciStyleBench, a unified three-component benchmark... (i) First, SciStyleStage...
SciStyleBench → containscomponent → SciStyleExtractor
confidence 95% · (iii) Third, SciStyleExtractor, a plug-and-play evaluation module...
LLM-as-Judge → suffersfrombias → Stylistic Bias
confidence 92% · direct LLM judges remain sensitive to writing style and struggle to distinguish scientific substance
SciStyleExtractor → increasesmetric → Adversarial Win Rate
confidence 90% · increasing... AWR from 0.554 to 0.899
SciStyleExtractor → increasesmetric → Substance Recognition Rate
confidence 90% · increasing SRR... to 0.759
SciStyleExtractor → reducesmetric → Style Bias Index
confidence 90% · SciStyleExtractor reduces SBI from 0.566 to 0.501
SciStyleStage → usessetting → No Context
confidence 88% · three settings no context, fixed-domain context, and open-domain retrieval context
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:However, whether these judges truly evaluate the scientific substance of ideas or are influenced by superficial stylistic presentation remains an open question. To address this question, we propose SciStyleBench, a unified three-component benchmark for diagnosing and mitigating stylistic bias in LLM-based idea evaluation: (i) First, SciStyleStage, a three-stage evaluation environment that applies controlled stylistic perturbations to fixed scientific content across three settings no context, fixed-domain context, and open-domain retrieval context, covering 600 scientific ideas and 15 style variants, with 9,000 evaluation instances per setting; (ii) Second, SciStyleMetrics, a set of quantitative measures, including Style Bias Index (SBI), Substance Recognition Rate (SRR), and Adversarial Win Rate (AWR), to characterize how stylistic variation affects scoring stability, substance discrimination, and ranking robustness; (iii) Third, SciStyleExtractor, a plug-and-play evaluation module that separates presentation style from scientific content by predicting style type and deviation before style-conditioned evaluation, enabling us to assess whether style awareness reduces stylistic bias. Experiments on SciStyleBench show that direct LLM judges remain sensitive to writing style and struggle to distinguish scientific substance. In contrast, SciStyleExtractor reduces SBI from 0.566 to 0.501 while increasing SRR and AWR from 0.504 and 0.554 to 0.759 and 0.899, respectively. These results suggest that robust idea evaluation requires invariance to stylistic variation without sacrificing sensitivity to scientific substance. Overall, SciStyleBench provides a systematic framework for identifying, quantifying, and mitigating stylistic bias in scientific idea evaluation.
Tags
Links
- Source: https://arxiv.org/abs/2608.01666v2
- Canonical: https://arxiv.org/abs/2608.01666v2
Trouble viewing inline? Open PDF directly →
Full Text
56,373 characters extracted from source content.
Expand or collapse full text
Style Wins, Substance Loses: A Diagnosis of LLM-as-Judge in Idea Generation Fengxian Ji1,2,4, Yuke Li1,311footnotemark: 1, Jingpu Yang411footnotemark: 1, Juanfan Wu1, Fan Zhang2, Zhexuan Cui6 Yu Xie5, Min Peng1, Qianqian Xie1, Xiuying Chen2, Zhuohan Xie222footnotemark: 2 Equal contribution.Corresponding author. Abstract With the rapid growth of LLM-based scientific agents, scientific idea generation has become a key component of AI-driven research, highlighting the need for reliable LLM-as-Judge systems. However, whether these judges truly evaluate the scientific substance of ideas or are influenced by superficial stylistic presentation remains an open question. To address this question, we propose SciStyleBench, a unified three component Benchmark for diagnosing and mitigating stylistic bias in LLM-based idea evaluation: (i) First, SciStyleStage, a three-stage evaluation environment that applies controlled stylistic perturbations to fixed scientific content across three settings no context, fixed-domain context, and open-domain retrieval context covering 600 scientific ideas and 15 style variants, with 9,000 evaluation instances per setting; (i) Second, SciStyleMetrics, a set of quantitative measures including Style Bias Index (SBI), Substance Recognition Rate (SRR), and Adversarial Win Rate (AWR) to characterize how stylistic variation affects scoring stability, substance discrimination, and ranking robustness; (i) Third, SciStyleExtractor, a plug-and-play evaluation module that separates presentation style from scientific content by predicting style type and deviation before style-conditioned evaluation, enabling us to assess whether style awareness reduces stylistic bias. Experiments on SciStyleBench reveal that direct LLM judges are sensitive to writing style and weak at substance discrimination, while SciStyleExtractor improves robustness by reducing SBI from 0.5660.566 to 0.5010.501 and increasing SRR/AWR from 0.504/0.5540.504/0.554 to 0.759/0.8990.759/0.899. These results suggest that robust idea evaluation requires invariance to stylistic variation without sacrificing sensitivity to scientific substance. Overall, SciStyleBench provides a systematic framework for identifying, quantifying, and mitigating stylistic bias in scientific idea evaluation. Introduction With the rapid development of AI Scientist systems and automated research agents, scientific idea generation is gradually shifting from a one-off writing assistance task to a key generative component in automated research workflows (Lu et al. 2024; Gottweis et al. 2026; Luo et al. 2025). Consequently, how to reliably evaluate and select from large pools of candidate ideas has become a central problem shaping the quality of downstream research processes (Guo et al. 2024; Qiu et al. 2025). In such systems, LLMs can continuously generate large numbers of candidate scientific ideas at very low cost(Si et al. 2024; Wang et al. 2024c; Ji et al. 2026c). However, downstream experimental validation, literature search, method implementation, paper writing, and human review all require substantial real-world resources (Gelles et al. 2024; Si et al. 2025; Ye et al. 2026). As a result, not every generated idea can be further executed (Jie et al. 2026; Liu et al. 2026b; Ji et al. 2026b; Yang et al. 2026b; Ji et al. 2026a). As shown in Fig. 1, stylistic transformations can change Top-K membership, causing ideas to enter or leave the selected set despite retaining the same underlying scientific substance. Therefore, idea evaluation is no longer merely an auxiliary step after generation; it is a critical filtering mechanism that determines which ideas receive downstream research resources. Figure 1: Style-induced changes in Top-K membership relative to the plain setting. Drop out and Enter denote leaving and entering the plain Top-K after transformation. Existing research on the evaluation of AI-generated scientific ideas has developed multiple technical directions, aiming to improve evaluation reliability from the perspectives of scoring format, evaluation dimensions, information sources, and human calibration. Representative approaches include ELO-based tournament-style pairwise ranking(Zheng et al. 2023; Li et al. 2025; Rabeyah et al. 2024), multi-dimensional direct scoring along dimensions such as novelty, feasibility, and effectiveness (Qiu et al. 2025), multi-agent specialized evaluation that incorporates external literature or specialized agents(Xiong et al. 2024; Liu et al. 2025b; Pu et al. 2025; Wang et al. 2024b), and human-in-the-loop validation supported by expert blind review or human verification(Si et al. 2024; Radensky et al. 2026; Afzal et al. 2026). However, despite these engineering improvements, most of these methods share the same structural premise: scalable idea evaluation still primarily relies on LLM-as-Judge. Behind this premise, a more fundamental question remains insufficiently answered (Wang et al. 2024a; Sinhahajari et al. 2026): do the scores produced by an LLM judge truly reflect the substantive value of a scientific idea, or are they shaped by surface-level effects introduced by its linguistic presentation style? Figure 2: Overview of the SciStyleBench framework. (a) Variant creation, (b) three-stage evaluation, (c) idea-variant taxonomy, (d) bias diagnosis with SBI, SRR, and AWR, and (e) style-aware judging with extracted style signals. To answer this question, it is not sufficient to simply observe whether an LLM judge’s scores change. What is needed is a complete evaluation framework that can measure, diagnose, and mitigate stylistic bias. However, existing scientific idea evaluation still has clear gaps at three levels: metrics, benchmarks, and judges. First, at the benchmark level, existing data typically entangle content with style and lack a diagnostic environment for controlled style perturbation under fixed content. Meanwhile, these benchmarks lack reliable evaluation ground-truth construction, making it difficult to effectively disentangle stylistic factors from scientific merit(Chen et al. 2026; Kon et al. 2025; Liu et al. 2026b). Second, at the metrics level, existing evaluations lack a formal metric system for quantifying stylistic bias, making it difficult to characterize how style affects judge scores, dimension-level scores, overall scores, and top-K selection(Liu et al. 2025a; Kulkarni et al. 2025). Finally, at the judge level, most LLM-as-Judge methods directly evaluate ideas from raw text, causing scientific substance and presentation style to become entangled. Existing approaches, including prompting, multi-dimensional scoring, and retrieval augmentation, do not explicitly model stylistic nuisance factors, making it difficult for judges to distinguish the effects of scientific merit from rhetorical presentation(Team 2024). To address these issues, we develop SciStyleBench, a benchmark for systematically studying stylistic bias in scientific idea evaluation consisting of three core components: First, SciStyleStage, a style-paired benchmark that establishes relative ground truth by holding scientific substance fixed while varying presentation style, covering 600 scientific ideas, 15 controlled variants, and three background settings, with 9,000 instances per setting. Second, SciStyleMetrics, characterizes how writing style affects the scores and ranking outcomes produced by LLM judges through three closely interrelated metrics SBI, ADR, and SRR which respectively measure stylistic sensitivity, signal loss at the aggregation layer, and the ability to recognize substantive content. Finally, SciStyleExtractor is a plug-and-play auxiliary judging module that identifies style-related nuisance signals and injects structured information about style type, deviation from neutral presentation, and bias-control guidance into a frozen judge. Experiments on SciStyleBench show that Idea evaluators remain sensitive to style and weak in substance discrimination, while SciStyleExtractor improves SBI/SRR/AWR from 0.566/0.504/0.5540.566/0.504/0.554 to 0.501/0.759/0.8990.501/0.759/0.899. These results show that most existing idea evaluators are influenced by writing style rather than relying solely on substantive content. Improving their ability to assess ideas based on substance is therefore essential. SciStyleBench provides a systematic framework for identifying, quantifying, and mitigating such stylistic biases. The main contributions of this paper are threefold. (1) First, SciStyleBench, a style-paired benchmark with controlled presentation perturbations that disentangles style and substance effects and reveals style-induced evaluation biases. (2) Second, SciStyleMetrics to quantify stylistic bias in scientific idea evaluation through metrics such as SBI, ADR, and SRR, with theoretical analysis of aggregation dilution. (3) Finally, SciStyleExtractor, a plug-and-play auxiliary module that extracts structured style signals and injects bias-control guidance into frozen LLM judges to improve robust scientific idea evaluation on SciStyleBench. Related Work LLM-as-Judge in Scientific Idea Generation. LLM-as-Judge has become a common approach for evaluating AI-generated scientific ideas. Existing methods typically use pairwise ranking or multi-dimensional scoring based on criteria such as novelty, feasibility, effectiveness, and falsifiability (Shahhosseini et al. 2025; Bao et al. 2026). Recent benchmarks further evaluate whether LLM judges can identify methodologically sound proposals or make temporally grounded judgments (Ho et al. 2026; Ye et al. 2026; Wang 2026). Other studies improve idea evaluation through literature retrieval, specialized agents, multi-agent review, and human calibration (Baek et al. 2025; Li et al. 2026; Keya et al. 2025; Dong et al. 2026; Su et al. 2025; Radensky et al. 2024; Liu et al. 2026a; Nigam et al. 2024). However, these approaches generally treat the linguistic realization of an idea as fixed and do not test whether the same scientific substance receives different evaluations under alternative presentation styles. Stylistic Bias in LLM-as-Judge. Prior work shows that LLM judges are influenced by position, length, verbosity, formatting, model identity, and other surface-level cues (Li et al. 2024; Shen et al. 2026; Koo et al. ; Ruan et al. 2026). Benchmarks such as LLMBar test whether judges resist superficially convincing responses, while subsequent studies examine stylistic preferences, systematize evaluation biases, and propose mitigation methods (Zeng et al. 2024; Wu and Aji 2025; Zhou et al. 2024, 2026; Yang et al. 2026a). These findings suggest that judge outputs may reflect presentation artifacts rather than task-relevant quality (Cao et al. 2025; Rasheed et al. 2026). However, existing studies mainly focus on general response evaluation and have not systematically examined content-preserving style interventions for complete scientific ideas or their effects on dimension-level scores, rankings, and Top-K selection across evidence settings. We address this gap with SciStyleStage, SciStyleMetrics, and SciStyleExtractor. SciStyleStage Problem Formulation To investigates whether LLM-based judges remain stable when the presentation style of a scientific idea changes while its scientific substance is preserved. We represent an idea as I=⟨S,T⟩I= S,T , where S denotes its scientific substance, including the research problem, methodological mechanism, variable relationships, and verifiable contributions, while T denotes its presentation style, including verbosity, rhetorical strength, narrative structure, confidence, and novelty framing. Given background information K, the judge’s evaluation is denoted by J(I,K)J(I,K). Under fixed background knowledge K, different presentations of the same scientific substance S should receive similar evaluations. We test this invariance through controlled style-only perturbations. Our core intervention is a style-only perturbation. A perturbation operation p transforms the original idea into Ip=p(I)=⟨S,Tp⟩I_p=p(I)= S,T_p , where Tp≠T_p≠ T. Thus, I and IpI_p preserve the same scientific substance S but differ in presentation style. We define the resulting evaluation shift as Δp(I,K)=J(Ip,K)−J(I,K). _p(I,K)=J(I_p,K)-J(I,K). (1) For a style-invariant judge, Δp(I,K) _p(I,K) should remain close to zero because the original idea and its style variant contain the same scientific substance. A systematic positive or negative shift indicates that the evaluation is influenced by presentation style rather than scientific content, which we refer to as style-induced bias. Style Perturbation Space We first selected 600 papers from NeurIPS and ICLR across several research domains, including Computer Science, Biology, Mathematics, Astronomy/Space Science, and Economics. We then used DeepSeek-V3 to generate style variants for each idea and validated their quality. As shown in Fig. 2 (a), We define the style perturbation space using two types of variants: single-style and mixed-style variants. The single-style space contains 11 perturbations across three categories. Category A includes three baseline and expression-control variants: A1 (Identity), which retains the original text; A2 (Paraphrase Only), which rephrases the text while preserving its meaning; and A3 (Plain Core), which removes rhetorical packaging. Category B includes five core style-only perturbations that preserve the scientific substance while modifying only the presentation style: B1 (Verbose), which expands the description with additional detail; B2 (Grand Narrative), which introduces broad and visionary framing; B3 (Overconfident), which expresses claims with greater certainty; B4 (Novelty Emphasis), which strengthens novelty-related wording; and B5 (Application Framing), which emphasizes practical value and potential impact. Category C includes three substance and logic contrast variants: C1 (Hollow), which removes substantive support while retaining persuasive presentation; C2 (Flawed), which introduces logical weaknesses; and C3 (Enriched), which adds substantive information. The mixed-style space contains four variants: H1 (Deceptive Hollow: B2 + B4 + C1), which combines hollow content with visionary and novelty framing; H2 (Confident Flaw: B3 + C2), which presents flawed reasoning with strong confidence; H3 (Verbose Enriched: B1 + C3), which combines detailed expression with substantive enrichment; and H4 (Ultimate Hype: B2 + B4 + B5), which jointly emphasizes vision, novelty, and application value. Three-Stage Evaluation Tasks As shown in Fig. 2 (b), SciStyleBench evaluates the same set of idea variants using identical judging prompts and scoring criteria under three evaluation settings that differ only in the literature context. Stage 1 provides no external literature, Stage 2 provides literature from a fixed domain-specific pool, and Stage 3 provides literature retrieved specifically for the source idea. This controlled design allows us to examine whether external evidence mitigates or amplifies the judge’s sensitivity to presentation style. Stage 1: No Background. The judge evaluates each idea variant without access to any external literature, relying solely on the information contained in the idea text. Stage 2: Fixed-Domain Background. The judge receives literature from a fixed pool corresponding to the scientific domain of the source idea. These domain-specific literature pools are constructed from ResearchBench (Liu et al. 2026b), Paperzilla 250 (Team 2024), and IdeaBench (Guo et al. 2024). ResearchBench supplies most scientific domains, Paperzilla 250 supplies Computer Science, and IdeaBench supplies Medicine. Stage 3: Idea-Specific Retrieval. We use the original, unperturbed source idea to construct retrieval queries covering its research question, methodology, and key concepts. Candidate papers are retrieved from OpenAlex, with Crossref used as a supplementary source when the OpenAlex results are insufficient. After deduplication and relevance ranking, we retain the 50 papers most relevant to each source idea. In Stages 2 and 3, the judge does not receive the full text of the selected papers. Instead, each paper is represented by its title and abstract, formatted as a structured list and appended to the idea variant as background context. All variants derived from the same source idea receive the same literature in the same order, preventing retrieval differences from introducing an additional confounding factor. SciStyleMetrics As shown in Fig. 2 (c), SciStyleMetrics evaluates whether an LLM judge is insensitive to stylistic variation while remaining sensitive to scientific substance. Let J(d)(Ii,p,Ki(m))J^(d)(I_i,p,K_i^(m)) denote the score of idea variant Ii,pI_i,p on dimension d under evaluation setting m. We measure three complementary and interdependent properties using SBI, SRR, and AWR. Style Bias Index. The Style Bias Index (SBI) measures whether the judge remains stable when the scientific substance is fixed but the presentation style changes. Using the Plain variant as the neutral reference and Paraphrase as a rewriting control, we define SBI(m)(p)=i,d[|(J(d)(Ii,p,Ki(m))−J(d)(Ii,plain,Ki(m)))−(J(d)(Ii,para,Ki(m))−J(d)(Ii,plain,Ki(m)))|] aligned SBI^(m)(p)=E_i,d [ |& (J^(d)(I_i,p,K_i^(m))-J^(d)(I_i,plain,K_i^(m)) )\\ &- (J^(d)(I_i,para,K_i^(m))-J^(d)(I_i,plain,K_i^(m)) ) | ] aligned (2) A lower SBI indicates that different expressions of the same scientific idea receive more consistent scores. Thus, SBI answers: when substance is unchanged, how much does the score change because of style? Substance Recognition Rate. The Substance Recognition Rate (SRR) measures whether the judge detects controlled changes in scientific quality. Let subQ_sub contain ordered pairs (a,b)(a,b) in which a has stronger substance than b, including Enriched over Plain, Plain over Hollow, and Plain over Flawed: SRR(m)=1|sub|∑(a,b)∈sub[J(a,K(m))>J(b,K(m))]. ^(m)= 1|Q_sub| _(a,b) _subI [J(a,K^(m))>J(b,K^(m)) ]. (3) A higher SRR indicates stronger sensitivity to substantive information, methodological validity, and logical soundness. Thus, SRR answers: can the judge recognize meaningful differences in scientific substance? Adversarial Win Rate. The Adversarial Win Rate (AWR) measures which signal dominates when style and substance conflict. Let advQ_adv contain pairs (h,l)(h,l), where h is a high-substance idea with plain presentation and l is a low-substance idea with persuasive stylistic packaging: AWR(m)=1|adv|∑(h,l)∈adv[J(h,K(m))>J(l,K(m))]. ^(m)= 1|Q_adv| _(h,l) _advI [J(h,K^(m))>J(l,K^(m)) ]. (4) A higher AWR indicates that the judge prioritizes scientific substance over verbosity, grand narratives, novelty claims, or overconfident language. Thus, AWR answers: when substance and presentation disagree, does the judge select the substantively stronger idea? Joint Interpretation. The three metrics must be interpreted jointly: robust evaluation requires low SBI, high SRR, and high AWR, corresponding to style invariance, substance sensitivity, and adversarial robustness. In particular, low style sensitivity does not necessarily imply strong evaluation ability. A judge that assigns nearly identical scores to all ideas may obtain a very low SBI, while still failing to distinguish Enriched from Plain, Plain from Hollow, or valid mechanisms from logically flawed ones. Therefore, minimizing SBI alone may reward score collapse rather than genuine robustness; it must be accompanied by high SRR and AWR. This distinction captures the central measurement principle of SciStyleMetrics: robust evaluation is not simple score invariance, but invariance to task-irrelevant nuisance variation while preserving sensitivity to the target scientific construct. Figure 3: Training and inference pipeline of SciStyleExtractor. The extractor learns teacher-aligned style signals during training and injects them into the judger at inference. SciStyleExtractor Scientific substance and presentation style are entangled in the raw idea text, making it difficult for an LLM judge to determine whether an apparent quality signal comes from the idea itself or from rhetorical packaging. SciStyleExtractor is therefore designed as an auxiliary judging module. It does not score or rewrite the idea; instead, it identifies style-related nuisance signals and injects structured bias-control information into a frozen judge. Style-Aware Judging Framework Given an idea variant Ip=⟨S,Tp⟩I_p= S,T_p and background context K, a standard judge directly produces pbase=Jθ(Ip,K)y^base_p=J_θ(I_p,K) and may mistake stylistic presentation for scientific quality. As shown in Fig. 3, SciStyleExtractor introduces an intermediate style signal before judging: p=fϕ(Ip,K)=(tp,δp,hp),^p=Jθ(Ip,K;p).z_p=f_φ(I_p,K)=(t_p, _p,h_p), y_p=J_θ(I_p,K;z_p). (5) Here, fϕf_φ is the trainable style extractor and JθJ_θ is the frozen judge. The extractor output pz_p contains the detected style type tpt_p, its deviation from neutral presentation δp _p, and a bias-control instruction hph_p. For example, it may identify an input as verbose, determine that the change is stylistic rather than substantive, and instruct the judge not to reward rhetorical elaboration. The final output ^p y_p contains the judge’s dimension-level scores, overall score, and evaluation feedback. Thus, pz_p serves only as auxiliary evidence and does not directly determine the evaluation result. Extractor-Aligned Training We train the extractor through teacher–student distillation. For each input (Ip,K)(I_p,K), the teacher judge receives an ideal style signal p⋆z_p , whereas the student judge receives the predicted signal p=fϕ(Ip,K)z_p=f_φ(I_p,K). Their output distributions are denoted by PT()=Pθ(∣Ip,K;p⋆)P_T(y)=P_θ(y I_p,K;z_p ) and PS()=Pθ(∣Ip,K;p)P_S(y)=P_θ(y I_p,K;z_p), respectively. We optimize the extractor using ℒKL=(Ip,K)∼[KL(PT()∥PS())].L_KL=E_(I_p,K) [KL (P_T(y)\,\|\,P_S(y) ) ]. (6) Minimizing this objective teaches the extractor to generate auxiliary style signals that make the student approximate the teacher’s debiased evaluation behavior. During inference, the trained extractor generates pz_p, which is injected into the frozen judge to support substance-focused evaluation. Judge Method No Background Fixed-domain Background Idea-specific Retrieval SBI ↓ SRR ↑ AWR ↑ SBI ↓ SRR ↑ AWR ↑ SBI ↓ SRR ↑ AWR ↑ Without SciStyleExtractor Direct Qwen3.5-4B 0.7800.0014070.780_0.001407 0.8600.0012040.860_0.001204 0.9400.0005640.940_0.000564 0.4760.0011800.476_0.001180 0.7000.0021000.700_0.002100 0.8500.0012750.850_0.001275 0.6600.0011330.660_0.001133 0.8400.0013440.840_0.001344 0.9100.0008190.910_0.000819 Direct Llama-3.1-8B 1.3540.0084071.354_0.008407 0.0200.0001960.020_0.000196 0.0100.0000990.010_0.000099 1.4180.0119081.418_0.011908 0.3000.0021000.300_0.002100 0.0900.0008190.090_0.000819 0.1650.0004260.165_0.000426 0.1100.0009790.110_0.000979 0.2500.0018750.250_0.001875 Direct Qwen3.5-27B 0.3080.0004430.308_0.000443 0.5100.0024990.510_0.002499 0.6300.0023310.630_0.002331 0.2620.0003680.262_0.000368 0.4400.0024640.440_0.002464 0.5200.0024960.520_0.002496 0.2650.0003650.265_0.000365 0.4200.0024360.420_0.002436 0.5500.0024750.550_0.002475 Direct DeepSeek-V3.2 0.4870.0007600.487_0.000760 0.7600.0018240.760_0.001824 0.7500.0018750.750_0.001875 0.4180.0006500.418_0.000650 0.6900.0021390.690_0.002139 0.7300.0019710.730_0.001971 0.2020.0004720.202_0.000472 0.4000.0024000.400_0.002400 0.4200.0024360.420_0.002436 Direct OpenReviewer 0.4940.0012480.494_0.001248 0.1600.0013440.160_0.001344 0.6100.0023790.610_0.002379 0.0780.0012210.078_0.001221 0.0000.0000000.000_0.000000 0.0000.0000000.000_0.000000 0.2380.0008370.238_0.000837 0.0200.0001960.020_0.000196 0.1700.0014110.170_0.001411 Direct AI-Scientist-Llama3.1-8B 0.4780.0009590.478_0.000959 0.4500.0024750.450_0.002475 0.7200.0020160.720_0.002016 0.1640.0007120.164_0.000712 0.1700.0014110.170_0.001411 0.1600.0013440.160_0.001344 0.1770.0003570.177_0.000357 0.3000.0021000.300_0.002100 0.2200.0017160.220_0.001716 Style-CoT Qwen3.5-4B 0.6650.0010080.665_0.001008 0.7700.0017710.770_0.001771 0.8300.0014110.830_0.001411 0.8600.0016220.860_0.001622 0.6800.0021760.680_0.002176 0.6800.0021760.680_0.002176 0.7140.0014270.714_0.001427 0.8000.0016000.800_0.001600 0.8100.0015390.810_0.001539 Style-CoT Llama-3.1-8B 1.3250.0125691.325_0.012569 0.2200.0017160.220_0.001716 0.1800.0014760.180_0.001476 0.7880.0082620.788_0.008262 0.2200.0017160.220_0.001716 0.1700.0014110.170_0.001411 0.2410.0004850.241_0.000485 0.0800.0007360.080_0.000736 0.0900.0008190.090_0.000819 Style-CoT Qwen3.5-27B 0.4840.0006360.484_0.000636 0.7600.0018240.760_0.001824 0.8500.0012750.850_0.001275 0.4520.0007820.452_0.000782 0.7700.0017710.770_0.001771 0.7600.0018240.760_0.001824 0.4510.0008720.451_0.000872 0.6500.0022750.650_0.002275 0.7100.0020590.710_0.002059 Style-CoT DeepSeek-V3.2 0.2570.0006760.257_0.000676 0.4800.0024960.480_0.002496 0.5300.0024910.530_0.002491 0.3630.0007110.363_0.000711 0.5800.0024360.580_0.002436 0.5200.0024960.520_0.002496 0.3750.0005000.375_0.000500 0.6000.0024000.600_0.002400 0.7200.0020160.720_0.002016 Style-CoT OpenReviewer 0.4720.0017910.472_0.001791 0.0600.0005640.060_0.000564 0.4300.0024510.430_0.002451 0.0080.0000290.008_0.000029 0.0000.0000000.000_0.000000 0.0000.0000000.000_0.000000 0.2820.0009760.282_0.000976 0.1100.0009790.110_0.000979 0.2100.0016590.210_0.001659 Style-CoT AI-Scientist-Llama3.1-8B 0.5150.0005580.515_0.000558 0.1000.0009000.100_0.000900 0.4500.0024750.450_0.002475 0.3280.0014300.328_0.001430 0.2800.0020160.280_0.002016 0.1600.0013440.160_0.001344 0.4090.0005430.409_0.000543 0.4300.0024510.430_0.002451 0.1600.0013440.160_0.001344 With SciStyleExtractor Qwen3.5-4B + LLM Style Injection 1.3310.0010811.331_0.001081 0.5700.0024510.570_0.002451 1.0000.0000001.000_0.000000 1.2750.0018761.275_0.001876 0.5300.0024910.530_0.002491 1.0000.0000001.000_0.000000 1.2320.0012151.232_0.001215 0.3900.0023790.390_0.002379 1.0000.0000001.000_0.000000 Llama-3.1-8B + LLM Style Injection 0.4670.0027040.467_0.002704 0.1600.0013440.160_0.001344 0.1700.0014110.170_0.001411 1.8030.0067901.803_0.006790 0.2800.0020160.280_0.002016 0.4000.0024000.400_0.002400 0.3500.0005520.350_0.000552 0.1400.0012040.140_0.001204 0.9000.0009000.900_0.000900 Qwen3.5-27B + LLM Style Injection 0.8140.0020720.814_0.002072 0.7000.0021000.700_0.002100 0.9900.0000990.990_0.000099 0.7600.0021460.760_0.002146 0.5600.0024640.560_0.002464 0.8300.0014110.830_0.001411 0.7260.0016620.726_0.001662 0.9200.0007360.920_0.000736 0.9500.0004750.950_0.000475 DeepSeek-V3.2 + LLM Style Injection 1.0090.0039171.009_0.003917 0.7800.0017160.780_0.001716 1.0000.0000001.000_0.000000 1.2510.0031941.251_0.003194 0.9100.0008190.910_0.000819 1.0000.0000001.000_0.000000 0.9670.0033320.967_0.003332 0.7800.0017160.780_0.001716 1.0000.0000001.000_0.000000 OpenReviewer + LLM Style Injection 0.5550.0015070.555_0.001507 0.2300.0017710.230_0.001771 0.4600.0024840.460_0.002484 0.0060.0000230.006_0.000023 0.0000.0000000.000_0.000000 0.0000.0000000.000_0.000000 0.5350.0017150.535_0.001715 0.2100.0016590.210_0.001659 0.3700.0023310.370_0.002331 AI-Scientist-Llama3.1-8B + LLM Style Injection 0.4190.0002500.419_0.000250 0.0900.0008190.090_0.000819 0.4800.0024960.480_0.002496 0.2510.0014230.251_0.001423 0.0500.0004750.050_0.000475 0.1900.0015390.190_0.001539 0.6180.0004920.618_0.000492 0.3600.0023040.360_0.002304 0.8800.0010560.880_0.001056 Qwen3.5-4B + Trained Style Extractor 0.3630.0003660.363_0.000366 0.9100.0008190.910_0.000819 0.9600.0003840.960_0.000384 0.3690.0003820.369_0.000382 0.9200.0007360.920_0.000736 0.9700.0002910.970_0.000291 0.4060.0002860.406_0.000286 0.8400.0013440.840_0.001344 0.9300.0006510.930_0.000651 Llama-3.1-8B + Trained Style Extractor 0.9120.0020470.912_0.002047 0.5500.0024750.550_0.002475 0.9900.0000990.990_0.000099 0.3920.0006270.392_0.000627 0.7400.0019240.740_0.001924 0.9600.0003840.960_0.000384 0.5680.0007120.568_0.000712 0.4200.0024360.420_0.002436 0.8800.0010560.880_0.001056 Qwen3.5-27B + Trained Style Extractor 0.5000.0004780.500_0.000478 0.8900.0009790.890_0.000979 0.9200.0007360.920_0.000736 0.5040.0007590.504_0.000759 0.8700.0011310.870_0.001131 0.8200.0014760.820_0.001476 0.5130.0003760.513_0.000376 0.8900.0009790.890_0.000979 0.8600.0012040.860_0.001204 DeepSeek-V3.2 + Trained Style Extractor 0.4150.0003800.415_0.000380 0.7300.0019710.730_0.001971 0.8800.0010560.880_0.001056 0.5980.0008050.598_0.000805 0.7100.0020590.710_0.002059 0.8600.0012040.860_0.001204 0.4690.0003900.469_0.000390 0.6400.0023040.640_0.002304 0.7600.0018240.760_0.001824 OpenReviewer + Trained Style Extractor 0.2450.0003480.245_0.000348 0.3700.0023310.370_0.002331 0.7800.0017160.780_0.001716 0.0080.0000120.008_0.000012 0.0200.0001960.020_0.000196 0.0300.0002910.030_0.000291 0.2800.0003340.280_0.000334 0.5600.0024640.560_0.002464 0.7400.0019240.740_0.001924 AI-Scientist-Llama3.1-8B + Trained Style Extractor 0.9200.0005140.920_0.000514 0.6900.0021390.690_0.002139 0.9900.0000990.990_0.000099 0.4520.0004490.452_0.000449 0.8400.0013440.840_0.001344 0.9600.0003840.960_0.000384 0.7020.0009370.702_0.000937 0.8100.0015390.810_0.001539 0.9400.0005640.940_0.000564 Table 1: Robustness comparison of judging methods across background settings. Each metric is computed from five evaluation dimensions (novelty, feasibility, significance, rigor, and clarity). Values report the mean performance, with subscripts denoting standard errors. Lower SBI and higher SRR/AWR indicate better robustness. Experiments Experimental Setup Dataset and evaluation. We conduct experiments on SciStyleBench, which contains 600 source scientific ideas, each paired with 15 controlled variants covering reference rewrites, style-only perturbations, substance and logical controls, and mixed-style perturbations. All variants are evaluated under three background settings: no background, fixed-domain background, and idea-specific retrieval background, resulting in 9,000 evaluation instances per setting and 27,000 instances in total. We evaluate judging robustness using SBI, SRR, and AWR, which jointly measure style invariance, substance sensitivity, and adversarial robustness. Judging methods. We evaluate six judges: four general-purpose models Qwen3.5-4B, Llama-3.1-8B, Qwen3.5-27B, and DeepSeek-V3.2 and two scientific-review models, OpenReviewer (Idahl and Ahmadi 2025) and AI-Scientist-Llama3.1-8B (Lu et al. 2024). OpenReviewer is obtained by supervised fine-tuning Llama-3.1-8B for scientific reviewing, enabling a controlled comparison between the original backbone and its domain-adapted counterpart. For each judge, we compare four configurations: direct judging, style-aware CoT prompting, LLM-based style-signal injection, and our trained SciStyleExtractor. Direct judging uses the raw idea, whereas style-aware CoT prompts the judge to consider stylistic influence before scoring. The two injection-based settings provide the frozen judge with structured auxiliary signals, including style type, deviation from neutral presentation, and a bias-control instruction, generated by either a strong LLM or SciStyleExtractor. All judges remain frozen, so performance differences reflect the effect of auxiliary style awareness rather than further judge adaptation. Figure 4: Comparison between direct judging and the trained SciStyleExtractor across six judge models. From left to right, the three column groups correspond to no background, fixed-domain background, and idea-specific retrieval. The top, middle, and bottom rows report SBI, SRR, and AWR, respectively; lower SBI and higher SRR/AWR indicate better judging robustness. Main Result We first evaluate whether LLM-as-a-Judge methods remain reliable under controlled style variations. We compare different judges across three background settings using SBI, SRR, and AWR, measuring style invariance, substance sensitivity, and adversarial robustness, respectively. Table 1 summarizes the results. Direct judges show substantial style sensitivity and limited substance discrimination, whereas SciStyleExtractor achieves a better balance across the three objectives, although residual ranking instability remains. (1) Direct judging is not robust. Across four general-purpose judges and three settings, Direct Judge achieves SBI/SRR/AWR scores of 0.566/0.504/0.5540.566/0.504/0.554, indicating substantial style sensitivity and limited substance discrimination. Dimension-level analysis shows that clarity (1.1851.185) and feasibility (0.7020.702) are most affected by presentation style. Additional context does not consistently improve robustness: retrieval reduces SBI but provides limited gains in SRR and AWR. (2) Low SBI does not necessarily imply robustness. Under fixed-domain background, Direct OpenReviewer obtains near-zero SBI but also near-zero SRR and AWR, suggesting score collapse rather than effective debiasing. With idea-specific retrieval and SciStyleExtractor, its scores improve to 0.280/0.560/0.7400.280/0.560/0.740. These results highlight that SBI, SRR, and AWR should be interpreted jointly. (3) SciStyleExtractor achieves the best balance. The trained SciStyleExtractor achieves 0.501/0.759/0.8990.501/0.759/0.899, outperforming Direct Judge and Style-CoT (0.581/0.551/0.5710.581/0.551/0.571). LLM-based style injection improves AWR (0.8530.853) but increases SBI to 0.9990.999, suggesting over-correction. The extractor’s gains mainly come from improved SRR and AWR rather than SBI reduction, indicating that it improves substance sensitivity while maintaining style robustness instead of simply flattening scores. (4) Ranking instability remains. The average absolute rank shifts are 2.1162.116, 2.6132.613, and 2.8172.817 for Direct Judge, Style-CoT, and the KL-trained SciStyleExtractor, respectively. Rank shifts are larger for substance/logic variants (2.9482.948) and hybrid variants (3.6933.693) than for rhetoric/framing variants (1.9401.940). The larger shift of SciStyleExtractor does not necessarily indicate worse robustness, as shifts caused by substantive changes reflect desirable sensitivity, while style-only shifts reveal residual bias. Hybrid variants remain the most challenging due to the combination of degraded substance and persuasive presentation. Figure 5: Score and rank shifts relative to the Plain reference. The x-axis denotes mean score delta and the y-axis denotes rank delta; lower rank shifts indicate more stable rankings. Additional Analysis Variant-level Perturbation Effects. To further unpack the main results, we examine how the effect of SciStyleExtractor varies across judge models and background settings. As shown in Fig. 4, we compare direct judging with the trained SciStyleExtractor across six judge models and three background settings. SciStyleExtractor generally improves SRR and AWR, indicating stronger substance recognition and adversarial robustness, while its effect on SBI varies across judges and settings. These results show that the extractor provides a more balanced evaluation overall, but does not uniformly eliminate style sensitivity. Score and Rank Shifts. We further analyze whether score changes translate into stable ranking behavior. As shown in Fig. 5, the relationship between mean score shifts and rank shifts varies substantially across judges and background settings. Several judges exhibit noticeable ranking changes even when their average score shifts are limited, showing that pointwise score stability does not necessarily guarantee ranking stability. The results further indicate that SciStyleExtractor improves overall robustness but does not completely remove style-induced changes in relative ranking. Top-K Membership Changes. For each style condition, we rank all 600 ideas and compare its Top-K set with the Plain reference. Drop out and Enter denote ideas leaving and entering the Plain Top-K after transformation. As shown in Fig. 1, the membership-change rate rises from 4.6%4.6\% at Top-5 to 15.9%15.9\% at Top-30, showing that style variation can alter candidate selection and lead to downstream screening or resource-allocation errors. Rewrite Validation First, to ensure the stylistic accuracy of the generated variants and the stability of the evaluation process, we further conduct a style consistency evaluation. Specifically, we manually annotate the source ideas and their corresponding generated variants, comparing the human evaluation results against the outcomes of the LLM judges, as well as performing cross-comparisons among different LLM judges. The evaluation results indicate that the target style of the generated variants can be consistently identified by various LLM judges, and the automated evaluation results maintain a high degree of agreement with human assessments. Therefore, this consistency manifested not only across different LLM judges but also between the LLM judges and human evaluators demonstrates the effectiveness of our generated variants in terms of style control and evaluation reliability. Comparison Pair Spearman (p) Overall Top-50 Five-score Top-5 ChatGPT vs Human 0.88 0.80 0.80 DeepSeek-V3.2 vs Human 0.84 0.78 0.40 Kimi vs Human 0.81 0.68 0.60 ChatGPT vs DeepSeek-V3.2 0.82 0.90 0.74 ChatGPT vs Kimi 0.92 0.93 0.82 DeepSeek-V3.2 vs Kimi 0.90 0.94 0.76 Table 2: Agreement Between Human and LLM Judges. Second, to validate that our perturbations modify presentation style while preserving scientific content, we randomly sampled 320 source-variant pairs for human evaluation. Two annotators assessed three aspects: (1) preservation of scientific substance, including the research question, mechanism, variables, constraints, methodology, and contribution; (2) consistency of perceived scientific quality; and (3) whether differences were primarily stylistic rather than content-related. Disagreements were resolved through adjudication without access to downstream evaluation results. As shown in Table 3, 93.1% of pairs preserved the original scientific substance, 90.9% maintained equivalent perceived quality, and 97.9% were identified as presentation-level variations. These results confirm that our perturbations primarily alter stylistic expression while preserving the underlying scientific value of the ideas. Variant #Pairs Substance Quality Style Preserve (%) Same (%) Only (%) Paraphrase 40 97.5 95.0 100.0 Plain Core 40 95.0 92.5 98.3 Verbose 40 92.5 90.0 98.3 Grand Narrative 40 90.0 87.5 96.7 Overconfident 40 92.5 90.0 98.3 Ultimate Hype 40 87.5 82.5 95.0 Overall 240 92.5 89.7 97.8 Table 3: Human validation of style-only transformations. Experts evaluate whether transformed variants preserve scientific substance, maintain perceived scientific quality, and differ primarily in presentation style. Ablation We fix Qwen3.5-4B as the target judge and compare five configurations: direct judging, Style-CoT, LLM-based style injection, and SciStyleExtractor trained with either SFT or SFT+KL. This comparison examines whether robustness gains come from generic prompting, externally generated style signals, or a learned extractor. Qwen3.5-27B provides teacher supervision, while the Qwen3.5-4B extractor is implemented as a LoRA adapter and trained on 2,000 training pairs for two epochs. For SFT and KL, the extractor is initialized with SFT and further aligned to the teacher’s output distribution through KL distillation. Results are averaged across the three background settings. As shown in Table. 4, Style-CoT performs worse than direct judging, indicating that simply prompting the judge to consider style is insufficient. LLM-based injection achieves a high AWR but substantially worsens SBI and SRR, suggesting over-correction caused by noisy or overly strong style signals. In contrast, both trained extractors improve the overall balance among the three metrics: SFT+KL achieves the lowest SBI, while SFT-only obtains slightly higher SRR and AWR. These results suggest that KL distillation mainly strengthens style invariance, whereas SFT better preserves substance discrimination and adversarial robustness. Style Signal Source SBI ↓ SRR ↑ AWR ↑ None / Direct Judge 0.639 0.800 0.900 Style-CoT Prompt 0.746 0.750 0.773 LLM-based Style Injection 1.279 0.497 1.000 Qwen3.5-4B Style Extractor (KL) 0.380 0.890 0.953 Qwen3.5-4B Style Extractor (SFT) 0.406 0.893 0.973 Table 4: Ablation of style-signal sources. We compare direct judging, style-aware prompting, LLM-based injection, and trained style extractors. SBI is averaged across three evaluation settings and style-only variants. Conclusion This work introduces SciStyleBench to diagnose and mitigate stylistic bias in scientific idea evaluation. Experiments across diverse judges and background settings show that direct LLM-as-a-Judge methods remain sensitive to presentation style and struggle to reliably distinguish substantive scientific quality. We further show that low style sensitivity alone is insufficient, as low SBI may result from score collapse rather than genuine robustness. SciStyleExtractor improves the balance among style invariance, substance recognition, and adversarial robustness, while residual ranking instability remains. Overall, robust scientific idea evaluation requires judges to suppress style-related nuisance signals while preserving sensitivity to scientific substance, logical validity, and practical feasibility. References O. M. Afzal, P. Nakov, T. Hope, and I. Gurevych (2026) Beyond" not novel enough": enriching scholarly critique with llm-assisted feedback. In Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), p. 2648–2671. Cited by: Introduction. J. Baek, S. K. Jauhar, S. Cucerzan, and S. J. Hwang (2025) ResearchAgent: iterative research idea generation over scientific literature with large language models. In Proceedings of NAACL 2025, Cited by: LLM-as-Judge in Scientific Idea Generation.. H. Bao, S. Wu, X. Liu, S. Li, S. Cao, and J. A. Evans (2026) Contemporary ai lacks the imagination to diverge or negate in science. arXiv preprint arXiv:2606.08251. Cited by: LLM-as-Judge in Scientific Idea Generation.. Q. Cao, X. Wang, Y. Yuan, Y. Liu, F. Luo, and R. Song (2025) Evaluating text creativity across diverse domains: a dataset and large language model evaluator. arXiv preprint arXiv:2505.19236. Cited by: Stylistic Bias in LLM-as-Judge.. H. Chen, M. Xiong, Y. Lu, W. Han, A. Deng, Y. He, J. Wu, Y. Li, Y. Liu, and B. Hooi (2026) Mlr-bench: evaluating ai agents on open-ended machine learning research. Advances in Neural Information Processing Systems 38. Cited by: Introduction. J. Dong, B. Li, and W. Lin (2026) Evolving idea graphs with learnable edits-and-commits for multi-agent scientific ideation. External Links: 2605.04922, Link Cited by: LLM-as-Judge in Scientific Idea Generation.. R. Gelles, V. Kinoshita, M. Musser, and J. Dunham (2024) Resource democratization: is compute the binding constraint on ai research?. Proceedings of the AAAI Conference on Artificial Intelligence 38 (18), p. 19840–19848. External Links: Document, ISSN 2159-5399, Link Cited by: Introduction. J. Gottweis, W. Weng, A. Daryin, and et al. (2026) Accelerating scientific discovery with co-scientist. Nature. External Links: Document Cited by: Introduction. S. Guo, A. H. Shariatmadari, G. Xiong, A. Huang, E. Xie, S. Bekiranov, and A. Zhang (2024) IdeaBench: benchmarking large language models for research idea generation. External Links: 2411.02429, Link Cited by: Introduction, Stage 2: Fixed-Domain Background.. S. Ho, M. Liu, H. Nghiem, and F. Huang (2026) SoundnessBench: can your ai scientist really tell good research ideas from bad ones?. arXiv preprint arXiv:2605.30329. Cited by: LLM-as-Judge in Scientific Idea Generation.. M. Idahl and Z. Ahmadi (2025) Openreviewer: a specialized large language model for generating critical scientific paper reviews. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (System Demonstrations), p. 550–562. Cited by: Judging methods.. F. Ji, Z. Xie, J. Yang, F. Zhang, Z. Song, and X. Chen (2026a) Parametric memory decoding for zero-shot routing in lora-based external parametric memory. arXiv preprint arXiv:2607.04118. Cited by: Introduction. F. Ji, J. Yang, Z. Song, L. Gao, J. Liang, Z. Chen, J. Zhang, and X. Chen (2026b) ServImage: an image generation and editing benchmark from real-world commercial imaging services. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 43504–43529. Cited by: Introduction. F. Ji, J. Yang, Z. Song, Y. Wang, Z. Cui, Y. Li, Q. Jiang, and X. Chen (2026c) FineState-bench: benchmarking state-conditioned grounding for fine-grained gui state setting. In Findings of the Association for Computational Linguistics: ACL 2026, p. 43073–43088. Cited by: Introduction. R. Jie, C. Chu, and Z. Wang (2026) Capability-aware early-stage research idea evaluation. arXiv preprint arXiv:2601.12473. Cited by: Introduction. F. Keya, G. Rabby, P. Mitra, S. Vahdati, S. Auer, and Y. Jaradeh (2025) SCI-idea: context-aware scientific ideation using token and sentence embeddings. External Links: 2503.19257, Link Cited by: LLM-as-Judge in Scientific Idea Generation.. P. T. J. Kon, J. Liu, X. Zhu, Q. Ding, J. Peng, J. Xing, Y. Huang, Y. Qiu, J. Srinivasa, M. Lee, et al. (2025) Exp-bench: can ai conduct ai research experiments?. arXiv preprint arXiv:2505.24785. Cited by: Introduction. [18] H. Koo, C. Jung, F. Wu, and J. Kim Auditing the judge: human-grounded bias discovery, quantification, and mitigation in llm judges. In Trustworthy AI for Good (AI4GOOD) Workshop@ ICML 2026, Cited by: Stylistic Bias in LLM-as-Judge.. A. Kulkarni, F. Alotaibi, X. Zeng, L. Wu, T. Zeng, B. M. Yao, M. Liu, S. Zhang, L. Huang, and D. Zhou (2025) Scientific hypothesis generation and validation: methods, datasets, and future directions. arXiv preprint arXiv:2505.04651. Cited by: Introduction. H. Li, Q. Dong, J. Chen, H. Su, Y. Zhou, Q. Ai, Z. Ye, and Y. Liu (2024) Llms-as-judges: a comprehensive survey on llm-based evaluation methods. arXiv preprint arXiv:2412.05579. Cited by: Stylistic Bias in LLM-as-Judge.. R. Li, C. Zhu, B. Xu, X. Wang, and Z. Mao (2025) Automated creativity evaluation for large language models: a reference-based approach. arXiv preprint arXiv:2504.15784. Cited by: Introduction. X. Li, H. Tu, and X. Han (2026) Graph2Idea:retrieval-augmented scientific idea generation with graph-structured contexts. External Links: 2606.09105, Link Cited by: LLM-as-Judge in Scientific Idea Generation.. H. Liu, S. Huang, J. Hu, Y. Zhou, and C. Tan (2025a) Hypobench: towards systematic and principled benchmarking for hypothesis generation. arXiv preprint arXiv:2504.11524. Cited by: Introduction. H. Liu, Y. Choi, S. Gautam, G. Jaffe, S. Y. Rieh, and M. Lease (2026a) Who owns creativity and who does the work? trade-offs in llm-supported research ideation. ArXiv.org. Cited by: LLM-as-Judge in Scientific Idea Generation.. Y. Liu, P. Sharma, M. Oswal, H. Xia, and Y. Huang (2025b) PersonaFlow: designing llm-simulated expert perspectives for enhanced research ideation. In Proceedings of the ACM Conference, Cited by: Introduction. Y. Liu, Z. Yang, T. Xie, J. Ni, B. Gao, Y. Li, S. Tang, W. Ouyang, E. Cambria, and D. Zhou (2026b) Researchbench: benchmarking llms in scientific discovery via inspiration-based task decomposition. In Findings of the Association for Computational Linguistics: ACL 2026, p. 13187–13207. Cited by: Introduction, Introduction, Stage 2: Fixed-Domain Background.. C. Lu, C. Lu, R. T. Lange, J. Foerster, J. Clune, and D. Ha (2024) The ai scientist: towards fully automated open-ended scientific discovery. External Links: 2408.06292, Link Cited by: Introduction, Judging methods.. Z. Luo, Z. Yang, Z. Xu, W. Yang, and X. Du (2025) LLM4SR: a survey on large language models for scientific research. External Links: 2501.04306, Link Cited by: Introduction. H. Nigam, M. Patwardhan, L. Vig, and G. Shroff (2024) Acceleron: a tool to accelerate research ideation. arXiv (Cornell University). External Links: Document Cited by: LLM-as-Judge in Scientific Idea Generation.. K. Pu, K. J. K. Feng, T. Grossman, T. Hope, B. Dalvi Mishra, M. Latzke, J. Bragg, J. C. Chang, and P. Siangliulue (2025) IdeaSynth: iterative research idea development through evolving and composing idea facets with literature-grounded feedback. In Proceedings of the CHI Conference on Human Factors in Computing Systems, Cited by: Introduction. Y. Qiu, H. Zhang, Z. Xu, M. Li, D. Song, Z. Wang, and K. Zhang (2025) AI idea bench 2025: ai research idea generation benchmark. External Links: 2504.14191, Link Cited by: Introduction, Introduction. A. A. Rabeyah, F. Góes, M. Volpe, and T. Medeiros (2024) Do llms agree on the creativity evaluation of alternative uses?. arXiv preprint arXiv:2411.15560. Cited by: Introduction. M. Radensky, S. Shahid, R. Fok, P. Siangliulue, T. Hope, and D. S. Weld (2024) Scideator: human-llm compound system for scientific ideation through facet recombination and novelty evaluation. arXiv preprint arXiv:2409.14634. Cited by: LLM-as-Judge in Scientific Idea Generation.. M. Radensky, S. Shahid, R. Fok, P. Siangliulue, T. Hope, and D. S. Weld (2026) Human-llm compound system for scientific ideation through facet recombination and novelty evaluation. External Links: 2409.14634, Link Cited by: Introduction. R. A. Rasheed, S. Banerjee, A. Mukherjee, and R. Hazra (2026) From fluent to verifiable: claim-level auditability for deep research agents. arXiv preprint arXiv:2602.13855. Cited by: Stylistic Bias in LLM-as-Judge.. K. Ruan, X. Wang, J. Hong, P. Wang, Y. Liu, and H. Sun (2026) Evaluating llms’ divergent thinking capabilities for scientific idea generation with minimal context. External Links: 2412.17596, Link Cited by: Stylistic Bias in LLM-as-Judge.. F. Shahhosseini, A. Marioriyad, A. Momen, M. S. Baghshah, M. H. Rohban, and S. H. Javanmard (2025) Large language models for scientific idea generation: a creativity-centered survey. arXiv preprint arXiv:2511.07448. Cited by: LLM-as-Judge in Scientific Idea Generation.. Y. Shen, M. Liu, D. Zhou, and L. Huang (2026) Navigating ideation space: decomposed conceptual representations for positioning scientific ideas. arXiv preprint arXiv:2601.08901. Cited by: Stylistic Bias in LLM-as-Judge.. C. Si, T. Hashimoto, and D. Yang (2025) The ideation-execution gap: execution outcomes of llm-generated versus human research ideas. arXiv preprint arXiv:2506.20803. Cited by: Introduction. C. Si, D. Yang, and T. Hashimoto (2024) Can llms generate novel research ideas? a large-scale human study with 100+ nlp researchers. External Links: 2409.04109, Link Cited by: Introduction, Introduction. S. Sinhahajari, N. Majumder, and S. Poria (2026) On the limits of llm-as-judge for scientific novelty assessment. arXiv preprint arXiv:2606.12071. Cited by: Introduction. H. Su, R. Chen, S. Tang, Z. Yin, X. Zheng, J. Li, B. Qi, Q. Wu, H. Li, W. Ouyang, P. Torr, B. Zhou, and N. Dong (2025) Many heads are better than one: improved scientific idea generation by a llm-based multi-agent system. External Links: 2410.09403, Link Cited by: LLM-as-Judge in Scientific Idea Generation.. P. Team (2024) Cited by: Introduction, Stage 2: Fixed-Domain Background.. P. Wang, L. Li, L. Chen, Z. Cai, D. Zhu, B. Lin, Y. Cao, L. Kong, Q. Liu, T. Liu, et al. (2024a) Large language models are not fair evaluators. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 9440–9450. Cited by: Introduction. Q. Wang, D. Downey, H. Ji, and T. Hope (2024b) SciMON: scientific inspiration machines optimized for novelty. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Bangkok, Thailand, p. 279–299. External Links: Document Cited by: Introduction. W. Wang, L. Gu, L. Zhang, Y. Luo, Y. Dai, C. Shen, L. Xie, B. Lin, X. He, and J. Ye (2024c) SciPIP: an llm-based scientific paper idea proposer. arXiv preprint arXiv:2410.23166. Cited by: Introduction. Y. Wang (2026) FirstResearch: auditable question formation for llm scientific discovery agents. arXiv preprint arXiv:2607.05682. Cited by: LLM-as-Judge in Scientific Idea Generation.. M. Wu and A. F. Aji (2025) Style over substance: evaluation biases for large language models. In Proceedings of the 31st International Conference on Computational Linguistics, p. 297–312. Cited by: Stylistic Bias in LLM-as-Judge.. G. Xiong, E. Xie, A. H. Shariatmadari, S. Guo, S. Bekiranov, and A. Zhang (2024) Improving scientific hypothesis generation with knowledge grounded large language models. arXiv preprint arXiv:2411.02382. Cited by: Introduction. H. Yang, R. Bao, C. D. Xiao, J. Ma, P. Bhatia, S. Gao, and T. Kass-Hout (2026a) Any large language model can be a reliable judge: debiasing with a reasoning-based bias detector. Advances in Neural Information Processing Systems 38, p. 6318–6362. Cited by: Stylistic Bias in LLM-as-Judge.. J. Yang, F. Ji, Z. Lai, Z. Cui, G. Ouyang, Q. Jiang, F. Zhang, M. Peng, Q. Xie, P. Nakov, et al. (2026b) LabGuard: grounding natural-language laboratory rules into runtime guards for embodied laboratory agents. arXiv preprint arXiv:2606.31045. Cited by: Introduction. B. Ye, S. Chen, J. Tu, C. Liu, Z. Xiong, S. Schmidgall, and D. S. Bitterman (2026) Proof of time: a benchmark for evaluating scientific idea judgments. arXiv preprint arXiv:2601.07606. Cited by: Introduction, LLM-as-Judge in Scientific Idea Generation.. Z. Zeng, J. Yu, T. Gao, Y. Meng, T. Goyal, and D. Chen (2024) Evaluating large language models at evaluating instruction following. In International Conference on Learning Representations, Vol. 2024, p. 40193–40219. Cited by: Stylistic Bias in LLM-as-Judge.. L. Zheng, W. Chiang, Y. Sheng, S. Zhuang, Z. Wu, Y. Zhuang, Z. Lin, Z. Li, D. Li, E. Xing, et al. (2023) Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in neural information processing systems 36, p. 46595–46623. Cited by: Introduction. H. Zhou, H. Huang, Y. Long, B. Xu, C. Zhu, H. Cao, M. Yang, and T. Zhao (2024) Mitigating the bias of large language model evaluation. In Proceedings of the 23rd Chinese National Conference on Computational Linguistics (Volume 1: Main Conference), p. 1310–1319. Cited by: Stylistic Bias in LLM-as-Judge.. H. Zhou, H. Huang, R. Zhang, K. Chen, B. Xu, C. Zhu, T. Zhao, and M. Yang (2026) Toward robust llm-based judges: taxonomic bias evaluation and debiasing optimization. arXiv preprint arXiv:2603.08091. Cited by: Stylistic Bias in LLM-as-Judge..