Paper deep dive
Knowing the Rules, Applying the Rules: Evaluating Language Models on Traditional Chinese Bazi
Jiulin Li, Ping Huang
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Knowing domain rules does not guarantee applying them to a case. We study this distinction in traditional Chinese Bazi through 3,000 Chinese multiple-choice questions spanning 14 Theory and 11 Case categories. Six endpoint systems are evaluated, with primary results reported on a 2,492-item model-informed refinement. Theory accuracy exceeds Case accuracy for every system, and gaps of 16.60-29.56 percentage points remain when invalid responses are excluded. The contrast is more specific than a general case-reasoning deficit. Across six systems, Twelve Stages and Nayin reach mean accuracies of 89.10% and 88.62%, while Shensha Basics reaches 75.96%. Within Case, Luck Pillars averages 84.62%, but Career and Family Relations average only 36.98% and 38.19%. Overall rankings also conceal different category strengths. On the original 3,000 items, paired DeepSeek native/disabled comparisons associate native configurations with Theory gains of 6.53 and 12.20 points for Flash and Pro, respectively; Case changes are -3.67 and +1.27 points. These are provider-configuration associations, not isolated causal effects of reasoning. The results motivate task-specific evaluation of cultural-domain applications rather than reliance on aggregate knowledge scores. The benchmark measures agreement with a model-generated, model-verified answer key, not real-world predictive validity. Final-set results are post-selection descriptions, and incomplete provenance and expert validation constrain their interpretation.
Tags
Links
- Source: https://arxiv.org/abs/2610.05682v1
- Canonical: https://arxiv.org/abs/2610.05682v1
Trouble viewing inline? Open PDF directly ā
Full Text
72,838 characters extracted from source content.
Expand or collapse full text
Knowing the Rules, Applying the Rules: Evaluating Language Models on Traditional Chinese Bazi Jiulin Li 1,* Ping Huang 2 1 Beijing Liuyi Guanhua Technology Co., Ltd. 2 State Key Laboratory of General Artificial Intelligence, BIGAI *Corresponding author:lijiulin@18trees.com September 2026 Abstract Knowing domain rules does not guarantee applying them to a case. We study this distinction in traditional Chinese Bazi through 3,000 Chinese multiple-choice questions spanning 14 Theory and 11 Case categories. Six endpoint systems are evaluated, with primary results reported on a 2,492-item model-informed refinement. Theory accuracy exceeds Case accuracy for every system, and gaps of 16.60ā29.56 percent- age points remain when invalid responses are excluded. The contrast is more specific than a general case-reasoning deficit. Across six systems, Twelve Stages and Nayin reach mean accuracies of 89.10% and 88.62%, while Shensha Basics reaches 75.96%. Within Case, Luck Pillars averages 84.62%, but Career and Family Relations average only 36.98% and 38.19%. Overall rankings also conceal different category strengths. On the original 3,000 items, paired DeepSeek native/disabled comparisons associate native configurations with Theory gains of 6.53 and 12.20 points for Flash and Pro, respectively; Case changes are -3.67 and +1.27 points. These are provider-configuration associations, not isolated causal effects of reasoning. The results motivate task-specific eval- uation of cultural-domain applications rather than reliance on aggregate knowledge scores. The benchmark measures agreement with a model-generated, model-verified answer key, not real-world predictive validity. Final-set results are post-selection descriptions, and incomplete provenance and expert validation constrain their interpretation. 1 Introduction Can a language model that knows a domainās vocabulary and rules use them reliably when several rules meet in a case? This distinction matters for systems that retrieve, explain, or apply specialized cultural knowledge. A strong overall score can conceal a weaker ability to answer contextual questions, while an undifferentiated reasoning label can hide tasks that remain comparatively easy. Traditional Chinese Bazi, or Four Pillars, provides a struc- turedsettingforevaluatingculturallyspecific knowledge. We use it as a cultural knowledge representation, not as an em- pirically validated forecasting system. Our benchmark pairs Theory questions about concepts and rule relations with Case questions that supply chart context and request an interpreta- tion. Figure 1summarizes the task families, category cov- Theory knowledge Case reasoning Concepts and rule relations 14 categories Chart-conditioned interpretation 11 categories Item distribution Original Final 1,5001,500 1,4541,038 3,000 2,492 Rule question Explicit rule or relation Four options: A / B / C / D Chart + interpretation Four pillars + question Four options: A / B / C / D Figure 1:Benchmark overview. Theory covers concepts and rule relations across 14 categories; Case covers chart-conditioned interpretation across 11 categories. Shared-scale bars show task composition in the original 3,000-item benchmark and final 2,492-item set. The miniature schematics illustrate the two question formats. erage, and original and final item distributions. This organi- zation supports task-specific comparisons of rule knowledge and contextual answers, while the two question lists do not isolate cognitive mechanisms. Our central finding is a selective, rather than uniform, ap- plication gap. Five of six systems exceed 90% on Twelve Stages and Nayin, yet all six remain below 45% on Career and Family Relations cases. Conversely, Luck Pillars cases score 78.21ā89.74%. Thus neither uniform Bazi competence nor a universal inability to solve cases describes the observed profile. The practical question is which parts of a domain an aggregate score represents. We make three contributions: 1.Asource-derivedbenchmarkspanning25categories, with separate Theory and Case tasks, an original 3,000-item release, and a documented 2,492-item evaluation subset. 2.A six-system analysis of task and category strengths, Case question subtypes, and invalid-response sensitivity, expos- ing limitations hidden by overall rankings. 3.Paired DeepSeek native/disabled configuration compar- isons on the original set, showing that Theory gains do not imply comparable Case gains. 1 arXiv:2610.05682v1 [cs.CL] 5 Oct 2026 Classical + in- structional sources + extraction Question creation + checking Original benchmark 3,000 items Model evaluation + model-informed refinement Final benchmark 2,492 items Chinese originals ā¢ English translations by the authors Refinement uses evaluated-model performance; the final set is not an independent holdout. Theory example Case example mq4_000001me4_000001 The WuāWei combination strengthens Wei Earth. Which relationship among the Five Elements does this illustrate? AMutual restraintBMutual punish- ment CMutual generationDMutual clash Case information: male Year Ding-Si Month Ren-Yin Day Xin-You Hour Yi-Wei Given the natal chart, what is this personās most likely career status? ABusinesspersonBOļ¬icialCArtist DOrdinary citizen Dataset key: CDataset key: D Figure 2:Construction pipeline and representative examples. Classical and instructional texts support extraction, question creation, and model-based checking. Evaluation of the original 3,000 items informs refinement to 2,492 retained items. The examples show Chinese questions with author English translations; keys are dataset labels. Refinement uses evaluated-model performance, so the final set is not an independent holdout. The answer key is model-generated and model-verified, and refinement used evaluated outcomes. These bound- aries remain explicit: our findings describe performance on this selected textual artifact, not verified expertise, a model- independent holdout, or the truth of Bazi predictions. Within that scope, the category profiles provide a more informative basis for evaluating domain applications than a single leader- board score. 2 Related Work Knowledge and cultural evaluation.MMLU, BIG-bench, TruthfulQA, and GPQA assess broad knowledge and chal- lenging QA [1,3,11,14]; CLUTRR targets relational in- ference [17]. CLUE, C-Eval, CMMLU, and GlobalBench broaden linguistic and cultural coverage [ 4,9,19,24]. The adjacent BaZi-Based Character Simulation Benchmark stud- ies temporal and persona reasoning [25]. Our contribution is a Theory/Case and category-level analysis within a four- choice task, not a first-Bazi-benchmark claim. Evaluation boundaries.Model-in-the-loop benchmarks and contrast sets motivate explicit selection histories [ 2,6, 13]; targeting and contamination studies motivate bounded score interpretation [5,12,15,16]. Our item diagnostics are panel conditional, not an item-response-theory calibration [ 7,8,21]. Chain-of-thought, self-consistency, test-time com- pute, and process-supervision research distinguish inference configurationfromanswercorrectness[10,18,20,22,23,26]. Accordingly, we report task-dependent configuration associ- ations without claiming an isolated reasoning mechanism. 3 Benchmark and Tasks What the benchmark asks.Each record has an identifier, a Chinese question or chart context, four options, an answer key, and task metadata. Category names organize content; they are not validated cognitive-skill labels. Figure 2traces the construction process from source extraction and question checking to model evaluation and refinement. Its represen- tative items make the task distinction concrete: the Theory item asks about a named rule relation, whereas the Case item suppliesafour-pillarchartbeforerequestinganinterpretation. Both use four-option answers. The examples retain their Chi- nese content and dataset keys, with English translations pro- vided by the authors. Source-linked construction.Classical and instructional texts are converted to passages and knowledge units, from which questions are generated and checked for schema and answer consistency. DeepSeek chat participates in extrac- tion, drafting, verification, and deduplication. Theory draws on 29 source works; 975/1,038 final Case items derive from four modern or instructional works. The public schema omits item-level source and school fields, and comprehensive ex- pert adjudication is not documented. We therefore describe the key as model-generated and model-verified. Appendix A supplies the full taxonomy and provenance limits. Model-informed refinement.After the original 3,000- item evaluation, item diļ¬iculty and discrimination informed category quotas and deletion priorities. The final set retains 2,492 items: 1,454 Theory and 1,038 Case. It removes 508 items (16.93%), comprising 477 answered correctly by all six native systems, 30 by five systems, and one by none. Four overlapping conflict/near-duplicate reasons do not add to that count. Retained identifiers, questions, options, and keys are unchanged, so final scores filter archived responses rather than rerun models. The procedure is explicitlymodel- 2 informed: it does not create an independent holdout. Final scores and intervals describe a selected list, with no demon- strated post-selection error guarantees. We retain original-set comparisons as the pre-selection reference. The change also increases the Theory share from 50% to 58.35%, motivating task and category breakdowns alongside overall scores. The original release, fixed-seed builder, and deletion manifestpre- serve the selection history. AppendixBgives the rules, be- fore/after distributions, and panel-conditional discrimination changes. 4 Evaluation Setup Systems and scoring.We evaluate Kimi-K3, GLM-5.3, Qwen3.8-Max, MiniMax-M3, DeepSeek-V4.1-Flash, and DeepSeek-V4-Pro-0813. We call themendpoint systems: four use mixed API routes and immutable model snapshots areunavailable. Eachitemreceivesazero-shotrequestattem- perature zero, with no tools, retrieval, demonstrations, or ex- plicit chain-of-thought request. The visible prompt requests one answer letter. Accuracy counts invalid or empty canon- ical answers as wrong; the four-choice random baseline is 25%. Comparisons.Main results use the final 2,492 items. Cate- gory means average six endpoint accuracies, computed from correct counts and denominators; category ranks are descrip- tive, not significance tests. Nominal Wilson intervals and paired McNemar tests describe the observed lists. Holm cor- rection covers 15 original-set model pairs. Source/chart clus- tering remains unmodeled, and final-set inference addition- ally conditions on model-informed selection. DeepSeek configuration comparisons instead use the original 3,000 items. Native means no thinking-disable parameter was sent; the counterpart sendsthinking: "type":"disabled"with the same visible prompt. Six Overall/Theory/Case tests form a separate Holm family. Ap- pendix Crecordsparsing, retries, routes, tokenlimits, andsta- tistical definitions. No new evaluation runs were performed for this analysis. 5 Results and Analysis 5.1 Overall scores conceal a task gap Table1reports final-set accuracy. Scores range from 63.20% to 75.56%, with Kimi-K3, GLM-5.3, and Qwen3.8- Max forming the highest-scoring group. These are opera- tional endpoint results under invalid-as-wrong scoring, not provider-neutral capability rankings. On the original 3,000 items, Holm-correctedpairedtestsdetectdifferencesin11/15 pairs but not among the top three or between the DeepSeek aliases; non-rejection does not establish equivalence. Full pairwise and original-set results appear in Appendix D. Every system scores higher on Theory than Case. Ex- cluding MiniMax, Theory spans 83.56ā86.04%, while Case spans 55.11ā61.75%; MiniMax scores 71.25% and 51.93%, respectively. Figure 3shows that the same aggregate direc- tion holds across all six endpoints, despite different absolute levels. Table 1:Post-selection results on the final benchmark candidate (%). Accuracy includes invalid answers; intervals are nominal Wilson 95% descriptions. SystemOverall95% CI Theory Case Invalid Kimi-K375.56 73.84ā77.21 86.04 60.89 4.98 GLM-5.375.08 73.34ā76.74 84.59 61.75 0.24 Qwen3.8-Max 74.84 73.10ā76.50 84.59 61.18 5.98 DS V4.1 Flash 72.43 70.64ā74.15 83.56 56.84 0.00 DS V4 Pro72.35 70.56ā74.07 84.66 55.11 0.00 MiniMax-M3 63.20 61.29ā65.07 71.25 51.93 1.08 5060708090 Final-set accuracy (%) Kimi-K3 GLM-5.3 Qwen3.8-Max DS V4.1 Flash DS V4 Pro MiniMax-M3 60.986.0 61.884.6 61.284.6 56.883.6 55.184.7 51.971.2 CaseTheory Figure 3:Final-set Theory and Case accuracy. Every endpoint has a lower Case score; invalid-response sensitivity retains the gap. Completion failures amplify, but do not account for, this contrast. Of 306 final-set invalid responses, 258 occur on Case items. Excluding invalids separately within each task leaves gaps of 16.60ā29.56 points. Kimi and Qwen then reach 68.77% and 69.47% on Case, still below their 86.34% and 86.07% Theory scores. This is a sensitivity analysis, not a replacement leaderboard: it conditions on successful com- pletion. Task composition also matters. Refinement increases the weight of the higher-scoring Theory task. Equal-weight task- macro scores are therefore 1.61ā2.47 points below micro scores; category-macro scores are 1.38ā2.20 points lower, withunchangedordering. Anapplication-orientedevaluation should report its task mix alongside any aggregate score. 5.2 Theory: unevencommandofdomainrules Figure4shows endpoint accuracies for all 25 categories in an annotated heatmap; the final column summarizes the six- model mean. Cells display one decimal place; the text retains two. Twelve Stages and Nayin are the highest-scoring The- ory categories, averaging 89.10% and 88.62%. Five systems exceed 90% on both; MiniMax scores 69.23% and 70.19%. Thus the panel-level strength is substantial but not universal. Annual and Luck Cycles also scores comparatively well at 85.62%. The lower-scoring Theory categories are Shensha Basics (75.96%), Pattern Basics (78.04%), and Strength and Weak- ness (79.01%). The 13.14-point difference between Twelve Stages and Shensha Basics shows why high Theory accu- racy should not be read as uniformly reliable rule knowledge. Even within the stronger task family, category-level checking remains necessary. 3 Kimi-K3GLM-5.3 Qwen3.8- Max DS V4.1 Flash DS V4 Pro MiniMax-M3Mean Twelve Stages (104) Nayin (104) Annual and Luck Cycles (102) Ten Gods (104) Combination and Clash (104) Ten Gods Relations (104) Kongwang (Void) (104) Useful and Adverse Gods (104) Five Elements (104) Six Relations (104) Stems and Branches (104) Strength and Weakness (104) Pattern Basics (104) Shensha Basics (104) 94.293.392.392.393.369.289.1 93.391.390.493.393.370.288.6 88.287.386.386.388.277.585.6 86.586.589.484.686.577.985.3 85.686.588.582.784.679.884.6 86.581.785.687.583.769.282.4 84.686.582.779.885.675.082.4 82.782.785.683.780.874.081.6 86.580.885.682.784.668.381.4 83.781.781.783.780.870.280.3 79.887.576.080.889.467.380.1 87.581.780.876.080.867.379.0 85.678.883.779.876.064.478.0 79.877.976.076.977.967.376.0 64.480.094.2 Kimi-K3GLM-5.3 Qwen3.8- Max DS V4.1 Flash DS V4 Pro MiniMax-M3Mean Luck Pillars (78) Personality (96) Misfortune and Legal (96) Year Timing (96) Chart Construction (96) Wealth (96) Health (96) Education (96) Romance (96) Family Relations (96) Career (96) 87.285.989.780.885.978.284.6 77.181.282.376.078.159.475.7 76.080.275.071.966.767.772.9 71.974.068.870.864.665.669.3 61.564.665.660.466.756.262.5 60.462.558.352.150.052.155.9 61.557.356.254.250.045.854.2 49.050.061.547.944.843.849.5 47.944.845.837.534.445.842.7 40.638.538.542.735.433.338.2 41.744.836.535.435.428.137.0 28.160.089.7 a) Theory Theory (%) b) Case Case (%) Figure 4:Final-set category accuracy (%), shown to one decimal place. The outlined Mean column reports the six-model mean computed from unrounded accuracies. Categories are sorted by six-model mean; numbers in parentheses indicate the number of items evaluated per model. These are descriptive post-selection results. Model differences further complicate an overall ranking. DeepSeek Pro reaches 89.42% on Stems and Branches, com- pared with Qwenās 75.96%, whereas Qwen leads the panel on Ten Gods at 89.42%. Kimi reaches 87.50% on Strength and Weakness and 85.58% on Pattern Basics. These are ob- served category point estimates, not evidence of statistically established specialization or a stable model-selection rule. Nor do Theory labels cleanly divide recall from reason- ing. The existing question-format tagreasoningyields scores of 75.35ā91.67%, whilejudgementyields 66.90ā83.62%. A label alone does not establish reasoning depth. Differences can reflect item content, source conventions, answer-key con- struction, or format; this benchmark does not isolate those mechanisms. 5.3 Case: selective rather than universal diļ¬i- culty Case performance varies more sharply. Luck Pillars aver- ages 84.62%, followed by Personality at 75.69% and Mis- fortune and Legal at 72.92%. In contrast, Career averages 36.98%, Family Relations 38.19%, and Romance 42.71%. All six systems remain below 45% on Career and Family Re- lations, whereas all exceed 78% on Luck Pillars. The range between the highest- and lowest-scoring Case categories is 47.64 points. This pattern rules out a simple summary that contextual questions are uniformly diļ¬icult. It also requires care in nam- ingthestrength: theLuckPillarscategoryincludesinterpreta- tions of luck periods, not only mechanical chart calculations. High scores there do not demonstrate a separately measured Table 2:Accuracy by existing Case label (%). āFactā and ātheoryā are dataset tags, not verified outcomes. Gap = theory minus fact, using unrounded values. System Fact (n= 464) Theory (n= 574) Gap (p) Kimi-K355.6065.16 +9.55 GLM-5.356.0366.38 +10.34 Qwen3.8-Max56.4764.98 +8.52 DS V4.1 Flash51.2961.32 +10.03 DS V4 Pro45.9162.54 +16.64 MiniMax-M345.0457.49 +12.45 calendrical algorithm, just as low Career scores do not iden- tify a particular failed reasoning step. There is no single category winner. Qwen reaches 61.46% onEducation, comparedwith43.75ā50.00%fortheothersys- tems, while Flash has the highest Family Relations point esti- mate at 42.71%. That latter distinction is modest in practical terms because the entire panel performs poorly on the cate- gory. The useful application-facing conclusion is the profile of limits, not the name of a winner. 5.4 Case subtypes and shared failures The Case metadata distinguishes questions taggedfactfrom those taggedtheory. We describe them asrecorded-outcome andrule-interpretationquestions: the former refer to events represented in the source case, whereas the latter ask for a domain-rule interpretation. These are dataset labels, not in- dependently verified facts or validated measures of two cog- nitive abilities. 4 Table 3:DeepSeek native versus disabled provider-configuration contrast on the original 3,000 items. Delta is native minus disabled accuracy in percentage points; Holm correction covers the six displayed tests. Native Flash records used 32,768 and later 65,536 maximum output tokens; native Pro used 65,536 throughout. System Subset Native DisabledāHolmp FlashOverall 76.9775.53 +1.430.1795 FlashTheory 84.0077.47 +6.531.66Ć10 ā8 FlashCase69.9373.60-3.670.00935 ProOverall 76.8370.10 +6.733.26Ć10 ā13 ProTheory 85.0772.87 +12.204.92Ć10 ā24 ProCase68.6067.33 +1.270.3663 All six systems score lower on the 464 recorded-outcome items than on the 574 rule-interpretation items (Table2). The gaps, computed before rounding, range from 8.52 to 16.64 points. Even a model that handles source-domain rules rela- tively well may not recover a caseās reported outcome. How- ever, the two subsets differ in content and category compo- sition; the association is not a controlled test of whether out- come questions themselves cause errors. Agreement statistics reveal another limitation of averages. The final set contains 1,118 items answered correctly by all six systems and 204 answered correctly by none: 44.86% and 8.19% of the release. Case has fewer unanimous-correct items (25.3%) and more unanimous-wrong items (13.2%) than Theory (58.8% and 4.6%). Career and Family Re- lations each have no unanimous-correct items and 21.88% unanimous-wrong items. Agreement with a shared wrong an- swer is not required by this statistic; it counts failure to match the key. These are candidate strata for expert review, not a causal error taxonomy. Without adjudicated answers and coded ex- planations, we cannot distinguish missing knowledge, com- peting school conventions, ambiguous questions, or flawed keys from answer vectors alone. For a domain application, separate validation of rule explanation, contextual interpreta- tion, and output completion is more informative than assum- ing that one high score transfers to all three. 6 Reasoning Configuration Analysis 6.1 Theory gains do not imply Case gains Table3compares DeepSeek native and disabled configura- tions on the original 3,000 items, before selection using na- tive outcomes. Flashās overall difference is +1.43 points and is not detected after Holm correction; Proās is +6.73 points. The aggregate changes conceal different task associations: native Flash gains 6.53 points on Theory but loses 3.67 on Case, whereas native Pro gains 12.20 on Theory and only 1.27 on Case. Proās Case difference is not detected after cor- rection. Both Theory contrasts are positive and detected, yet nei- ther system shows a correspondingly detected positive Case difference. Additional native reasoning behavior is therefore not a uniform remedy for the observed application gap. This result supports reporting task-specific configuration compar- isons instead of interpreting a total-score gain as improve- ment across a domain. The paired records also change answers in both directions. Flash matches the key only in the native configuration on 328 items but only in the disabled configuration on 285; Pro has 465 versus 263. These transitions explain the net gains with- out identifying why a particular answer changed. No individ- ual failure mechanism is inferred from them. 6.2 Whattheconfigurationcontrastcanestab- lish The visible prompt is shared, but the requests are not an isolated intervention on a reasoning mechanism. Native in- put counts contain fixed offsets of 26 tokens per Flash item and 79 per Pro item. Their contents are not exposed in the canonical table. Native Flashās maximum output limit changed from 32,768 to 65,536 during execution, and im- mutable provider snapshots are absent. We therefore report provider-configuration associations, not causal reasoning ef- fects. Comparing native Flash with native Pro would instead be a cross-system comparison. Operational costs are also asymmetric. On the original set, median native/disabled request latency is 10.68/1.43 seconds for Flash and 20.95/2.07 seconds for Pro. Those durations include network and queueing as well as provider execution; configurations ran at different times. They are not compute- only estimates or monetary costs. Appendix D.4retains the token, latency, and route evidence. For application design, these observations motivate a bounded evaluation strategy: compare configurations on the actual task categories and check whether additional latency accompanies useful answer improvements. This study does not establish an optimal configuration policy. It shows why assuming that a reasoning-enabled configuration helps every kind of cultural-domain question is not supported by the ob- served results. 7 Limitations Representation and validation.The model-generated, model-verified key lacks comprehensive expert adjudication. Contested school conventions, incomplete public item prove- nance, concentrated Case sources, and unknown contamina- tion limit interpretation. Scores measure agreement with this representation, not metaphysical truth, predictive validity, or general cultural competence. Applications involving conse- quential real-world decisions are outside the supported scope. Selection and statistical scope.Final items were selected using the evaluated panel, not held out independently. Cate- gories and Case tags are observational groupings; source and chart reuse remain unmodeled. Point estimates and nominal intervals do not establish population-level category effects. Shared generation/evaluation model families and single runs at temperature zero add uncertainty. Endpoint limitations.Mixed routes, missing immutable snapshots, and configuration offsets restrict capability and ef- ficiency comparisons. 5 8 Conclusion Bazi evaluation reveals a more specific picture than a sin- gle knowledge or reasoning score. Models score well on some Theory categories but unevenly on others; Case perfor- mance ranges from strong Luck Pillars scores to shared weak- nesses in Career and Family Relations. Paired DeepSeek configuration changes improve Theory more consistently than Case. These findings favor task-specific validation for cultural-domain applications. The benchmark supplies a doc- umented setting for that analysis, while model-informed se- lection and answer-key limitations keep the conclusions con- ditional on the evaluated artifact. Disclosures Resources and data availability.BaZi2500 is publicly available onHugging Faceunder C BY-NC 4.0. Evalua- tion code and reproducibility resources are available in the GitHubrepository; benchmarkresultsanddocumentationare provided on theproject page. The authors have confirmed source redistribution rights and completed privacy review. This textual study recruited no participants. Scores must not support consequential real-world decisions (AppendixE). Funding and competing interests.This work was sup- ported by Beijing Liuyi Guanhua Technology Co., Ltd. The authors declare no competing interests. AIuse.Modelsassistedbenchmarkconstructionandprepa- ration of this manuscript. They did not provide independent expert validation. Numerical results derive from checked analysis artifacts. Implications for Application The results suggest a practical evaluation rule for systems that work with culturally specific knowledge: test retrieval of named rules separately from their use in a structured situ- ation. A system can be highly reliable on a definition or rela- tion while failing when several chart fields must be combined before selecting an answer. In this benchmark, Twelve Stages and Nayin are the clearest Theory strengths, whereas Shen- sha Basics, Pattern Basics, and Strength and Weakness are less uniform. The Case profile is sharper still: Luck Pillars is comparatively strong, but Career, Family Relations, and Ro- mance remain diļ¬icult across the evaluated systems. These are descriptive diagnostics for the released answer key, not evidence that any category is a valid real-world predictor. For an application designer, a single aggregate score should therefore be treated as a routing signal rather than a readiness certificate. A useful evaluation dashboard should expose task family, category, invalid-response rate, and whether the prompt requests a factual relation or an interpre- tation. The native/disabled comparisons reinforce this point: additional provider-side reasoning behavior was associated with larger Theory gains than Case gains, but the configura- tions also differed in token limits, routes, and execution time. The benchmark can reveal that mismatch and motivate tar- geted tests; it cannot identify a universal reasoning mecha- nism or prescribe a production setting. Finally, the final list is a model-informed refinement of the original 3,000 items. It is useful for describing the re- tained panel, but it is not an independent holdout. Applica- tions should report which release, prompt, answer key, and endpoint configuration they use, preserve invalid outputs for audit, and avoid treating a correct multiple-choice label as expert certification. What the benchmark can diagnose.The category pro- file is most useful as a prioritization device. High scores on Twelve Stages or Nayin indicate that the model can often map a named concept to one of four released labels. They do not show that it can reconcile competing relations, iden- tify which chart fields are relevant, or explain an answer in a new context. Conversely, low scores on Career or Family Relations identify a stress test for chart-conditioned interpre- tation, buttheydonotbythemselvesseparatemissingdomain knowledge from ambiguity in the key. Those hypotheses re- quire item-level adjudication and controlled prompt variants, which are outside this release. Responsible use. The benchmark is appropriate for com- paring text systems, monitoring regressions, and selecting cases for expert review. It is not a certification test for prac- titioners, a basis for advice about a person, or evidence that a model has learned a metaphysical system in a historically complete sense. The answer key is a property of this re- lease. Any downstream application should expose that status to users, retain the underlying question and answer trace, and route high-stakes or contested interpretations to qualified hu- man review. These practices turn the observed TheoryāCase mismatch into an engineering signal without converting a de- scriptive benchmark result into a claim about people or the world. Reading a benchmark score.A score measures released- key agreement, not metaphysical truth, forecasting validity, or expert certification. Theory emphasizes rule knowledge; Case adds chart-conditioned interpretation, but these are not paired tests isolating cognitive mechanisms. Invalid re- sponses count as wrong; excluding them conditions accuracy on completion. The 2,492 retained items were selected from theoriginal3,000usingmodeloutcomes. Final-setscoresare therefore post-selection descriptions; the original list remains the pre-selection reference. Minimum reporting record.Bind Theory/Case scores and category denominators to the release, key version, item IDs, prompt, parser, retry rule, and endpoint configura- tion. Include invalid and empty-response rates, token lim- its, run dates, and response traces showing how retries be- come scored records. Configuration contrasts also need a stated difference direction and correction family. This record makes scores auditable; an aggregate alone cannot replace task-specific diagnostics. 6 References [1]BIG-bench authors. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models.Transactions on Machine Learning Research, 2023. doi: 10.48550/arXiv.2206.04615. URLhttps://arxiv.org/abs/2206.04615. [2]Matt Gardner, Yoav Artzi, Victoria Basmov, Jonathan Berant, Ben Bogin, Sihao Chen, Pradeep Dasigi, Dheeru Dua, Yanai Elazar, Ananth Gottumukkala, Nitish Gupta, Hannaneh Hajishirzi, Gabriel Ilharco, Daniel Khashabi, Kevin Lin, Jiangming Liu, Nelson F. Liu, Phoebe Mulcaire, Qiang Ning, Sameer Singh, Noah A. Smith, Sanjay Subramanian, Reut Tsarfaty, Eric Wallace, Ally Zhang, and Ben Zhou. Evaluating modelsā local decision boundaries via contrast sets. In Trevor Cohn, Yulan He, and Yang Liu, editors, Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1307ā1323, Online, November 2020. Association for Computational Linguistics. doi: 10.18653/v1/2020.findings-emnlp.117. URL https://aclanthology.org/2020. findings-emnlp.117/. [3]Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. InInternational Conference on Learning Representations, 2021. doi: 10.48550/arXiv.2009.03300. URL https://arxiv.org/abs/2009.03300. [4]Yuzhen Huang, Yuzhuo Bai, Zhihao Zhu, Junlei Zhang, Jinghan Zhang, Tangjun Su, Junteng Liu, Chuancheng Lv, Yikai Zhang, Jiayi Lei, Yao Fu, Maosong Sun, and Junxian He. C-eval: A multi-level multi-discipline chinese evaluation suite for foundation models. InAdvances in Neural Information Processing Systems, 2023. doi: 10.48550/arXiv.2305.08322. URL https://arxiv.org/abs/2305.08322. [5]Alon Jacovi, Avi Caciularu, Omer Goldman, and Yoav Goldberg. Stop uploading test data in plain text: Practical strategies for mitigating data contamination by evaluation benchmarks. In Houda Bouamor, Juan Pino, and Kalika Bali, editors,Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 5075ā5084, Singapore, December 2023. Association for Computational Linguistics. doi: 10.18653/v1/2023.emnlp-main.308. URL https: //aclanthology.org/2023.emnlp-main.308/. [6]Douwe Kiela, Max Bartolo, Yixin Nie, Divyansh Kaushik, Atticus Geiger, Zhengxuan Wu, Bertie Vidgen, Grusha Prasad, Amanpreet Singh, Pratik Ringshia, Zhiyi Ma, Tristan Thrush, Sebastian Riedel, Zeerak Waseem, Pontus Stenetorp, Robin Jia, Mohit Bansal, Christopher Potts, and Adina Williams. Dynabench: Rethinking benchmarking in nlp. In Kristina Toutanova, Anna Rumshisky, Luke Zettlemoyer, Dilek Hakkani-Tur, Iz Beltagy, Steven Bethard, Ryan Cotterell, Tanmoy Chakraborty, and Yichao Zhou, editors,Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4110ā4124, Online, June 2021. Association for Computational Linguistics. doi: 10.18653/v1/2021.naacl-main.324. URLhttps: //aclanthology.org/2021.naacl-main.324/. [7]John P. Lalor, Hao Wu, and Hong Yu. Building an evaluation scale using item response theory. In Jian Su, Kevin Duh, and Xavier Carreras, editors,Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 648ā657, Austin, Texas, November 2016. Association for Computational Linguistics. doi: 10.18653/v1/D16-1062. URL https://aclanthology.org/D16-1062/. [8]John P. Lalor, Hao Wu, Tsendsuren Munkhdalai, and Hong Yu. Understanding deep learning performance through an examination of test set diļ¬iculty: A psychometric case study. In Ellen Riloff, David Chiang, Julia Hockenmaier, and Junāichi Tsujii, editors,Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing , pages 4711ā4716, Brussels, Belgium, October 2018. Association for Computational Linguistics. doi: 10.18653/v1/D18-1500. URL https://aclanthology.org/D18-1500/. [9]Haonan Li, Yixuan Zhang, Fajri Koto, Yifei Yang, Hai Zhao, Yeyun Gong, Nan Duan, and Timothy Baldwin. Cmmlu: Measuring massive multitask language understanding in chinese. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors,Findings of the Association for Computational Linguistics: ACL 2024, pages 11260ā11285, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10.18653/v1/2024.findings-acl.671. URL https: //aclanthology.org/2024.findings-acl.671/. [10]Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Letās verify step by step.arXiv preprint arXiv:2305.20050, 2023. doi: 10.48550/arXiv.2305.20050. URL https://arxiv.org/abs/2305.20050. [11]Stephanie Lin, Jacob Hilton, and Owain Evans. Truthfulqa: Measuring how models mimic human falsehoods. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio, editors,Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3214ā3252, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10.18653/v1/2022.acl-long.229. URL https: //aclanthology.org/2022.acl-long.229/. [12]Inbal Magar and Roy Schwartz. Data contamination: From memorization to exploitation. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio, editors,Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: 7 Short Papers), pages 157ā165, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10.18653/v1/2022.acl-short.18. URLhttps: //aclanthology.org/2022.acl-short.18/. [13]Yixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela. Adversarial nli: A new benchmark for natural language understanding. In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel Tetreault, editors,Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4885ā4901, Online, July 2020. Association for Computational Linguistics. doi: 10.18653/v1/2020.acl-main.441. URLhttps: //aclanthology.org/2020.acl-main.441/. [14]David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R. Bowman. Gpqa: A graduate-level google-proof q&a benchmark.arXiv preprint arXiv:2311.12022, 2023. doi: 10.48550/arXiv.2311.12022. URL https://arxiv.org/abs/2311.12022. [15]Oscar Sainz, Jon Campos, Iker GarcĆa-Ferrero, Julen Etxaniz, Oier Lopez de Lacalle, and Eneko Agirre. Nlp evaluation in trouble: On the need to measure llm data contamination for each benchmark. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Findings of the Association for Computational Linguistics: EMNLP 2023, pages 10776ā10787, Singapore, December 2023. Association for Computational Linguistics. doi: 10.18653/v1/2023.findings-emnlp.722. URL https://aclanthology.org/2023. findings-emnlp.722/ . [16]David Schlangen. Targeting the benchmark: On methodology in current natural language processing research. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli, editors,Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pages 670ā674, Online, August 2021. Association for Computational Linguistics. doi: 10.18653/v1/2021.acl-short.85. URL https: //aclanthology.org/2021.acl-short.85/. [17]Koustuv Sinha, Shagun Sodhani, Jin Dong, Joelle Pineau, and William L. Hamilton. Clutrr: A diagnostic benchmark for inductive reasoning from text. In Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan, editors,Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 4506ā4515, Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 10.18653/v1/D19-1458. URL https://aclanthology.org/D19-1458/. [18]Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. Scaling llm test-time compute optimally can be more effective than scaling model parameters. arXiv preprint arXiv:2408.03314, 2024. doi: 10.48550/arXiv.2408.03314. URL https://arxiv.org/abs/2408.03314. [19]Yueqi Song, Simran Khanuja, Pengfei Liu, Fahim Faisal, Alissa Ostapenko, Genta Winata, Alham Fikri Aji, Samuel Cahyawijaya, Yulia Tsvetkov, Antonios Anastasopoulos, and Graham Neubig. Globalbench: A benchmark for global progress in natural language processing. In Houda Bouamor, Juan Pino, and Kalika Bali, editors,Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 14157ā14171, Singapore, December 2023. Association for Computational Linguistics. doi: 10.18653/v1/2023.emnlp-main.875. URLhttps: //aclanthology.org/2023.emnlp-main.875/. [20]Mirac Suzgun, Nathan Scales, Nathanael SchƤrli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc Le, Ed Chi, Denny Zhou, and Jason Wei. Challenging big-bench tasks and whether chain-of-thought can solve them. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki, editors, Findings of the Association for Computational Linguistics: ACL 2023, pages 13003ā13051, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10.18653/v1/2023.findings-acl.824. URL https: //aclanthology.org/2023.findings-acl.824/. [21]Clara Vania, Phu Mon Htut, William Huang, Dhara Mungra, Richard Yuanzhe Pang, Jason Phang, Haokun Liu, Kyunghyun Cho, and Samuel R. Bowman. Comparing test sets with item response theory. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli, editors,Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 1141ā1158, Online, August 2021. Association for Computational Linguistics. doi: 10.18653/v1/2021.acl-long.92. URL https: //aclanthology.org/2021.acl-long.92/. [22]Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V. Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models.International Conference on Learning Representations, 2023. doi: 10.48550/arXiv.2203.11171. URL https://arxiv.org/abs/2203.11171. [23]Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc V. Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models.arXiv preprint arXiv:2201.11903, 2022. doi: 10.48550/arXiv.2201.11903. URL https://arxiv.org/abs/2201.11903. [24]Liang Xu, Hai Hu, Xuanwei Zhang, Lu Li, Chenjie Cao, Yudong Li, Yechen Xu, Kai Sun, Dian Yu, Cong Yu, Yin Tian, Qianqian Dong, Weitang Liu, Bo Shi, 8 Yiming Cui, Junyi Li, Jun Zeng, Rongzhao Wang, Weijian Xie, Yanting Li, Yina Patterson, Zuoyu Tian, Yiwen Zhang, He Zhou, Shaoweihua Liu, Zhe Zhao, Qipeng Zhao, Cong Yue, Xinrui Zhang, Zhengliang Yang, Kyle Richardson, and Zhenzhong Lan. Clue: A chinese language understanding evaluation benchmark. In Donia Scott, Nuria Bel, and Chengqing Zong, editors,Proceedings of the 28th International Conference on Computational Linguistics, pages 4762ā4772, Barcelona, Spain (Online), December 2020. International Committee on Computational Linguistics. doi: 10.18653/v1/2020.coling-main.419. URL https: //aclanthology.org/2020.coling-main.419/ . [25]Siyuan Zheng, Pai Liu, Xi Chen, Jizheng Dong, and Sihan Jia. Bazi-based character simulation benchmark: Evaluating ai on temporal and persona reasoning. arXiv preprint arXiv:2510.23337, 2025. doi: 10.48550/arXiv.2510.23337. URL https://arxiv.org/abs/2510.23337. WordPlay Workshop 2025. [26]Shu Zhou, Rui Ling, Junan Chen, Xin Wang, Tao Fan, and Hao Wang. When more thinking hurts: Overthinking in llm test-time compute scaling. In Maria Liakata, Viviane P. Moreira, Jiajun Zhang, and David Jurgens, editors,Findings of the Association for Computational Linguistics: ACL 2026, pages 23967ā23977, San Diego, California, United States, July 2026. Association for Computational Linguistics. doi: 10.18653/v1/2026.findings-acl.1199. URL https://aclanthology.org/2026. findings-acl.1199/. 9 Table 4:Dataset statistics for the original and final releases. StatisticOriginalFinal Total items3,0002,492 Theory items1,5001,454 Case items1,5001,038 Theory categories1414 Case categories1111 Options per item44 Removed itemsā 508 (16.93%) A Dataset and Provenance Table4summarizes the original 3,000-item benchmark and its final 2,492-item subset. Table5gives the final English category names used in the figures. Theoryās āSix Relationsā tests conceptual kinship relations, whereas Caseās āFamily Relationsā asks for case interpretation. Category-level analy- ses are descriptive unless otherwise marked; no family-wise hypothesis test is claimed for the heatmaps. A.1 Construction audit The original benchmark evaluated here contains 3,000 multiple-choice items: 1,500 Theory questions and 1,500 Case questions. Its construction process converts books into chapter text, extracts fine-grained source passages, drafts questions with provenance, and checks schema and answer uniqueness. The pipeline requires source evidence for an ad- mitted item and records links between an item and its source paragraph or knowledge unit. A language model (DeepSeek chat) participates in gating, extraction, drafting, verification, and deduplication. The evidence audit found no comprehen- sive expert-human validation pass for all gold answers. We therefore use āmodel-generated and model-verifiedā rather than āexpert-annotatedā as the operative description. The present paper audits and evaluates this existing 3,000- item benchmark and its deletion-only refinement. It does not claim that the paper package alone can regenerate the histor- ical artifact from raw books. Historical prompts and registry versions are not complete in the available evidence. The source mix is heterogeneous. The Theory portion spans 29 source works, including classical texts. In the fi- nal Case portion, 975 of 1,038 items (93.93%) come from four modern or instructional works. The public JSONL schema strips per-item source and school fields, so a user of the release can reproduce the task and metadata but can- not reconstruct the full item-to-source map from the release alone. We therefore describe the resource as source-linked during construction and source-derived with incomplete pub- licprovenance, ratherthanclaimingthateveryitemisdirectly grounded in a classical passage. A.2 Category coverage and schema The taxonomy contains 14 Theory categories and 11 Case categories. Theory categories cover Five Elements, Stems and Branches, TenGods, relations, combinations and clashes, strength, patterns, useful and adverse gods, auxiliary signs, Nayin, void periods, Twelve Stages, annual/luck cycles, and family relations. Case categories cover career, education, wealth, romance, health, personality, year timing, chart con- struction, luckpillars, misfortune/legalmatters, andfamilyre- lations. The final release contains 104 items in each Theory category except Annual and Luck Cycles (102 after forced defect removal), and 96 items in ten Case categories; Luck Pillars contains all 78 available items. No text is edited during release hardening. The original identifiers are retained, and the clean release is a strict subset of the original 3,000 records. This permits direct filtering of the original model responses rather than a second, potentially non-comparable model run. Thepublicreleasehasnear-balancedgoldanswerpositions (A: 646, B: 607, C: 624, D: 615), and each native run has 2,492uniqueidentifiers, zeromissingrecords, andzerocross- model metadata disagreements. These structural checks do not validate gold-answer correctness. B Hardening Audit B.1 Selection rule and retained release Hardening follows full evaluation of the original 3,000 items. Diļ¬iculty is one minus the fraction of six native systems an- swering correctly; discrimination combines item-rest corre- lation and panel response spread (AppendixC.2). Both diag- nostics are conditional on this small model panel. After removing confirmed conflicts and near duplicates, the script targets 104 items per Theory category and 96 per Case category, retaining underfilled categories. Within an overfull category, deletion priority is six-of-six correct, five-of-six, one-of-six, two-to-four, then zero-of-six. Seed 20260922 breaks ties within bands. Near-duplicate retention also uses model-derived discrimination. The original release, fixed-seed script, and deletion manifest preserve this process. The final release removes 508 unique items (16.93%). The mutually exclusive removal-band distribution is 477 six-of- six correct, 30 five-of-six correct, and one zero-of-six cor- rect. The same manifest includes four forced data-quality re- movals: two records with an identical stem but conflicting gold labels and two near-duplicate variants of a retained item. Those reason labels are embedded in the performance-band counts and must not be added as disjoint counts. Unanimous- correct items provide no discordant evidence for any model pairās McNemar test, motivating their priority. Because other bands are also deleted, we do not claim that every deletion is inferentially neutral. Twocategoryexceptionsremainafterquotabalancing. An- nual and Luck Cycles has 102 Theory items because two con- firmed defective records were removed, and Luck Pillars has 78 Case items because the original category contained only 78. Every other Theory category has 104 items and every other Case category has 96. B.2 Observed shifts and inference limits Table6reports an increase in mean empirical diļ¬iculty from 0.2326 to 0.2776 and in mean discrimination from 0.2182 to 0.2574; the observed number of Holm-significant pairs re- mains11/15. Filteringcachedresponseslowersallsixaccura- cies by 3.94ā5.87 percentage points (Table 8). Category quo- tas reduce large Case categories, changing the task mix from 50/50to58.35%Theoryand41.65%Case. Theremovalsum- 10 Table 5:Final taxonomy and category sizes. ID Theory categorynID Case categoryn T01 Five Elements104 C01 Career96 T02 Stems and Branches104 C02 Education96 T03 Ten Gods104 C03 Wealth96 T04 Ten Gods Relations104 C04 Romance96 T05 Combination and Clash104 C05 Health96 T06 Strength and Weakness104 C06 Personality96 T07 Pattern Basics104 C07 Year Timing96 T08 Useful and Adverse Gods104 C08 Chart Construction96 T09 Shensha Basics104 C09 Luck Pillars78 T10 Nayin104 C10 Misfortune and Legal96 T11 Kongwang (Void)104 C11 Family Relations96 T12 Twelve Stages104 T13 Annual and Luck Cycles102 T14 Six Relations104 Table 6:Observed changes under model-informed hardening. Final-set test counts are post-selection descriptions. MetricOriginal Final Items3,000 2,492 Mean empirical diļ¬iculty 0.2326 0.2776 Mean discrimination0.2182 0.2574 Six-model all-correct1,595 1,118 Six-model all-wrong205204 Holm-significant pairs11/15 11/15 Table 7:Actual deletion-band composition of the final hardening. Item bandRemoved Six of six correct477 Five of six correct30 Two to four correct0 One of six correct0 Zero of six correct1 Total508 mary in Table7uses mutually exclusive performance bands; overlapping data-quality reasons are described above. These changes establish a harder selected list for this panel, not greater validity or universal diļ¬iculty. Outcome- informed inclusion can retain panel-specific weaknesses and distort subsequent comparisons. Final-set intervals and tests have no demonstrated post-selection coverage or error con- trol; weusethemdescriptivelyandretainoriginal-setcompar- isons as the pre-selection reference. An independent future evaluation requires inclusion decisions made before observ- ing the evaluated systems. The hardening report also includes exploratory schemes A, B,andC.Thoseschemesaresimulations, notthefinalrelease. Separate threshold-sensitivity experiments showed that delet- ing five-of-six items could reduce detected pairs from 11/15 to 8/15 or 6/15, whereas the quota-based final subset retains 11/15 nominally detected pairs. Simulated sizes and scores are not used as final benchmark results. C Evaluation Protocol and Statistical Definitions The main and disabled-configuration runs use the following user-facing template, with the question and four options sub- Table 8:Observed endpoint-system accuracy before and after hardening. Original scores are recomputed from the outcome matrix used by hardening; final scores filter those responses by retained ID. ModelOriginal Final Change Kimi-K379.50 75.56-3.94 GLM-5.379.13 75.08-4.05 Qwen3.8-Max78.97 74.84-4.13 DS V4.1 Flash76.97 72.43-4.54 DS V4 Pro76.83 72.35-4.48 MiniMax-M369.07 63.20-5.87 stituted for each independent request: You are answering a multiple-choice question about traditional Chinese Bazi theory. Select the single correct answer. Output only one letter: A, B, C, or D. Do not provide any explanation or additional text. Question: question A. option_A B. option_B C. option_C D. option_D Table 9:Run configuration. āNativeā means no thinking-disable parameter was sent. Run family ModelsReasoning Source V1 native Six listed models NativeMixed routes V2 FlashDS V4.1 FlashDisabled DeepSeek oļ¬icial V2 ProDS V4 ProDisabled DeepSeek oļ¬icial All runs use temperature zero, no tools, no retrieval, no demonstrations, and one independent request per item. Orig- inal runs cover 3,000 items per system; the final table filters these to 2,492 identifiers. Disabled runs allow 65,536 output tokens and send this parameter: thinking: "type":"disabled" Native Flash began with a 32,768-token limit and later used 65,536; native Pro used 65,536. No canonical response has a lengthfinish reason. Provider model names and API iden- tifiers are retained when available, but not all systems have immutable snapshot identifiers. The parser contract and full route/configuration manifest are paper-local files: evidence/canonicalization_contract.md 11 Table 10:Per-system run and route manifest. Raw counts include transport retries; canonical counts are the item records used by the analysis. Exact route segments, API model IDs, base-path classes, response-file paths, and complete hashes are retained in the recovered CSV manifest. Endpoint configurationObserved route(s)Mode/maxUTC spanRaw SetCanonical SHA prefix DS V4 Pro (native)DeepSeek oļ¬icial (3004)native; 6553613:32ā16:21 3004 final2492 a380761e51f DS V4.1 Flash (native)DeepSeek oļ¬icial (3002)native; 32768/65536 09:07ā10:02 3002 final2492 2225eb6179df MiniMax-M3 (native)SCNet (1807); early/unknown (1340)native; 32768/65536 07:45ā13:24 3147 final2492 981c54a982f4 Kimi-K3 (native)SCNet (1816); micuapi (1424); early/unknown (919) native; 32768/65536 07:46ā17:00 4159 final2492 c8b94ded91c9 Qwen3.8-Max (native)SCNet (1636); micuapi (1589); early/unknown (883) native; 32768/65536 07:45ā17:35 4108 final2492 248c1ba7d807 GLM-5.3 (native)SCNet (1489); micuapi (1706); early/unknown (913) native; 32768/65536 07:45ā14:39 4108 final2492 ce25699f48ab DS V4.1 Flash (disabled) DeepSeek oļ¬icial (3000)disabled; 6553614:45ā14:47 3000 original3000 25d9a60a6a5e DS V4 Pro (disabled)DeepSeek oļ¬icial (3000)disabled; 6553614:35ā14:37 3000 original3000 27e44a4c73f1 tables/source/run_route_manifest.csv The manifest records hashes and raw/canonical counts. C.1 Parsing, retries, and canonicalization The parser removes balanced reasoning tags and accepts a single AāD letter or an explicit answer marker. Empty, mal- formed, or multiple-answer outputs are invalid. Transport er- rors are excluded; among non-error records, the latest non- empty parsed answer is selected, otherwise the latest empty response. Raw retries remain in the manifest; accuracy uses one canonical record per item and endpoint configuration. Headline values are recomputed from per-item records rather than runtime counters or progress files, which con- tained audited inconsistencies. A paper-local derivation script recomputes the original leaderboard, paired tests, con- figuration contrasts, and eļ¬iciency values from the hardening matrix and raw DeepSeek records. C.2 Statistical definitions For a subset withkkey-matching responses amongnrecords, accuracy is100k/n. Invalid final answers remain innand contribute zero tok. Wilson score intervals usez= 1.96as nominal finite-list descriptions. For two systems evaluated on the same item set,bcounts items correct only for the first system andccounts items correct only for the second. The exact two-sided McNemar test treatsbas a binomial variate conditional onb+c, and HolmāBonferroni correction is ap- plied across the 15 original-set system pairs. The six original- set native/disabled Overall/Theory/Case tests form a separate Holm family. Final-set tests are post-selection summaries because the native outcomes helped determine inclusion; no selective-inference correction is claimed. For itemi, empirical diļ¬iculty is d i = 1ā k i 6 , wherek i is the number of correct answers among the six na- tive models. The project-specific discrimination diagnostic is q i = 1 2 max(0, r i,rest ) + 1 2 4k i (6āk i ) 36 , wherer i,rest is the item-rest point-biserial correlation. With only six model observations per item, this score is a panel- conditional diagnostic, not a stable psychometric item param- eter. The paper therefore reports it descriptively and does not claim an item-response-theory calibration. Table 11:Final-set micro and macro scores. Task macro is the equal-weight mean of Theory and Case accuracy; category macro is the equal-weight mean over 25 category accuracies. SystemMicro Task macro Category macro Kimi-K375.5673.4673.69 GLM-5.375.0873.1773.38 Qwen3.8-Max 74.8472.8973.13 DS V4.1 Flash 72.4370.2070.41 DS V4 Pro72.3569.8870.15 MiniMax-M3 63.2061.5961.82 D Secondary Results Invalid-exclusion sensitivity is reported in Table 14. The top- three order changes from Kimi-K3, GLM-5.3, Qwen3.8-Max undertheoperationalinvalid-as-wrongruletoQwen3.8-Max, Kimi-K3, GLM-5.3 among valid responses. This alternative denominator is not a replacement leaderboard because it con- ditions on successful gateway completion. The answer-position audit finds no significant deviation from a uniform predicted A/B/C/D distribution for any main model (pā„0.127). This test does not exclude other anno- tation artifacts. The structural audit reports longest-option frequencies and substantial chart reuse in the original Case set. D.1 Full aggregate results Tables11,12,13, and14preservemacroscoring, theoriginal- set reference, task contrasts, and invalid-exclusion sensitivity. Task-macro scores are 1.61ā2.47 points lower than final mi- cro scores; category-macro scores are 1.38ā2.20 points lower, with unchanged ordering. Figures6and7give the final-set accuracy visualization and complete pairwise difference matrix. Their nominal in- tervals and significance markers are descriptive after model- informed selection. D.2 Fine-grained analysis D.2.1 Category structure The final benchmark has nearly equal Theory category sizes (102ā104 items) and mostly equal Case category sizes (96 items, with 78 Luck Pillars items). Balancing reduces the weight of the largest source category in broad comparisons while retaining categories below the target size. Across models, Twelve Stages and Nayin are the easi- est Theory categories (mean empirical diļ¬iculty 0.109 and 12 0.02.55.07.510.012.515.0 Thousands per item Flash Pro Reported reasoning tokens 01020304050 Thousands per item Kimi GLM Qwen Flash Pro MiniMax Reasoning-content characters Median 90th percentile Figure 5:Reasoning-effort summaries: median and 90th percentile per item. The token panel includes only the two direct DeepSeek runs with reliable positive token reports. The character panel covers all six systems and treats absent reasoning content as zero; characters remain a route-dependent proxy. Table 12:Pre-selection reference results on the original 3,000 items. Intervals are nominal Wilson 95% intervals and do not account for source/chart clustering. Endpoint alias Overall95% CI Theory Case Kimi-K379.50 78.02ā80.91 86.40 72.60 GLM-5.379.13 77.64ā80.55 85.00 73.27 Qwen3.8-Max 78.97 77.47ā80.39 85.00 72.93 DS V4.1 Flash 76.97 75.43ā78.44 84.00 69.93 DS V4 Pro76.83 75.29ā78.31 85.07 68.60 MiniMax-M369.07 67.39ā70.70 72.07 66.07 Table 13:Theory and Case accuracy and primary gap (percentage points). ModelTheory Case Gap Kimi-K386.04 60.89 25.15 GLM-5.384.59 61.75 22.84 Qwen3.8-Max84.59 61.18 23.42 DS V4.1 Flash 83.56 56.84 26.72 DS V4 Pro84.66 55.11 29.56 MiniMax-M371.25 51.93 19.32 Table 14:Sensitivity to excluding invalid responses from each modelās denominator. ModelPrimary Excl. invalid Change Kimi-K375.5679.52 +3.96 GLM-5.375.0875.26 +0.18 Qwen3.8-Max74.8479.60 +4.76 DS V4.1 Flash72.4372.430.00 DS V4 Pro72.3572.350.00 MiniMax-M363.2063.89 +0.69 0.114), whereasShenshaBasics, PatternBasics, andStrength and Weakness are the hardest. On Case items, Career and Family Relations are the most diļ¬icult (mean diļ¬iculty 0.630 and 0.618, with no six-model-all-correct items), followed by Romance; Luck Pillars is the easiest Case category (0.154). These summaries describe agreement with the released gold answers, not the truth of competing interpretive schools. D.2.2 Diļ¬iculty and agreement At the item level, 1,118 questions (44.86%) are answered correctly by all six models and 204 (8.19%) by none. The distribution is therefore bimodal rather than smoothly diļ¬i- cult. Case items have an all-correct rate of 25.3% and an 6065707580 Accuracy (%) Ā· nominal Wilson 95% interval Kimi-K3 GLM-5.3 Qwen3.8-Max DS V4.1 Flash DS V4 Pro MiniMax-M3 75.56 75.08 74.84 72.43 72.35 63.20 Figure 6:Final-set micro accuracy with nominal 95% Wilson intervals. Values are percentages; intervals are descriptive after outcome-informed selection. all-wrong rate of 13.2%, compared with 58.8% and 4.6% for Theory. The pattern motivates the hardening procedureās fo- cus on concordant denominator mass, but it also warns that a model-informed subset can alter the diļ¬iculty spectrum. D.2.3 Answer-position and structural checks Thetargetanswerdistributionisapproximatelybalanced, and chi-square goodness-of-fit tests on model predictions do not reject a uniform A/B/C/D distribution (pā„0.127; Cohenās wā¤0.048). No marginal answer-position imbalance was detected; this does not exclude item-conditioned position ef- fects or other annotation artifacts. A separate structural au- dit finds that the correct option is uniquely the longest in 13 Kimi GLM Qwen Flash Pro MiniMax Kimi GLM Qwen Flash Pro MiniMax ā+0.5+0.7+3.1*+3.2*+12.4* -0.5ā+0.2+2.6*+2.7*+11.9* -0.7-0.2ā+2.4*+2.5*+11.6* -3.1*-2.6*-2.4*ā+0.1+9.2* -3.2*-2.7*-2.5*-0.1ā+9.1* -12.4*-11.9*-11.6*-9.2*-9.1*ā * Holm-adjusted p < .05 (exploratory) Figure 7:Final-set pairwise accuracy differences in percentage points (row minus column). Asterisks mark nominal exact McNemarp<.05after Holm correction across 15 pairs; these post-selection markers are exploratory. Table 15:Most and least diļ¬icult categories by six-model empirical diļ¬iculty. TaskCategorynDiļ¬iculty Theory Shensha Basics1040.240 Theory Pattern Basics1040.220 Theory Strength and Weakness 1040.210 Theory Nayin1040.114 Theory Twelve Stages1040.109 CaseCareer960.630 CaseFamily Relations960.618 CaseRomance960.573 CasePersonality960.243 CaseLuck Pillars780.154 765/3,000 original items (25.50%; 284 Theory and 481 Case) and 564/2,492 final items (22.63%; 278 Theory and 286 Case), as shown in Table16. These frequencies are struc- tural diagnostics; they alone do not establish an exploitable answer-length heuristic. No item text was altered to change these patterns. The Case portion also reuses chart contexts. The origi- nal 1,500 Case items contain 274 distinct charts, with 1,449 records belonging to a chart used at least twice and a maxi- mum reuse of 33. The final 1,038 Case items contain 249 dis- tinct charts, 974 shared-chart records, and a maximum reuse of 25. These counts motivate treating McNemar and Wilson calculations as fixed-list diagnostics rather than population inference; no cluster bootstrap is claimed. D.3 Configuration contrasts and answer tran- sitions Flash has 328 native-key-match/disabled-mismatch original items versus 285 in the reverse direction; Pro has 465 versus 263. These counts explain the net gains without implying a Kimi GLM Qwen Flash Pro MiniMax Twelve Stages Nayin Annual and Luck Cycles Ten Gods Combination and Clash Ten Gods Relations Kongwang (Void) Useful and Adverse Gods Five Elements Six Relations Stems and Branches Strength and Weakness Pattern Basics Shensha Basics 949392929369 939190939370 888786868877 878789858778 868788838580 878286888469 858783808675 838386848174 878186838568 848282848170 808876818967 888281768167 867984807664 807876777867 Figure 8:Theory-category accuracy across the six models. Vector labels show rounded percentages; full category names follow Table5. Table 16:Structural-cue and repeated-chart audit. The cue is present when the correct option is uniquely the longest. Chart statistics apply to Case items; āshared itemsā counts Case records whose chart occurs at least twice. Release TasknCuen(%) Charts Max reuse Original Theory 1,500 284 (18.93)ā Original Case1,500 481 (32.07)27433 Original Total3,000 765 (25.50)ā FinalTheory 1,454 278 (19.12)ā FinalCase1,038 286 (27.55)24925 FinalTotal2,492 564 (22.63)ā causal failure mechanism. Figure11visualizes the original- set contrast, while Table 17preserves the stronger selected- final diagnostic; the latter conditions on native-arm-informed selection and is not the primary ablation. D.4 Eļ¬iciency and failure statistics D.4.1 Reasoning eļ¬iciency On the original 3,000 items, native Pro is associated with 202 net additional key matches for 14,378,326 reported reasoning tokens, or 71,180 tokens per net match and 1.405 accuracy points per 1,000 additional reasoning tokens per item. Flash has 43 net matches for 15,219,104 tokens, or 353,933 tokens per net match and 0.283 points on the same measure. These unstable descriptive ratios divide by net gain; Flashās overall difference is not detected. They are not monetary costs and are not comparable to mixed-route systems whose reasoning- token fields are incomplete or zero-filled. On the original set, Flashās mean latency was 28.22 sec- onds in the native configuration and 1.53 seconds in the dis- abled configuration. Proās corresponding means were 91.39 and 2.22 seconds. Native/disabled medians were 10.68/1.43 seconds for Flash and 20.95/2.07 seconds for Pro, giving ra- tios of medians of 7.44Ćand 10.13Ć. Request durations include network delays, queueing, and provider execution. The configurations ran at different times, 14 Kimi GLM Qwen Flash Pro MiniMax Luck Pillars Personality Misfortune and Legal Year Timing Chart Construction Wealth Health Education Romance Family Relations Career 878690818678 778182767859 768075726768 727469716566 616566606756 606258525052 615756545046 495061484544 484546383446 413939433533 424536353528 Figure 9:Case-category accuracy across the six models. Vector labels show rounded percentages; the larger spread in several categories is consistent with the aggregate Theory-to-Case gap. 0123456 Number of six models matching the key 0 200 400 600 800 1000 1200 Items 204 175 157 187 211 440 1118 Figure 10:Final-set item counts by number of models matching the released key, pooled across Theory and Case (2,492 items). so their latency contrast describes an operational association rather than a causal or compute-only effect. Table 18:Token and request-latency eļ¬iciency for the paired DeepSeek configurations on the original 3,000 items.āCis the native-minus- disabled net key-match count; p/1k is accuracy-point gain per 1,000 additional reasoning tokens per item. Latency is native/disabled seconds; the ratio divides the two medians. AliasāCTok./āCpp/1k Lat. med. N/D Ratio Flash 43 353,933 0.28310.68/1.43 7.44Ć Pro 202 71,180 1.40520.95/2.07 10.13Ć The six V1 native runs show that key-mismatching an- swerscontainmorereasoningtextthankey-matchinganswers for every system, making length a poor proxy for success- ful deliberation. On the original DeepSeek runs, Case items consume more output tokens than Theory: 7,535.8 versus 2,614.2 per item for Flash and 8,003.2 versus 1,586.3 for Pro. Latency should not be used as a cross-system speed ranking. 657075808590 Original-set accuracy (%) Flash Overall Flash Theory Flash Case Pro Overall Pro Theory Pro Case +1.4 +6.5 -3.7 +6.7 +12.2 +1.3 DisabledNative Figure 11:Native and disabled-configuration accuracy on the original 3,000 items. Connected dots show paired provider configurations; displayed deltas are native minus disabled. Table 17:Selected-final-set native/disabled diagnostic. Because inclusion uses the native-arm outcomes, these deltas and nominal adjustedp-values are post-selection summaries, not confirmatory effects. Model Subset Native Disabledā Flash Overall 72.4371.79 +0.64 Flash Theory 83.5677.03 +6.53 Flash Case56.8464.45-7.61 ProOverall 72.3566.29 +6.06 ProTheory 84.6672.49 +12.17 ProCase55.1157.61-2.50 D.4.2 Invalid responses There are 306 invalid responses among the 14,952 six-system final-set evaluations (2.05%). Qwen3.8-Max has 149, Kimi- K3 124, MiniMax-M3 27, and GLM-5.3 6; both DeepSeek native runs have zero. By recorded source, the micuapi route has 278 invalids in 3,263 records (8.52%), compared with 15/3,804 for SCNet and 13/2,901 for an early segment whose endpoint is not recorded. The oļ¬icial DeepSeek route has 0/4,984. Of the invalid records, 292 contain positive-length reasoning content and 14 Kimi/micuapi records do not; finish reasons are blank for 266,errorfor seven, andstopfor 33, with nolengthrecords. These observations support an in- terface or route diagnosis, but do not prove that an underlying model is intrinsically incapable of answering an item. D.4.3 Failure concentration The 204 final-set items unmatched by all six answer vectors are concentrated in Career, Romance, Chart Construction, andEducationCasecategories. Ontheoriginalset, thepaired configurations create bidirectional reversals: 285 Flash items and 263 Pro items change from a disabled-configuration key match to a native-configuration mismatch. We do not infer a causal failure mechanism. Without human-coded examples, these are failure statistics and candidate strata for expert ad- judication, not a qualitative error analysis. D.4.4 Source and latency sensitivity Routes covary with task order. The micuapi segment is mostly Case; early and SCNet segments are mostly Theory. 15 024681012141618 Mean reasoning content (1,000 characters / item) 60.0 62.5 65.0 67.5 70.0 72.5 75.0 77.5 Accuracy (%) Kimi GLM Qwen Flash Pro MiniMax Figure 12:Overall accuracy versus mean reasoning-content length over all final-set records, including zeros for absent content. Characters are used because reported reasoning tokens are not comparable across routes. This descriptive cross-system plot is confounded by route and task composition. Table 19:Invalid responses in the six-model main evaluation. ModelInvalidnInvalid (%) Qwen3.8-Max1495.98 Kimi-K31244.98 MiniMax-M3271.08 GLM-5.360.24 DS V4.1 Flash00.00 DS V4 Pro00.00 Total3062.05 Accuracy differences between segments thus combine route and item-composition effects. We use source primarily to di- agnose invalid responses and restrict eļ¬iciency interpretation to clean direct comparisons. E Reproducibility and Release The canonical final analysis contains eight complete runs (six native and two disabled), yielding8Ć2,492 = 19,936 retained-item evaluations. The archived raw response files cover the original 3,000-item run for each of the same eight configurations. The paper package retains CSV snapshots for all reported tables and PDF/PNG copies of deterministic fig- ures. The original analysis scripts remain the source of truth; paper-local derivation helpers accept their root path rather than embedding a machine-specific location. The release history required to interpret the paper is: 1.archived original JSONL files with 1,500 Theory and 1,500 Case items; 2.per-item main and ablation responses; 3.item-metric and hardening analyses; 4.the fixed-seed final subset builder; 5.final JSONL files and the 508-row deletion manifest. BaZi2500 is the public benchmark release under C BY- NC 4.0; resource links are provided in the main-text disclo- sures. E.1 Responsible use and ethics This artifact evaluates culturally situated textual and rule- based QA. It does not establish or endorse real-world predic- tive validity and should not inform medical, legal, financial, 0246 Invalid canonical answers (%) Kimi-K3 GLM-5.3 Qwen3.8-Max DS V4.1 Flash DS V4 Pro MiniMax-M3 4.98% 0.24% 5.98% 0.00% 0.00% 1.08% Figure 13:Invalid response rates on the final set. Empty answers are counted as wrong in the primary accuracy metric. employment, relationship, or other consequential decisions. Historical, school-specific, and culturally contested claims may appear in the source representation; model performance does not adjudicate them. The study evaluates archived textual QA artifacts and re- cruited no participants or interventions. The authors have manually reviewed and addressed privacy in the public data. E.2 Data availability and author disclosures The project preserves original and final releases, the dele- tion manifest, analysis tables, and deterministic figures. BaZi2500 is publicly released under C BY-NC 4.0; the au- thors have confirmed that its classical and traditional source materials have no modern copyright restrictions and may be redistributed in this form. Author names and aļ¬iliations are provided on the title page. Contact: lijiulin@18trees.com. This work was supported by Beijing Liuyi Guanhua Technol- ogy Co., Ltd. The authors declare no competing interests. Language models assisted upstream source gating, extrac- tion, drafting, verification, and deduplication and were sub- jects of evaluation. An agent-assisted workflow assembled this manuscript from project evidence. No model acted as an independent human expert validator; numerical results come from deterministic analysis outputs checked against source artifacts. 16