Paper deep dive
CIPHER: Benchmarking Cross-record Inference over Privacy-Hardened Evidence Records
Suparno Roy Chowdhury, Manan Roy Choudhury, Dhruv Madhwal, Vivek Gupta
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 91%
Last extracted: 9/13/2026, 3:42:04 AM
Summary
The paper introduces CIPHER, a benchmark for evaluating cross-record inference over privacy-hardened evidence records. It combines structured attributes with free-text narratives from consumer-finance, clinical, and law-enforcement domains. The study evaluates various AI systems (retrieval, prompting, hybrid) under native redaction and surrogate-based evidence restoration, finding that most errors stem from incorrect record selection and predicate interpretation rather than arithmetic execution.
Entities (19)
Relation Signals (18)
CIPHER â containsdomains â CFPB
confidence 95% · CIPHER covers law-enforcement, consumer-finance, and clinical records... CFPB We sample 400 records from the public CFPB Consumer Complaint Database
CIPHER â containsdomains â MIMIC-IV
confidence 95% · MIMIC From the MIMIC-IV v3.1 note module... we draw 500 discharge notes.
CIPHER â containsdomains â FIR
confidence 95% · FIR We collect 322 First Information Reports... CIPHER covers law-enforcement... records.
CIPHER â evaluates â Llama-3.3-70B
confidence 92% · We evaluate five model checkpoints: Llama 3.3 70B... These models span proprietary and open-weight families...
CIPHER â evaluates â gpt-oss-120b
confidence 92% · We evaluate five model checkpoints: ... GPT-OSS 120B...
CIPHER â evaluates â Gemini 2.0 Flash
confidence 92% · We evaluate five model checkpoints: ... Gemini 2.0 Flash...
CIPHER â evaluates â Qwen 3 Next 80B
confidence 92% · We evaluate five model checkpoints: ... Qwen 3 Next 80B...
CIPHER â evaluates â WEAVER
confidence 90% · Hybrid reasoning We include WEAVER (Khoja et al., 2025)...
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Reasoning over privacy-constrained records requires combining structured attributes with evidence from free-text narratives. We introduce CIPHER (Cross-record Inference over Privacy-Hardened Evidence Records), a benchmark of expert-validated questions from consumer-finance, clinical, and law-enforcement records. The questions cover common tabular operations and include executable SQL supervision. We evaluate retrieval, prompting, table-specialist, and hybrid symbolic-neural systems under native redaction and surrogate-based evidence restoration. All system families exhibit substantial failures even when supporting records are provided. Most errors arise from incorrect record selection and predicate interpretation rather than arithmetic execution. Privacy transformations have non-uniform effects, sometimes obscuring necessary evidence and sometimes reducing distraction. CIPHER provides a reproducible testbed for diagnosing these failures and assessing how transformations of sensitive text affect reasoning over hybrid records.
Tags
Links
- Source: https://arxiv.org/abs/2609.07022v1
- Canonical: https://arxiv.org/abs/2609.07022v1
Trouble viewing inline? Open PDF directly â
Full Text
69,802 characters extracted from source content.
Expand or collapse full text
CIPHER: Benchmarking Cross-record Inference over Privacy-Hardened Evidence Records Suparno Roy Chowdhury â Manan Roy Choudhury â Dhruv MadhwalVivek Gupta Arizona State University srchowd3,mroycho1,dmadhwal,vgupt140@asu.edu Abstract Reasoning over privacy-constrained records re- quires combining structured attributes with evi- dence from free-text narratives. We introduce CIPHER (Cross-record Inference over Privacy- Hardened Evidence Records), a benchmark of expert-validated questions from consumer- finance, clinical, and law-enforcement records. The questions cover common tabular oper- ations and include executable SQL supervi- sion. We evaluate retrieval, prompting, table- specialist, and hybrid symbolicâneural systems under native redaction and surrogate-based ev- idence restoration. All system families ex- hibit substantial failures even when support- ing records are provided. Most errors arise from incorrect record selection and predicate interpretation rather than arithmetic execution. Privacy transformations have non-uniform ef- fects, sometimes obscuring necessary evidence and sometimes reducing distraction. CIPHER provides a reproducible testbed for diagnosing these failures and assessing how transforma- tions of sensitive text affect reasoning over hy- brid records. 1 Introduction Questions over real-world records often require evidence from both structured attributes and free- text narratives. A date may appear in a dedicated column, for example, while the event associated with it is described only in text. Answering such questions may also require identifying every qual- ifying record before computing a count, ranking, trend, or co-occurrence. Systems must therefore ground question constraints across representations and compose evidence across multiple records, as illustrated in Figure 1. Existing resources only partially evaluate this setting. Spider (Yu et al., 2019) focuses on exe- cutable reasoning over relational schemas, while * Equal contribution. IDProductConsumer complaint narrative 7 Checking or savings account On X X around X PST... my husband says, ?I can't access our mobile banking?... the representative says, ?your accounts has been frozen due to fraudulent activity?... ?we don't have access to any funds' 45 Checking or savings account ?X I went to the Wells Fargo bank ATM located at X X X X X X X MN X... proceeded to insert X $X.X bills. The ATM took my cash... a provisional credit of $X.X will be credited to my account.? 32 Vehicle loan or lease ?I purchased and financed a X X X on X/X/X... neither my name nor my phone number was correct in their system...? 37 Credit card or prepaid card ?On X/X/2019... I received a call... asking if I tried to make a X purchase... she would close the account due to fraudulent activity and send me a new card..? Answer: ID 7 (Incorrect) CFPB ID 7 is a checking account with restricted access LLM Output Retrieves case of all checking or savings accounts Analyzes consumer narratives for restricted account access Fails to retrieve ID 45. Provisional credits are credited to restricted accounts Reason: Provisional credits are awarded to bankers with disputed or restricted access to bank accounts when potentially suspected of fraudulent activity. Gold Truth Answer: ID 7, ID 45 (Correct) Question: What are the checking or savings account which the consumer reports having restricted account access? Figure 1: Language models must jointly reason over structured attributes and free-text narratives in privacy- constrained hybrid records. HybridQA (Chen et al., 2021b) combines tables and text drawn primarily from regular web-based sources. RUST-BENCH (Abhyankar et al., 2025a) studies tables containing unstructured text under increasing scale and reasoning complexity. How- ever, these resources provide limited support for diagnosing whether errors arise from record selec- tion, narrative interpretation, cross-representation grounding, or operation execution. This distinction is especially important for privacy-sensitive records. Clinical notes, financial complaints, and public-sector reports may contain generalized values, placeholders, partially masked spans, or removed text. Such transformations can hide relevant evidence or weaken links between structured attributes and narrative mentions. They may also remove distracting surface cues. Their arXiv:2609.07022v1 [cs.CR] 7 Sep 2026 effects on reasoning are therefore not necessarily uniform across domains or systems. We present CIPHER, a resource for evaluat- ing language models over privacy-constrained hy- brid records. It contains 3,427 questions derived from consumer-finance complaints, clinical dis- charge summaries, and law-enforcement reports. The questions cover filtering, aggregation, ranking, trend analysis, and co-occurrence, with every in- stance requiring structured and narrative evidence. Each question is paired with an executable SQL specification, a reference answer, and supporting- record provenance. These annotations are retained for evaluation but withheld from the model. They enable reference answers to be regenerated, con- trolled contexts to include all required evidence, and failures to be traced to particular stages of evi- dence use. The resource includes aligned evidence con- ditions based on native de-identification and surrogate-based surface realizations. Surrogates are contextually plausible replacements rather than recovered private values, and the transformations are not claimed to provide formal privacy guaran- tees. They instead support within-domain compar- isons of how changes to visible text affect model behavior. Our primary evaluation places all supporting records in the model context together with same- domain distractors, separating reasoning failures from missing-support failures. A complementary full-table setting evaluates evidence selection over the complete candidate collection. We compare retrieval, prompting, table-specialist, and hybrid symbolicâneural systems using answer-level met- rics and a diagnostic failure taxonomy. The results show that providing the required records does not make the task reliable. Errors arise primarily from selecting irrelevant records and misinterpreting narrative predicates rather than from arithmetic execution. Fixed-depth retrieval is particularly weak for questions requiring ev- idence from several records. Performance also changes non-monotonically across evidence condi- tions: restoring plausible surface forms helps some configurations while degrading others. Our contributions are: âąA multi-domain hybrid-reasoning resource. CIPHER contains 3,427 validated questions grounded in structured attributes and nar- ratives from financial, clinical, and law- enforcement records. âąExecutable supervision with evidence provenance. Each question includes a regen- erable reference answer and identifiers for the records contributing to that answer. âąControlled evidence conditions and eval- uation settings. Aligned transformations, support-complete distractor contexts, and full- table experiments enable comparisons of evi- dence interpretation and selection behavior. âą Diagnostic evaluation across system fami- lies. Experiments reveal that record selection and narrative-predicate interpretation remain major sources of failure even when the re- quired evidence is available. Subject to source-data licenses and access re- quirements, we release the benchmark representa- tions, templates, executable supervision, prompts, transformation metadata, and evaluation code. 2 Related Work Prior work on tabular and hybrid QA largely fo- cuses on clean, web-derived tables, especially from Wikipedia. Benchmarks such as WikiTableQues- tions (Pasupat and Liang, 2015), WikiSQL (Zhong et al., 2017), and Spider (Yu et al., 2019) target neural semantic parsing and text-to-SQL, while Hy- bridQA (Chen et al., 2021b), OTT-QA (Chen et al., 2021a), TAT-QA (Zhu et al., 2021), and FeTaQA (Nan et al., 2021) extend this to joint reasoning over tables and text. However, these datasets typically exhibit regular schemas and low noise, so strong performance may exploit annotation artifacts or distributional overlap rather than robust reasoning (Shaw et al., 2021). More recent work examines LLMs under long- context and tool-based reasoning settings. RUST- BENCH (Abhyankar et al., 2025a) shows that per- formance degrades as tables grow, domains be- come more heterogeneous, and reasoning chains lengthen, with similar trends in long-context and tool-augmented studies (Fu et al., 2023) with no study on the effects of privacy driven performances. Relative to this literature, CIPHER is the first resource to combine noisy, institutionally redacted narratives with structured attributes across three distinct domains, and to pair every question with ex- ecutable supervision and supporting-record prove- nance that lets errors be traced to a specific stage of I DPr od u ctCon su m er com p l ai n t n ar r at i v e 7 Check i n g or savi n gs account On X X ar oun d X PST... ? 45 Check i n g or savi n gs account X I w ent to the Wel l s Far go ban k ATM l ocated at ... ...... Dat aset M et a-Dat a De-Red act i on St r u ct u r ed Dat a ar e data poi nts f r om the t ab l e h ead er s. Un st r u ct u r ed Dat a ar e data poi nts f r om the n ar r at i v es. Qu est i on Cr eat i on Syn t h et i c De-Red act i on On Th u r sd ay t h e 26t h ar oun d 11 PST... Hum an r evi ew er s i n depen dently assess each syntheti c span Pr oduct = "Ch eck i n g or Sav i n gs Accou n t" Legal Pr ocess I nvolved = "d i d n ot i n v ol v e an y l egal p r ocess" <st r u ct u r ed .Pr od u ct > <u n st r u ct u r ed . l egal _p r ocess_i n v ol v ed > Text Text Tem p l at e Qu est i on Tem p l at e SQL Qu est i on s W i t h Val u es I n ser t ed SQL W i t h Val u es I n ser t ed Qu est i on Ver i f i cat i on I n st an t i at ed Qu est i on SQL W i t h Val u es I n ser t ed Gr ou n d A n sw er H u m an Rev i ew Ph ase Revi ew ed by NLP exper ts i n or der to val i date questi on s an d gr oun d tr uth an sw er s Ap p r ov e Rew r i t eRem ov e Ap p r ov e (or Cor r ect ed ) Qu est i on s Questi on s that ar e appr oved (or r ew or ked an d appr oved) ar e added to the ben chm ar k I DPr od u ct 7 Ch eck i n g or sav i n gs accou n t 45 Ch eck i n g or sav i n gs ... ... Un st r u ct u r ed JSONS Con su m er com p l ai n t On X X ar oun d X PST... ? X I w ent to the Wel l s Far go ban k ATM l ocated at ... Con su m er com p l ai n t On X X ar oun d X PST... ? X I w ent to the Wel l s Far go ban k ATM l ocated at ... On X X ar oun d X PST... ? On X X ar oun d X PST... St r u ct u r ed JSONS De-r edacted span s ar e stor ed i n JSONs for r epl i cati on an d ver i f i cati on Questi on s an d Gr oun d Tr uth m atch Questi on s an d Gr oun d Tr uth n eed adjustm ent Questi on s an d Gr oun d Tr uth n eed r em oval SQLs ar e ver i f i ed agai n st the m et a d at a t ab l e, acti n g as the gr oun d tr uth. Un st r u ct u r ed M et a-Dat a St r u ct u r ed M et a-Dat a Red act ed Sp an Ex t r act i on De-Red act ed Sp an s St or ed De-Red act i on H u m an Rev i ew Figure 2: CIPHER construction pipeline. Source records combine structured attributes with narrative text. Paired question and executable SQL templates produce benchmark instances, which are then reviewed, revised when necessary, or removed. evidence use rather than only scored at the answer level. 3 Task Definition 3.1 Hybrid Questions Figure 2 summarizes the construction pipeline de- scribed in this section and in Section 4: source records are combined with paired question and SQL templates, then reviewed, revised, or removed to produce the final benchmark instances. A hybrid recordz j = âšb j ,x j â©pairs schema- aligned attributesb j , such as dates, categories, iden- tifiers, locations, or numeric values, with the associ- ated free textx j . A question is hybrid when resolv- ing it requires at least one constraint grounded in b j and at least one grounded inx j . A question may restrict records by a year stored in a structured field, for example, while determining from the narratives whether a particular event occurred. Questions may require one or several qualifying records and cover filtering, aggregation, ranking, trend analysis, and co-occurrence. Under evidence conditionc, a system receives the questionuand a model-visible record col- lectionV (c) u , and returns a predicted answer ey (c) u = h(u,V (c) u ). Every question also carries an executable SQL specification and an aligned, construction-only annotation view; executing the specification regenerates both the reference answer y (c) u and the identifiers of its supporting records. The specification, annotations, operation label, and support identifiers are never shown to the system. They are retained for reference generation, support- controlled context construction, and evaluation. 3.2 Redacted and Restored Evidence Each question is evaluated under two aligned con- ditions. The redacted condition,c = red, contains the native de-identification or post-hoc masking associated with its domain. The restored condi- tion,c = rest, exposes an aligned version in which marked spans carry additional surface information. Appendix E reports the reviewer assessment of those surrogates. For CFPB and MIMIC the restored spans hold contextually plausible surrogate values, because the original private values are unavailable, so these variants must not be interpreted as recovered or gen- uinely unredacted records. Where post-hoc mask- ing leaves the source text available, the restored condition may preserve the corresponding source span. In both cases the comparison measures sen- sitivity to the information shown to the model, not privacy protection or re-identification risk. Questions and record identifiers are held fixed across conditions, while the visible collection, ref- erence answer, and supporting records may dif- fer. For an answer metricM, we report the effect of restoration asâM = M (rest) â M (red) , so a positive value indicates higher performance under restored evidence and a negative value indicates that the native redacted condition is stronger. The paired evaluation includes only questions whose reference answers remain supported by the model- visible records in both conditions. 4 The CIPHER Benchmark 4.1 Domains CIPHER covers law-enforcement, consumer- finance, and clinical records. The three domains differ in schema, narrative style, record length, and de-identification provenance. FIR We collect 322 First Information Re- ports from public Indian law-enforcement web- sites (Anti-Corruption Bureau, Government of Haryana, 2024). The PDF reports are processed using OCR, layout analysis, and field extraction. Schema-aligned attributes include report number, police station, date, legal section, and location, while complaint statements are retained as narrative evidence. Because the source reports are not uni- formly de-identified, Microsoft Presidio (Mendels et al., 2018) is used to replace detected personally identifiable information with typed placeholders. CFPB We sample 400 records from the public CFPB Consumer Complaint Database (Consumer Financial Protection Bureau, 2026). Each record includes attributes such as product, issue, com- pany, submission date, and response status together with a consumer-authored complaint. We retain the source schema with limited normalization and use the narratives in their institutionally de-identified form. MIMIC From the MIMIC-IV v3.1 note mod- ule (Johnson et al., 2023), we draw 500 discharge notes. MIMIC is institutionally de-identified, and all derived materials remain subject to PhysioNet credentialing, licensing, and redistribution require- ments. 4.2 Benchmark Statistics CIPHER contains 3,427 questions: 1,571 from FIR, 1,356 from CFPB, and 500 from MIMIC. Questions cover filtering, aggregation, ranking, trend analysis, and co-occurrence and produce nu- meric, textual, and list-valued answers. Table 1 summarizes the resource. Average narra- tive length ranges from 630 words in FIR to 1,018 words in MIMIC. The domains also differ in their operation and answer-type distributions. FIR and CFPB contain substantial proportions of list-valued answers, whereas MIMIC is dominated by numeric and textual outputs. FIR CFPB MIMIC Questions # Questions1,571 1,356500 Avg. question length13.724.420.0 Operation category (%) Aggregation25.525.020.2 Trend15.816.318.2 Ranking11.217.814.6 Co-occurrence20.49.725.2 Filtering27.131.221.8 Record representation # Structured columns272318 # Narrative columns416 Avg. narrative words6308131,018 Answer type (%) Number41.730.554.6 String15.025.642.8 List43.344.02.6 Table 1: Table statistics for the benchmark in regards to question lengths, operation categories, records, and answer types. Figure 3: Template-validation outcomes by domain 4.3 Template Audit The resource is audited at the template-family level. Each reviewer sees the seed specification, its gener- ated variants, the source records, the SQL specifi- cation, the reference output, and the supporting- record identifiers. APPROVE marks a package that satisfies all three requirements as seen in C, REWRITE a repairable issue such as an invalid pa- rameter, variant mismatch, or incomplete prove- nance trace, and REMOVE a family that cannot meet the requirements without changing its in- tended operation or evidence. Reviewers decide independently, and disagreements are resolved by inspecting the affected question, execution trace, and source records. Figure 3 reports the disposi- tions: 88.0% to 94.5% of templates are retained per domain, combining direct approvals with those kept after revision. Agreement is computed before adjudication, and pairwise CohenâsÎșranges from 0.93 to 0.96 (Table 2). Reviewer pair CohenâsÎș Jaccard α 1,2 0.930.93 α 2,3 0.960.96 α 3,1 0.940.94 Table 2: CohenâsÎșcalculated among three NLP experts. 5 Experimental Setup 5.1 Models Weevaluatefivemodelcheckpoints: Llama 3.3 70B(Llama Team, 2024), GPT- OSS 120B and GPT-OSS 20B (OpenAI, 2025), Gemini 2.0 Flash (Gemini Team, 2025b), and Qwen 3 Next 80B (Qwen Team, 2025). These models span proprietary and open-weight families and support the context lengths required by the evaluation inputs. All models use greedy decoding with temper- ature0.0and a maximum output length of 8,192 tokens. Questions and record serializations are held constant across configurations whenever model in- terfaces permit. Exact checkpoints, API versions, prompts, and inference parameters are included in the released configuration. 5.2 Baselines We compare prompting, explicit reasoning, retrieval-augmented generation,and hybrid symbolicâneural baselines. Prompting and reasoning The zero-shot base- line asks the model to return an answer directly from the provided records. The few-shot base- line adds two examples that demonstrate reasoning across structured attributes and narratives (Brown et al., 2020). We additionally evaluate three methods that elicit intermediate reasoning: chain-of-thought (CoT) (Wei et al., 2023), program-of-thought (PoT) (Chen et al., 2023), and least-to-most de- composition (LtM) (Zhou et al., 2023). CoT requests an explicit reasoning trace, PoT encour- ages program-like execution, and LtM decomposes the question into smaller intermediate steps. All five configurations receive the complete support- controlled context. Retrieval-augmented generation Retrieval op- erates over a per-question corpus in which every candidate record is an independent unit, serialized as one line of keyâvalue pairs,[field 1 : value 1 , . . . , field n : value n ] ; the same serialization is used when records enter the prompt. Our first setup encodes records and questions with BGE- large-en-v1.5 (Xiao et al., 2024), using the raw question as the query without BGEâs instruction prefix. Final-layer embeddings are mean-pooled under the attention mask andL 2 -normalized, so in- ner product gives cosine similarity; since each ques- tion carries its own corpus, records are re-encoded per question and scored exhaustively. Our second setup rescores the returned candidates with BGE- reranker-large (Xiao et al., 2024), a cross-encoder that jointly encodes the question and each record and emits a single relevance logit, reordering the pool before truncation. We sweepk â1, 3, 5, 10 and reportk = 5, drawing everykfrom a common pool of ten candidates. The top-krecords are con- catenated in rank order beneath the question, each under a positional header (Table i), with the data- point context before and JSON format instructions after, plus a directive to emit valid JSON. An LLM then reads the question and selected records and writes its answer in the requested JSON form. Hybrid reasoningWe include WEAVER (Khoja et al., 2025) and BlendSQL (Glenn et al., 2024). WEAVER uses a language model to plan symbolic table operations, while BlendSQL produces SQL- like programs that combine schema-level execution with language-model calls over narrative fields. Table specialists We include TableGPT2 (Su et al., 2024) and TAT-LLM 70B (Zhu et al., 2024). TableGPT2 pairs a dedicated table encoder with a language model to capture schema and cell-level structure, while TAT-LLM is fine-tuned for discrete reasoning over tabular and textual input through a step-wise extract, reason, and execute pipeline. Both are complete systems rather than methods ap- plied to a backbone, so each reports a single score per dataset in Table 5. 5.3 Evaluation Support-controlled contexts The primary eval- uation supplies every record identified by the ex- ecutable specification together with same-domain distractors. Each context contains between 39 and 42 records. Distractors are selected from outside the verified support set and checked to ensure that they do not change the reference output. The reference answer is regenerated using the same record identifiers included in the model- visible context. This design separates missing- evidence failures from errors in record selec- tion, predicate interpretation, cross-representation grounding, and operation execution. Answer metrics Because reference outputs in- clude numbers, strings, ordered sequences, and sets, we report three complementary metrics. Token overlap,M 1 , measures normalized lexical agree- ment between the prediction and reference. Seman- tic acceptance,M 2 , is a binary decision produced by Gemini 2.5 Flash (Gemini Team, 2025a) us- ing the question, reference, and candidate answer. The complete judge instructions are provided in Appendix A.6. Both are averaged across instances and expressed as percentages, soM 1 is the mean overlap score andM 2 is the acceptance rate of a configuration. The primary metric,M 3 , combines them at the level of the individual question, M (i) 3 = 2M (i) 1 M (i) 2 M (i) 1 + M (i) 2 , M 3 = 1 N N X i=1 M (i) 3 , withM (i) 3 set to zero whenever either component is zero. BecauseM (i) 2 is binary,M (i) 3 is zero for rejected answers and a monotone rescaling ofM (i) 1 for accepted ones. A prediction therefore earns credit only when the judge accepts it and its content agrees with the reference: an answer judged plausi- ble while omitting qualifying records or reporting a different value cannot be rescued by surface agree- ment, and an answer that overlaps the reference lexically earns nothing if it does not answer the question. We prefer this to the arithmetic mean be- cause it penalizes imbalance rather than averaging it away. Scoring each question before aggregating also makesM 3 an ordinary per-instance quantity, so it can be decomposed by operation category and answer type rather than only read at the configu- ration level. We retainM 1 andM 2 alongsideM 3 because the two can disagree, and the appendix reports all three for every configuration. Judge validation We compareM 2 with human decisions on 750 outputs, sampled evenly across CFPB, FIR, and MIMIC. Two NLP researchers independently determine whether each candidate should be accepted given its question and reference output. Agreement between the human decisions and Gemini 2.5 Flash reaches an overall Cohenâs Îșof 0.91, with domain-level values ranging from 0.88 to 0.93. The complete agreement table and annotation procedure are provided in Appendix D. Gemini 2.0 Flash is one of the evaluated systems, whereas Gemini 2.5 Flash is used for semantic eval- uation; these are distinct model versions. Because the judge may still exhibit wording or model-family preferences,M 2 is treated as a complementary sig- nal rather than the sole measure of performance. 6 Results and Analysis We organize the analysis around three questions. First, we examine how well current systems answer hybrid questions under both support-controlled and full-table settings. Second, we study whether re- placing redacted spans with plausible surrogate content consistently improves performance. Third, we analyze the failure modes underlying the ob- served results. Unless stated otherwise, the discus- sion focuses onM 3 , our relaxed combined answer evaluator. CompleteM 1 ,M 2 , andM 3 results are provided in Appendix B. 6.1 RQ1: How well do current systems reason over hybrid records? Figure 4 shows that supplying the required records does not make hybrid reasoning reliable. Across the 135 modelâmethodâdomain configurations, the medianM 3 is 26.5, 85 (63.0%) fall below 30, and none reaches 50, with a maximum of 48.6. The bot- tleneck is not retrieving an answer-bearing record but deciding which records satisfy the question and interpreting conditions expressed in narrative text. Averaged over models and methods, CFPB is least difficult at 25.4, then MIMIC at 22.6 and FIR at 19.9, where typed placeholders from post- hoc masking remove surface cues linking narrative mentions to structured fields. Method family matters more than backbone. Prompting averages 29.6 against 25.2 for table- based methods and 2.8 for retrieval, but none dom- inates: LtM leads on CFPB (36.8) and MIMIC (36.3), CoT on FIR (35.5), while zero-shot nearly matches CoT on CFPB (34.7 against 34.8) yet col- lapses to 13.7 on FIR. The strongest cells are 48.6 and 44.8 for Qwen3-Next with LtM on FIR and CFPB, and 40.1 for GPT-OSS-120B with Blend- SQL on MIMIC, the highest MIMIC score in the table. Table-based methods are not uniformly weaker: BlendSQL averages 34.8 on MIMIC, driven by semantic acceptance (M 2 of 65 to 72 for four of five backbones) rather than lexical overlap. Retrieval falls far below every other family, aver- aging 2.9M 3 for RAG and 2.7 for RAG+Rerank at k = 5, with the best cell at 6.1 and the weakest at 0.2. Performance is lowest on MIMIC (1.6 and 1.6), Gemini 2.0 Flash Qwen3 Next Llama 3.3 GPT-OSS 120B GPT-OSS 20B CoT PoT LtM Zero-shot Few-shot WEAVER BlendSQL RAG RAG+Rerank 43.534.331.034.034.7 13.433.910.15.929.2 9.648.632.833.328.5 11.15.69.812.030.2 42.437.332.834.722.8 19.033.619.531.112.7 23.710.714.323.34.2 3.04.34.25.34.8 3.63.53.33.86.1 FIR Gemini 2.0 Flash Qwen3 Next Llama 3.3 GPT-OSS 120B GPT-OSS 20B 24.627.526.528.236.4 17.621.523.421.330.9 37.336.538.334.634.9 23.933.826.726.631.5 23.125.825.826.232.3 20.324.421.826.817.8 35.435.335.940.127.5 0.21.00.62.24.2 0.81.50.41.53.6 MIMIC Gemini 2.0 Flash Qwen3 Next Llama 3.3 GPT-OSS 120B GPT-OSS 20B 39.738.837.130.727.8 34.028.225.030.029.9 43.644.833.632.629.3 40.539.434.930.928.0 35.628.037.432.727.1 33.814.026.436.423.1 29.725.621.738.528.1 3.53.52.43.10.9 3.23.12.33.11.3 CFPB 0 10 20 30 40 50 Instance-level harmonic score (%) Instance-level harmonic score (M 3 ) Figure 4:M 3 accuracy across models, methods, and domains under the support-controlled setting. Every input contains the records required to answer the question together with same-domain distractors. intermediate on CFPB (2.7 and 2.6), and highest on FIR (4.3 and 4.1). Reranking shifts each domain average by less than half a point and never over- takes the bi-encoder, so the cross-encoder does not recover support ranked below the retrieval cutoff. Appendix B.1 compares Gemini 2.5 Pro and Qwen 3-80B on a 30% sample of CFPB and FIR, where Gemini 2.5 Pro leads by 34.8 and 37.1 M 3 points under BlendSQL. Semantic acceptance again drives the gap, suggesting stronger back- bones can narrow these failures even though no model in the main sweep does so. Table 3 evaluates Gemini 2.0 Flash on the com- plete CFPB and FIR collections. Scores stay within the range observed under support-controlled con- texts, with no configuration exceeding 42.0. The bestM 3 results are 42.0 (CoT on CFPB) and 39.1 (LtM on FIR); no method dominates all three mea- sures on CFPB, although LtM does on FIR. The separation is informative: LtM leadsM 1 on CFPB (54.5) but its judge acceptance of 32.5 is the lowest of the three, leaving it behind CoT on the combined score, so partial lexical agreement does not guaran- tee a complete answer. BlendSQL is the most bal- anced method on FIR, with all three metrics within 0.2, suggesting executable decomposition helps or- ganize evidence over larger collections. Failures therefore arise both in locating relevant records and in interpreting the narrative conditions that deter- mine whether they qualify. 6.2 RQ2: Does restoring masked surface information consistently improve reasoning? Figure 5 compares native redacted records with aligned records carrying plausible surrogate sur- face forms. The surrogates do not recover the origi- nal private values, but approximate how additional visible information changes model behavior. Restoration does not yield uniform gains. Across the 21 method and domain configurations,M 3 in- creases in 12 cases, decreases in eight, and is un- changed in one, with mean changes of+7.6points for FIR,+3.7for MIMIC, andâ5.5for CFPB. The largest improvement, zero-shot prompting on FIR at+23.2points from 9.8 to 33.0, is specific to a weak starting point: the other three prompting strategies on FIR already exceed 31 under redaction and move by at most 1.8 points. PoT and WEAVER improve in all three domains, PoT by 14.8 points on FIR, 6.7 on CFPB, and 6.2 on MIMIC. The domains diverge sharply. On CFPB, mean M 1 rises by 5.3 points while meanM 2 falls by 7.6, so five of seven configurations lose ground, led by few-shot atâ16.2and CoT atâ15.8: restoration can increase surface overlap without improving se- mantic adequacy, and plausible replacements may introduce distracting entities into institutionally de- identified complaints. MIMIC moves the other way, with six of seven configurations improving and ev- ery prompting strategy gaining between 6.2 and 8.4 points. Method sensitivity is equally uneven, as BlendSQL gains 14.5 points on FIR while losing 11.6 on MIMIC and 10.4 on CFPB. Sensitivity to 1020304050 M 3 (%) CoT PoT LtM Zero-shot Few-shot WEAVER BlendSQL +0.1 +14.8 -1.4 +23.2 -1.8 +3.9 +14.5 FIR 1020304050 M 3 (%) +7.7 +6.2 +8.4 +7.7 +6.9 +0.4 -11.6 MIMIC 1020304050 M 3 (%) -15.8 +6.7 -2.5 -10.0 -16.2 +10.0 -10.4 CFPB Redacted vs surrogate-restored evidence: instance-level M 3 (Llama 3.3 70B) Restoration improvesRedacted stronger Figure 5: Effect of surrogate restoration on instance-levelM 3 for Llama 3.3 70B. Each arrow runs from the score under native redacted evidence (open circle) to the score under surrogate-restored evidence (arrowhead). Green arrows point right, where restoration improves performance; red arrows point left, where the native redacted condition is stronger. CFPBFIR MethodM 1 M 2 M 3 M 1 M 2 M 3 BlendSQL 30.8 42.6 35.8 33.2 33.4 33.3 CoT42.8 41.2 42.0 23.9 29.0 26.2 LtM54.5 32.5 40.7 35.8 43.0 39.1 Table 3: Long-table results with Gemini 2.0 Flash. evidence transformations is thus partly determined by how a method converts narrative content into record-level decisions. Appendix F contains the full scope of results including M1 and M2 results and their respective deltas. 6.3 RQ3: Where do current systems fail? To characterize failure modes, two of the authors independently labeled each majority-failed ques- tion in the 30% domain-stratified sample (seed 42) with a single dominant failure category from five options: evidence selection, predicate interpreta- tion, aggregation/count, entity grounding, and oth- er/structural. Disagreements were resolved by dis- cussion. Figure 6 summarizes the dominant failure mode for each majority-failed question in the 30% domain-stratified sample. The largest concentra- tion is evidence-selection error: it accounts for 72.5% of aggregation failures, 73.8% of ranking failures, 50.4% of trend failures, and 48.1% of co- occurrence failures. Filtering questions show a dif- ferent pattern, with predicate-interpretation errors forming the largest group (40.4%). These results in- dicate that failures generally occur while deciding which records satisfy the question and how narra- tive conditions should be interpreted, rather than during the final computation. Evidence selection Predicate interpretation Aggregation / count Entity grounding Other / structural Primary failure mode Filtering Aggregation Ranking Trend Co-occurrence Operation category 29.440.45.39.615.4 72.57.36.90.512.8 73.83.56.43.512.8 50.424.810.17.07.8 48.120.64.45.021.9 Dominant failure mode per question (30% seed-42 sample) 0 20 40 60 Share of majority-failed questions (%) Figure 6: Question-level distribution of primary failure modes across operation categories. Percentages are computed over majority-failed questions in the 30% seed-42 sample; each question contributes one dominant failure label. 7 Conclusion We introduced CIPHER, a benchmark for language-modelreasoningoverprivacy- constrained records that combine structured fields and free-text narratives. It covers three real world domains and five analytical operation of aggregation, filtering, co-occurence, trend study- ing, and ranking types under multiple evidence transformations. Our question-level analysis shows that systems fail mainly when selecting records and interpreting predicates, even when supporting evidence is present. Privacy transformations have method- and domain-dependent effects: they may remove necessary cues or suppress misleading ones. CIPHER provides executable supervision. By isolating record selection and predicate interpretation as the dominant failure modes, CIPHER provides a foundation for developing models that reason reliably over sensitive records. 8 Limitations CIPHER evaluates how privacy-related transforma- tions affect LLM reasoning over hybrid records, comparing redacted and surrogate-restored evi- dence across FIR, CFPB, and MIMIC. Three lim- itations bound the conclusions we draw. First, the redacted-versus-restored results (Figure 5) use Llama 3.3 70B only; because restoration already proves method- and domain-dependent, it may be backbone-dependent as well, and establishing this requires repeating the comparison across models. Second, our baseline and setting coverage is par- tial: we do not evaluate H-Star or Binder (Ab- hyankar et al., 2025b; Cheng et al., 2023), nor large reasoning models, which may bring stronger se- mantic understanding than the baselines here, and Appendix B.1 gives only a smaller-scale indica- tion; the full-table setting (Table 3) is likewise re- stricted to one model and two domains, as scaling it further requires specialized pipelines. Third, CI- PHER targets five analytic operation types with nu- meric, short-categorical, and list answers, and does not cover open-ended explanation, justification, or long-form summarization, while template-level re- view (Section 4.3) does not guarantee that every instantiated question is free of residual errors. In future work, we plan to extend this analysis along these dimensions and to construct methods that explicitly improve reasoning over redacted evi- dence. 9 Ethics Statement Data sourcing and intended use.CIPHER is re- leased as a diagnostic research benchmark intended to improve the safety, robustness, and accountabil- ity of hybrid textâknowledge reasoning systems operating on real-world data. All datasets used in this work are derived from publicly available, de- identified, or appropriately licensed sources, includ- ing government-released FIR records, the CFPB Consumer Complaint Database, and the MIMIC- IV clinical dataset accessed under PhysioNetâs data use agreement. Any personally identifiable information (PII) present in the original sources was systematically handled using established de- identification or redaction pipelines, and no new personal identifiers were introduced at any stage of dataset construction. The redacted and unredacted settings are explicitly treated as evaluation con- ditions rather than deployment recommendations, and we emphasize that strong benchmark perfor- mance should not be interpreted as certification for real-world use in high-stakes legal, financial, or medical settings. License and release. The benchmark will be re- leased under a research-only, non-commercial license, with access conditioned on agreement to ethical usage terms prohibiting adversarial training, generation of synthetic sensitive records, or deploy- ment without qualified human oversight. Release materials will include a datasheet documenting data sources, preprocessing, redaction procedures, annotation protocols, known biases, and limita- tions, as well as all evaluation code, prompts, and templates for full reproducibility. Bias and scope.We acknowledge that the bench- mark reflects the institutional, geographic, and lin- guistic characteristics of its source domains, and may therefore inherit biases present in those data. These limitations are documented transparently, and we encourage future extensions that broaden coverage across jurisdictions, languages, and popu- lations. In future work, we plan to further explore these dimensions at a more granular level. Human and AI involvement.Human validation of question templates and annotations was con- ducted on a voluntary basis by domain-informed re- searchers based in the United States and India (for FIRs) to ensure semantic correctness and realism, without exposure to sensitive personal identifiers. Large language models were used only in a lim- ited, controlled manner for assistance with data nor- malization, question instantiation, and manuscript editing; AI-assisted tools were used selectively for language refinement, grammatical improvements, and editorial support, while all core ideas, anal- yses, and scientific contributions remain entirely the authorsâ own. Overall, CIPHER is intended to support responsible AI research by making privacy- sensitive failure modes visible and by promoting more cautious, transparent evaluation of hybrid rea- soning systems in real-world settings. Acknowledgments This work was conducted with the Complex Data Reasoning and Analysis Lab (CoRAL) at Arizona State University, led by Vivek Gupta. We thank the members of CoRAL for their valuable insights and suggestions. We are also grateful to the participants who reviewed and validated the data for their time, careful assessment, and helpful feedback. References Nikhil Abhyankar, Purvi Chaurasia, Sanchit Kabra, Ananya Srivastava, Vivek Gupta, and Chandan K. Reddy. 2025a. Rust-bench: Benchmarking llm rea- soning on unstructured text within structured tables. Preprint, arXiv:2511.04491. Nikhil Abhyankar, Vivek Gupta, Dan Roth, and Chan- dan K. Reddy. 2025b. H-star: Llm-driven hybrid sql-text adaptive reasoning on tables.Preprint, arXiv:2407.05952. Anti-Corruption Bureau, Government of Haryana. 2024. Ambala range, anti-corruption bureau, haryana. https://acb.haryana.gov.in/division/ ambala-range/. Accessed: January 2026. Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, and 12 others. 2020.Lan- guage models are few-shot learners.Preprint, arXiv:2005.14165. Wenhu Chen, Ming-Wei Chang, Eva Schlinger, William Wang, and William W. Cohen. 2021a. Open ques- tion answering over tables and text.Preprint, arXiv:2010.10439. Wenhu Chen, Xueguang Ma, Xinyi Wang, and William W. Cohen. 2023.Program of thoughts prompting: Disentangling computation from rea- soning for numerical reasoning tasks.Preprint, arXiv:2211.12588. Wenhu Chen, Hanwen Zha, Zhiyu Chen, Wenhan Xiong, Hong Wang, and William Wang. 2021b. Hybridqa: A dataset of multi-hop question answering over tabular and textual data. Preprint, arXiv:2004.07347. Zhoujun Cheng, Tianbao Xie, Peng Shi, Chengzu Li, Rahul Nadkarni, Yushi Hu, Caiming Xiong, Dragomir Radev, Mari Ostendorf, Luke Zettlemoyer, Noah A. Smith, and Tao Yu. 2023. Binding lan- guage models in symbolic languages.Preprint, arXiv:2210.02875. ConsumerFinancialProtectionBureau.2026. Consumercomplaintdatabase.https: //w.consumerfinance.gov/data-research/ consumer-complaints/#get-the-data.Ac- cessed: January 2026. Yao Fu, Hao Peng, Ashish Sabharwal, Peter Clark, and Tushar Khot. 2023. Complexity-based prompting for multi-step reasoning. Preprint, arXiv:2210.00720. Gemini Team. 2025a. Gemini 2.5: Pushing the fron- tier with advanced reasoning, multimodality, long context, and next generation agentic capabilities. Preprint, arXiv:2507.06261. Gemini Team. 2025b. Gemini: A family of highly capa- ble multimodal models. Preprint, arXiv:2312.11805. Parker Glenn, Parag Pravin Dakle, Liang Wang, and Preethi Raghavan. 2024. Blendsql: A scalable dialect for unifying hybrid question answering in relational algebra. Preprint, arXiv:2402.17882. Alistair E. W. Johnson, Lucas Bulgarelli, Lu Shen, Alvin Gayles, Ayad Shammout, Steven Horng, Tom J. Pol- lard, Sicheng Hao, Benjamin Moody, Brian Gow, Li-wei H. Lehman, Leo A. Celi, and Roger G. Mark. 2023. Mimic-iv, a freely accessible electronic health record dataset. Scientific Data, 10:1â18. Rohit Khoja, Devanshu Gupta, Yanjie Fu, Dan Roth, and Vivek Gupta. 2025. Weaver: Interweaving SQL and LLM for table reasoning. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 28282â28308, Suzhou, China. Association for Computational Linguistics. Llama Team. 2024.The llama 3 herd of models. Preprint, arXiv:2407.21783. Omri Mendels, Coby Peled, Nava Vaisman Levy, Sharon Hart, Tomer Rosenthal, Limor Lahiani, et al. 2018. Microsoft Presidio: Context-aware, plug- gable and customizable PII anonymization service for text and images.https://microsoft.github. io/presidio. Microsoft. Linyong Nan, Chiachun Hsieh, Ziming Mao, Xi Victoria Lin, Neha Verma, Rui Zhang, Wojciech Kry Ì sci Ì nski, Nick Schoelkopf, Riley Kong, Xiangru Tang, Murori Mutuma, Ben Rosand, Isabel Trindade, Renusree Bandaru, Jacob Cunningham, Caiming Xiong, and Dragomir Radev. 2021. Fetaqa: Free-form table ques- tion answering. Preprint, arXiv:2104.00369. OpenAI. 2025. gpt-oss-120b & gpt-oss-20b model card. Preprint, arXiv:2508.10925. OpenAI. 2026. Openai gpt-5 system card. Preprint, arXiv:2601.03267. Panupong Pasupat and Percy Liang. 2015. Compo- sitional semantic parsing on semi-structured tables. Preprint, arXiv:1508.00305. Qwen Team. 2025. Qwen3 technical report. Preprint, arXiv:2505.09388. Peter Shaw, Ming-Wei Chang, Panupong Pasupat, and Kristina Toutanova. 2021. Compositional general- ization and natural language variation: Can a se- mantic parsing approach handle both?Preprint, arXiv:2010.12725. Aofeng Su, Aowen Wang, Chao Ye, Chen Zhou, Ga Zhang, Gang Chen, Guangcheng Zhu, Haobo Wang, Haokai Xu, Hao Chen, Haoze Li, Haoxuan Lan, Jiaming Tian, Jing Yuan, Junbo Zhao, Jun- lin Zhou, Kaizhe Shou, Liangyu Zha, Lin Long, and 14 others. 2024. Tablegpt2: A large multi- modal model with tabular data integration. Preprint, arXiv:2411.02059. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. 2023. Chain-of-thought prompting elic- its reasoning in large language models. Preprint, arXiv:2201.11903. Shitao Xiao, Zheng Liu, Peitian Zhang, Niklas Muen- nighoff, Defu Lian, and Jian-Yun Nie. 2024. C-pack: Packed resources for general chinese embeddings. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Infor- mation Retrieval, SIGIR â24, page 641â649, New York, NY, USA. Association for Computing Machin- ery. Tao Yu, Rui Zhang, Kai Yang, Michihiro Yasunaga, Dongxu Wang, Zifan Li, James Ma, Irene Li, Qingn- ing Yao, Shanelle Roman, Zilin Zhang, and Dragomir Radev. 2019. Spider: A large-scale human-labeled dataset for complex and cross-domain semantic pars- ing and text-to-sql task. Preprint, arXiv:1809.08887. Victor Zhong, Caiming Xiong, and Richard Socher. 2017.Seq2sql: Generating structured queries from natural language using reinforcement learning. Preprint, arXiv:1709.00103. Denny Zhou, Nathanael SchĂ€rli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuurmans, Claire Cui, Olivier Bousquet, Quoc Le, and Ed Chi. 2023. Least-to-most prompting enables complex reasoning in large language models.Preprint, arXiv:2205.10625. Fengbin Zhu, Wenqiang Lei, Youcheng Huang, Chao Wang, Shuo Zhang, Jiancheng Lv, Fuli Feng, and Tat-Seng Chua. 2021. Tat-qa: A question answering benchmark on a hybrid of tabular and textual content in finance. Preprint, arXiv:2105.07624. Fengbin Zhu, Ziyang Liu, Fuli Feng, Chao Wang, Moxin Li, and Tat-Seng Chua. 2024. Tat-llm: A spe- cialized language model for discrete reasoning over tabular and textual data. Preprint, arXiv:2401.13223. Appendix This appendix contains the prompts used for all prompting-based baselines, the information extrac- tion prompt, the LLM-as-a-Judge prompt used dur- ing evaluation, and the complete benchmark re- sults. A Prompt Templates A.1 Prompt A: Chain-of-Thought You are a data analysis assistant for tabular <domain> data. You will be given a table of <domain> and a question about that table. Each row is one FIR and the first row contains column names. COLUMN_DEFINITIONS The main narrative field is <unstructured narrative>. Some fields are unstructured text and some tokens are redacted like <PERSON_000001>; treat them as opaque strings. Your task: Think through the steps and think carefully. Display the final answer in a simple one word or phrase format A.2 Prompt B: Few-Shot You are a data analysis assistant that learns from examples. You will be given several examples consisting of: - a small table of <domain records> - a natural language question about that table, - the correct answer. Then you will get a new table and question. Answer it in the same style. Each row is one <domain report> and the first row contains column names. COLUMN_DEFINITIONS The main narrative field is <unstructured narrative>. Other columns provide structured metadata such as station, date, year, sections, and parties involved. Your task on the NEW question: 1. Read and interpret the table and question. 2. Identify which rows and fields are relevant, using both the narrative and structured columns. 3. Produce a concise answer in the format the question requires. This is seen in the question 4. If the question cannot be answered from the table, reply with NONE or [] as appropriate. 5. Unless the question requires it, do not include the narrative. Only include the relevant answers for every question noted in answers. Here is two examples: Example 1 Table (with header row): row id,police station,StatementOfComplaint 1,BEHAL,"The complainant reports that his daughter left home in the evening to go to the market and did not return. The family searched the village and bus stand but she is still missing. The complainant suspects that an unknown person has kidnapped her." 2,BEHAL,"The complainant reports that he slipped from his bicycle on the road and injured his leg. He was taken to the government hospital for treatment. No person is missing or unaccounted for in this incident." Question: How many FIRs registered at Behal police station involve a missing person case? Answer: "1" ... A.3 Prompt C: Least-to-Most You are a data analysis assistant for tabular FIR data. You will be given a table of <domain> and a question about that table. Each row is one FIR and the first row contains column names. COLUMN_DEFINITIONS The main narrative field is <unstructured narrative> Use it as the primary source of information about the incident, and use other columns when the question refers to them. Use least-to-most reasoning for each question: 1. Under Decomposition: break the question into a numbered list of simpler sub-questions (e.g., identifying relevant filters, fields, and computations). 2. Under Solving: answer each sub-question in order, referring to rows or row_ids and the relevant columns. 3. Under Final answer: output a single concise answer in the format the question requires (e.g., an integer, a specific value, a list of row_ids, or a short text ). - If the question truly cannot be answered from the table, use Final answer: NONE or Final answer: []. - Unless the question requires it, do not include the narrative. Only include the relevant answers for every question noted in answers. Your output MUST follow this structure exactly: Decomposition: 1. ... 2. ... ... Solving: 1. ... 2. ... ... Final answer: <your final answer> A.4 Prompt D: Program-of-Thought You are an expert at translating questions about <domain> tables into small Python programs. The data is stored in a conceptual table called temp. Column names match the header row. COLUMN_DEFINITIONS Most questions require: - Reading and interpreting the narrative field. - Returning whatever value the question asks for For each question, write a short Python 3 program that returns the information requested. Example: row_id,police_station, StatementOfComplaint_pseudo 1,BEHAL,"The complainant reports that his daughter left home in the evening to go to the market and did not return. The family searched the village and bus stand but she is still missing. The complainant suspects that an unknown person has kidnapped her." Question: How many FIRs registered at Behal police station involve a missing person case? Correct Python program of thought: row = [(1, BEHAL)] print(row) A.5 Prompt E: Information Extraction You are an information extraction assistant. Your task: Read the FIR / CFPB complaint / MIMIC narrative given to you as input and extract structured information into a single JSON object that follows the schema and data types defined below. GENERAL INSTRUCTIONS - Output MUST be valid JSON. - Output ONLY the JSON object (no explanations or extra text). - Use exactly the field names and structure specified below. - Do not add new fields or rename any fields. - If a data point is not mentioned or cannot be inferred with reasonable certainty, set it to: - null for scalar fields (string, number), - false for boolean flags (unless the narrative clearly implies true), - [] for lists/arrays. Do not hallucinate or guess values that are not grounded in the text. A.6 Prompt F: LLM-as-a-Judge You are an LLM-as-a-judge evaluating answers to questions. You will receive a: "id": A string table identifier for the row (for example: "pmet1", "pmet2"). "question": The question asked. "gold answer": The ground truth. "candidates": A dictionary of model answers to evaluate. The keys are model names such as "gemini answer", "llama answer", "gptoss answer", and "qwen answer". Your task: For EACH candidate in EACH case, decide if it is very close to GOLD ANSWER. However, please be lenient. If the answers are not super close, try to see if they are partially correct. Criteria for "yes": Counts perfectly match or are off by a margin of 5% Candidate includes most gold info for IDs if requested. For example, if the gold is [283, 285] and the predicted answer is [1,2,283], say "yes" since it's partially correct. Same top-ranked entities for "highest/top" questions or entities in order. Gold is null/empty/zero and Candidate says "NONE", "no answer", [] empty brackets, 0 or anything where the gold answer has nothing and the predicted answers gravitate towards that. Incidents or strings make sense semantically, e.g. "Blunt force trauma", "deep head injury". JSONs perfectly match Criteria for "no": IDs or dates contradict the gold completely. Answer is irrelevant or hallucinated especially when a gold label is empty. Answer does not make semantic sense to the question being asked. Discrepancies between counts is too hig B Complete Benchmark Results Table 5 reports the complete evaluation results across all datasets, methods, and models, providing the full quantitative basis for the analyses in Sec- tion 6.M 1 denotes the lexical overlap score,M 2 the LLM-as-a-Judge semantic acceptance rate, and M 3 the mean over instances of the per-question har- monic mean ofM 1 andM 2 . Tab. denotes hybrid table-reasoning methods, Ret. retrieval-augmented methods, and Spec. table-specialist models; the latter are complete systems rather than methods applied to a backbone, so they report a single score per dataset spanning the model columns. Boldface marks the best score for a given metric within each dataset and method category. The method families separate far more sharply than the models do. Averaged over all datasets and backbones, prompting reaches 29.6, hy- brid table reasoning 25.2, and retrieval only 2.8. Within prompting, least-to-most decomposition is strongest at 34.6, followed by chain-of-thought at 33.0 and few-shot at 30.9, while zero-shot (25.7) and program-of-thought (23.6) trail both. The two table-based methods are closer than the prompting spread, BlendSQL averaging 26.3 against 24.1 for WEAVER, and BlendSQLâs advantage is concen- trated on MIMIC, where high semantic acceptance (M 2 between 65 and 72 for four of five backbones) lifts it to between 27.5 and 40.1. Its best cell there, 40.1 with GPT-OSS-120B, is the highest MIMIC score in the table. Retrieval is weakest everywhere. RAG (2.9) and RAG+Rerank (2.7) are indistin- guishable, and the lowest cell in the table is 0.2 for RAG with Gemini 2.0 Flash on MIMIC. By contrast, the five general-purpose models span barely two and a half points: Qwen3-next leads at 23.9, followed by GPT-OSS-120B at 23.3, Gemini 2.0 Flash at 22.8, GPT-OSS-20B at 21.8, Dataset MethodGemini 2.5 ProQwen 3-80B M 1 M 2 M 3 M 1 M 2 M 3 CFPB BlendSQL52.4% 77.0% 62.4%41.3% 20.7% 27.6% CoT51.0% 73.0% 60.0%55.3% 43.0% 48.4% FIR BlendSQL59.0% 87.0% 70.3%29.3% 38.4% 33.2% CoT57.0% 86.0% 68.6%23.8% 39.6% 29.7% Table 4: Gemini 2.5 Pro vs. Qwen 3. Best in-slot results are highlighted in bold. Results are on a 30% sample. and Llama 3.3 at 21.4. Scaling within a family is not consistently helpful: GPT-OSS-20B exceeds GPT-OSS-120B on MIMIC (24.3 against 23.1) while falling behind on FIR (19.2 against 20.4) and CFPB (21.7 against 26.4). The choice of reasoning strategy therefore matters considerably more than the choice of backbone. Datasets remain ordered as in the main results, with CFPB at 25.4, MIMIC at 22.6, and FIR at 19.9. Method stability varies as much as method qual- ity. Chain-of-thought is the most consistent across backbones, with a spread of at most 12.5 points within a dataset, whereas least-to-most swings 39.0 points on FIR alone (9.6 for Gemini 2.0 Flash against 48.6 for Qwen3-next), program-of-thought swings 28.0, and zero-shot 24.6. The strongest configurations overall are Qwen3-next with least- to-most on FIR (48.6) and CFPB (44.8), and Gem- ini 2.0 Flash with least-to-most on CFPB (43.6). No configuration among the 135 reaches 50, while 85 remain below 30 and the median is 26.5. The table-specialist models are strong on exactly one dataset each, TableGPT2 on FIR (16.7) and TAT- LLM 70B on CFPB (18.3), and fall to between 4.2 and 11.1 elsewhere; even at their best they sit below every prompting-family average on the dataset in question, with the single exception of TableGPT2 on FIR against zero-shot (13.7). B.1 Gemini 2.5 Pro vs. Qwen 3-80B Table 4 reports an additional experiment on a ran- dom 30% sample of CFPB and FIR, comparing Gemini 2.5 Pro with Qwen 3-80B under the two method families that perform best in the main ta- ble, BlendSQL and CoT. Because this experiment uses a sample rather than the full evaluation set, its scores are not directly comparable with those in Table 5. Within the comparison, Gemini 2.5 Pro is consistently stronger on both datasets and for both methods. The margin is widest for BlendSQL, where Gemini 2.5 Pro reaches 62.4 on CFPB and 70.3 on FIR against 27.6 and 33.2 for Qwen 3- 80B. Semantic acceptance drives this gap, as Gem- ini 2.5 Pro is judged correct on 77.0% of CFPB items and 87.0% of FIR items whereas Qwen 3- 80B reaches only 20.7% and 38.4%. Qwen 3-80B is competitive only on CFPB with CoT, where its lexical overlap of 55.3 exceeds the 51.0 obtained by Gemini 2.5 Pro, but it remains behind on both the judge-based and the combined metric. Stronger semantic matching and final-answer correctness, rather than surface phrasing, therefore account for most of the difference. C Annotation and Validation Guidelines This appendix describes the annotation protocol used to validate question templates, instantiated QA pairs, and gold answers in CIPHER. The ob- jective of annotation is to ensure that all questions are natural, answerable from the table content, and supported by correct structured and unstructured evidence, while respecting privacy constraints. Annotator roles and qualifications. Annota- tions were performed by NLP researchers with prior experience in dataset validation and famil- iarity with semi-structured data. Domain-specific context (legal, clinical, and consumer finance) was provided through short primers. All participation was voluntary, and annotators were instructed to treat the benchmark as an evaluation artifact rather than a decision-support resource. The annotators worked on a voluntary basis seeing the good cause of the paper. Annotation inputs.For each template and its in- stantiated examples, annotators were shown: (i) the full semi-structured table, (i) the natural-language question, (i) the instantiated parameters, (iv) the corresponding SQL query, and (v) the gold answer produced by executing the query on the table. Annotation procedure. For each instantiated QA pair, annotators followed a fixed sequence: 1.Assess whether the question is natural and reflects a realistic analytical intent. 2.Verify that sufficient evidence exists in the table (structured fields, narrative text, or both) to support an answer. 3.Check that the SQL query correctly retrieves the gold answer from the table. Each instance was assigned one of three labels: Approve (question and answer fully supported), DatasetCat.Method Gemini 2.0 FlashQwen3-nextLlama 3.3GPT-OSS-120BGPT-OSS-20B M 1 M 2 M 3 M 1 M 2 M 3 M 1 M 2 M 3 M 1 M 2 M 3 M 1 M 2 M 3 FIR Prompting CoT48.5 63.6 43.544.4 46.8 34.340.2 48.9 31.039.9 48.4 34.046.5 58.0 34.7 PoT42.8 21.1 13.456.4 40.8 33.939.8 19.3 10.121.0 17.05.943.9 49.5 29.2 LtM24.0 16.89.661.6 60.6 48.643.2 47.2 32.839.6 45.6 33.344.2 48.1 28.5 Zero-shot15.3 30.3 11.19.418.45.617.5 23.99.816.6 24.4 12.044.4 49.9 30.2 Few-shot47.4 61.7 42.443.4 54.4 37.341.1 48.9 32.839.5 48.7 34.742.0 37.4 22.8 Tab. WEAVER31.7 27.2 19.040.5 46.5 33.629.0 30.2 19.535.2 42.4 31.112.9 19.5 12.7 BlendSQL32.6 45.6 23.718.2 20.9 10.725.3 32.8 14.331.3 44.9 23.323.8 38.94.2 Ret. RAG11.3 11.93.09.013.64.312.18.44.29.914.65.312.8 10.64.8 RAG+Rerank12.4 11.03.68.812.83.511.37.83.38.512.53.817.0 11.86.1 Spec. TableGPT2M 1 = 20.5, M 2 = 41.3, M 3 = 16.7 TAT-LLM 70BM 1 = 4.1, M 2 = 14.3, M 3 = 4.2 MIMIC Prompting CoT21.7 46.0 24.625.0 49.4 27.524.2 48.6 26.525.8 48.6 28.238.8 57.0 36.4 PoT15.7 37.0 17.619.9 39.8 21.521.7 44.4 23.419.4 42.2 21.337.3 53.4 30.9 LtM41.6 51.9 37.341.5 50.9 36.538.7 49.7 38.335.8 47.7 34.640.8 52.4 34.9 Zero-shot21.3 45.7 23.935.3 52.8 33.823.8 49.2 26.723.6 49.4 26.635.7 52.4 31.5 Few-shot20.5 46.2 23.123.5 51.2 25.823.7 49.0 25.823.6 49.8 26.235.1 55.5 32.3 Tab. WEAVER18.3 43.6 20.322.7 47.4 24.420.4 45.4 21.824.5 50.2 26.816.3 37.8 17.8 BlendSQL31.7 65.1 35.431.7 66.0 35.332.2 65.4 35.936.1 71.8 40.128.6 43.2 27.5 Ret. RAG8.41.00.22.45.61.03.02.00.63.85.82.217.05.04.2 RAG+Rerank7.61.80.83.46.61.53.02.60.53.75.01.520.24.43.6 Spec. TableGPT2M 1 = 7.9, M 2 = 31.6, M 3 = 7.6 TAT-LLM 70BM 1 = 11.1, M 2 = 47.4, M 3 = 11.1 CFPB Prompting CoT53.4 53.5 39.752.0 55.4 38.849.0 47.4 37.139.2 46.4 30.740.4 40.2 27.8 PoT39.2 45.4 34.035.2 41.1 28.230.9 36.4 25.037.8 42.9 30.042.8 52.0 29.9 LtM66.0 51.3 43.666.3 52.5 44.853.9 41.1 33.643.1 46.0 32.644.0 42.1 29.3 Zero-shot52.7 53.1 40.554.1 51.0 39.452.3 44.3 34.940.5 47.3 30.942.3 39.3 28.0 Few-shot47.3 51.9 35.639.5 44.6 28.047.3 50.1 37.441.2 49.2 32.741.3 40.0 27.1 Tab. WEAVER42.7 45.1 33.828.3 20.0 14.035.9 40.0 26.443.5 53.5 36.425.6 31.0 23.1 BlendSQL39.7 41.1 29.740.7 36.4 25.632.9 35.8 21.744.8 57.2 38.543.5 40.7 28.1 Ret. RAG16.86.83.515.97.23.516.26.42.415.37.03.118.01.00.9 RAG+Rerank15.46.43.214.56.03.114.35.22.314.56.83.119.91.31.3 Spec. TableGPT2M 1 = 6.4, M 2 = 7.8, M 3 = 4.4 TAT-LLM 70BM 1 = 38.4, M 2 = 27.0, M 3 = 18.3 Table 5: Results on CIPHER.M 1 is token-overlap score,M 2 is judge acceptance rate, andM 3 is the mean instance-level harmonic score. Rewrite (minor fixes needed, e.g., SQL or word- ing), or Remove (unsupported, ambiguous, or un- realistic). Quality control and agreement. A subset of templates and instances was independently re- viewed by multiple annotators to measure inter- annotator agreement. CohenâsÎșwas used for cate- gorical decisions, and disagreements were resolved through adjudication. Systematic issues identified during review triggered template-level revisions. Privacy handling. Annotators worked exclu- sively with de-identified or redacted data unless explicitly authorized. Any instance suspected of containing residual personally identifiable informa- tion was flagged and excluded until re-redaction was performed. Annotators were prohibited from attempting re-identification or inference of masked entities. Deliverables. For each annotated item, anno- tators provided: (i) the final label (Approve/Re- work/Remove), (i) a brief justification for Rework or Remove decisions, and (i) suggested correc- tions where applicable. All decisions were logged with timestamps for auditability. Reproducibility.The dataset release includes the annotation rubric, validation scripts, inter-annotator agreement statistics, and documentation describing the full annotation workflow. These materials en- able independent reproduction and extension of the benchmark while maintaining ethical and privacy safeguards. FamilyMethod CFPBFIRMIMIC M 1 M 2 M 3 M 1 M 2 M 3 M 1 M 2 M 3 Prompting CoT50.4 (+1.4)29.7 (-17.7) 21.3 (-15.8)42.9 (+2.7)43.6 (-5.3)31.0 (+0.0)32.0 (+7.8)51.6 (+3.0)34.2 (+7.7) PoT47.6 (+16.7) 47.6 (+11.2) 31.7 (+6.7)38.0 (-1.8)35.8 (+16.5) 24.9 (+14.8)31.3 (+9.6) 55.9 (+11.5) 29.6 (+6.2) LtM57.6 (+3.7)38.9 (-2.2)31.1 (-2.5)46.4 (+3.2)40.9 (-6.3)31.3 (-1.5)49.8 (+11.1) 63.2 (+13.5) 46.6 (+8.3) Zero-shot51.0 (-1.3)33.2 (-11.1) 24.8 (-10.1)43.5 (+26.0) 47.3 (+23.4) 33.0 (+23.2)31.8 (+8.0)51.4 (+2.2)34.4 (+7.7) Few-shot49.0 (+1.7)30.3 (-19.8) 21.2 (-16.2)42.4 (+1.3)43.8 (-5.1)31.0 (-1.8)30.7 (+7.0)52.4 (+3.4)32.8 (+7.0) Table-based WEAVER44.1 (+8.2)48.1 (+8.1) 36.4 (+10.0)32.8 (+3.8)33.9 (+3.7)23.4 (+3.9)20.8 (+0.4)52.2 (+6.8)22.2 (+0.4) BlendSQL39.9 (+7.0)14.1 (-21.7) 11.3 (-10.4)43.5 (+18.2) 38.2 (+5.4) 28.8 (+14.5)23.6 (-8.6)39.3 (-26.1) 24.3 (-11.6) Table 6: Performance under surrogate-restored evidence on CIPHER, with redacted-to-restored deltas for Llama 3.3 70B. D LLM-as-a-Judge Validation Many CIPHER answers are semantically equiva- lent while differing in surface form, so a purely lexical criterion would reject predictions that a reader would accept. We therefore use an LLM-as- a-Judge forM 2 . The judge is instructed to grant credit when a candidate conveys the reference an- swer under a different wording, ordering, or level of formatting detail, and to withhold it when the candidate is hallucinated, contradicts the reference, or reports quantities that do not match. The com- plete instructions are given in Appendix A.6. To establish that this signal is reliable, two NLP researchers independently reviewed 750 judged out- puts, drawn evenly across the three domains at 250 per domain, and recorded whether they agreed with the decision produced by Gemini 2.5 Flash. Table 7 reports the resulting agreement. CohenâsÎșis 0.91 overall and remains between 0.88 and 0.93 in every domain, so the judge tracks human acceptance deci- sions closely even though the three domains differ substantially in narrative length and terminology. One evaluated system, Gemini 2.0 Flash, be- longs to the same model family as the judge. The two are distinct checkpoints, and Gemini 2.5 Flash is the substantially more capable of the pair. Agree- ment is also uniformly high across all three do- mains rather than concentrated on the outputs of any single system, which indicates that the judge is not simply favouring answers written by a related model. We nonetheless treatM 2 as one signal among three rather than as a standalone measure of correctness. Table 7: Agreement between the human annotators and the LLM-as-a-Judge decisions, reported by domain. Domain# PairsCohenâs Îș CFPB2500.91 FIR2500.88 MIMIC2500.93 Overall7500.91 Table 8: Agreement between the two reviewers on the quality of the 895 surrogate replacements, reported per criterion. CriterionAgreement (%) Fluency and naturalness84.4 Semantic consistency87.1 Matching intent94.6 E Quality of the Surrogate De-redaction In order to gauge the efficacy of redaction on CFPB and MIMIC which do not come with their own unredacted spans, surrogates are generated with GPT-5.5 (OpenAI, 2026), and each replacement is logged in a fixed JSON record together with the affected field, the semantic type of the masked value, and the substituted text. The resulting 895 unique replacements make the restored variants reproducible from the redacted records, and they also drive the transformation-style ablation, since the same logged metadata can be reapplied as entity masking, span masking, or complete redaction. A surrogate is only useful if it reads as ordinary domain text, stays consistent with the rest of the record, and does not alter what the narrative as- serts. Two NLP experts therefore reviewed all 895 replacements against three criteria. Fluency and naturalness asks whether the substituted span is id- iomatic for the domain. Semantic consistency asks whether the surrogate conflicts with any other struc- tured field or narrative mention in the same record. Matching intent asks whether the sentence contain- ing the surrogate still supports the same reading as the redacted original. Table 8 reports reviewer agreement on each cri- terion. Agreement is highest for matching intent at 94.6%, which is the property that matters most for our purpose, since a surrogate that preserves the meaning of the span keeps the reference answer and its supporting records unchanged across con- ditions. The two lower figures, 84.4% for fluency and naturalness and 87.1% for semantic consis- tency, reflect the more subjective nature of those judgements, where reviewers differed mainly on whether a plausible but unusual surrogate should count as idiomatic. These surrogates remain con- textually plausible substitutes rather than recovered private values, so the restored condition measures sensitivity to the information visible to the model and not the recovery of the underlying record. F Redacted vs Unredacted table Table 6 contains the M1, M2, and M3 (harmonic mean) of all the results using Llama-3.3 70bn in the surrogate-entity vs redacted experiment per- formed with FIR, CFPB, and MIMIC. Green in- dicates that surrogate restoration improves perfor- mance; red indicates that the native redacted condi- tion is stronger. Aggregating within domains, restoration is domain-dependent: meanâM 3 is+7.6for FIR, +3.7for MIMIC, andâ5.5for CFPB. Of the 21 methodâdomain cells, 12 improve, 8 decline, and 1 (CoT/FIR) is unchanged at +0.0. The pattern is driven byM 2 , notM 1 : lexical overlap rises in 18 of 21 cells regardless of out- come, but semantic acceptance moves in both di- rections and disagrees withM 1 âs sign in 8 cells (mostly CFPB). In each of those 8,M 3 tracksM 2 , notM 1 â exactly as intended, sinceM (i) 3 â 0 when the judge rejects regardless of surface over- lap. CFPB is the clearest case: CoT, LtM, and Few-shot all gain lexical overlap while losing 2â20 points of judge acceptance, andM 3 shows the loss. Method sensitivity varies independently of ac- curacy. PoT and WEAVER improve in all three domains (PoT:+6.7/ + 14.8/ + 6.2; WEAVER: +10.0/ + 3.9/ + 0.4), while BlendSQL is the most volatile â+14.5on FIR butâ11.6MIMIC and â10.4CFPB. CoT and Few-shot post the two largest single losses in the table, both on CFPB (â15.8,â16.2). The single biggest swing over- all is zero-shot on FIR (+23.2), best explained by FIRâs placeholders being unusually uninformative when redacted, rather than a general zero-shot ef- fect. Overall, restoration reliably boosts surface over- lap, but its effect on judged correctness is domain- and method-dependent â net positive for FIR and MIMIC, net negative for CFPB, and safest for decomposition-style methods (WEAVER) versus single-pass or SQL-heavy ones (CoT, Few-shot, BlendSQL).