Paper deep dive
Privacy-Preserving Data Drift Detection and Recovery for Large-Scale LLM Applications via Proxy Representations
Michael Levit, Josh Ledgard, Haoyu Dong, Vishwas Suryanarayanan, Eyal Kolman, Sharon Tan, Qiang Gan, Vishal Chowdhary
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 94%
Last extracted: 8/12/2026, 1:10:44 AM
Summary
The paper introduces ProxyDrift, a privacy-preserving framework for detecting data drift and generating synthetic evaluation datasets for large-scale LLM applications. By using non-PII proxy representations derived from LLM-based classification of user interactions, the system measures drift via a redundancy-aware alignment score and generates synthetic data using a Chow-Liu tree-based conditional sampler. The framework is deployed in a major cloud productivity suite serving hundreds of millions of users, demonstrating high alignment with production traffic and strong roundtrip consistency.
Entities (6)
Relation Signals (5)
ProxyDrift → developedby → Microsoft Corporation
confidence 99% · Michael Levit ... Microsoft Corporation
ProxyDrift → appliedto → LLM
confidence 95% · Privacy-Preserving Data Drift Detection and Recovery for Large-Scale LLM Applications
ProxyDrift → uses → Chow-Liu Tree
confidence 95% · Our approach operates entirely on non-PII proxy representations... a conditional sampler that generates synthetic proxies... Chow–Liu tree-based conditional sampler
ProxyDrift → uses → Redundancy-Aware Alignment Score
confidence 92% · We introduce (1) a chance-calibrated, redundancy-aware (RA) alignment score that aggregates per-dimension drift measurements
ProxyDrift → uses → Jensen-Shannon distance
confidence 90% · Among several candidate distance metrics... we selected Jensen–Shannon distance (JSD) as the default
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:LLM applications deployed at scale face a fundamental challenge: privacy constraints prevent direct inspection of user interactions, making it difficult to obtain any representative evaluation dataset or to track the ongoing evolution of production traffic. We present ProxyDrift, a framework that (i) identifies and measures drift between production traffic and offline evaluation sets, and (ii) constructs and refreshes those evaluation sets accordingly; all without access to raw user data. Our approach operates entirely on non-PII proxy representations: structured, multi-dimensional descriptors derived from LLM-based classification of user interactions. We introduce (1) a chance-calibrated, redundancy-aware (RA) alignment score that aggregates per-dimension drift measurements via mutual information; (2) a conditional sampler that generates synthetic proxies respecting inter-dimensional dependencies; (3) a roundtrip consistency analysis that exposes generator/classifier disagreements and guides proxy taxonomy refinement; and (4) a feedback-linkage analysis that ties per-dimension and per-value proxy distributions to user satisfaction, surfacing actionable failure and success modes. Serving hundreds of millions of users, ProxyDrift enables continuous drift monitoring and targeted synthetic data generation without exposing sensitive user data. Experiments confirm strong roundtrip consistency, discriminator-level indistinguishability of synthetic queries from human queries, and tight end-to-end alignment (RA~0.9) with production.
Tags
Links
- Source: https://arxiv.org/abs/2608.08245v1
- Canonical: https://arxiv.org/abs/2608.08245v1
Trouble viewing inline? Open PDF directly →
Full Text
66,437 characters extracted from source content.
Expand or collapse full text
Privacy-Preserving Data Drift Detection and Recovery for Large-Scale LLM Applications via Proxy Representations Michael Levit Josh Ledgard Haoyu Dong Vishwas Suryanarayanan Eyal Kolman Sharon Tan Qiang Gan Vishal Chowdhary Microsoft Corporation mlevit, ledgardjosh, hadong, visuryan, eyalko, shartan, qigan, vishalc@microsoft.com Abstract LLM applications deployed at scale face a fundamental challenge: privacy constraints prevent direct inspection of user interactions, making it difficult to obtain any representative evaluation dataset or to track the ongoing evolution of production traffic. We present ProxyDrift, a framework that (i) identifies and measures drift between production traffic and offline evaluation sets, and (i) constructs and refreshes those evaluation sets accordingly; all without access to raw user data. Our approach operates entirely on non-PII proxy representations: structured, multi-dimensional descriptors derived from LLM-based classification of user interactions. We introduce (1) a chance-calibrated, redundancy-aware (RA) alignment score that aggregates per-dimension drift measurements via mutual information; (2) a conditional sampler that generates synthetic proxies respecting inter-dimensional dependencies; (3) a roundtrip consistency analysis that exposes generator/classifier disagreements and guides proxy taxonomy refinement; and (4) a feedback-linkage analysis that ties per-dimension and per-value proxy distributions to user satisfaction, surfacing actionable failure and success modes. Serving hundreds of millions of users, ProxyDrift enables continuous drift monitoring and targeted synthetic data generation without exposing sensitive user data. Experiments confirm strong roundtrip consistency, discriminator-level indistinguishability of synthetic queries from human queries, and tight end-to-end alignment (RA≈0.9RA≈ 0.9) with production. 1 Introduction LLM applications deployed in privacy-sensitive settings face a fundamental obstacle to offline evaluation: strict compliance constraints prevent direct access to user queries, system responses, and grounding documents. Raw interaction data cannot leave the production environment, so practitioners often have no eyes-on view of real traffic at all. This makes it difficult to assemble a representative offline evaluation dataset in the first place, and even harder to keep one aligned with traffic as user populations shift, new workflows emerge, and interaction patterns change Gama et al. (2014); Lu et al. (2018). Without a way to characterize production usage from telemetry alone, evaluation datasets remain unverified static snapshots whose relevance to live traffic is unknown and quietly degrades over time. This undetected misalignment directly distorts the decisions that offline evaluation is meant to inform. An offline set that under- or over-represents a scenario systematically inflates or deflates aggregate quality scores, hides real user pain points, and tunes model and prompt iterations against a non-existent user. Detecting and quantifying this drift is therefore a prerequisite for trustworthy offline evaluation. Traditional drift-detection methods that compare distributions over raw inputs Gama et al. (2014); Lu et al. (2018) are inapplicable in this setting: the inputs themselves cannot be inspected. We address this with ProxyDrift, a framework that enables both (i) privacy-preserving comparison of production and offline distributions, and (i) the construction and refresh of offline evaluation datasets that match production – all without exposing raw user data. The key insight is to replace sensitive interaction data with structured, multi-dimensional proxy representations: non-PII descriptors that capture the behavioral and linguistic characteristics of user interactions (e.g., intent type, communication style, domain, output expectations) without retaining any verbatim content. Individual dimensions of these proxies are either observed directly (e.g., query length), or generated by an LLM classifier operating within the production compliance boundary and can be safely persisted in telemetry. Given proxy representations for both production traffic and offline evaluation datasets, ProxyDrift provides four integrated capabilities: (1) drift identification and measurement via a chance-calibrated, redundancy-aware alignment score that quantifies divergence per dimension and aggregates with redundancy discounting; (2) evaluation dataset construction via a Chow–Liu tree-based conditional sampler Chow and Liu (1968) that generates synthetic proxies preserving pairwise dimensional dependencies, followed by LLM-based hydration into natural-language queries; (3) continuous monitoring through an end-to-end pipeline integrating telemetry collection, histogram aggregation, alignment scoring, and dashboards; and (4) taxonomy and quality diagnostics from intrinsic consistency analysis (driving schema refinement) and from linking per-dimension and per-value distributions to user feedback. ProxyDrift has been deployed in a major cloud-based productivity suite serving hundreds of millions of users, where it monitors multiple application scenarios continuously and generates synthetic evaluation data on a weekly cadence. The optimized taxonomy of dimensions and values exhibits high drift alignment (RA ≈0.9≈ 0.9), high roundtrip consistency, and generates synthetic queries that an unguided LLM discriminator cannot reliably tell apart from human queries. 2 Related Work Data Drift and Distribution Shift. The problem of distribution shift in deployed ML systems has been extensively studied under various names: concept drift Gama et al. (2014); Lu et al. (2018), dataset shift Rabanser et al. (2019), and label shift Lipton et al. (2018). Most prior work assumes direct access to input features or model predictions for statistical testing, a luxury unavailable under production privacy constraints. ProxyDrift differs in detecting drift indirectly, through aggregate comparisons over structured proxy representations rather than raw data, and in using the same substrate to construct production-aligned evaluation sets. Synthetic Data Generation. Synthetic tabular data has been approached via GANs Xu et al. (2019), VAEs, and more recently LLMs Borisov et al. (2023); Jordon et al. (2022). Our conditional sampler targets discrete, multi-dimensional proxy objects with a known schema, using Chow–Liu trees Chow and Liu (1968) that are interpretable and outperform neural generators on the cross-dimension dependency structure we care about. Architecturally closest to our solution is Aug-PE Xie et al. (2024) which produces differentially private synthetic text via inference-time API access alone; ProxyDrift shares the principle of keeping private data inside the trust boundary and pushing only structured aggregates out, but it uses LLM-classified proxies to characterize the production distribution itself rather than to release DP synthetic text for training. LLM-Based Classification and Generation. Using LLMs for structured classification and controlled generation Ouyang et al. (2022); Zhao et al. (2026) is well-established; our framework chains the two and uses the generation–classification cycle (§7.2) as a novel intrinsic measure of how faithfully proxy information is preserved end to end. Clio Tamkin et al. (2024) characterizes millions of Claude conversations via AI-assistant-driven clustering with downstream applications to economic-task usage Handa et al. (2025) and longitudinal adoption Appel et al. (2025); WildChat Zhao et al. (2024) releases a 1M-conversation opt-in ChatGPT corpus; and TnT-LLM Wan et al. (2024) iteratively induces a label taxonomy over chat-style conversations. ProxyDrift fixes a 21-dimension schema rather than inducing one, and consumes classifier outputs as a chance-calibrated distributional fingerprint for drift measurement and synthetic-data alignment rather than as qualitative insights or features for downstream supervised tasks. 3 Proxy Representation and Classification The foundation of ProxyDrift is a structured proxy representation that captures the behavioral properties of user interactions without retaining verbatim content. A proxy assigns each interaction a value (or set of values) along D categorical dimensions defined by a JSON schema; in our deployment, D=21D=21 dimensions organized along the following axes: Classified vs. Observable. Classified dimensions are obtained by running the interaction through an LLM classifier and require semantic understanding of the query text. For example, the user’s intent, communication style, domain, or expected output format. Observable properties are computed deterministically from interaction metadata without an LLM (e.g., word-count bucket, query language, and explicit grounding-file types). Nominal vs. Ordinal. Nominal dimensions have unordered categorical values (e.g., for output format: Word, PowerPoint, PDF). Ordinal dimensions have a meaningful value ordering (e.g., for output length: Brief << Moderate << Detailed << Extensive). The distinction matters for distance computation (cf. §4). Single-valued vs. multi-valued. Single-valued dimensions assign exactly one label per interaction (e.g., for communication formality: one of Formal, Neutral, or Informal). Multi-valued dimensions assign a (confidence ordered) list of labels. Only nominal dimensions can be multi-valued (e.g., the intent dimension might assign both Document Creation and Research to a query that requests both). Every label carries a confidence score s∈0,1,…,5s∈\0,1,…,5\, where s=0s=0 is reserved for the special value Unknown (classifier unable to determine a label) and s=5s=5 indicates maximum confidence or deterministic observation. Multi-valued dimensions list their labels in decreasing confidence order. Observable properties always carry maximum confidence. For presentation convenience, we also group the dimensions into five semantic categories as indicated in Table 2. The LLM-based classifier turns user queries into structured proxy objects. It operates within the production compliance boundary via a dedicated augmentation workflow. Given a user interaction and a structured prompt defining the taxonomy (allowed values, format constraints, scoring guidelines), the LLM produces a JSON object conforming to the proxy schema. A self-correction loop ensures schema compliance: if the LLM’s output violates structural constraints (invalid values, malformed tuples, missing dimensions), the specific violations are described in natural language and fed back for correction. This loop runs up to K=10K=10 attempts, progressively resolving syntax errors without human intervention. Additional extensions were introduced that generate proxies either for the entire multi-turn interaction or for individual queries given the context of all previous turns while also taking discourse break into account. 4 Drift Measurement Given proxy distributions for production traffic and an offline evaluation dataset, ProxyDrift quantifies drift at three levels: aggregate and per-dimension alignments, as well as actionable per-value diagnostics. Throughout, P=(p1,…,pn)P=(p_1,…,p_n) denotes the production (reference) distribution and O=(o1,…,on)O=(o_1,…,o_n) denotes the evaluation (observed) distribution over n categories. Among several candidate distance metrics (Total Variation, Wasserstein, Euclidean, …), we selected Jensen–Shannon distance (JSD) as the default for categorical distributions because it reacts strongly to support mismatches. Indeed, a category heavily used in production but absent from the evaluation set (or vice versa) incurs a large penalty. For reference distribution P and observed distribution O over n categories, the normalized JSD is defined as JSD(P,O)=12DKL(P∥P+O2)+12DKL(O∥P+O2)ln2,JSD(P,O)= 12D_KL\! (P\, \|\, P+O2 )+ 12D_KL\! (O\, \|\, P+O2 ) 2, (1) and is bounded in [0,1][0,1]. Since raw distance values are difficult to interpret in isolation, we provide a universally comparable, chance-corrected alignment score by calibrating against a permutation baseline. Given distance d(P,O)d(P,O), we estimate the expected distance under random label assignment by repeatedly shuffling O: ^π[d(P,π(O))]=1T∑t=1Td(P,πt(O))(T=50,000) E_π[d(P,π(O))]= 1T _t=1^Td(P, _t(O)) (T=50,000) (2) The alignment score is then: a(P,O)=max(0, 1−d(P,O)^π[d(P,π(O))])a(P,O)= \! (0,\;1- d(P,O) E_π[d(P,π(O))] ) (3) Identical distributions receive a=1a=1; a=0a=0 means the evaluation is no closer to production than a random reshuffling of probability mass. Holding P fixed and shuffling only O preserves P’s real-world sparsity as the reference structure, asking “how much better than chance is O at reproducing P?” An earlier design that drew O from a Dirichlet distribution (always dense) severely underestimated the baseline distance for sparse production distributions, yielding misleadingly low alignment scores. When multiple dimensions are evaluated, each dimension i yields its own alignment score aia_i. To produce a single overall score, we use a redundancy-discounted weighted average that accounts for three factors: (1) importance weight wiw_i; (2) entropy HiH_i of the reference distribution; and (3) pairwise mutual information MI(i,j)MI(i,j) between dimensions. The importance weights wiw_i are set by domain experts and encode the intuition that not all dimensions matter equally for downstream quality (for example, drift in user intent is far more consequential than drift in surface formality). Dimensions are processed in decreasing order of wi⋅Hiw_i· H_i. For each dimension i, the redundancy discount is: ρij _ij =MI(i,j)/Hi =MI(i,j)/H_i (overlap with earlier dim j) (4) ri r_i =∑j≺iρij = _j i _ij (total redundancy) (5) gi g_i =max(0, 1−λri) = (0,\;1-λ\,r_i) (discount, λ=0.8λ=0.8) (6) The effective dimension weight is then set to w~i=gi⋅wi w_i=g_i· w_i, and the redundancy-aware (RA) alignment is: ARA=∑iw~i⋅ai∑iw~iA_RA= _i w_i· a_i _i w_i (7) This formulation ensures that highly correlated dimensions are progressively down-weighted, preventing a cluster of redundant dimensions from dominating the aggregate score. When all dimensions are independent, it reduces to an ordinary weighted average. Because both aia_i and ARAA_RA are chance-corrected on [0,1][0,1], raw values can be turned into qualitative labels by partitioning the range; we use equal thirds (bad <1/3<1/3, average ≥1/3≥ 1/3, good ≥2/3≥ 2/3) uniformly throughout the paper. 5 Synthetic Dataset Generation Data generation is a key component of the ProxyDrift that can be used to augment existing eyes-on datasets with identified drift or to generate a new production-aligned one. Since reusing real production traffic verbatim is not an option due to privacy concerns, and hand-crafting new sets is slow and inherits the blind spots that motivated this work, our solution is to generate a synthetic set that matches production at the level of distribution statistics over proxy dimensions. This method was chosen over mirroring individual production queries to avoid overtuning. The pipeline proceeds in four stages: (1) as a part of production workflow, user queries are classified into structured proxy objects and saved along with their observed properties in telemetry; (2) proxies are aggregated into per-dimension marginals and pairwise co-occurrence counts; (3) the conditional sampler draws synthetic proxies from these statistics; (4) an LLM generator hydrates each proxy into a natural-language query; The remainder of this section details the Chow–Liu tree model that we used for sampling and two finite-sample corrections that account for limited data. A baseline independence sampler treats each dimension independently, drawing from per-dimension marginals. This discards all correlations and routinely produces implausible combinations (e.g., Legal Document output type with a Casual tone) – artefacts that exercise the model on inputs no real user would issue, biasing quality estimates and wasting evaluation budget. We therefore need a sampler that respects the joint structure of the dimensions. We model the joint distribution over D dimensions using the Chow-Liu optimal tree approximation Chow and Liu (1968): the maximum-weight spanning tree over dimensions with mutual information as edge weights. For every dimension pair (Xi,Xj)(X_i,X_j), with vocabularies iV_i and iV_i we maintain a contingency table of co-occurrence counts from raw proxy records. For multi-valued dimensions, every combination of values from the two dimensions in the same record contributes a count. Joint-derived marginals ensure consistency: p(vi)=∑vjp(vi,vj),MI(Xi;Xj)=∑vi,vjp(vi,vj)lnp(vi,vj)p(vi)p(vj)p(v_i)= _v_jp(v_i,v_j), (X_i;X_j)= _v_i,v_jp(v_i,v_j) p(v_i,v_j)p(v_i)\,p(v_j) (8) The Chow-Liu theorem states that the optimal tree-structured approximation is the maximum-weight spanning tree with MI as edge weights. We construct it via Kruskal’s algorithm with union-find (rank compression and path halving), yielding O(D2logD)O(D^2 D) complexity. The undirected tree is then rooted at the node with highest degree: the “hub” that directly conditions the most children, providing a short path to a large fraction of the dimensions and limiting the number of conditional steps over which any single estimation error can propagate. The rooted tree defines: P(X1,…,XD)=P(Xr)∏d≠rP(Xd∣Xπ(d))P(X_1,…,X_D)=P(X_r) _d≠ rP(X_d X_π(d)) (9) where r is the root and π(d)π(d) the parent of dimension d. See an example tree in Figure 1. Figure 1: Chow–Liu maximum spanning tree for one application scenario, built from the initial dimension taxonomy (prior to the refinement described in §7.3). Edge labels are sigmoid-dampened mutual information (§5). Color indicates semantic category: intent & style, tone & affect, output, content, observable. The textbook Chow–Liu construction assumes that both the pairwise MI values and the per-parent conditionals can be read off the data directly. In practice, neither is true: many dimension pairs are supported by a modest number of co-occurring records, and many parent values are themselves rare, so plug-in estimates are noisy in opposite ways: MI is over-estimated, while conditionals are often under-determined. We address each with a tailored correction. With NijN_ij as the number of co-occurring records for the pair (Xi,Xj)(X_i,X_j), MI computed by plug-in from finite samples exhibits a positive bias Brillinger (2004) scaling as (|i|−1)(|j|−1)/(2Nij)(|V_i|-1)(|V_j|-1)/(2N_ij). Left uncorrected, this bias inflates the apparent dependence between high-cardinality dimensions and can promote spurious edges into the maximum spanning tree. Rather than subtracting an analytical correction that can overshoot and yield negative MI estimates, we apply a multiplicative sigmoid dampening: MI^(Xi;Xj)=MI(Xi;Xj)⋅σ(lnNijc⋅|i|⋅|j|), MI(X_i;X_j)=MI(X_i;X_j)·σ\! ( N_ijc·|V_i|·|V_j| ), (10) where σ is the logistic sigmoid and c=5c=5 follows Cochran’s rule Cochran (1954) for minimum expected cell counts. The factor smoothly suppresses MI when sample size is small relative to the table area |i|⋅|j||V_i|·|V_j| and approaches 11 when evidence is plentiful. By construction it preserves zeros (I^=0 I=0 when I=0I=0), is strictly less than I for any finite sample (asymptotically approaching I as Nij→∞N_ij→∞), and adapts automatically across pairs of differing cardinality. Once the tree is fixed, each non-root dimension XcX_c must be sampled from P(Xc∣Xp=vp)P(X_c X_p=v_p) for every value vpv_p of its parent. Parent values vary widely in support: rare ones yield near-degenerate empirical conditionals that lock in spurious dependencies the data do not actually justify. We therefore shrink each conditional toward the marginal prior P0(Xc)P_0(X_c) by a coefficient that reflects how strongly the data itself argues for a difference from the prior: Step 1: We start with the empirical conditional: P^(Xc=vc∣Xp=vp)=c(vp,vc)/nvp. P(X_c=v_c X_p=v_p)=c(v_p,v_c)/n_v_p. (11) Step 2: We test whether the empirical conditional differs from the marginal prior P0(Xc)P_0(X_c) using Pearson’s χ2χ^2: χ2=∑vc(Ovc−Evc)2Evc,Evc=nvp⋅P0(vc).χ^2= _v_c (O_v_c-E_v_c)^2E_v_c, E_v_c=n_v_p· P_0(v_c). (12) We apply a relaxed variant of Cochran’s rule Cochran (1954), requiring at least one expected count per cell on average (nvp≥|c|n_v_p≥|V_c|); when this fails the χ2χ^2 approximation is unusable and we set γ=1γ=1 (pure prior). Step 3: The shrinkage coefficient is then set to: γvp=p-value. _v_p= p-value. (13) The square root softens the transition: significant conditionals (p≪1p~ ~1) keep γ≈0γ≈ 0 and trust the data, borderline cases (p≈0.1p≈ 0.1) receive moderate shrinkage, and non-significant slices (p→1p→ 1) collapse back to the prior. Step 4: The final interpolated conditional: P(Xc=vc∣Xp=vp)=(1−γvp)P^(vc∣vp)+γvpP0(vc).P(X_c=v_c X_p=v_p)=(1- _v_p)\, P(v_c v_p)+ _v_p\,P_0(v_c). (14) Together, the two described corrections ensure that the sampler only commits to a dependency (either as an edge in the tree or as a sharp conditional within an edge) when the data demonstrably supports it, and otherwise falls back to the prior estimate. Given the fitted model above, sampling follows the tree factorization in topological order: sample the root from its prior, then each child from its conditional given its parent’s sampled value. Each sample produces a complete proxy object in O(D)O(D) time. The sampled proxy is then converted into a natural-language query by an LLM generator. The generator de-facto reverses the work of proxification: it receives the proxy as structured input with the same schema as the LLM classifier (except for observable properties) and produces a natural language query. 6 System Deployment and Data This section describes the production system in which ProxyDrift operates and the datasets used in the evaluation that follows. ProxyDrift is deployed within a large-scale agentic chat system that serves hundreds of millions of users across a cloud-based productivity suite, orchestrating multiple specialized scenarios that include generating, editing documents (presentations, reports, spreadsheets, etc.) on the user’s behalf. These two skills are the focus of the present work. Within the production compliance boundary, a dedicated workflow fires after each session: the LLM classifier (§3) emits a proxy object that is persisted in telemetry – the only representation of user interactions that leaves the live environment. Scheduled pipelines aggregate proxies daily into per-dimension marginals and pairwise co-occurrence tables, one set per scenario; these aggregates feed both the drift detector (§4) and the synthetic data generator (§5). The evaluation in §7 draws on multiple datasets: (0) legacy hand-curated sets, one per artifact-type scenario, assembled over the product’s lifetime for functional coverage rather than distributional fidelity. In §7.1 they are used as the baseline against which synthetic datasets are compared; (1) a designed synthetic set of 450 queries that is representative of production marginals and covers all dimension values, including rare ones, from our taxonomy. It is used in the roundtrip experiments (§7.2–§7.3); (2) two eyes-on user-feedback sets with production interactions and binary thumbs-up/thumbs-down labels. (2a) for document generation: 5.3K interactions (70.5% positive; used in §7.5) and (2b) for comprehension: 2.3K interactions (26.8% positive; used in §7.6); and (3) a balanced 200-query discrimination set (100 from dataset (1) and 100 from dataset (2a)) used in §7.4. 7 Evaluation We evaluate ProxyDrift along four complementary axes: (1) end-to-end drift alignment between legacy/synthetic and production proxy distributions; (2) generation+classification roundtrip consistency of the proxy representation and its use for taxonomy refinement; (3) indistinguishability of synthetic queries from human queries; and (4) discriminative power of proxy dimensions for predicting user satisfaction. We deliberately discuss alignment first because it motivated the schema-refinement loop in (2): the very gap between the legacy offline test set and production traffic is what prompted us to build the (conditional) sampler whose proxies are then put through the roundtrip experiments. 7.1 End-to-End Drift Alignment This experiment measures whether the full ProxyDrift pipeline can produce a synthetic dataset whose proxy distribution matches production. We use the redundancy-aware (RA) alignment score of §4 and report results for the Excel artifact-creation scenario as the results for other document types (e.g., PowerPoint) are almost identical. This evaluation predates the taxonomy-refinement loop of §7.3: even against the unrefined schema, it shows that distribution-aware sampling is worth pursuing in the first place. Setup. Reference distributions were computed from a week of production traffic (2026-04-02 to 2026-04-08). Two samplers each produced a synthetic dataset of ∼1,000 1,000 proxies: the basic (dimensionally independent) sampler, drawing each dimension from its production marginal, and the conditional (Chow-Liu) sampler of §5. As a third point of reference we re-scored the pre-existing hand-curated Excel test set (§6, dataset (0)) against the same production baseline. Table 1: RA alignment (§4) for the Excel document generation scenario. Dataset RA score Band Legacy hand-curated set 0.284 bad Basic (independent) sampler 0.897 good Conditional (Chow-Liu) sampler 0.917 good Results. As Table 1 shows, both synthetic samplers land squarely in the good band (cf. §4), with the conditional sampler outperforming the basic one (RA 0.9170.917 vs. 0.8970.897). The basic sampler scores high because the RA metric evaluates per-dimension marginals and the basic sampler is calibrated directly to them; the conditional sampler edges past it by additionally preserving cross-dimension structure (e.g., intent-output co-occurrences) without sacrificing marginal fidelity. By contrast, the legacy hand-curated test set fails on most dimensions, with individual per-dimension alignment scores typically below 0.400.40 and the overall alignment score in the bad range. This gap also impacts evaluation outcomes: in the Excel agent scenario, scores on the distribution-aligned synthetic dataset are consistently lower than on the legacy offline set (e.g., Accuracy 4.29→ 3.71, Reliability 3.85→ 3.44; avg. ∼ 0.3 decrease).111Each (query, generated document) pair is scored by an LLM-as-a-judge Zheng et al. (2023); Kim et al. (2024) on a 1–5 scale across five rubric dimensions, including Accuracy (correctness of query interpretation and implementation), Completeness (coverage of requirements), Usefulness (practical value for user goals), Reliability (technical consistency and robustness), and Aesthetics (formatting and presentation quality). This suggests systematic score inflation in conventional offline evaluation, directly supporting the decision distortion described in the introduction. This gap is the practical justification for the rest of the pipeline: distribution-aware synthetic data closes a quality gap that manual curation cannot. 7.2 Roundtrip Consistency The roundtrip experiment investigates internal consistency of the proxy representation: does a proxy, when used to generate a synthetic query and then reclassified, reconstruct itself? We designed the experiment with two purposes in mind: as a diagnostic that quantifies how faithfully the generation-classification cycle preserves proxy information, and as the optimization signal that drives the taxonomy refinement loop of §7.3: each iteration of the schema and prompts is judged by the reduction in roundtrip distance and in the per-dimension confusion patterns it produces. Protocol. The M input proxies are obtained by running the LLM classifier on the designed synthetic dataset (§6, dataset (1)) and joining each predicted proxy with the observable properties measured directly from the corresponding query (language, word-length buckets, explicit grounding markers). For each of these input proxies: (1) the proxy is given to the LLM generator, which produces N=3N=3 synthetic queries; (2) each query is independently classified back into a proxy (with its observable properties re-measured); (3) for each (original, reconstructed) pair, a per-dimension distance is computed and aggregated into a scalar d∈[0,1]d∈[0,1]. Per-dimension distances are confidence-weighted and account for value identity, ordinal proximity (for sorted dimensions), and – for multi-valued dimensions – set overlap. Initial results. The experiment was first conducted against an initial, intuition-driven taxonomy on the M=450M=450 input proxies derived from dataset (1). Of these, 31 (6.9%) failed to complete the generation-classification cycle typically due to the LLM classifier failure to produce a schema-compliant output for the generated query within the self-correction budget. For the remaining 419 roundtrips (3 generations each), the mean reconstruction distance was μ¯=0.291 μ=0.291, with the within-proxy standard deviation (σ¯=0.055 σ=0.055) substantially smaller than the across-proxy standard deviation (sμ=0.1s_μ=0.1). This pattern revealed that the residual mismatch was systematic rather than random: specific dimension values were consistently confused, pointing to actionable taxonomy deficiencies rather than general noise. 7.3 Taxonomy Refinement via Confusion Analysis By computing per-dimension confusion matrices from the roundtrip outputs, we identified which dimensions and values were conflated during the cycle. Each matrix is built by proportional-redistribution mass accounting that preserves both original and reconstructed marginals, and is summarized by a size-adjusted diagonal-concentration score: the share of mass on (or, for ordinal dimensions, near) the main diagonal, rescaled to compensate for matrix size. This analysis directly guided iterative improvements to the proxy schema. We: • retired dimensions that carried little signal and were consistently confused (e.g., domain expertise, intent categories); • revised value sets, i.e., merged values too similar for the LLM to discriminate and split overly broad categories; • introduced new dimensions for constructs that were conflated within existing ones (e.g., separating content specificity from communication constraints); • rewrote definitions and examples in both classifier and generator prompts, adding disambiguation guidance for borderline cases identified from the confusion matrices. The resulting taxonomy (the post-optimization schema) organizes dimensions into the categories shown in Table 2. Category Dimension |||V| Type Ord. Intent & Style intent_types 25 multi — query_formality 4 single ✓ query_structure 4 single — query_technical 4 single ✓ query_engagement 5 multi — query_constraints 5 single ✓ Tone & Affect tone_sentiment 6 single ✓ tone_emotion 9 multi — tone_urgency 5 single ✓ Output output_type 11 single — output_format 6 single — output_length 5 single ✓ output_components 6 multi — output_creativity 4 single ✓ Content domain_names 21 multi — content_specificity 4 single ✓ content_embedded 5 multi — Observable char_count_bucket 6 single ✓ word_count_bucket 6 single ✓ query_language 11 single — grounding_explicit 20 multi — Table 2: Post-optimization proxy schema (single-turn). D=21D=21 dimensions: 17 LLM-classified and 4 observable. |||V| is vocabulary size. “Ord.” stands for ordinal (has a meaningful value ordering). Table 3: Roundtrip consistency before and after taxonomy refinement and for the two multi-turn variants (M=450M=450 input proxies, N=3N=3 generations each) Lower μ¯ μ is better. Single-turn Multi-turn Statistic Before After Inter. Query Failed roundtrips 31 2 4 3 Mean proxy distance (μ¯ μ) 0.282 0.158 0.186 0.153 Std. dev. of means (sμs_μ) 0.100 0.076 0.084 0.085 Mean within-proxy std (σ¯ σ) 0.055 0.042 0.046 0.041 Table 3 shows the aggregate roundtrip statistics before and after taxonomy refinement, on the same set of 450 input proxies. Refinement cut the mean reconstruction distance from 0.282 to 0.158 (a 44% relative reduction) and dropped the roundtrip failure rate from 6.9% to 0.4%: evidence that the revised schema is far easier for the LLM to apply consistently. Formerly problematic dimensions also improved sharply (query_engagement: 0.51 → 0.81 macro diagonal concentration; query_structure: 0.65 → 0.81), while already-strong dimensions held (output_type: 0.89 → 0.95). n Enthusiasm Curiosity Frustration Impatience Confusion Uncertainty Neutral Other Enthusiasm 75 71% 29% Curiosity 17 35% 65% Frustration 77 75% 8% 13% Impatience 9 11% 89% Confusion 27 78% 15% Uncertainty 33 21% 9% 30% 36% Neutral 1087 95% Other 6 83% 17% (a) tone_emotion (multi-valued, 8 values). Macro conc.: 0.79. n Unknown Minimal Moderate Detailed Comprehensive Unknown 9 89% 11% Minimal 589 35% 56% 8% Moderate 575 58% 40% Detailed 131 89% 5% Comprehensive 27 7% 93% (b) query_constraints (ordinal, 5 values). Macro conc.: 0.85. Figure 2: Row-normalized confusion matrices for two representative dimensions. (a) Multi-valued: common emotions recovered well; rare ones collapse into Neutral. (b) Ordinal: neighbor-smearing (Minimal ↔ Moderate) reflects genuine construct ambiguity. Cf. output_type (nominal, macro conc. 0.95) for the high-diagonal nominal case. Figure 2 shows representative confusion matrices for two dimension archetypes: tone_emotion (Fig. 2(a)), a multi-valued dimension capturing emotional tone, in which high-frequency emotions reconstruct reliably while rarer ones (Curiosity, Uncertainty) collapse into Neutral; and query_constraints (Fig. 2(b)), an ordinal dimension whose neighbor-smearing pattern (Minimal often reconstructed as Moderate) reflects genuine ambiguity in the underlying construct rather than model failure. For comparison, output_type (nominal, 11 values) achieves macro diagonal concentration 0.95, a near-perfect document-type recovery. Encouragingly, Table 3 also shows that both multi-turn schema variants (§3) behave consistently with the single-turn baseline with the roundtrip yields mean reconstruction distances of μ¯=0.186 μ=0.186 at the interaction level and μ¯=0.153 μ=0.153 at the query level – both within 0.030.03 of the single-turn μ¯=0.158 μ=0.158, and per-dimension diagonal-concentration factors also falling in the same range as their single-turn counterparts. This suggests that the proxy representation extends naturally to multi-turn settings without loss of fidelity. 7.4 Naturalness Test To assess naturalness of the synthetic data, we submitted 100 human queries (from production feedback logs) and 100 synthetic queries to an independent LLM discriminator under two conditions: a guided prompt (enumerating known synthetic tells such as overly polished grammar and balanced structure) and an unguided prompt (verdict only). Table 4: Naturalness test: LLM accuracy at distinguishing human vs. synthetic queries (n=200n=200, chance = 50%). Condition Accuracy p-value Human rec. Synth rec. Guided 0.650 1.3×10−51.3× 10^-5 0.91 0.39 Unguided 0.540 0.144 0.92 0.16 As shown in Table 4, without explicit guidance the discriminator cannot reliably distinguish synthetic from human queries (p=0.14p=0.14). With guidance, accuracy rises to 65%, but the gain comes entirely from identifying human queries via surface markers (typos, code-switching, implicit context); synthetic recall remains low (39%), meaning most synthetic queries are still classified as human. 7.5 Predictive Signal Analysis Beyond measuring drift, we ask: which proxy dimensions carry actionable information about system quality? We evaluate this using the feedback signal dataset (2a) with binary user satisfaction (thumbs-up/down) as the target signal Y. For each dimension X we compute the held-out conditional log-likelihood (CLL) gain of predicting Y from X versus the marginal under stratified 5-fold cross-validation, with conditionals estimated under Laplace smoothing Lidstone (1920). Ordinal dimensions optionally use pool-adjacent-violators (PAV) refinement Zadrozny and Elkan (2002), with the nested PAV-vs-raw tiebreaker on the training fold settled by chance-corrected AMI Vinh et al. (2010) with Kish’s effective-sample-size correction Kish (1965). We call a dimension’s signal stable when its per-fold mean μ¯ μ is positive and at least twice the per-fold standard deviation (SNR μ¯/σ≥2 μ/σ≥ 2). Applied to 5,293 interactions, seven dimensions clear the SNR ≥2≥ 2 bar: sentiment, intent type, emotion, domain, output creativity, formality, and query word count, with CLL gains in [0.0019,0.0152][0.0019,0.0152] nats per fold. A second cluster (output type, output format, query language) has positive means dominated by a handful of rare classes, so per-fold variance is high; the remaining dimensions carry essentially no signal. A complementary per-value diagnostic, the lift P^(Y=1∣X=v)−P^(Y=1) P(Y=1 X=v)- P(Y=1) with significance assessed by a one-vs-rest binomial z-contrast, reveals that dimension-level scores are diluted by dominant uninformative modes. Individual values carry stronger signal in both directions, e.g., queries laced with Frustration elicit thumbs-down 93% of the time (z=−7.5z=-7.5), while Summarization requests lift thumbs-up to 51% (z=+5.2z=+5.2, vs. a 27% baseline). These per-value scores directly inform which dimensions to prioritize in drift monitoring and which production segments to investigate or amplify when quality shifts. 7.6 Telemetry-Driven Copilot Improvement for Question Answering (QA) over Excel Files The feedback-linkage analysis in §7.5 shows that proxy dimensions can expose user-facing failure modes without access to raw user content. We next demonstrate its portability by applying the diagnostic to QA over Excel files in Microsoft Copilot. Given an Excel workbook and a query, Copilot must answer by understanding various workbook semantics. Many queries depend on formulas, filters, sorting state, charts, formatting, or data validation. We therefore define Excel-specific binary proxy dimensions: Formula=yes, Filter/Sort=yes, Chart=yes, and Cell-Styling=yes, indicating whether each attribute is needed for question answering. These proxy values turn heterogeneous spreadsheet requirements into measurable, actionable signals. We instantiate the diagnostic on 2,3462,346 Copilot ask-queries over Excel files with thumb-up/down feedback (dataset (2b)), where thumbs-down is treated as dissatisfaction (DSAT) and thumbs-up as satisfaction (SAT). Overall, 29.5%29.5\% of queries receive negative feedback (DSAT). The data is stratified into matched train/test splits of 1,1741,174/1,1721,172 queries. On the training split, we apply the per-value binomial z-contrast from §7.5, comparing each yes value against the global DSAT baseline. The held-out test split is reserved for measuring how many DSAT cases become satisfactory after refinement. As shown in Table 5, all four proxy values have elevated DSAT rates, suggesting that specific workbook semantics challenges Excel QA. Table 5: Case study on QA over Excel files. The left block reports training-split DSAT diagnostics for Excel-specific proxy values. The right block reports post-refinement gains on sampled held-out telemetry DSAT cases and on sampled QA pairs from public spreadsheet benchmarks. DSAT Diagnostic Post-refinement Gain Proxy value n DSAT z DSAT→ Public Acc. Filter/Sort = yes 43 41.9% +1.77+1.77 40.0% +50.0+50.0 pts Formula = yes 176 34.7% +1.49+1.49 60.0% +55.6+55.6 pts Chart = yes 110 34.5% +1.15+1.15 20.0% +16.7+16.7 pts Cell-Styling = yes 26 34.6% +0.57+0.57 40.0% +33.4+33.4 pts We used this signal to refine the Excel encoding consumed by Copilot. Inspection showed that production Excel encoding dropped specific semantics to save context length. We augmented the encoding to preserve formulas, sort/filter state, data validation, and merged-cell structure, making these properties explicit to the model. For held-out telemetry validation, we then sample five DSAT cases per proxy category and assess whether the refined encoding flips DSAT outcomes to SAT. The refinement recovers 60.0%60.0\%, 40.0%40.0\%, 20.0%20.0\%, and 40.0%40.0\% of Formula, Filter/Sort, Chart, and Cell-Styling cases, respectively, illustrating how ProxyDrift turns privacy-preserving proxy diagnostics into engineering improvements for Microsoft Copilot. It also yields large gains on public spreadsheet benchmarks sampled to match the telemetry feature distribution, including RealHiTBench Wu et al. (2025), SheetBench Wang et al. (2026), and MiMoTable Li et al. (2025). Accuracy improves by +55.6+55.6, +50.0+50.0, +16.7+16.7, and +33.4+33.4 points on Formula, Filter/Sort, Chart, and Cell-Styling queries, respectively. 8 Conclusion We presented ProxyDrift, a privacy-preserving framework for detecting, measuring, and mitigating data drift in large-scale LLM applications. Operating entirely on structured proxy representations, the system enables continuous distributional monitoring under strict compliance constraints via a permutation-calibrated alignment score and a Chow–Liu conditional sampler with adaptive chi-squared shrinkage that respects inter-dimensional dependencies. A production deployment serving hundreds of millions of users demonstrates high roundtrip consistency, synthetic queries an unguided LLM discriminator cannot reliably distinguish from human queries, strong end-to-end alignment (RA ≈0.9≈ 0.9), and proxy dimensions that carry measurable signal about user satisfaction. Future work includes extending the framework to grounding documents and generated artifacts, correlating proxy dimensions with runtime quality metrics, and using synthetic data for agent optimization beyond evaluation. GenAI Usage Disclosure Large language models (GPT-5.X, Claude Opus 4.X) were used as tools during the research: (1) as the classification and generation engines within the ProxyDrift pipeline itself (the subject of this paper); (2) for light editing and grammar checking of the manuscript text. All experimental design, algorithmic contributions, analysis, and interpretation are the work of the human authors. References R. Appel, P. McCrory, A. Tamkin, M. McCain, T. Neylon, and M. Stern (2025) Anthropic economic index report: uneven geographic and enterprise ai adoption. External Links: 2511.15080, Link Cited by: §2. V. Borisov, K. Sessler, T. Leemann, M. Pawelczyk, and G. Kasneci (2023) Language models are realistic tabular data generators. In The Eleventh International Conference on Learning Representations, External Links: Link Cited by: §2. D. R. Brillinger (2004) Some data analyses using mutual information. Brazilian Journal of Probability and Statistics, p. 163–182. Cited by: §5. C. K. Chow and C. N. Liu (1968) Approximating discrete probability distributions with dependence trees. IEEE Transactions on Information Theory 14 (3), p. 462–467. Cited by: §1, §2, §5. W. G. Cochran (1954) Some methods for strengthening the common χ2χ^2 tests. Biometrics 10 (4), p. 417–451. Cited by: §5, §5. J. Gama, I. Žliobaitė, A. Bifet, M. Pechenizkiy, and A. Bouchachia (2014) A survey on concept drift adaptation. ACM computing surveys (CSUR) 46 (4), p. 1–37. Cited by: §1, §1, §2. K. Handa, A. Tamkin, M. McCain, S. Huang, E. Durmus, S. Heck, J. Mueller, J. Hong, S. Ritchie, T. Belonax, K. K. Troy, D. Amodei, J. Kaplan, J. Clark, and D. Ganguli (2025) Which economic tasks are performed with ai? evidence from millions of claude conversations. External Links: 2503.04761, Link Cited by: §2. J. Jordon, L. Szpruch, F. Houssiau, M. Bottarelli, G. Cherubin, C. Maple, S. N. Cohen, and A. Weller (2022) Synthetic data–what, why and how?. arXiv preprint arXiv:2205.03257. Cited by: §2. S. Kim, J. Suk, S. Longpre, B. Y. Lin, J. Shin, S. Welleck, G. Neubig, M. Lee, K. Lee, and M. Seo (2024) Prometheus 2: an open source language model specialized in evaluating other language models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, p. 4334–4353. Cited by: footnote 1. L. Kish (1965) Survey sampling. John Wiley & Sons, New York. Cited by: Appendix C, §7.5. Z. Li, Y. Du, M. Zheng, and M. Song (2025) MiMoTable: a multi-scale spreadsheet benchmark with meta operations for table reasoning. In Proceedings of the 31st International Conference on Computational Linguistics, Abu Dhabi, UAE, p. 2548–2560. External Links: Link Cited by: §7.6. G. J. Lidstone (1920) Note on the general case of the Bayes–Laplace formula for inductive or a posteriori probabilities. Transactions of the Faculty of Actuaries 8, p. 182–192. Cited by: §7.5. Z. Lipton, Y. Wang, and A. Smola (2018) Detecting and correcting for label shift with black box predictors. In International conference on machine learning, p. 3122–3130. Cited by: §2. J. Lu, A. Liu, F. Dong, F. Gu, J. Gama, and G. Zhang (2018) Learning under concept drift: a review. IEEE transactions on knowledge and data engineering 31 (12), p. 2346–2363. Cited by: §1, §1, §2. L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, et al. (2022) Training language models to follow instructions with human feedback. Advances in neural information processing systems 35, p. 27730–27744. Cited by: §2. S. Rabanser, S. Günnemann, and Z. Lipton (2019) Failing loudly: an empirical study of methods for detecting dataset shift. Advances in Neural Information Processing Systems 32. Cited by: §2. A. Tamkin, M. McCain, K. Handa, E. Durmus, L. Lovitt, A. Rathi, S. Huang, A. Mountfield, J. Hong, S. Ritchie, M. Stern, B. Clarke, L. Goldberg, T. R. Sumers, J. Mueller, W. McEachen, W. Mitchell, S. Carter, J. Clark, J. Kaplan, and D. Ganguli (2024) Clio: privacy-preserving insights into real-world ai use. External Links: 2412.13678, Link Cited by: §2. N. X. Vinh, J. Epps, and J. Bailey (2010) Information theoretic measures for clusterings comparison: variants, properties, normalization and correction for chance. Journal of Machine Learning Research 11 (95), p. 2837–2854. External Links: Link Cited by: Appendix C, §7.5. M. Wan, T. Safavi, S. K. Jauhar, Y. Kim, S. Counts, J. Neville, S. Suri, C. Shah, R. W. White, L. Yang, R. Andersen, G. Buscher, D. Joshi, and N. Rangan (2024) TnT-llm: text mining at scale with large language models. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, KDD ’24, New York, NY, USA, p. 5836–5847. External Links: ISBN 9798400704901, Link, Document Cited by: §2. Z. Wang, J. Su, M. Zhou, H. Zeng, M. Jia, X. Lv, H. Dong, X. Ma, S. Han, and D. Zhang (2026) SheetBrain: a neuro-symbolic agent for accurate reasoning over complex and large spreadsheets. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 40, Palo Alto, CA, p. 33800–33808. External Links: Document Cited by: §7.6. P. Wu, Y. Yang, G. Zhu, C. Ye, H. Gu, X. Lu, R. Xiao, B. Bao, Y. He, L. Zha, W. Ye, J. Zhao, and H. Wang (2025) RealHiTBench: a comprehensive realistic hierarchical table benchmark for evaluating LLM-based table analysis. In Findings of the Association for Computational Linguistics: ACL 2025, Vienna, Austria, p. 7105–7137. External Links: Document, Link Cited by: §7.6. C. Xie, Z. Lin, A. Backurs, S. Gopi, D. Yu, H. Inan, H. Nori, H. Jiang, H. Zhang, Y. T. Lee, B. Li, and S. Yekhanin (2024) Differentially private synthetic data via foundation model apis 2: text. In Proceedings of the 41st International Conference on Machine Learning, ICML’24, p. 54531–54560. Cited by: §2. L. Xu, M. Skoularidou, A. Cuesta-Infante, and K. Veeramachaneni (2019) Modeling tabular data using conditional GAN. Advances in neural information processing systems 32, p. 7335–7345. Cited by: §2. B. Zadrozny and C. Elkan (2002) Transforming classifier scores into accurate multiclass probability estimates. In Proceedings of the eighth ACM SIGKDD international conference on Knowledge discovery and data mining, p. 694–699. External Links: Document Cited by: §7.5. W. X. Zhao, K. Zhou, J. Li, T. Tang, X. Wang, Y. Hou, Y. Min, B. Zhang, J. Zhang, Z. Dong, Y. Du, C. Yang, Y. Chen, Z. Chen, J. Jiang, R. Ren, Y. Li, X. Tang, Z. Liu, P. Liu, J. Nie, and J. Wen (2026) A survey of large language models. External Links: 2303.18223, Link Cited by: §2. W. Zhao, X. Ren, J. Hessel, C. Cardie, Y. Choi, and Y. Deng (2024) WildChat: 1m chatgpt interaction logs in the wild. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024, External Links: Link Cited by: §2. L. Zheng, W. Chiang, Y. Sheng, S. Zhuang, Z. Wu, Y. Zhuang, Z. Lin, Z. Li, D. Li, E. Xing, et al. (2023) Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in neural information processing systems 36, p. 46595–46623. Cited by: footnote 1. Appendix A Proxy Schema: Full Dimension Reference Table LABEL:tab:schema-reference expands the compact summary of Table 2 into a full reference for the post-optimization proxy schema (single-turn): D=21D=21 dimensions across five categories, with 17 LLM-classified and 4 directly observable dimensions. In the Properties column, “single” / “multi” indicates whether the dimension carries one value or a set, “ord.” / “nom.” distinguishes ordinal from nominal value sets, and |||V| counts all admissible values (including Unknown for single-valued dimensions; multi-valued dimensions encode Unknown as the empty set). Each classified value carries a 11–55 confidence; observable dimensions always carry confidence s=5s=5 when known. In the Values column, ordinal value lists are written in rank order (low → high). The multi-turn schema variants used in §7.2–§7.3 share the same categories and definitions; they add a small number of turn-level dimensions whose value sets are direct generalizations of those listed here. Table 6: Post-optimization proxy schema (single-turn): full description and value sets for all D=21D=21 dimensions, organized by the five categories of Table 2. Group Dimension Description Properties Values Intent & Style intent_types What action the user is requesting; multiple intents may co-occur in a single query. multi, nom., ||=25|V|=25 Factual Inquiry; Conceptual Explanation; Procedural Guidance; Definitional Query; Comparative Analysis; Recommendation Seeking; Troubleshooting; Verification; Task Execution; Content Generation; Document Creation; Document Edit; Template Generation; Structured Output; Code Generation; Data Transformation; Summarization; QnA; Research; Greeting; Clarification Request; Follow Up Question; Scope Modification; Capability Inquiry; Other. query_formality Register of the query text on a colloquial-to-elevated spectrum. single, ord., ||=4|V|=4 Unknown; Informal; Neutral; Formal. query_structure Surface organization of the query: terse imperative, flowing prose, or explicit organizational markers (lists, headers). single, nom., ||=4|V|=4 Unknown; Direct; Verbose; Structured. query_technical Sophistication of the language used in the query (an observable linguistic property, not an inference about the user). single, ord., ||=4|V|=4 Unknown; Non Technical; Intermediate; Expert. query_engagement How the user positions themselves toward the agent: requesting a deliverable, asking for advice, delegating, or seeking to understand. multi, nom., ||=5|V|=5 Unknown; Directive; Consultative; Delegating; Inquisitive. query_constraints How tightly the user constrains the response, measured by the count and explicit organization of stated requirements. single, ord., ||=5|V|=5 Unknown; Minimal; Moderate; Detailed; Comprehensive. Tone & Affect tone_sentiment Primary affective polarity of the query. single, ord., ||=6|V|=6 Unknown; Strongly Negative; Negative; Neutral; Positive; Strongly Positive. tone_emotion Emotional undertones inferable from the language; multiple emotions may co-occur. multi, nom., ||=9|V|=9 Unknown; Enthusiasm; Curiosity; Frustration; Impatience; Confusion; Uncertainty; Neutral; Other. tone_urgency Time pressure signalled by the query, from no temporal context to explicit emergency markers. single, ord., ||=5|V|=5 Unknown; Low; Medium; High; Critical. Output output_type For document-producing intents, the kind of artifact requested (Unknown when no document is expected). single, nom., ||=11|V|=11 Unknown; Report; Proposal; Presentation; Spreadsheet; Plan; Documentation; Letter; Email; Tutorial; Other. output_format Concrete file format for the requested artifact. single, nom., ||=6|V|=6 Unknown; Word; PowerPoint; Excel; PDF; Other. output_length User-stated or implied expectation for the response/artifact length. single, ord., ||=5|V|=5 Unknown; Brief; Moderate; Detailed; Extensive. output_components Specific components the user expects in the output (only meaningful when a document is being produced). multi, nom., ||=6|V|=6 Code; Examples; Sources; Visuals; Actionable; Other. output_creativity Expected stylistic register of the output: factual, balanced (default for most document requests), or imaginative. single, ord., ||=4|V|=4 Unknown; Factual; Balanced; Creative. Content domain_names Subject-matter domain(s) of the query; multiple domains may apply. multi, nom., ||=21|V|=21 Technology Computing; Engineering; Natural Sciences; Mathematics; Medical Health; Psychology Cognitive; Business Economics; Law Legal; Education Pedagogy; Arts Humanities; Creative Arts; Social Sciences; Language Linguistics; Lifestyle Personal; Entertainment Media; Sports Recreation; Food Culinary; Travel Geography; Environment Sustainability; Spirituality Religion; Other. content_specificity How concretely the query is anchored to real entities, numbers, and references. Generic descriptors (“our company”, “the team”) do not count as grounding. single, ord., ||=4|V|=4 Unknown; Abstract; Grounded; Highly Specific. content_embedded Non-natural-language artifacts embedded inline in the query text. multi, nom., ||=5|V|=5 Code; Formula; Data; URL; Other. Observable char_count_bucket Query length in characters, bucketed. Measured directly from the query text. single, ord., ||=6|V|=6 Unknown; 1–50; 50–100; 100–200; 200–500; ≥500≥ 500. word_count_bucket Query length in words, bucketed. Measured directly from the query text. single, ord., ||=6|V|=6 Unknown; 1–10; 10–20; 20–40; 40–100; ≥100≥ 100. query_language Natural language of the query, detected from the text. single, nom., ||=11|V|=11 Unknown; English; Spanish; French; German; Italian; Portuguese; Japanese; Mandarin; Hindi; Other. grounding_explicit Types of grounding artifacts (attached files, referenced entities) the user provides alongside the query. multi, nom., ||=20|V|=20 pptx; docx; xlsx; pdf; csv; md; txt; json; html; aspx; png; image; file; page; loop; meeting; email; chat; people; Other. Appendix B Roundtrip Metric Formulae This appendix gives the full definitions of the three families of quantities used in the roundtrip analysis (§7.2, §7.3): the per-dimension proxy distance, the proportional-redistribution confusion matrix, and the size-adjusted diagonal-concentration score. All classifier confidence scores are integers in [1,S][1,S], with S the schema’s maximum confidence (typically S=5S=5); the distinguished value Unknown always carries score 0. B.1 Per-Dimension Distance A proxy assigns each dimension either a single [value, score] pair or, for multi-valued dimensions, a list of such pairs. Single-valued dimensions. Let (v1,s1)(v_1,s_1) and (v2,s2)(v_2,s_2) be the original and reconstructed pairs. (i) one side is Unknown: d=sother/Sd=s_other/S (and d=0d=0 if both are Unknown). (i) v1=v2v_1=v_2: d=|s1−s2|/Sd=|s_1-s_2|/S (pure confidence mismatch). (i) v1≠v2v_1≠ v_2, both real: d=min(1,fvalue⋅fscore),fscore=s1+s22S,d= \! (1,\;f_value· f_score ), f_score= s_1+s_22S, (15) where fvalue=1f_value=1 for nominal dimensions and, for ordinal dimensions of arity n, fvalue=|rank(v1)−rank(v2)|(n−1)/2.f_value= |rank(v_1)-rank(v_2)|(n-1)/2. (16) Adjacent ordinal values therefore incur only a partial penalty, while values more than half the range apart saturate to 1; high-confidence mismatches are penalized more than low-confidence ones. Multi-valued dimensions. Let O and R be the original and reconstructed value–score sets, with score lookups svO,svRs^O_v,s^R_v. Partition the value union into shared, original-only, and reconstructed-only labels. If shared=∅shared= and at least one side is non-empty, d=1d=1. Otherwise, d=min(1,∑v∈shared|svO−svR|S+∑v∈orig-onlysvOS+∑v∈recon-onlysvRS|shared|).d= \! (1,\; _v |s^O_v-s^R_v|S+ _v -only s^O_vS+ _v -only s^R_vS|shared| ). (17) Dividing by the number of shared values lets correct matches buy tolerance for additional mismatches. Proxy-level aggregation. The scalar proxy distance d∈[0,1]d∈[0,1] used in Table 3 is a weighted mean of the per-dimension distances across all non-observable dimensions, using the same raw importance weights wiw_i that enter the RA score (§4), i.e. before the redundancy discount gig_i of (7) is applied. Per-proxy aggregation has no notion of cross-dimension overlap to discount, so only the operator-supplied importance weights are used. B.2 Confusion-Matrix Construction For a single roundtrip line with original values (vk,sk)k=1K\(v_k,s_k)\_k=1^K and reconstructed values (ul,tl)l=1L\(u_l,t_l)\_l=1^L, mass is added to the per-dimension matrix C∈ℝ≥0|V|×|V|C _≥ 0^|V|×|V| in four steps: 1. Score floor for Unknown. Replace each Unknown’s score by max(s,1) (s,1) so that it contributes visible mass; weights elsewhere are wk=skw_k=s_k. 2. Balance. Let SO=∑kwkS_O= _kw_k and SR=∑lwl′S_R= _lw _l. Append a synthetic <blank> pseudovalue with weight |SO−SR||S_O-S_R| to the lighter side; the total mass to distribute is M=2max(SO,SR)M=2 (S_O,S_R). 3. Diagonal (matches). For each value v present on both sides with weights (w,w′)(w,w ), add 2min(w,w′)2 (w,w ) to Cv,vC_v,v and subtract min(w,w′) (w,w ) from each side’s residual. 4. Off-diagonal (proportional). Distribute the remaining mass MremM_rem across all (original, reconstructed) pairs in proportion to the residual marginals: Cvk,ul+=Mremwk∑k′wk′wl′∑l′wl′.C_v_k,u_l +=M_rem\, w_k _k w_k \, w _l _l w _l . (18) After accumulating across all M×NM\!×\!N roundtrip samples, each row is normalized so that ∑jCij=1 _jC_ij=1. B.3 Diagonal Concentration Each confusion matrix is summarized by a scalar concentration around the main diagonal. We first compute a raw score and then apply a size adjustment. Raw score. For nominal dimensions the raw score is the diagonal fraction, sraw=∑iCii∑i,jCij.s_raw= _iC_i _i,jC_ij. (19) For ordinal dimensions of arity n each cell receives a proximity weight: wij=1−|i−j|n−1,both ordered values at ranks i,j,1,both Unknown,0.5,exactly one side Unknown,0,either side is <blank>,w_ij= cases1- |i-j|n-1,&both ordered values at ranks i,j,\\ 1,&both Unknown,\\ 0.5,&exactly one side Unknown,\\ 0,&either side is <blank>, cases (20) yielding sraw=∑i,jCijwij∑i,jCij.s_raw= _i,jC_ij\,w_ij _i,jC_ij. (21) Blank entries (lost or hallucinated labels) carry the maximum penalty; Unknown on only one side is treated as a half mismatch, reflecting partial (not definitive) information. Size adjustment. Larger matrices spread mass across more cells, making high raw scores harder to achieve; we compensate with a size-dependent exponent, sadj=sraw 1/n,s_adj=s_raw^\,1/ n, (22) which leaves 0 and 11 fixed and boosts intermediate scores in proportion to n n. At n=10n=10, a raw score of 0.600.60 maps to 0.810.81. Macro vs. micro averaging. The micro variant applies (19)/(21) to the full mass-weighted matrix, so frequent values dominate. The macro variant computes the per-row diagonal share for each value and averages them with equal weight, exposing how rare values behave. All “macro diagonal concentration” figures in §7.3 use the macro variant. Appendix C Unbiased Mutual Information for Discriminative Power The discriminative-power evaluation of §7.5 requires safeguards against the upward bias of mutual information on finite samples. Disjoint training and validation folds already control CLL gain. However, for ordinal dimensions, we must additionally choose between the raw partition and a PAV-merged coarsening on the same training fold; this nested comparison cannot be settled by held-out CLL alone, so we use an Adjusted Mutual Information (AMI) tiebreaker. Need for chance correction. Under X⟂YX \!\!\! Y, the plug-in estimator MI^(X;Y)=∑ijp^ijlog(p^ij/(p^i⋅p^⋅j)) MI(X;Y)= _ij p_ij ( p_ij/( p_i· p_· j)) is strictly positive in expectation, with bias roughly (R−1)(C−1)/(2N)(R-1)(C-1)/(2N) that grows with the partition cardinality R×CR\!×\!C. For nested candidates the data-processing inequality already pins the direction (the finer partition wins in the population), so the test is one-sided and any uncorrected positive bias deterministically flips the verdict toward the finer partition. Vinh–Epps–Bailey adjusted MI. We adopt the closed-form expectation under a fixed-margin hypergeometric null derived in Vinh et al. [2010]: [MI∣,,N]=∑i,j∑nij=ℓijuijnijNlogNnijaibjℙ[Nij=nij∣ai,bj,N],E[MI ,b,N]= _i,j _n_ij= _ij^u_ij n_ijN\, \! N\,n_ija_ib_j\,P[N_ij=n_ij a_i,b_j,N], (23) where (ℓij,uij)=(max(0,ai+bj−N),min(ai,bj))( _ij,u_ij)=( (0,a_i+b_j-N), (a_i,b_j)) and ℙ[⋅]P[·] is the central hypergeometric mass. The chance-corrected score is AMImax(X;Y)=MI(X;Y)−[MI]max(H(X),H(Y))−[MI],AMI_ (X;Y)= MI(X;Y)-E[MI] (H(X),H(Y))-E[MI], (24) clipped to [0,1][0,1]. We use the max normalizer because it is the only one Vinh et al. prove respects the unit interval under the hypergeometric null. The inner sum is evaluated in log-space via a precomputed logΓ table to avoid factorial overflow. Kish correction for fractional weights. Equation (23) requires integer margins and an integer total N. Our contingencies are weighted by classifier-confidence scores we∈(0,1]w_e∈(0,1], so N=∑eweN= _ew_e is generally non-integer. We replace it with Kish’s effective sample size Kish [1965], Neff=(∑ewe)2∑ewe2,N_eff= ( _ew_e )^2 _ew_e^2, (25) rounded to the nearest integer. Intuitively, NeffN_eff shrinks when a few events carry most of the weight; the null distribution should be wider in that regime, and the AMI adjustment correspondingly more aggressive. Weighted margins are scaled by Neff/N_eff/N and rounded with the largest-remainder method to obtain integer pseudo-margins (∗,∗)(a^*,b^*) summing to NeffN_eff. These are substituted for (,,N)(a,b,N) throughout the right-hand side of (23). Only [MI]E[MI] is computed this way; the observed MI(X;Y)MI(X;Y) and the entropies in (24) come from the original weighted contingency. The substitution is the conventional adjustment when applying a counting null to a weighted sample and errs conservatively (slightly inflating [MI]E[MI] and so deflating AMIAMI). Limited application for AMI. We use AMI only as the train-fold tiebreaker between the raw and PAV-refined partitions for ordinal dimensions, where a like-for-like comparison that controls for partition cardinality is essential. However, the operational question “how much would knowing X help me predict Y on a new interaction?” is answered directly by held-out CLL gain, which is what §7.5 reports.