Paper deep dive
Glyph: A Multi-Strategy Agentic System for Column Description and Sensitivity-Ontology Tagging of Enterprise Data Catalogs
Kostia Kudriavtsev, Parvez Rafi, Sha Sundaram
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 92%
Last extracted: 9/10/2026, 2:12:37 AM
Summary
The paper introduces Glyph, a production-grade multi-agent system designed to automate column description generation and sensitivity-ontology tagging for enterprise data catalogs. Glyph addresses the challenge of 'documentation debt' in large-scale data lakes by using two cooperating LLM agents: the Descriptor, which generates column descriptions grounded in pipeline source code via active Retrieval-Augmented Generation (RAG), and the Tagger, which assigns labels from a 275-leaf Data Classification Ontology. The Tagger employs a multi-strategy approach, running a description-based LLM tagger, a regex-based tagger, and a metadata-based tagger (using a fine-tuned MiniLM encoder) in parallel, then fusing their outputs using Reciprocal Rank Fusion (RRF). The system is designed to be privacy-preserving by operating on metadata and code rather than cell values, ensuring compliance with enterprise governance policies.
Entities (8)
Relation Signals (8)
Glyph → composedof → Descriptor
confidence 95% · Glyph is assembled from components... The Descriptor... The Tagger
Glyph → composedof → Tagger
confidence 95% · Glyph is assembled from components... The Tagger
Glyph → developedby → Apple
confidence 95% · Apple Seattle, WA and Cupertino, CA, USA k_kudriavtsev@apple.com
Tagger → usesmethod → Reciprocal Rank Fusion
confidence 93% · The Tagger... fuses their ranked outputs with Reciprocal Rank Fusion (RRF).
Tagger → targets → Data Classification Ontology
confidence 92% · The Tagger assigns labels from a governed 275-leaf Data Classification Ontology
Descriptor → usesmethod → active Retrieval-Augmented Generation
confidence 92% · The Descriptor grounds generation in the pipeline source code... via a reasoning--acting tool loop (active Retrieval-Augmented Generation).
Tagger → usesmodel → MiniLM
confidence 90% · We fine-tune a 6-layer MiniLM metadata encoder... relative to the stock base encoder.
Glyph → builtwith → LangGraph
confidence 88% · Glyph is a graph-structured agent application... built with LangGraph
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Enterprise data lakes accumulate tables faster than human stewards can document or classify them, leaving columns with missing descriptions and unassigned governance labels. This documentation debt undermines data discovery, access control, and regulatory compliance. We present Glyph, a production system that frames two coupled problems, column description generation and column type annotation for data classification, as cooperating LLM agents orchestrated as stateful graphs. The Descriptor grounds generation in the pipeline source code that produces each column, retrieved on demand from an enterprise GitHub via a reasoning--acting tool loop (active Retrieval-Augmented Generation). The Tagger assigns labels from a governed 275-leaf Data Classification Ontology by running three complementary strategies in parallel (a description tagger, a line-of-business regex tagger, and a metadata tagger backed by a fine-tuned contrastive encoder over a vector database), then fuses their ranked outputs with Reciprocal Rank Fusion (RRF). We fine-tune a 6-layer MiniLM metadata encoder with an in-batch contrastive objective, lifting same-tag retrieval on an in-distribution held-out split from NDCG@10 0.55 to 0.92 (MAP@100 $0.19 \rightarrow 0.90$) relative to the stock base encoder. We report end-to-end multi-label tagging quality under a recall-weighted F2 objective across three evaluation groups, an ablation isolating each strategy and the RRF fusion, and the engineering decisions that distinguish Glyph from prior column-type-annotation work and from commercial value/regex sensitivity scanners: value-free and code-grounded design, per-tag provenance, and graceful degradation. Together these make multi-agent LLM cataloging auditable and operable as a production service.
Tags
Links
- Source: https://arxiv.org/abs/2609.10430v1
- Canonical: https://arxiv.org/abs/2609.10430v1
Trouble viewing inline? Open PDF directly →
Full Text
53,477 characters extracted from source content.
Expand or collapse full text
Glyph: A Multi-Strategy Agentic System for Column Description and Sensitivity-Ontology Tagging of Enterprise Data Catalogs Kostia Kudriavtsev Parvez Rafi Sha Sundaram Abstract Enterprise data lakes accumulate tables faster than human stewards can document or classify them, leaving columns with missing descriptions and unassigned governance labels. This documentation debt undermines data discovery, access control, and regulatory compliance. We present Glyph, a production system that frames two coupled problems, column description generation and column type annotation for data classification, as cooperating LLM agents orchestrated as stateful graphs. The Descriptor grounds generation in the pipeline source code that produces each column, retrieved on demand from an enterprise GitHub via a reasoning–acting tool loop (active Retrieval-Augmented Generation). The Tagger assigns labels from a governed 275-leaf Data Classification Ontology by running three complementary strategies in parallel (a description tagger, a line-of-business regex tagger, and a metadata tagger backed by a fine-tuned contrastive encoder over a vector database), then fuses their ranked outputs with Reciprocal Rank Fusion (RRF). We fine-tune a 6-layer MiniLM metadata encoder with an in-batch contrastive objective, lifting same-tag retrieval on an in-distribution held-out split from NDCG@10 0.55 to 0.92 (MAP@100 0.19→0.900.19→ 0.90) relative to the stock base encoder. We report end-to-end multi-label tagging quality under a recall-weighted F2F_2 objective across three evaluation groups, an ablation isolating each strategy and the RRF fusion, and the engineering decisions that distinguish Glyph from prior column-type-annotation work and from commercial value/regex sensitivity scanners: value-free and code-grounded design, per-tag provenance, and graceful degradation. Together these make multi-agent LLM cataloging auditable and operable as a production service. Apple Seattle, WA and Cupertino, CA, USA k_kudriavtsev@apple.com, p_rafi@apple.com, shas@apple.com 1 Introduction Modern analytics organizations operate data catalogs containing tens of thousands of tables and millions of columns. Two questions must be answered for every column before the data can be safely used: what the column means (documentation) and what category of data it holds (classification and governance). In practice both are answered manually, if at all: descriptions are missing or stale, and sensitive-data tagging, needed for access control and privacy compliance, lags discovery by months. The cost is concrete: undocumented personally identifiable information (PII) is discovered reactively after incidents rather than proactively, and new engineers cannot reason about datasets they did not build. One central challenge is the scale of dataset tagging. In practice, datasets are not confined to a single storage technology: the same logical catalog may span Apache Iceberg tables, relational databases (RDBMS), and Cassandra column-family stores. Consistently tagging metadata across these heterogeneous backends, at the volume encountered in a production environment, is intractable by hand. A single organization’s production catalog spans ∼ 3.4M columns across ∼ 98,000 tables, distributed over four lines of business and the three storage backends above. At this scale, manual annotation is not merely slow but economically infeasible: even a highly optimistic rate of one column per minute would require roughly 30 person-years of expert effort for this catalog alone, and the catalog drifts faster than any team can keep pace. This combination of volume and backend heterogeneity, neither of which is addressed by point solutions tied to one storage engine, is precisely what motivates an automated, learning-based approach: only a system that generalizes across backends and improves without per-table human labeling can bring documentation and sensitivity tagging to catalog scale. The second task is an instance of column type annotation (CTA), the assignment of a semantic label to a column. It is a problem studied for over fifteen years, from joint graphical models over web tables (Limaye et al. 2010) through supervised deep classifiers (Hulsebos et al. 2019; Zhang et al. 2020), pre-trained table encoders (Suhara et al. 2022; Deng et al. 2021), and, most recently, large language models (Korini and Bizer 2023; Feuer et al. 2024; Kayali et al. 2024). Glyph differs from this lineage in three ways that define its contribution. First, its target is not an open-domain type vocabulary (DBpedia/Schema.org) but a governed enterprise sensitivity ontology: 275 leaf annotations organized by and sensitivity level, where the dominant class (internal non-sensitive) covers roughly 72% of columns. Second, Glyph is privacy-preserving by design: it never reads cell values. The encoder sees only the structural key catalog.database.table.column, and the Descriptor grounds in the producing source code rather than the data. Reasoning about sensitive tables without ever accessing their data is a first-class capability, not a limitation: it satisfies enterprise privacy and data-access policy by construction. Third, Glyph couples heterogeneous-strategy rank fusion with an LLM-as-Judge self-refinement loop and per-tag provenance, so every label carries the source, score, and reasoning behind it, a finer-grained provenance than the rule logs of commercial scanners or the bare labels of academic CTA systems. Four requirements shaped the design and distinguish Glyph from a naive “ask the model” prototype: (1) grounding over guessing. Ambiguous names, such as a column name that may denote either a PII-sensitive person name or a misnomer for a movie title, must be resolved from authoritative evidence rather than parametric guesses. (2) Defense in depth. Since no single signal suffices, Glyph combines descriptions, structural metadata, historical assignments, and naming conventions in a principled way. (3) Measurable, self-improving quality. Output is verified automatically against an explicit bar and refined when it falls short. (4) Production discipline: stateless horizontal scaling, deterministic decoding, per-request authentication, structured tracing, and reproducible training. Contributions. (a) A two-agent architecture that separates meaning generation (Descriptor) from category assignment (Tagger), connected by a shared state and ontology, and, to our knowledge, the first published account of multi-strategy CTA targeting a governed enterprise sensitivity ontology with sensitivity levels. (b) A multi-strategy tagger that fuses dense retrieval, LLM reasoning, and rule-based signals with Reciprocal Rank Fusion (RRF) (Cormack et al. 2009) and a candidate-constrained LLM re-ranker, with full per-tag provenance. (c) A contrastive fine-tuning recipe (Henderson et al. 2017) for a metadata-specific encoder, improving in-distribution candidate recall of the metadata tagger (NDCG@10 0.55→0.920.55→ 0.92, MAP@100 0.19→0.900.19→ 0.90) on a per-tag, column-level held-out split of the training corpus. (d) An end-to-end evaluation methodology built on steward-curated, per-line-of-business sets under strict and expanding regimes, plus a strategy ablation, and a documented production stack that turns the prototype into a reliable service. 2 Related Work Glyph sits at the intersection of two literatures: semantic column type annotation and enterprise metadata/governance catalogs, which we survey in turn. 2.1 Semantic Column Type Annotation Classical and supervised CTA. Assigning a category label to a column was pioneered by Limaye et al. (2010), whose graphical model jointly annotates cells, column types, and inter-column relations against a knowledge base via collective inference; T2K Match (Ritze et al. 2015) is the reference unsupervised matcher to DBpedia. Sherlock (Hulsebos et al. 2019) reframed CTA as supervised learning over character-, word-, and statistics-level features of column values, predicting one of 78 types at support-weighted F1 ≈0.89≈ 0.89 on VizNet, and Sato (Zhang et al. 2020) added table context and a CRF for ≈0.92≈ 0.92. Pham et al. (2016) rank candidate labels by similarity of an unseen column to labeled examples, and Meimei (Takeoka et al. 2019) is one of the few classical methods that is explicitly multi-label per column. Pre-trained table encoders. TURL (Deng et al. 2021), TaBERT (Yin et al. 2020), and TABBIE (Iida et al. 2021) pre-train structure-aware Transformers over web tables; Doduo (Suhara et al. 2022) serializes an entire table and fine-tunes a single BERT in a multi-task setup to predict column types and relations, reaching ≈0.92≈ 0.92 micro-F1 on VizNet and remaining the canonical fine-tuned baseline. RECA (Sun et al. 2023) enriches a target column with related tables retrieved by entity-schema matching, and Watchog (Miao and Wang 2024) pre-trains column representations contrastively for label efficiency. Every system in this family consumes cell values, though Doduo and TURL read headers alongside them. LLM-era CTA. Korini and Bizer (2023) frame CTA as text classification and prompt ChatGPT, finding that few-shot, label-narrowing prompts approach fine-tuned baselines on SOTAB. ArcheType (Feuer et al. 2024) maps free-form LLM output back to a fixed vocabulary by label remapping, reaching near-SOTA with open models. CHORUS (Kayali et al. 2024) shows one foundation model can subsume table-class detection, CTA, and join discovery via prompt assembly, and RACOON (Wei et al. 2024) retrieval-augments LLM CTA with a knowledge graph. Table-GPT (Li et al. 2024) and TableLlama (Zhang et al. 2024) instead instruction-tune LLMs for table tasks. How Glyph differs. Glyph’s metadata tagger (Section 4.3.3) descends from the similarity-to-labeled-examples line but uses a fine-tuned neural encoder and ANN search over a curated corpus rather than hand-engineered features; its regex tagger (Section 4.3.2) is a T2K-style matcher (Ritze et al. 2015); and its description tagger (Section 4.3.1) is a retrieval-constrained, candidate-validated variant of single-prompt LLM CTA (Korini and Bizer 2023) with ArcheType-style label grounding (Feuer et al. 2024). Three differences are structural. Generation is constrained to retrieved ontology candidates rather than remapped post hoc. Three heterogeneous strategies are fused by RRF rather than one prompt being trusted, a per-column trade against the collective inference of Limaye et al. (2010) and Sato (Zhang et al. 2020) that we revisit in Section 6. And none of these systems targets a governed sensitivity taxonomy from metadata alone, without cell values. 2.2 Benchmarks The dominant CTA benchmarks measure open-domain types: VizNet (Hu et al. 2019) supplies the 78-type Sherlock/Sato corpus; SOTAB (Korini et al. 2022) labels web tables with 91 Schema.org types; GitTables (Hulsebos et al. 2023) annotates ∼ 1M repository-mined tables and documents that CTA accuracy drops on realistic relational tables; SemTab (Jiménez-Ruiz et al. 2019) runs the annual CEA/CTA/CPA challenge against DBpedia/Wikidata. None measures a sensitivity/data-classification taxonomy, none is multi-label in Glyph’s sense, and all treat cell values as the primary signal even where headers are present. Because SOTAB and GitTables do ship column headers, they remain a usable out-of-distribution probe for Glyph’s metadata-only encoder (Section 7) — concrete near-term work. 2.3 Industrial Catalogs and Data-Security Posture Goods (Halevy et al. 2016) is the canonical account of automatic enterprise metadata enrichment at scale, crawling billions of datasets to infer schema, provenance, and summaries, but not LLM-generated descriptions or governed sensitivity labels. Aurum (Castro Fernández et al. 2018) and Table-Union search (Nargesian et al. 2018) link columns by content signatures for discovery; Glyph’s fine-tuned encoder plus ANN search is the neural analogue used for tagging rather than join discovery. Open catalogs such as DataHub (LinkedIn 2019) and Amundsen (Grover and others 2020) store column tags and lineage but rely on rule/keyword classifiers and human-authored descriptions, and Apache Atlas (Apache Software Foundation 2015) propagates sensitivity classifications along lineage without inferring the seed label. Commercial governance and value-scanning tools (Microsoft 2021; Informatica 2018; Microsoft 2018; Google Cloud 2017; BigID, Inc. 2016; Immuta, Inc. 2015) auto-apply classifications predominantly via regex, dictionaries, and value-pattern matching over cell values. Glyph is the enrichment engine that produces the descriptions and seed labels those catalogs store and propagate; by reasoning from the producing source code and structural metadata it can classify columns whose sensitivity is contextual rather than format-detectable — a generic recommendations column holding a list of recommended apps, which a value scanner sees only as high-entropy noise. 3 Problem Formulation and the Data Classification Ontology We formalize the two problems Glyph solves, description generation and multi-label sensitivity classification, and then describe the governed ontology that makes the classification task fundamentally unlike open-domain column-type annotation. Task. Let a column be identified by its metadata key x=⟨LOB,database,table,column⟩x= ,database,table,column (1) and (optionally) a natural-language description dxd_x, where LOB stands for line of business (a representation of the company’s enterprise hierarchy). Glyph solves two problems: (i) description generation, producing dxd_x from authoritative evidence; and (i) multi-label classification (Tsoumakas and Katakis 2007), assigning a subset Yx⊆Y_x of an ontology O of sensitivity annotations, each with a confidence and a justification. The ontology. O is the Data Classification Ontology, a governed taxonomy of 275 leaf annotations (e.g., USER_NAME, USER_ID, EMAIL_ADDR)11 1 Ontology tag names shown throughout this paper (e.g., USER_ID, EMAIL_ADDR, NON_SENSITIVE) are illustrative placeholders; the production Data Classification Ontology uses different internal codes, which are withheld for confidentiality. grouped under classification categories. The categories form a hierarchy from generic (e.g., the PII category) to specific (e.g., the EMAIL_ADDR leaf annotation). Each annotation additionally carries several dimensions, such as a sensitivity-level scale, so the ontology encodes not only what a column is but how sensitive it is and to whom. This makes Glyph’s target fundamentally unlike the open-domain type vocabularies of VizNet (Hu et al. 2019), SOTAB (Korini et al. 2022), GitTables (Hulsebos et al. 2023), and SemTab (Jiménez-Ruiz et al. 2019): the labels are governance decisions, the task is multi-label, and the distribution is severely skewed. The internal non-sensitive class (NON_SENSITIVE) alone accounts for ≈72%≈ 72\% of columns. This imbalance is the central learning challenge and motivates both class-balanced training and an evaluation that does not let the majority class mask minority-class errors (Sections 4.4 and 5). Why values are off-limits. Reading cell values to classify enterprise columns is, for many of the most sensitive datasets, precisely the action governance forbids before classification exists, a chicken-and-egg constraint absent from value-driven CTA (Hulsebos et al. 2019; Zhang et al. 2020; Suhara et al. 2022). Glyph therefore operates on metadata and code (which engineers may read) rather than data, trading a signal for deployability on restricted-access tables where Sherlock/Doduo-style scanners cannot run. 4 Approach 4.1 System Overview Glyph is a graph-structured agent application: each agent is a directed graph of nodes (pure functions over a typed state) connected by ordinary and conditional edges, built with LangGraph (Chase and LangChain Inc. 2024). Nodes are produced by a builder pattern that injects dependencies (catalog client, LLM handles, search services) through an ApplicationConfig, decoupling node logic from I/O. State is a Pydantic (Colvin and others 2023) model, validated at every hop, and configuration is layered in three tiers (environment defaults → per-deployment AgentConfig → per-request HyperParams). Fig. 1 depicts the two agents and their nodes. Descriptor fetch_schemaanalyze_batchGitHub RAGgrade_resultsLLM-as-Judgegenerate_dataset_descriptionformat_outputretry w/ feedback Tagger prepare_columnsllm_desc_taggerdescription-basedregex_taggerrule-basedllm_metadata_taggermetadata-onlyevaluate_metricsLLM-as-Judgemerge_all_tagsRRF k=60k=60tag_normalizertier-weightedformat_outputretry failed cols Figure 1: Glyph as two independently deployable LangGraph agents. The Descriptor grounds column descriptions in the producing source code via an active-RAG tool loop, with an LLM-as-Judge refinement cycle. The Tagger runs three complementary strategies in parallel and fuses their ranked tag lists with Reciprocal Rank Fusion (k=60k=60), then re-scores them by sensitivity tier; a separate judge refines the description tagger. Orange boxes are LLM-as-Judge stages, grey boxes are I/O. The two agents are independently deployable REST services but compose naturally: the Descriptor’s generated descriptions are the ideal input to the description tagger. Both decode at temperature 0.0 for determinism, cap token usage, and create LLM handles per request so model choice and credentials can vary by call. Glyph is assembled from components whose behavior is well characterized — active retrieval-augmented generation, contrastive dense retrieval, multi-label classification, rank fusion, and LLM-as-Judge critique. This is a deliberate choice for a governed sensitivity ontology, where auditable components with known failure modes matter more than novel machinery. The contribution is the composition: which signals are fused, how they are validated against the ontology, and what it takes to serve them. Each mechanism is described below with its lineage cited in place. 4.2 The Descriptor Agent The Descriptor answers “what does this column mean?” as a four-stage pipeline (top panel of Fig. 1). Schema Discovery. fetch_schema queries the enterprise data registry (the organization’s internal metadata catalog) to collect table details: column names, types, existing descriptions, etc. Source-grounded generation (RAG). analyze_batch is the core. To avoid the latency and inconsistency of per-column calls, all columns are analyzed in a single LLM invocation, in batches of at most 50 columns. The model is given the schema and a tool, GitHub Enterprise code search, that it may call autonomously; the evidence is code, not data — the Descriptor never reads column values. This realizes active RAG (Lewis et al. 2020; Yao et al. 2023), with the interleaved reasoning of chain-of-thought prompting (Wei et al. 2022) and in contrast to a fixed dense retriever (Karpukhin et al. 2020): the LLM decides which queries to run (a default budget of 3 tool calls per column, capped per batch with forced finalization), reads context around each match, and grounds descriptions in the application code that actually produces each column.22 2 Code search is rate-limit aware: a proactive 2 s pause after 25 calls in a 60 s window, plus exponential back-off (30 s initial, doubling, up to 5 attempts). A heuristic confidence score is assigned to each description from observable signals (length, presence of technical and business terms, whether the producing job was identified, and PII awareness), capped at 1.0. PII flagging. A first-class objective: columns such as user_id are prefixed with explicit PII warnings, and the autograder scores PII awareness directly. Whereas value-level scanners such as Presidio (Microsoft 2018) and Cloud DLP (Google Cloud 2017) detect PII from recognizable value formats, Glyph reasons about sensitivity from the code and metadata, catching identifiers whose values are opaque hashes that format detectors miss. Dataset description. A generate_dataset_description node synthesizes a table-level summary of business purpose from the per-column results, giving downstream consumers a self-documenting catalog entry. Grading and refinement. When enabled, grade_results invokes a judge model over the descriptions in batches. The judge scores four criteria — plausibility, specificity, completeness, and PII awareness — aggregated with fixed weights (0.350.35, 0.250.25, 0.250.25, 0.150.15), and is configured by default to be a different model family from the generator, since LLM judges exhibit self-enhancement bias when scoring their own output (Zheng et al. 2023). Columns that fall below threshold are returned to analyze_batch with the judge’s per-criterion feedback; on retry the agent re-searches GitHub for the worst-scoring columns. The loop runs up to a small fixed cap (2 passes by default). 4.3 The Tagger Agent The Tagger answers “what category of data is this?” by assigning ontology tags (e.g., USER_ID, EMAIL_ADDR) with confidence scores and reasoning. Its defining feature is multi-strategy parallelism with principled fusion: three strategies run concurrently over the columns (each fanned out into worker batches), and their ranked tag lists are merged by a dedicated fusion node (bottom panel of Fig. 1). The three are chosen so that their failure modes are uncorrelated, allowing fusion to reconcile them. 4.3.1 Description Tagger The description tagger works from the meaning of a column and is retrieval-constrained: for each column it forms the query "name: description" (where the description is the column’s human-authored entry in the enterprise data registry when one exists, and the column name alone otherwise), retrieves the most similar ontology annotations from semantic search above a cosine threshold C, and asks the LLM to select and rank the best tags from those candidates, with reasoning. Because candidates are constrained to real ontology entries and validated afterward, hallucinated tags are rejected. This is a retrieval-constrained, candidate-validated variant of single-prompt LLM CTA (Korini and Bizer 2023) with ArcheType-style label grounding (Feuer et al. 2024); constraining generation to retrieved candidates (rather than ArcheType’s post-hoc remapping) is consistent with RACOON’s finding that retrieval improves LLM CTA (Wei et al. 2024). The ontology annotations (for example, EMAIL_ADDR: an electronic mail address, typically written as user@domain) are rich natural-language records, so this search uses a general-purpose all-MiniLM-L6-v2 (Reimers and Gurevych 2019; Wang et al. 2020) (384-dim) sentence encoder as-is. Grading and refinement. When grading is enabled, an evaluate_metrics node scores the description tagger’s tags with a combination of zero-cost deterministic metrics and a batched LLM-as-Judge (Zheng et al. 2023); columns whose tags fall below the configured threshold are returned to llm_desc_tagger for another attempt, a bounded Self-Refine (Madaan et al. 2023; Shinn et al. 2023) loop. The refined output then joins the regex and metadata taggers at fusion (Section 4.3.4); only the description tagger passes through this judge. 4.3.2 Regex Tagger The regex tagger encodes institutional knowledge by applying domain expert defined line-of-business-specific pattern rules over column names; for example, a column email has a hardcoded rule that assigns the tag EMAIL_ADDR. The strategy provides high-precision anchors in the T2K matching lineage (Ritze et al. 2015) that complement the learned signals. 4.3.3 Metadata Tagger The metadata tagger works from structural metadata alone, the fully qualified key catalog.database.table.column, without needing a human description, a metadata-only descendant of value-feature classifiers (Hulsebos et al. 2019) and similarity-to-example rankers (Pham et al. 2016). It is itself a RAG system (Lewis et al. 2020): the metadata key is encoded with a fine-tuned sentence encoder (Section 4.4) and searched against an exact inner-product (flat) vector index of verified training-corpus assignments built from human labeled corpus. Vectors are L2-normalized so inner product equals exact cosine similarity; candidates above a configured similarity S1 carry the tags of their nearest historical neighbors. Long, deeply nested types are truncated to 200 characters at clean delimiter boundaries to match training-time formatting and reduce embedding noise. A worker then presents the retrieved candidates, with inline similarity attribution, to the LLM, which selects and ranks the final tags from these candidates. Each selected tag carries a reasoning record noting exactly which source contributed at what confidence (e.g., [sources: ann=0.92]). To our knowledge, this per-tag, multi-source provenance is absent from prior CTA systems (Hulsebos et al. 2019; Zhang et al. 2020; Suhara et al. 2022; Deng et al. 2021; Yin et al. 2020; Iida et al. 2021; Sun et al. 2023; Miao and Wang 2024; Korini and Bizer 2023; Feuer et al. 2024; Kayali et al. 2024; Wei et al. 2024; Li et al. 2024; Zhang et al. 2024), which emit labels without source attribution, and finer-grained than the rule-hit logs of commercial scanners (Microsoft 2021; Informatica 2018; Microsoft 2018; Google Cloud 2017; BigID, Inc. 2016; Immuta, Inc. 2015). 4.3.4 Cross-Strategy Fusion with Reciprocal Rank Fusion The three strategies produce independent ranked lists per column on incomparable score scales (an ANN cosine, an LLM confidence, a binary rule hit). To combine them without calibration, merge_all_tags applies Reciprocal Rank Fusion (Cormack et al. 2009), which needs no score calibration — and which, unlike self-consistency ensembling (Wang et al. 2023), fuses ranked lists rather than voting over samples. For a tag appearing at rank rir_i (1-indexed) in each active source’s list, rrf(t)=∑i∈lists containing t1k+ri,k=60,rrf(t)= _i\,∈\,lists containing t 1k+r_i, k=60, normalized by the theoretical maximum nactive/(k+1)n_active/(k+1) (the score of a tag ranked #1 by every active source), yielding a confidence in [0,1][0,1] that equals 1.0 only under unanimous top agreement. When several strategies nominate the same tag, a fixed priority order (llm_desc_tagger >> llm_metadata_tagger >> regex_tagger) selects whose provenance fields (reasoning, ontology, grades) are inherited, while the confidence is the cross-strategy RRF score. The fused list is deduplicated by annotation name, with the priority order above deciding which source’s provenance is retained, and sorted with a deterministic tie-break on the annotation name. Finally, tag_normalizer re-scores the fused list against the governed sensitivity tiers of Section 3: it blends the RRF confidence c with the annotation’s ontology tier weight w as αc+(1−α)wα\,c+(1-α)\,w, where untiered and higher-sensitivity annotations are unweighted (w=1.0w=1.0) and lower-sensitivity tiers (levels 2, 3, 4) carry weights 0.750.75, 0.50.5, 0.250.25, respectively. Tags below a minimum normalized score are dropped and the remainder re-sorted, so the final ranking reflects both cross-strategy agreement and governed sensitivity level. Table 1 works this scoring through for a single column. Source INTERNAL COUNTRY NON_SENSITIVE Regex tagger ✓ Description tagger ✓ Metadata tagger ✓ ✓ RRF score 0.333 0.667 0.328 Table 1: Tagger votes and fused confidence for a country column — “The 2-letter ISO 3166-1 alpha-2 country code.” The fused row is the normalized RRF confidence (k=60k=60); the metadata tagger ranks COUNTRY above NON_SENSITIVE. 4.4 Domain-Specific Embedding Fine-Tuning The metadata tagger’s ANN search (Section 4.3.3) is the one component of the Tagger that a general-purpose encoder cannot serve well. Its queries are metadata keys that are terse, abbreviation-laden, and non-linguistic, and a stock model does not know that uat.core_db.users.uid and prod.analytics.events.user_id denote the same concept; abbreviations, catalog prefixes, and type variants are opaque to it, and the off-the-shelf model leaves substantial same-tag recall unrealized in our training-corpus retrieval diagnostic (MAP@100 ≈0.19≈ 0.19; Table 2, pre-tuning). We therefore fine-tune a compact all-MiniLM-L6-v2 encoder (a 6-layer distillation of BERT (Devlin et al. 2019), a Transformer (Vaswani et al. 2017) with 384 hidden units, ∼ 90 MB) with an in-batch contrastive objective. This encoder is specific to the metadata index; the description tagger’s ontology search uses a stock encoder unchanged (Section 4.3.1). Training data. A training corpus of metadata_key → tags assignments is extracted from a production system and represents already tagged and verified assignments. Pairs are constructed within tag groups: any two columns sharing a tag form an (anchor, positive) pair. Because a few tags (notably NON_SENSITIVE, ≈ 72%) dominate, we cap pairs per tag (3 000 for the reported encoder) and require a minimum group size, yielding 70 948 balanced training pairs. Objective. We use MultipleNegativesRankingLoss (Henderson et al. 2017) (scale 20, cosine similarity): for each (anchor, positive) pair, all other positives in the batch act as negatives, pulling same-tag columns together and pushing different-tag columns apart — the in-batch-negatives recipe that also underpins dense passage retrieval (Karpukhin et al. 2020). Evaluation. A per-tag random 10% column-level held-out query set, with each tag’s contribution capped at 500 keys to bound memory (9,242 queries against an 83,229-entry corpus), is scored with an Information-Retrieval evaluator (Reimers and Gurevych 2019; Järvelin and Kekäläinen 2002) using NDCG, MRR, MAP, and Accuracy@k. Because the hold-out is at column granularity, columns from the same table may appear in both the query set and the corpus, so these numbers are an in-distribution upper bound that does not control for within-table leakage (Kaufman et al. 2012). Table-disjoint splitting, at (catalog, db, table) granularity, with floors of ≥30≥\!30 test / ≥90≥\!90 train samples per tag, is applied separately to the training corpus, to index the vector-search corpus; the encoder fine-tuning itself uses the column-level hold-out described above. Table 2 quantifies how well the encoder surfaces same-tag historical neighbors and therefore upper-bounds the candidate recall of the metadata tagger; end-to-end tagging accuracy on unseen tables (Section 5) and out-of-distribution generalization (Section 7) are addressed separately. The relevant set for a query is all same-tag corpus entries, so its size varies enormously by tag: huge for NON_SENSITIVE, a handful for rare tags. This is why recall@k is small while accuracy@k stays high (≥0.92≥\!0.92 post-tuning). MAP@100 is the most informative line because it rewards recovering the full same-tag pool regardless of its size: its +0.71+0.71 gain shows the model surfaces far more of the relevant neighbors that the metadata tagger’s LLM then re-ranks. Metric Pre-tuning Post-tuning Δ NDCG@10 0.5497 0.9225 +0.3728+0.3728 MRR@10 0.7875 0.9469 +0.1594+0.1594 Accuracy@1 0.7138 0.9229 +0.2091+0.2091 Accuracy@5 0.8812 0.9797 +0.0986+0.0986 MAP@100 0.1888 0.9009 +0.7121+0.7121 Table 2: In-distribution retrieval quality of the metadata candidate-generation encoder on a per-tag, column-level held-out split of the training corpus, before and after contrastive fine-tuning. These are retrieval, not tagging, metrics. Both the encoder and the vector index are versioned and published to S3-compatible object storage under timestamped paths; the vector index is rebuilt from the same training corpus (embedding the metadata_key without the tag suffix to avoid train/serve skew) and loaded read-only on container cold start. We use an exact inner-product (flat) index because the metadata tagger runs as a stateless, ephemeral container at modest scale (10210^2–10510^5 vectors); scaling past ∼ 10610^6 vectors would require a quantized or graph-based index. 5 End-to-End Evaluation Table 2 measures retrieval, not tagging. This section reports the metric that is comparable in spirit to the CTA literature: end-to-end multi-label tagging quality of the full Tagger (all three taggers plus RRF) against curated expectations. F1F_1 F2F_2 Configuration G1 G2 G3 Overall G1 G2 G3 Overall p50 (s) Regex tagger only 0.062 0.103 0.128 0.077 0.040 0.067 0.085 0.050 0.30 Description tagger only, Claude Sonnet 4.5† 0.122 0.326 0.411 0.205 0.250 0.325 0.590 0.238 47.31 Metadata tagger only 0.890 0.969 0.956 0.912 0.853 0.957 0.937 0.880 40.41 Fusion, Claude Sonnet 4.5 0.878 0.951 0.932 0.897 0.878 0.961 0.938 0.890 57.80 † description tagger is evaluated only over columns that carry a description; columns without one are out of scope for this strategy. Table 3: Measured micro-averaged F1F_1 and F2F_2 per curated dataset group, overall F1F_1/F2F_2, and p50 request latency for each configuration (higher F1F_1/F2F_2 / lower latency is better). All runs cover the same 4 evaluation files. Per-group F1F_1/F2F_2 is aggregated from per-file confusion counts. The best F2F_2 per group is in bold. Harness and metrics. Glyph ships a tag-evaluation harness that compares predicted tags against curated expectations per column over the universe of candidate tags, computing the confusion quantities and the derived accuracy, precision, recall, and the FβF_β family (Manning et al. 2008). While we report precision, recall, F0.5F_0.5, and F1F_1 for completeness, we adopt F2F_2 as the primary selection metric. The choice of β in Fβ=(1+β2)PRβ2P+RF_β=(1+β^2)\, PRβ^2P+R encodes a cost model, and sensitivity tagging has a sharply asymmetric one. A false negative, a sensitive or PII-bearing column left untagged, silently defeats access control and privacy compliance and is typically discovered only reactively, after an incident. A false positive, an over-applied tag, is cheap to remediate and dismissed by a data steward during review. We therefore select on F2F_2 (β=2β=2), which weights recall twice as heavily as precision and aligns the headline number with the governance reality that missing a sensitive column is far costlier than over-tagging a benign one. Evaluation sets. These sets are curated by line of business and span multiple LOBs, each a collection of column metadata, expected-tags cases authored with data stewards. The evaluation sets are table-disjoint from the training corpus of Section 4.4: none of their 1,905 tables appears in the 15,205-table vector index, though the two draw on overlapping catalogs and databases, so no evaluated column was seen at training or indexing time. 5.1 Per-Dataset Tagging Quality (F2F_2) Table 3 reports measured, micro-averaged F2F_2 per curated dataset group (Group 1–3) for four end-to-end configurations, isolating (i) each strategy in isolation, and (i) the full multi-strategy fusion. Per-group figures are micro-averaged from the per-file confusion counts within each group (F2=5TP/(5TP+4FN+FP)F_2=5\,TP/(5\,TP+4\,FN+FP)); the table also reports the overall F2F_2 and the p50 request latency of each configuration. We lead with F2F_2 (Section 5): the ranking of configurations by F2F_2 is the ranking that minimizes missed sensitive-data classifications. Three findings stand out. First, no single strategy is sufficient: the regex tagger alone collapses to near-zero F2F_2 across every group (it fires on almost nothing outside its hand-written patterns), and the strongest lone strategy, the metadata tagger, reaches only 0.8800.880 overall. Second, fusion ranks above every single strategy under F2F_2: the best fused configuration lifts overall F2F_2 to 0.8900.890. This ordering is metric-dependent, and deliberately so: the three metrics disagree on which system to ship — under F1F_1 the fusion and the metadata tagger are tied, and under F0.5F_0.5 the single strategy leads — so the choice of β is a deployment decision, not a cosmetic one. Third, quality trades against latency: fusion’s F2F_2 comes at a p50 of 57.857.8 s. The description tagger is the outlier: at 47.347.3 s it spends 82%82\% of fusion’s latency to reach only 0.2380.238, below the metadata tagger’s 0.8800.880, so it is not a viable standalone configuration and earns its place only as an input to the fusion. The operative deployment choice is therefore metadata-only versus full fusion. Dimension Glyph (this work) Sherlock Sato Doduo Korini & Bizer Input signal Metadata only; no cell values Cell values Values + table topic Serialized table values Values + name Learning paradigm Fine-tuned contrastive encoder + rules + commercial LLM Supervised deep neural network (DNN) DNN + CRF Fine-tuned PLM (multi-task) Zero-/few-shot LLM Target taxonomy 275 curated tags 78 open types 78 open types 78 open types SOTAB types Fusion RRF none CRF joint multi-task head none Deployment commercial research research research research Reported eval F1 ≈ 0.897 and F2 ≈ 0.890 F1 ≈ 0.89 (VizNet) F1 ≈ 0.92 micro-F1 ≈ 0.92 F1 ≈ 0.85 (SOTAB) Table 4: Positioning of Glyph against representative CTA systems. Systems, in column order: Sherlock (Hulsebos et al. 2019), Sato (Zhang et al. 2020), Doduo (Suhara et al. 2022) and Korini and Bizer (2023). 5.2 Deployed Impact: Steward Acceptance Glyph runs in production behind the governance workflow: human stewards adjudicate its suggestions, and each accepted or rejected column–tag pair is written back to a data store. The training pipeline re-exports that store, re-fine-tunes the metadata encoder (Section 4.4) — a human-feedback loop that closes on a weekly cadence. 5075100accepted %63.3%99.8%123456consecutive week Figure 2: Weekly steward acceptance of Glyph’s tag suggestions over six consecutive weeks of production operation. The vertical axis begins at 50%. Figure 2 plots every steward decision to date: acceptance rises from 63.3%63.3\% to 99.8%99.8\% across ∼24,000 24,000 adjudicated pairs. Decision volume grows 21×21× over the same window and the population under review broadens alongside it, so we report the trend as an association rather than a causal effect. Because a steward only ever sees tags the system emitted, acceptance is a precision-like measure and is blind to false negatives; it complements rather than replaces the recall-weighted F2F_2 of Table 3. 5.3 Positioning Against the CTA Literature These numbers are not directly comparable to published CTA F1 (Sherlock (Hulsebos et al. 2019), Sato (Zhang et al. 2020), Doduo (Suhara et al. 2022), and the near-fine-tuned accuracy of ArcheType (Feuer et al. 2024) with open models across several label spaces), because the taxonomy (a curated enterprise list of tags with sensitivity levels vs. 78–91 open-domain types without sensitivity levels), the label cardinality (multi-label, dominated by a few specific tags), and the information available (metadata + code vs. cell values) all differ; prior-work figures are reproduced from each paper’s best-reported configuration and serve only as in-spirit reference points. Table 4 positions Glyph against representative CTA systems and commercial sensitivity scanners across the dimensions that matter for enterprise deployment. 6 Discussion and Future Work Glyph’s central bet is that redundancy plus principled fusion beats any single clever signal: a rule engine that is precise but brittle, a dense retriever that generalizes but drifts, and an LLM that reasons but hallucinates, reconciled by RRF and an ontology validator, with an autograder converting open-ended generation into a measurable, self-improving task. We are candid about what this paper does and does not establish. Operational lessons. Two design choices proved decisive in production and generalize beyond Glyph. (1) Value-free, code-grounded design (Section 1) lets Glyph run at all on restricted-access tables that value/regex scanners (Microsoft 2021; Informatica 2018; Microsoft 2018; Google Cloud 2017; BigID, Inc. 2016; Immuta, Inc. 2015) cannot touch. (2) Batching for cost: analyzing a whole table’s columns in one LLM call rather than one call per column is the ratio that dominates latency and cost at catalog scale. Future work. Two directions to extend Glyph. (1) Collective inference. Glyph fuses per-column signals and ignores the inter-column and inter-tag correlation exploited since Limaye et al. (2010), Sato (Zhang et al. 2020), and Meimei (Takeoka et al. 2019); evaluating such joint models under our NON_SENSITIVE tag dominance is a promising direction. (2) A reinforcement-learning loop for the Tagger: treating tag selection as a learned policy (Schulman et al. 2017) optimized against a reward model (Christiano et al. 2017; Ouyang et al. 2022), so that a persistent policy learns to rank rare tags, attacking class imbalance, and internalizes steward feedback rather than re-deriving quality per request. 7 Conclusion Glyph shows that automated, auditable data cataloging at enterprise scale is achievable by composing well-understood techniques inside a disciplined agentic framework, and by adapting the column-type-annotation tradition to a governed sensitivity ontology under an enterprise privacy constraint that forbids reading values. By grounding descriptions in source code, fusing complementary value-free tagging signals, validating against an authoritative ontology, and grading its own output, Glyph produces descriptions and tags that are not only useful but explainable and auditable: every tag carries its provenance, confidence, and reasoning. We have been deliberate about the boundary between what is demonstrated (strong in-distribution retrieval, a deployed multi-strategy system) and what remains to be shown (public-benchmark generalization, judge–human agreement), and we see closing that gap as the natural next step toward substantially reducing the manual stewardship burden of enterprise data governance. References Apache Software Foundation (2015) Apache Software Foundation Apache Atlas: data governance and metadata framework. Note: https://atlas.apache.org Cited by: §2.3. BigID, Inc. (2016) BigID, Inc. BigID data discovery and classification platform. Note: Product documentation Cited by: §2.3, §4.3.3, §6. Castro Fernández et al. (2018) R. Castro Fernández, Z. Abedjan, F. Koko, et al. Aurum: a data discovery system. In Proc. IEEE ICDE, External Links: Document Cited by: §2.3. Chase and LangChain Inc. (2024) H. Chase and LangChain Inc. LangGraph: building stateful, multi-actor applications with LLMs. Note: Software framework Cited by: §4.1. Christiano et al. (2017) P. F. Christiano, J. Leike, T. B. Brown, et al. Deep reinforcement learning from human preferences. In Advances in Neural Information Processing Systems (NeurIPS), External Links: 1706.03741 Cited by: §6. Colvin et al. (2023) S. Colvin et al. Pydantic: data validation using python type hints. Note: https://docs.pydantic.dev Cited by: §4.1. Cormack et al. (2009) G. V. Cormack, C. L. A. Clarke, and S. Büttcher Reciprocal rank fusion outperforms Condorcet and individual rank learning methods. In Proc. ACM SIGIR, External Links: Document Cited by: §1, §4.3.4. Deng et al. (2021) X. Deng, H. Sun, A. Lees, Y. Wu, and C. Yu TURL: table understanding through representation learning. Proc. VLDB Endowment (PVLDB) 14 (3), p. 307–319. External Links: 2006.14806 Cited by: §1, §2.1, §4.3.3. Devlin et al. (2019) J. Devlin, M. Chang, K. Lee, and K. Toutanova BERT: pre-training of deep bidirectional transformers for language understanding. In Proc. NAACL-HLT, External Links: 1810.04805 Cited by: §4.4. Feuer et al. (2024) B. Feuer, Y. Liu, C. Hegde, and J. Freire ArcheType: a novel framework for open-source column type annotation using large language models. Proc. VLDB Endowment (PVLDB) 17. External Links: 2310.18208 Cited by: §1, §2.1, §2.1, §4.3.1, §4.3.3, §5.3. Google Cloud (2017) Google Cloud Sensitive data protection (cloud DLP): infoType detection. Note: Product documentation Cited by: §2.3, §4.2, §4.3.3, §6. Grover et al. (2020) M. Grover et al. Amundsen: a data discovery and metadata platform. Note: Lyft / open source, https://w.amundsen.io Cited by: §2.3. Halevy et al. (2016) A. Halevy, F. Korn, N. F. Noy, et al. Goods: organizing Google’s datasets. In Proc. ACM SIGMOD, External Links: Document Cited by: §2.3. Henderson et al. (2017) M. Henderson, R. Al-Rfou, B. Strope, et al. Efficient natural language response suggestion for smart reply. Note: arXiv preprint arXiv:1705.00652 Cited by: §1, §4.4. Hu et al. (2019) K. Hu, S. Bako, J. Hofman, et al. VizNet: towards a large-scale visualization learning and benchmarking repository. In Proc. ACM CHI, External Links: 1905.04616 Cited by: §2.2, §3. Hulsebos et al. (2023) M. Hulsebos, Ç. Demiralp, and P. Groth GitTables: a large-scale corpus of relational tables. Proc. ACM on Management of Data (SIGMOD) 1 (1). External Links: 2106.07258 Cited by: §2.2, §3. Hulsebos et al. (2019) M. Hulsebos, K. Hu, M. Bakker, et al. Sherlock: a deep learning approach to semantic data type detection. In Proc. ACM SIGKDD, External Links: 1905.10688 Cited by: §1, §2.1, §3, §4.3.3, §4.3.3, §5.3, Table 4. Iida et al. (2021) H. Iida, D. Thai, V. Manjunatha, and M. Iyyer TABBIE: pretrained representations of tabular data. In Proc. NAACL-HLT, External Links: 2105.02584 Cited by: §2.1, §4.3.3. Immuta, Inc. (2015) Immuta, Inc. Immuta sensitive data discovery and policy enforcement. Note: Product documentation Cited by: §2.3, §4.3.3, §6. Informatica (2018) Informatica CLAIRE: ai-powered metadata intelligence and automatic data classification. Note: Product documentation Cited by: §2.3, §4.3.3, §6. Järvelin and Kekäläinen (2002) K. Järvelin and J. Kekäläinen Cumulated gain-based evaluation of IR techniques. ACM Transactions on Information Systems 20 (4), p. 422–446. Cited by: §4.4. Jiménez-Ruiz et al. (2019) E. Jiménez-Ruiz, O. Hassanzadeh, V. Efthymiou, J. Chen, and K. Srinivas SemTab: tabular data to knowledge graph matching (CEA/CTA/CPA). In SemTab @ ISWC / ESWC, Cited by: §2.2, §3. Karpukhin et al. (2020) V. Karpukhin, B. Oğuz, S. Min, et al. Dense passage retrieval for open-domain question answering. In Proc. EMNLP, External Links: 2004.04906 Cited by: §4.2, §4.4. Kaufman et al. (2012) S. Kaufman, S. Rosset, C. Perlich, and O. Stitelman Leakage in data mining: formulation, detection, and avoidance. ACM Transactions on Knowledge Discovery from Data 6 (4), p. 1–21. Cited by: §4.4. Kayali et al. (2024) M. Kayali, A. Lykov, I. Fountalis, et al. CHORUS: foundation models for unified data discovery and exploration. Proc. VLDB Endowment (PVLDB) 17. External Links: 2306.09610 Cited by: §1, §2.1, §4.3.3. Korini and Bizer (2023) K. Korini and C. Bizer Column type annotation using ChatGPT. In VLDB Workshops (TaDA), External Links: 2306.00745 Cited by: §1, §2.1, §2.1, §4.3.1, §4.3.3, Table 4. Korini et al. (2022) K. Korini, R. Peeters, and C. Bizer SOTAB: the WDC Schema.org table annotation benchmark. In SemTab @ ISWC, CEUR-WS Vol-3320, Cited by: §2.2, §3. Lewis et al. (2020) P. Lewis, E. Perez, A. Piktus, et al. Retrieval-augmented generation for knowledge-intensive NLP tasks. In Advances in Neural Information Processing Systems (NeurIPS), External Links: 2005.11401 Cited by: §4.2, §4.3.3. Li et al. (2024) P. Li, Y. He, D. Yashar, et al. Table-GPT: table fine-tuned GPT for diverse table tasks. Proc. ACM on Management of Data (SIGMOD). External Links: 2310.09263 Cited by: §2.1, §4.3.3. Limaye et al. (2010) G. Limaye, S. Sarawagi, and S. Chakrabarti Annotating and searching web tables using entities, types and relationships. Proc. VLDB Endowment (PVLDB) 3 (1–2), p. 1338–1347. Cited by: §1, §2.1, §2.1, §6. LinkedIn (2019) LinkedIn DataHub: a generalized metadata search & discovery tool. Note: https://datahubproject.io Cited by: §2.3. Madaan et al. (2023) A. Madaan, N. Tandon, P. Gupta, et al. Self-refine: iterative refinement with self-feedback. In Advances in Neural Information Processing Systems (NeurIPS), External Links: 2303.17651 Cited by: §4.3.1. Manning et al. (2008) C. D. Manning, P. Raghavan, and H. Schütze Introduction to information retrieval. Cambridge University Press. Cited by: §5. Miao and Wang (2024) Z. Miao and J. Wang Watchog: a light-weight contrastive learning based framework for column annotation. In Proc. EDBT, Cited by: §2.1, §4.3.3. Microsoft (2018) Microsoft Presidio: context-aware, pluggable PII detection and anonymization. Note: https://github.com/microsoft/presidio Cited by: §2.3, §4.2, §4.3.3, §6. Microsoft (2021) Microsoft Microsoft Purview data map: automated data classification and sensitivity labeling. Note: Product documentation Cited by: §2.3, §4.3.3, §6. Nargesian et al. (2018) F. Nargesian, E. Zhu, K. Q. Pu, and R. J. Miller Table union search on open data. Proc. VLDB Endowment (PVLDB) 11 (7), p. 813–825. Cited by: §2.3. Ouyang et al. (2022) L. Ouyang, J. Wu, X. Jiang, et al. Training language models to follow instructions with human feedback. In Advances in Neural Information Processing Systems (NeurIPS), External Links: 2203.02155 Cited by: §6. Pham et al. (2016) M. Pham, S. Alse, C. A. Knoblock, and P. Szekely Semantic labeling: a domain-independent approach. In Proc. Int. Semantic Web Conf. (ISWC), Cited by: §2.1, §4.3.3. Reimers and Gurevych (2019) N. Reimers and I. Gurevych Sentence-BERT: sentence embeddings using siamese BERT-networks. In Proc. EMNLP-IJCNLP, External Links: 1908.10084 Cited by: §4.3.1, §4.4. Ritze et al. (2015) D. Ritze, O. Lehmberg, and C. Bizer Matching HTML tables to DBpedia. In Proc. Int. Conf. on Web Intelligence, Mining and Semantics (WIMS), External Links: Document Cited by: §2.1, §2.1, §4.3.2. Schulman et al. (2017) J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov Proximal policy optimization algorithms. External Links: 1707.06347, Link Cited by: §6. Shinn et al. (2023) N. Shinn, F. Cassano, E. Berman, et al. Reflexion: language agents with verbal reinforcement learning. In Advances in Neural Information Processing Systems (NeurIPS), External Links: 2303.11366 Cited by: §4.3.1. Suhara et al. (2022) Y. Suhara, J. Li, Y. Li, et al. Annotating columns with pre-trained language models. In Proc. ACM SIGMOD, External Links: 2104.01785 Cited by: §1, §2.1, §3, §4.3.3, §5.3, Table 4. Sun et al. (2023) Y. Sun, H. Xin, and L. Chen RECA: related tables enhanced column semantic type annotation framework. Proc. VLDB Endowment (PVLDB) 16 (6), p. 1319–1331. Cited by: §2.1, §4.3.3. Takeoka et al. (2019) K. Takeoka, M. Oyamada, S. Nakadai, and T. Okadome Meimei: an efficient probabilistic approach for semantically annotating tables. In Proc. AAAI, Cited by: §2.1, §6. Tsoumakas and Katakis (2007) G. Tsoumakas and I. Katakis Multi-label classification: an overview. International Journal of Data Warehousing and Mining 3 (3), p. 1–13. Cited by: §3. Vaswani et al. (2017) A. Vaswani, N. Shazeer, N. Parmar, et al. Attention is all you need. In Advances in Neural Information Processing Systems (NeurIPS), External Links: 1706.03762 Cited by: §4.4. Wang et al. (2020) W. Wang, F. Wei, L. Dong, et al. MiniLM: deep self-attention distillation for task-agnostic compression of pre-trained transformers. In Advances in Neural Information Processing Systems (NeurIPS), External Links: 2002.10957 Cited by: §4.3.1. Wang et al. (2023) X. Wang, J. Wei, D. Schuurmans, et al. Self-consistency improves chain-of-thought reasoning in language models. In Proc. ICLR, External Links: 2203.11171 Cited by: §4.3.4. Wei et al. (2022) J. Wei, X. Wang, D. Schuurmans, et al. Chain-of-thought prompting elicits reasoning in large language models. In Advances in Neural Information Processing Systems (NeurIPS), External Links: 2201.11903 Cited by: §4.2. Wei et al. (2024) L. L. Wei, G. Xiao, and M. Balazinska RACOON: an LLM-based framework for retrieval-augmented column type annotation with a knowledge graph. Note: arXiv preprint arXiv:2409.14556 Cited by: §2.1, §4.3.1, §4.3.3. Yao et al. (2023) S. Yao, J. Zhao, D. Yu, et al. ReAct: synergizing reasoning and acting in language models. In Proc. ICLR, External Links: 2210.03629 Cited by: §4.2. Yin et al. (2020) P. Yin, G. Neubig, W. Yih, and S. Riedel TaBERT: pretraining for joint understanding of textual and tabular data. In Proc. ACL, External Links: 2005.08314 Cited by: §2.1, §4.3.3. Zhang et al. (2020) D. Zhang, Y. Suhara, J. Li, M. Hulsebos, Ç. Demiralp, and W. Tan Sato: contextual semantic type detection in tables. Proc. VLDB Endowment (PVLDB) 13 (12). External Links: 1911.06311 Cited by: §1, §2.1, §2.1, §3, §4.3.3, §5.3, Table 4, §6. Zhang et al. (2024) T. Zhang, X. Yue, Y. Li, and H. Sun TableLlama: towards open large generalist models for tables. In Proc. NAACL, External Links: 2311.09206 Cited by: §2.1, §4.3.3. Zheng et al. (2023) L. Zheng, W. Chiang, Y. Sheng, et al. Judging LLM-as-a-judge with MT-Bench and Chatbot Arena. In NeurIPS Datasets and Benchmarks Track, External Links: 2306.05685 Cited by: §4.2, §4.3.1.