Paper deep dive
QCATS: Query Context-Aware Transformer Slicing for Efficient Predictive Query Processing
Yueying Li, Zhongle Xie, Ke Chen, Lidan Shou
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 88%
Last extracted: 10/11/2026, 3:28:55 AM
Summary
The paper introduces QCATS, a framework for efficient in-database predictive query processing that utilizes query context-aware transformer slicing. Unlike existing methods that perform dense inference or token-level Mixture-of-Experts (MoE) routing, QCATS leverages query predicates and metadata statistics to pre-select specific Feed-Forward Network (FFN) slices (experts) before execution. This approach reduces latency by up to 4.42x while maintaining accuracy, using offline expert construction and lightweight query-level routing.
Entities (7)
Relation Signals (6)
QCATS → outperforms → dense baselines
confidence 95% · QCATS achieves up to 4.42x latency reduction while preserving prediction accuracy comparable to dense baselines.
QCATS → evaluatedon → BERT-base
confidence 90% · Experiments on four predictive-query workloads with BERT-base and Qwen-0.6B show that QCATS achieves up to 4.42x latency reduction
QCATS → evaluatedon → Qwen-0.6B
confidence 90% · Experiments on four predictive-query workloads with BERT-base and Qwen-0.6B show that QCATS achieves up to 4.42x latency reduction
QCATS → uses → FFN
confidence 90% · QCATS exploits FFN activation sparsity to extract sparse expert slices from a fine-tuned dense Transformer.
QCATS → replaces → token-level routing
confidence 85% · Instead of performing routing inside the model, QCATS shifts routing decisions to the query level.
FastText → usedby → QCATS
confidence 80% · we employ a lightweight embedding model (FastText) to extract embeddings and apply K-Means clustering
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:In-database predictive query processing increasingly applies Transformer-based models within relational pipelines. However, existing in-database inference typically exposes only tuple-level model inputs to the inference runtime, leaving relational predicates and metadata statistics invisible to neural execution planning. In this paper, we propose QCATS, a query context-aware transformer slicing framework that enables efficient sparse inference inside database systems. QCATS executes at query granularity: instead of routing individual tokens or tuples during inference, it uses query predicates and metadata statistics to pre-select context-aligned FFN slices before model execution. The framework comprises offline expert construction and lightweight query-level routing that dynamically selects experts during execution. QCATS further introduces system optimizations, including asynchronous CPU-GPU pipelines and routing-aware batching. Experiments on four predictive-query workloads with BERT-base and Qwen-0.6B show that QCATS achieves up to 4.42x latency reduction while preserving prediction accuracy comparable to dense baselines.
Tags
Links
- Source: https://arxiv.org/abs/2610.09894v1
- Canonical: https://arxiv.org/abs/2610.09894v1
Trouble viewing inline? Open PDF directly →
Full Text
83,738 characters extracted from source content.
Expand or collapse full text
QCATS: Query Context-Aware Transformer Slicing for Efficient Predictive Query Processing Yueying Li 1 , Zhongle Xie 1,2, * , Ke Chen 1,2 , Lidan Shou 1,2 1 The State Key Laboratory of Blockchain and Data Security, Zhejiang University; 2 Hangzhou High-Tech Zone (Binjiang) Institute of Blockchain and Data Security zjuyueyingli, xiezl, chenk, should@zju.edu.cn Abstract—In-database predictive query processing increasingly applies Transformer-based models within relational pipelines. However, existing in-database inference typically exposes only tuple-level model inputs to the inference runtime, leaving rela- tional predicates and metadata statistics invisible to neural execu- tion planning. In this paper, we propose QCATS, a query context- aware transformer slicing framework that enables efficient sparse inference inside database systems. QCATS executes at query granularity: instead of routing individual tokens or tuples during inference, it uses query predicates and metadata statistics to pre- select context-aligned FFN slices before model execution. The framework comprises offline expert construction and lightweight query-level routing that dynamically selects experts during execu- tion. QCATS further introduces system optimizations, including asynchronous CPU–GPU pipelines and routing-aware batching. Experiments on four predictive-query workloads with BERT-base and Qwen-0.6B show that QCATS achieves up to 4.42× latency reduction while preserving prediction accuracy comparable to dense baselines. Index Terms—In-database Learning, Model Slicing, Query Processing I. INTRODUCTION Modern analytical databases integrate Transformer-based predictive operators into SQL pipelines to process unstructured text alongside relational data. Known as in-database predictive querying [1], [2], this paradigm supports semantic analytics through cloud inference APIs or UDFs [3]–[5]. These queries typically produce short outputs, such as labels or scores, for many SQL-selected tuples, making accumulated tuple-level inference, rather than autoregressive decoding, a dominant cost of query execution. Opportunities to reduce this cost arise from the structure of predictive queries: inference operates on subsets selected by SQL predicates. These predicates, together with subset statistics, define a semantic space that often differs from the global data distribution, which we refer to as query context. For Transformer models, different query contexts induce dis- tinct activation patterns in feed-forward networks (FFNs) [6]. As illustrated in Fig. 1, sentiment analysis over food reviews activates a different subset of neurons from the same task on clothing or electronics reviews, suggesting that only a fraction of model capacity is required for many query workloads. We further empirically verify this phenomenon in Fig. 2 using eight queries, details are provided in the appendix. For each * Corresponding author. category...review ......... Clothes Predictive Query 2 Inference on PQ2 SELECT 퐀䠀퐀砀퐀鰀䠀퐀砀(review) FROM T WHERE category = ‘Food’; Predictive Query 1 Inference on PQ1 Clothes Food Food Table T Query Context1 Query Context2 Semantic Space Non-FFN FFN Non-FFN FFN SELECT 퐀䠀퐀砀퐀鰀䠀퐀砀(review) FROM T WHERE category = ‘Clothes’; Semantic Mismatch Leads to Waste Low-activation Neurons (can be pruned) Active Neurons Fig. 1: FFN Activation Varies Across Query Contexts. Neurons Q1 Q2 Q3 Q4 Q5 Q6 Q7 Q8 0.11 0.12 0.13 0.14 0.15 Log Activation Fig. 2: Average FFN neuron activations for text tuples selected by eight queries (Q1–Q8). query, we feed its selected text tuples into the fine-tuned BERT model and compute the average activation of each FFN neuron, producing one row of the heatmap. Each column represents an FFN neuron, and darker colors indicate higher average activation. A small subset of neurons remains highly activated across the queries and is therefore placed on the left. Beyond this shared subset, highly activated regions appear at different neuron positions for different query contexts, indicating that distinct queries activate different subsets of neurons. This observation motivates selecting context-aligned FFN slices according to the query context. However, existing database systems typically execute infer- ence through black-box user-defined functions (UDFs) [3]– [5], transmitting only raw text data to the model while discarding relational constraints and metadata statistics. Even when the inference engine applies generic batching or kernel- level acceleration, these optimizations are unaware of the re- lational context that determines which tuples are semantically arXiv:2610.09894v1 [cs.DB] 7 Oct 2026 similar. Consequently, models operate in a context-agnostic manner and execute dense inference for every input tuple. This mismatch leads to substantial waste in computation and memory resources in predictive query processing. That is, the computation repeatedly activates the full model despite the query-level locality of predictive workloads. From a systems perspective, the inefficiency motivates con- ditional execution, where only computation relevant to the current query context is activated during inference. Analogous to predicate pushdown [7]–[9] and column pruning [10], [11], which avoid irrelevant data movement and attribute access, query-aware neural execution should avoid irrelevant parame- ter access and activation computation. Unfortunately, existing neural inference pipelines lack mechanisms to translate query semantics into deterministic and database-friendly conditional model execution. SparseactivationtechniqueslikeMixture-of-Experts (MoE) [12], [13] partially address this issue by enabling conditional computation within neural networks. However, they rely on token-level, layer-wise routing during inference, which introduces control-flow divergence and dynamic data redistribution overheads, degrading GPU utilization [14], [15]. In database workloads, this overhead is amplified because tuples selected by the same query are naturally processed in large batches; token-level routing breaks this homogeneity and prevents the executor from constructing a single static inference path for the batch. Furthermore, it introduces additional buffering overhead for fragmented tokens across experts, causing inflated activation memory and unpredictable resource demands. Despite extensive research on sparse inference and expert routing, existing approaches [16]–[18] are primarily designed for standalone model serving and do not exploit the semantic structure exposed by database queries. To our knowledge, no prior system leverages query-level context to enable deter- ministic, database-integrated sparse execution of Transformer models. This gap prevents current solutions from achieving both high inference efficiency and tight integration with rela- tional execution pipelines. Beyond model-level limitations, predictive query processing also introduces system-level challenges. CPU-side preprocess- ing, including tokenization and data preparation, can become a bottleneck when GPU inference is accelerated on-the-fly. Meanwhile, long-tailed text length distributions [19] cause ex- cessive padding overhead under conventional batching, while synchronous execution pipelines provide insufficient CPU– GPU overlap, leading to pipeline stalls and underutilized hardware resources. In this work, we propose QueryContext-AwareTransformer Slicing (QCATS), a framework that rethinks sparse inference for Transformer-based models from a database-first perspec- tive. Instead of performing routing inside the model, QCATS shifts routing decisions to the query level. By leveraging the query context, QCATS pre-determines the subset of model parameters that will be activated before inference begins. This design enables static, deterministic routing, avoids runtime control-flow divergence, and aligns sparse execution with the granularity of database workloads. In this sense, QCATS treats model inference as a query physical operator whose execution path can be specialized using optimizer-visible relational con- text before tuple-level inference begins. QCATS consists of two tightly integrated components. First, it transforms a fine-tuned dense model into a library of context-aligned expert slices by exploiting FFN activation sparsity, enabling sparse inference without retraining models from scratch. Second, it introduces a lightweight context- aware router that maps query contexts directly to expert combinations. By decoupling routing from execution, QCATS ensures contiguous memory access via homogeneous batching, thereby avoiding the extra buffering overhead and reducing peak activation memory. To further address system-level bot- tlenecks, we design an asynchronous CPU–GPU execution pipeline with hierarchical bucketing that masks preprocessing overhead and reduces padding-induced waste. We summarize our main contributions as follows: • We formulate query context-aware transformer slicing for in-database predictive query processing and propose QCATS, which uses query predicates and metadata statistics to specialize neural inference execution paths. • We introduce a parameter decomposition strategy that par- titions FFN layers into shared and semantic components, enabling efficient sparse inference from a fine-tuned dense Transformer without constructing separate expert networks from scratch. • We design a lightweight query-level routing mechanism that maps query contexts to sparse expert combinations, shifting conditional computation from token-level dynamic routing to query-level static routing. • We integrate QCATS into a database-oriented execution pipeline with routing-aware batching and asynchronous CPU–GPU execution. Experiments on four workloads and two Transformer backbones show significant latency and memory reductions while preserving prediction accuracy. The rest of the paper is organized as follows. §I defines predictive queries and formulates query context-aware trans- former slicing. §I presents QCATS and §IV introduces database-oriented execution optimizations. §V reports the ex- perimental evaluation. §VI reviews related work, and §VII concludes the paper. I. FOUNDATIONS In this section, we establish the foundations required to reason about query-aware model slicing in database systems. A. Predictive Query We consider predictive queries in analytical databases, which integrate relational filtering with model inference. Let a relationT consist of text attributesX text and structured metadata X meta . The metadata describes the context of each text record, such as product category, brand, or user profile in e-commerce scenarios. Crucially, our framework is broadly applicable to any scenario where the model input is shaped by Model M 퐀 text Token FFN FFN Layer Predictive QueryResult Execution Engine Query Parser UDF Runtime (a) With Dense Model Model M 퐀 text Token Expert 0 Expert 1 Expert 2 Expert 3 MoE Layer Predictive QueryResult Execution Engine Query Parser UDF Runtime Router (b) With MoE Model Context-Aware Router Expert 0 Expert 1 Expert 2 Expert 3 MoE Layer Token Model M 퐀 text Predictive QueryResult Execution Engine Query Parser UDF Runtime (c) With QCATS (Ours) Fig. 3: Predictive Query Processing in Database. Transformer Execution Engine Query Parser Context -Aware Router Target Data Expert Library ℰ Full Data E 0 E 1 E 2 Predictive Query 퐀 meta Stage I (§I-A) Stage I (§I-B) Expert Weight g Activate 퐀 meta 퐀 Finetuned Model 퐀 query 퐀 text 퐀 text UDF Runtime E 3 E 0 E 1 E 2 E 3 Trainable (§I-C) Fig. 4: Overview of QCATS. SQL evaluation over the metadata. Given relational constraints Φ over X meta , a predictive query Q first obtains an interme- diate subset T Φ(X meta ) through standard relational operators, including WHERE, JOIN, and GROUP BY, and then applies a Transformer-based model M to X text. Formally, we define the predictive query Q as an extended projection over the metadata-constrained subset: T scored = π ∗,M(X text )→Y T Φ(X meta ) (1) where T Φ(X meta ) denotes the intermediate subset materialized by the database engine under metadata constraints Φ, and M evaluates X text to append a prediction column Y . We define query contextC Q as a composite representation: C Q = Φ, S(T Φ ) (2) where Φ denotes explicit relational constraints, and S(T Φ ) captures statistics over the post-constraint subsetT Φ . The con- text C Q specifies the semantic scope of model inference and can be materialized into dense features via feature engineering or lightweight encoders, as seen in §I-B. Note that existing UDF-based inference processes tuples independently and does not expose such relational context to the inference pipeline. B. Core Idea and Goal Transformer feed-forward networks (FFNs) dominate model parameters and inference cost [13], [20]. Yet, their activations are highly sparse: prior studies show that retaining a modest subset of neurons can preserve most output magnitude and predictive performance [21]–[23]. This redundancy motivates sparse FFN execution. Conditional computation mechanisms such as Mixture-of-Experts (MoE) exploit this redundancy by replacing dense FFNs with expert networks E 1 ,...,E N controlled by a routing function G(·). Formally, for an input hidden state x, the output is: y = N X i=1 G(x)· E i (x).(3) Existing MoE architectures are primarily designed for large- scale pre-training and rely on token-level, layer-wise routing, in which routing decisions are computed dynamically for individual tokens. While effective for scaling model capacity, this fine-grained routing mechanism introduces non-trivial dynamic scheduling and dispatching overhead and fails to exploit the semantic coherence exposed by database query execution. As illustrated in Fig. 3, our key insight is that predictive queries naturally enable query-level conditional execution. Different from token-level conditional computation in common MoE design (Fig. 3b), which adapts execution paths inde- pendently for each input, query-level conditional execution (Fig. 3c) leverages the semantic consistency of query-selected subsets and applies a unified execution configuration across all tuples in T Φ . Under this paradigm, the query context acts as a high- level routing control signal for model inference activation. Routing signal can be computed before inference, enabling static execution paths, homogeneous batch construction, and tighter integration with relational execution pipelines. Goal. We focus on predictive queries that apply Transformer-based models independently to many tuples. Our objective is to reduce the accumulated inference cost over T Φ rather than improving autoregressive generation and serving- specific mechanisms of the Transformer itself. We formalize our task as Query Context-Aware Trans- former Slicing problem. Let θ denote the parameter set of a Transformer model M. Given a predictive query Q with constraints Φ, our goal is to generate a sliced model M Φ that selects a sparse parameter subset θ Φ ⊂ θ for efficient query execution. The sliced model should satisfy two goals: • Execution Efficiency (G1). The sliced model should re- duce inference latency and peak memory footprint when integrated into the database query pipeline. • Accuracy Preservation (G2). The activated slice should preserve the predictive performance of the full model on the target subset T Φ within a small tolerance margin. I. QCATS We propose Query Context-Aware Transformer Slicing (QCATS), a framework that shifts model inference from dense execution to query-adaptive sparse execution. Unlike tradi- tional MoE with token-level routing, QCATS adopts query- level routing to pre-activate semantic parameter subsets, reduc- ing computation and VRAM usage while preserving predictive accuracy. As shown in Fig. 4, QCATS consists of two stages: offline expert construction and query-time dynamic assembly. Note that the offline construction is performed once while the online path performs only lightweight routing and sliced inference. Stage I: Offline Sparsity-Based Expert Construction. QCATS exploits FFN activation sparsity to extract sparse expert slices from a fine-tuned dense Transformer. These slices form an expert library E that supports conditional execution while preserving the model’s semantic capability, as elaborated on §I-A. Stage I: Query-Time Context-Aware Inference. At query time, QCATS extracts relational constraints and metadata statistics as query context, maps them to expert weights through a lightweight router, and executes inference using the selected slices. The inference pipeline, as detailed in §I-B consists of Query Processing, Query Context Featurization, Context-Aware Expert Routing, and Sliced Model Execution. A. Stage I: Expert Construction QCATS begins with a one-off structural reconfiguration step performed during system initialization to prepare the dense transformer model for query-level conditional execution. By analyzing activation patterns in FFNs, we decompose the orig- inal model into a library of physically isolated Expert Slices as illustrated in Fig. 5. This offline transformation converts monolithic FFN layers into modular computation units that can be selectively activated at query time. The resulting Expert Library provides a stable execution substrate for query-level routing and selective parameter activation, enabling efficient, deterministic, and database-friendly sparse inference. To construct specialized experts, QCATS first identifies distinct semantic regions from the text data X text . Directly using all data samples introduces noise due to overlapping topics and ambiguous semantic boundaries, which can blur the specialization signals required for expert construction. Instead, we choose to extract prototypical samples. Specifically, we employ a lightweight embedding model (FastText [24]) to ex- tract embeddings and apply K-Means clustering to partition the data into N semantic groups D 1 ,...,D N 1 . For each group, we retain only the samples closest to the corresponding cluster centroid to reduce noise. These representative samples provide stable and discriminative activation patterns for subsequent neuron analysis, ensuring that each expert slice is anchored to a well-separated semantic domain. 1 We set the sample size of each group to 500 as a trade-off between effectiveness and cost. Using these samples, we profile the internal activation behavior of the fine-tuned dense model. We perform forward passes on data from each semantic group D n and collect neuron activation statistics across FFN layers. This profiling step maps semantic variations in the input space to structured patterns in the neuron activation space, producing the activa- tion signals required to identify different types of neurons. Neurons in FFN layers exhibit heterogeneous functional roles [6]: some encode general linguistic patterns that are broadly useful across domains, while others respond selec- tively to specific semantic regions. To preserve both general- ization capability and domain specialization, QCATS explicitly separates these two types of neurons. Concretely, we partition FFN neurons into a shared set U shared , which captures univer- sally useful computation, and multiple semantic sets U (n) unique , each corresponding to a distinct semantic group D n . First, to capture general knowledge shared across different domains, we aim to identify neurons that remain consistently active regardless of the specific semantic group. To quantify the universal utility, we calculate General Importance Score S global j : S global j = 1 N N X n=1 A (n) j (4) where A (n) j denotes the average activation magnitude of neu- ron j in semantic group n, and N is the total number of groups. Neurons with the highest scores are then selected to form the shared set U shared , ensuring the retention of fundamental model capabilities while providing a stable foundation for subsequent semantic specialization. Subsequently, to capture semantic knowledge, we need to identify neurons that exhibit high specificity. Such neurons are highly active in a specific semantic group but suppressed in other groups. To quantify this contrastive property, we define Semantic Score R j,n : R j,n = A (n) j 1 N−1 P m̸=n A (m) j + ε (5) where the denominator represents the background noise level from other groups. By filtering for neurons with top-ranked R j,n values, we construct the semantic set U (n) unique for each semantic group, ensuring that each expert specializes in its corresponding group. Based on the selected neuron sets, we physically reconstruct the original dense parameter matrix W into N isolated Expert Slices, namely the tensor setW (1) ,..., W (N ) . For standard Transformer FFN layers composed of up projection W up , gate projection W gate , and down projection W down , we apply neuron selection consistently across all three matrices to ensure structural integrity. Specifically, the n-th expert slice W (n) is composed of the union of the shared set and the corresponding semantic set, i.e., U shared ∪ U (n) unique . This design allows each expert slice to independently retain both general linguistic capabilities and semantic expertise. KNN E0E1 FFN Shared Neurons 푈 shared Low-activation Neurons 푋 text 풟 1 Activation Profiling Expert Construction . . . Clustering 풟 0 Semantic Neurons 푈 unique (0) 푈 unique (1) Fig. 5: Illustration of Expert Construction. B. Stage I: Context-Aware Inference At query time, QCATS extracts the query context C Q from each incoming query. By matching it against the N pre- established semantic groups, the system routes the query to the corresponding expert for efficient inference. Specifically, we represent C Q as a dense context embedding e ctx through feature engineering and a lightweight encoder, then use it to drive expert routing. This process comprises the following four key steps: 1) Query Processing: Raw predictive queries often inter- leave lightweight relational operators (e.g., JOIN, GROUP BY) with computationally expensive neural UDFs. Because the data fed into the neural model is strictly pre-filtered by these SQL operators, QCATS must first parse the query to identify these filtering boundaries. We formally extract these relational filtering signals as explicit constraints, denoted as Φ. This query parsing and semantic-aware logical rewriting serve two main purposes: minimizing the data subset fed into the inference stage and providing highly selective semantic context Φ for downstream routing. A major challenge in extracting a clean Φ arises when neural UDFs are embedded within predicates. In database query processing, the query parser translates an input SQL into a logical query plan represented as an Abstract Syntax Tree (AST), where relational predicates and UDF invocations coexist as distinct operator nodes. A conventional optimizer may treat neural UDF predicates similarly to scalar predicates or lack sufficient cost information to delay them appropriately. To prevent this, QCATS introduces a deferred evaluation rule. As shown in Fig. 6a, it rewrites the AST by moving the UDF predicate (σ) upward, past lightweight scalar constraints and JOINs, delaying its execution. This deferral naturally isolates all preceding relational operators into the explicit constraints Φ. Consequently, the UDF acts as a final semantic filter evaluated on a reduced subset, while Φ provides a precise semantic context for routing. Crucially, features are always extracted from the subset that the UDF actually processes, so the routing input remains consistent with the true inference domain regardless of how the rewrite reshapes the data. For queries involving logical partitioning (e.g., GROUP BY), QCATS abandons standard tuple-at-a-time inference (Fig. 6b). Instead, the execution engine physically parti- tions intermediate tuples into dense semantic subsets prior to model inference. The context-aware router then treats each pre-aggregated subgroup as an independent batch, activating distinct sliced models. To balance context-awareness with overhead, QCATS ranks the pre-aggregated groups by their cardinalities in the query-filtered subset, assigns dedicated expert slices only to the few dominant groups, and merges the remaining tail groups into a shared routing path. Ultimately, this parsing and rewriting phase outputs a refined logical plan alongside the clearly isolated explicit constraints Φ. These elements define the exact semantic boundaries of the target data, providing a precise, noise-free foundation for the subsequent featurization phase. 2) Query Context Featurization: Discrete SQL constraints and structured metadata are semantically heterogeneous and cannot be directly processed by a neural router. To bridge the gap between relational structures and continuous embeddings, QCATS performs a two-phase preparation procedure: Feature Selection, which identifies informative metadata attributes, and Context Representation, which transforms them into unified vector representations suitable for downstream routing. One-time Feature Selection. Database tables often contain redundant or weakly informative metadata that contributes little to semantic understanding. Feeding all columns to the router introduces noise and computational overhead. To ad- dress this issue, we design a dual-view scoring mechanism that identifies a compact subset of highly discriminative attributes. Importantly, this process is executed as a one-off offline task and incurs no overhead during query-time inference. To quantify the global discriminative power of each column, we train a lightweight classifier using metadata attributes as input features. Specifically, we measure the global relevance score of each column using Gini importance derived from a Random Forest model [25] trained to predict the context groups D n built in §I-A. In ambiguity zones, textual features alone are insufficient. We must identify columns that distinguish samples where text fails to do so. To this end, we construct a KNN [26] index over the training set of the target downstream task using text embeddings, and retrieve each sample’s k nearest neighbors. Here, the downstream task labels are the ground-truth labels provided in that training set (e.g., sentiment labels in a sentiment analysis dataset of product reviews). A sample pair is marked as a conflict pair if the two samples are close in embedding space yet carry different labels from this training set, indicating that textual similarity alone cannot reliably discriminate them. We then compute the average value discrepancy of each metadata column across all identified conflict pairs, and adopt this discrepancy as the local disambiguation score, prioritizing columns that differ systematically across such pairs and there- fore may help resolve ambiguities left by text alone. Finally, we select the Top-M columns based on a weighted average of these scores, defining them as the effective context columns. Schema-Agnostic Featurization. To ensure the router gener- alizes across diverse database tables, we require a standardized input format that abstracts away schema differences. We SELECT U.gender FROM Product P JOIN User U ON P.id = U.id WHERE P.category = 'music' AND Sentiment(U.review)=’positive’ Sentiment User 퐀 ⋈ Π U.gender Product 퐀 category=’music’ User ⋈ Π U.gender Product 퐀 category=’music’ Sentiment 퐀 subset sliced model Context -Aware Router label activate (a) Query Rewriting SELECT AVG(Sentiment(U.review)) FROM Product P JOIN User U on P.id = U.id WHERE U.gender = 'man' GROUPBY P.category gender=’man’ User 퐀 Product ⋈ P.category, Sentiment(U.review) as score Π γ P.category, AVG(score) subsets for each group sliced models Context -Aware Router scores for each group activate (b) Grouped Query Processing Fig. 6: Specialized Query Processing in QCATS. Metadata Features 퐀 meta Query Features 퐀 query Self-Attention FFN MLP Expert Weight g E 0 E 1 E 2 E 3 1 Cross-Attention FFN STESTE Fig. 7: Architecture of Context-Aware Router. introduce an input featurization mechanism that maps diverse columns into a unified embedding space. Numerical values are processed via discretization-based encoding, while categorical fields utilize frequency-based encoding. The function F (·) generates two dense feature sets that enable the model to infer explicit intent against global data distributions without relying on fixed position rules: (a) Implicit Metadata Features F meta : This feature set captures the statistical characteristics of the selected context columns under the current dataset distribution. It encodes both column semantics and distributional profiles as: F meta =e (m) col , e (m) stat M m=1 (6) where M denotes the number of selected context columns. For the m-th column, e (m) col represents the column name embed- ding, and e (m) stat encodes its statistical distribution properties. (b) Explicit Predicate Features F query : This feature set captures the user’s specific filtering intent expressed in SQL predicates. To minimize noise from irrelevant attributes, we retain only predicates involving the selected context columns. If the query contains J such predicates, the representation is constructed as: F query =e (j) col , e (j) val J j=1 (7) where J is the number of valid predicates. For the j-th predicate, e (j) col and e (j) val denote the embeddings of the target column and filtering value, respectively. Note that if there exist no relevant predicates, this feature set is empty, i.e., J = 0. 3) Context-Aware Expert Routing: To transform the query context features into specific expert configurations, we de- sign a customized Context-Aware Router. Unlike traditional MoE architectures that perform token-level, layer-wise routing inside the model, our router operates as a decoupled and lightweight look-ahead module that determines expert acti- vation prior to inference. Our design is inspired by hybrid attention architectures [27], but is specifically adapted to effectively process the non-sequential and structured nature of database content, as depicted in Fig. 7. Since database columns are permutation-invariant, standard positional encodings are unsuitable as they impose artificial order. To preserve the native structure of tabular inputs, we discard absolute positioning and introduce Structural Type Embedding (STE), enabling the router to capture relational semantics without spurious sequential dependencies. Building on this structured input, we employ a specialized Transformer model interaction to distill semantic context. First, to capture global correlations among columns (e.g., the joint distribution between Brand and Price), we utilize a 2-layer Transformer encoder to process the implicit metadata features F meta , generating a contextually consistent global representation H meta . Next, to filter out redundant signals and focus solely on the query-specific intent, we modify a 2-layer Transformer decoder by repurposing the Multi-Head Cross-Attention (MHCA) mechanism. We initialize the query state C (0) using the explicit predicate features F query , or employ a learnable global intent embedding e global to autonomously discern salient metadata when predicates are absent. With the encoded global metadata H meta serving as the key and value, the decoder dy- namically attends to metadata attributes that are most relevant to the current query context: C (l) = LayerNorm(C (l−1) + MHCA(C (l−1) | z Query , H meta | z Key , H meta | z Value )) (8) where C (l) denotes the context representation at decoder layer l. The final stage transforms the variable-length semantic context into a discrete, efficient execution plan. We first apply Masked Mean Pooling on the decoder output to produce a fixed-length context embedding e ctx . To explicitly bound inference cost, we then employ a Top-K Gating Network, which projects e ctx through an MLP and activates only the Top-K most relevant expert slices: g = Top-K(Softmax(MLP(e ctx )))(9) where g ∈R N (with only K non-zero entries) represents the sparse gating vector containing the routing weights for the selected experts. The hyperparameter K explicitly gov- erns the trade-off between model capacity and execution efficiency: a smaller K prioritizes computational sparsity, while a larger K preserves richer semantic nuances. This design effectively translates semantic understanding into a deterministic, hardware-friendly execution path, eliminating redundant parameter computations. 4) Sliced Model Execution: In the inference phase, once the context-aware router generates routing decisions, the system directly retrieves the activated Top-K expert tensors W (k) | k ∈E active from the Expert Library. The output y of the sliced Transformer FFN layer is computed as: y = X k∈E active g k · E k (x)(10) where x denotes the input hidden state and g k is the expert weight. As mentioned previously, unlike token-level MoE execution, our sliced model execution is performed at query granular- ity and uses pre-constructed expert slices, avoiding dynamic control flow and runtime expert construction. To be specific, the design yields two key efficiency benefits. First, QCATS reduces computation associated with inactive expert slices by restricting FFN matrix multiplications to the selected parame- ter subset. Second, peak VRAM reduction is achieved because expert slices have substantially smaller dimensionality than the original dense FFN. Consequently, the size of intermediate activations decreases proportionally, eliminating the need to allocate large GPU memory buffers for the full-width network and thereby enabling memory-efficient deployment. Discussion. The practicality of QCATS is demonstrated through its robust design and seamless system integration. Its architecture provides adaptive semantic coverage: queries with strong semantic signals are routed to dominant special- ized experts, while weakly constrained queries benefit from weighted multi-expert collaboration. The shared neuron set U shared preserves fundamental linguistic capabilities, ensuring a stable performance floor across diverse query contexts. It is worth noting that QCATS complements general- purpose model-serving platforms and low-level execution op- timizations. While vLLM [28] and SGLang [29] optimize autoregressive serving through KV-cache management and continuous batching, QCATS uses query context to reduce short-output inference costs across database tuples. QCATS also supports attention acceleration methods, such as FlashAt- tention [30], FastMoE [14], and quantization [31]–[33]. De- coupling the expert library from the router enables efficient ex- ecution under stable data distributions. As data and workloads gradually evolve, incremental fine-tuning of the lightweight router can support long-term deployment in dynamic OLAP scenarios without rebuilding the expert library. C. Decoupled Training Strategy To operationalize QCATS, we must optimize two distinct components: the Context-Aware Router ψ and the Expert Library E . Since the router is initialized from scratch, while experts are physically derived from a dense backbone, naive joint optimization may lead to unstable training dynamics. Specifically, the router requires specialized experts to learn discriminative routing policies, whereas expert specialization depends on stable and consistent routing decisions. To address this problem, we adopt a decoupled two-phase training strategy. Phase I establishes a reliable routing prior through teacher-guided semantic alignment, while Phase I leverages the fixed routing policy to drive physical expert spe- cialization. This separation stabilizes optimization and enables effective coordination between routing and expert learning. 1) Phase I: Router Semantic Alignment via Activation Distillation: In this phase, we train Context-Aware Router by constructing supervised routing targets derived from the internal activation behavior of the fine-tuned dense model. While Section I-A determines the physical composition of expert slices, the semantic correspondence between query contexts and experts remains undefined. We bridge this gap by treating the dense model as a teacher and using its FFN activations as oracle signals to align the expert with the query context. Given a query-selected subset D q , we run the dense model and aggregate activation magnitudes over the unique neurons of each expert: s (n) raw = 1 |D q | X x∈D q X j∈U (n) unique |h j (x)|(11) where h j (x) is the post-activation output of neuron j. Note that shared neurons are excluded because they are active across most inputs and provide weak routing signals. Even when focusing on unique neurons, disparities in ab- solute activation magnitudes persist across different experts. To cope with this issue, we propose a calibration strategy that acts as a weighted balance between the relative improvement and the raw activation magnitude: s (n) cal = s (n) raw − μ n μ n + ε + η· s (n) raw (12) where μ n denotes the average activation magnitude of expert n across the training corpus and η is a coefficient controlling the weight of the original signal. This combination preserves the importance of strong raw signals while emphasizing context- specific spikes through the relative term. g ∗ n = ( Normalize(Softmax(s cal /τ )) n if n∈E ∗ 0otherwise (13) where g ∗ n denotes the ground truth routing weight for the n-th expert, τ is a temperature hyperparameter controlling the sharpness of the probability distribution. This formulation ensures that the ground truth labels not only identify the most suitable experts E ∗ but also accurately reflect their relative importance as determined by the teacher model. 2) Phase I: Context-Driven Joint Expert Training: In the second phase, the context-aware router ψ is frozen and serves as a fixed coordinator that determines data-to-expert assignment. The optimization objective shifts to the physical differentiation and joint training of the expert pool E . By de- coupling router learning from expert optimization, we establish a stable training environment in which each expert is forced to specialize in the semantic subspaces identified during Phase I. a) Router-Guided Physical Specialization: To transform the dense FFN layers into a collection of specialized experts, we decompose the original model into structurally isolated expert modules E 1 ,...,E N . During forward propagation, expert execution strictly follows the routing decisions pro- duced by the frozen router. Only experts selected by the active index setE active participate in computation and parameter updates: y = X k∈E active g k · E k (x;θ k )(14) where x and y denote the input and output hidden states, re- spectively, and g k represents the fixed routing weight assigned to expert E k . Under this selective activation scheme, gradients are restricted to the parameters θ k of active experts, enforcing strong functional separation and driving each expert to adapt exclusively to its designated context. Recent studies on MoE refactoring [6] show that rout- ing patterns exhibit strong correlation across consecutive Transformer layers, rendering independent layer-wise routing largely redundant. Following this observation, we adopt a global routing consistency strategy, where the single routing vector g generated by ψ is broadcast to all FFN layers. This constructs vertically aligned expert pathways that preserve semantic coherence throughout the network depth and enable consistent processing of specialized representations. b) Distillation-Enhanced Expert Specialization (Op- tional): Directly transitioning from dense to sparse execu- tion may introduce temporary performance degradation. To accelerate expert adaptation and preserve the teacher model’s representational capacity, we incorporate the original dense model as a persistent reference and apply dual-granularity distillation [34]. Specifically, we introduce two auxiliary objectives: 1) Hidden State Alignment. We minimize the mean squared error between student (S) and teacher (T ) hidden states across all Transformer layers. An adaptive normalization functionN (·) is applied to account for architectural differ- ences (e.g., Pre-Norm vs. Post-Norm) and improve gradient stability: L hid = L X l=1 MSE(N (H (l) S ),N (H (l) T ))(15) where L denotes the total number of layers, and H (l) S and H (l) T represent the corresponding hidden states. 2) Logit Alignment. We apply symmetric KL divergence to align output probability distributions, encouraging the sparse experts to preserve the predictive behavior of the dense teacher. The overall optimization objective is defined as: L total =L task + α (L hid +L logit )(16) where L task denotes the primary task loss (e.g., the task in the predictive query), and α controls the strength of distilla- tion regularization. This auxiliary supervision stabilizes sparse training and enables QCATS to retain dense-model accuracy under constrained execution budgets. IV. SYSTEM-LEVEL OPTIMIZATION In large-scale query scenarios, batching data from multiple queries is standard practice to maintain high GPU utilization. However, the diverse query contexts within mixed batches trigger divergent expert activation paths, which may fragment GPU parallelism and thereby degrade hardware efficiency. Simultaneously, database text data typically exhibits a long- tailed length distribution, which makes static batching ineffi- cient due to excessive zero padding. Furthermore, as QCATS significantly boosts GPU inference efficiency, traditional syn- chronous execution pipelines could become increasingly CPU- bound, causing preprocessing latency to emerge as a major bottleneck for overall system performance. To address the aforementioned inefficiencies, we propose a system-level optimization framework that adopts an asyn- chronous producer-consumer pipeline architecture. As illus- trated in Fig. 8, we decouple the inference lifecycle into two separate modules: The CPU acts as the producer, re- sponsible for preparing routing-aware batched inputs, while the GPU serves as the consumer, dedicated to performing compute-intensive sparse inference computations. A thread- safe prefetch buffer connects the two stages, allowing data preparation and model execution to proceed concurrently and overlap in time. The core of the producer module, as shown in the left part in Fig. 8, is hierarchical bucketing. It reorganizes query data through a two-level process to jointly optimize execution-path consistency and hardware efficiency. The first level focuses on aligning model execution behavior by grouping inputs according to routing decisions, while the second level targets memory and computation efficiency by minimizing padding overhead through length-aware batching. At the first level, QCATS leverages the context-aware router to precompute expert assignments before batch construction. Text data are grouped based on their routing outcomes, such that all texts within the same group activate an identical group of experts during inference. This routing-aware group- ing ensures execution-path homogeneity within each batch and eliminates control-flow divergence on GPUs. At the second level, text data within each routing group is further reorganized based on its length. Text entries with similar lengths are placed into the same bucket to reduce padding overhead. To maintain stable GPU utilization across varying input sizes, the system dynamically adjusts batch sizes according to the representative sequence length of each bucket and the available GPU resource budget. Subsequently, the bucketed text entries are tokenized and collated into batches, which are then enqueued into a prefetch buffer for immediate processing by the consumer module in the GPU. In summary, the two-level reorganization enables QCATS to construct routing-consistent and length-homogeneous batches that maximize sparse execution efficiency while minimiz- ing wasted computation caused by padding. Combined with Context-Aware Router Hierarchical Bucketing CPU (Producer) Group[E0&E1] Group [E2&E3] ... E 0 E 1 E 2 E 3 E 0 E 1 E 2 E 3 GPU (Consumer) Inference 0 Inference i Prefetch Buffer 퐀 0 퐀 퐀 . . . Expert Weight g PQ 퐀 meta 퐀 text . . . Fig. 8: QCATS Processing with Pipeline. the asynchronous producer–consumer pipeline, hierarchical bucketing effectively overlaps CPU-side preprocessing with GPU inference, eliminates pipeline stalls, and ensures that system throughput is primarily bounded by model execution performance rather than data preparation overhead. V. EXPERIMENTS In this section, we conduct extensive experiments to evaluate QCATS against the design objectives mentioned in §I-B. The evaluation mainly focuses on the following Research Questions (RQs): • RQ1 (System Efficiency): For G1, how much improvement does QCATS achieve in terms of end-to-end latency and peak GPU memory footprint? • RQ2 (System Effectiveness): For G2, can QCATS repro- duce the predictive accuracy of the original fine-tuned dense baselines regarding the reduction in activated parameters? • RQ3 (Ablation & Breakdown): What is the contribution of each component to the goals? Specifically, how does the latency breakdown demonstrate the effectiveness of our proposed optimization strategies? Meanwhile, what is the contribution of each system component to accuracy preservation? • RQ4 (Sensitivity & Robustness): How sensitive is the model performance to key hyperparameters, and how is the neuron activation pattern aligned with the query context? In addition, does QCATS maintain robustness when transfer- ring to similar domains for data analysis? A. Experimental Setup 1) Datasets and Tasks: To evaluate QCATS across diverse domains, we use four real-world datasets summarized in Table I. • Amazon Review (E-commerce) [35] contains user reviews and item metadata with 17 structured columns. The task is to predict product ratings from user reviews. • Kickstarter (Crowdfunding Finance) [36] tracks crowd- funding campaigns and includes 7 structured metadata columns. The task is to predict campaign funding status from the project name and blurb. • OSHA (Industrial Safety) [37] records industrial accidents with 12 structured metadata columns. The task is to classify injury severity from accident abstracts. TABLE I: Summary of dataset statistics and task definitions. DatasetTotal SizeContext Columns Amazon1,908,006[maincategory, subcategory, price] Kickstarter202,835[maincategory, subcategory, goal] OSHA42,645[occupation, accidenttype, age] CVE90,847[cwe, privilegesrequired, productcount] • CVE (Cybersecurity) [38] describes software vulnerabili- ties and contains 11 structured metadata columns. The task is to predict CVSS severity from vulnerability descriptions. 2) Query Workload Generation: Public datasets rarely in- clude query logs combining text inputs, structured metadata, and predictive UDFs, so we randomly generate 50 SQL tem- plates with one to three predicates each for the datasets from the aforementioned widely-used benchmarks. Context-column predicates use categorical equality or numerical ranges, com- bined only with AND; context columns may also appear in GROUP BY clauses. Non-context predicates are randomly sampled from LIKE, IN, equality, and range conditions, optionally connected by OR. Templates may include valid joins sampled from predefined joinable table pairs. These templates represent common analytical prediction queries involving cat- egory/range filtering, metadata joins, and grouped prediction aggregation. 3) Baselines and Implementation: To validate generality and effectiveness, we evaluate QCATS on BERT and Qwen architectures, and compare three groups of methods: 1) Dense Baselines: Original unpruned BERT-base [39] and Qwen-base [40] models. 2) MoE Baselines: MoEfication [23], which partitions FFN neurons into logical experts without changing weights, and MoEBERT [34], which converts a fine-tuned model into a physical MoE through adaptation and distillation. 3) QCATS (Ours): QCATS applied to BERT and Qwen, denoted as QCATS (BERT) and QCATS (Qwen). All sparse methods activate 25% of FFN parameters during inference, while dense baselines use all parameters. Since no standard MoEfication implementation exists for Qwen, we compare Qwen-base directly with QCATS (Qwen). QCATS is implemented in Apache Spark [41] 4.0.1 with PyTorch 2.5 and CUDA 11.8, using BERT-base (110M parameters) and Qwen- 0.6B on an Intel Xeon Gold 5218R CPU and one NVIDIA A5000 GPU (24GB). Note that we use compact models to reflect the resource constraints of predictive database systems and to isolate the benefits of query-aware sparsification. Unless otherwise stated, QCATS uses the N4K1 configuration (N = 4, K = 1), with the hidden dimension split between shared and unique neurons. The router uses the top M = 3 attributes, a 512-hidden Transformer with 4 heads, and a 3-layer 256-dimensional MLP. During training, Phase I optimizes the router with lr = 5e− 4, η = 0.02, and τ = 1.0, while Phase I fine-tunes the experts with α = 5.0, and learning rates of 3e−5 for BERT and 1e−5 for Qwen, respectively. Additional hyperparameters are reported in the appendix. 2565121024 Batch Size 0 5 10 15 20 Peak Memory (GB) BERT-base MoEfication MoEBERT QCATS (a) CVE (BERT) 51210242048 Batch Size 0 5 10 15 Peak Memory (GB) BERT-base MoEfication MoEBERT QCATS (b) Kickstarter (BERT) 3264128 Batch Size 0 5 10 15 Peak Memory (GB) Qwen-base QCATS (c) CVE (Qwen) 128256512 Batch Size 0 5 10 15 Peak Memory (GB) Qwen-base QCATS (d) Kickstarter (Qwen) Fig. 9: Peak Memory usage comparison. Offline Construction Cost. With 100k training examples, activation profiling, clustering, and expert-slice construction take a few minutes. The expert fine-tuning dominates offline cost, taking approximately 30 minutes for BERT and 2 hours for Qwen, or 1.5 and 6 hours with distillation, respectively. 4) Evaluation Settings and Metrics: We evaluate efficiency by executing approximately 600 SQL queries per dataset, derived from the 50 query templates, retrieving 1 million rows in total. We report end-to-end latency and peak GPU memory usage, with the average batch size aligned across methods. We evaluate effectiveness using accuracy and weighted F1-score on the aggregated test set formed by merging query results from all workloads. B. System Efficiency (RQ1) To answer RQ1, we assess the system-level efficiency in end-to-end response time and peak memory. 1) End-to-End Latency: Under the efficiency evaluation setting (§V-A4) on the 1-million-row workload, Table I reports the end-to-end latency of all methods. All baselines and QCATS w/o pipeline use serial execution, whereas QCATS adopts the proposed pipelined architecture. The results show that QCATS achieves speedup by using query-level routing to effectively reduce execution overhead; on top of this, the pipeline further amplifies the gain by overlapping CPU preprocessing with GPU inference. In contrast, MoEfication only introduces logical sparsity, which does not necessarily translate into efficient physical execution on GPUs. 2) Peak GPU Memory (VRAM) Consumption: As shown in Fig. 9, we evaluate peak GPU memory usage under the efficiency evaluation setting (§V-A4). Given that the static weights of BERT and Qwen occupy only about 220 MB and 1.2 GB, respectively, the substantial reduction in peak memory usage achieved by QCATS indicates that its memory savings mainly come from reduced activation memory. QCATS allo- cates memory only to the activated expert subsets and reduces DB OverheadRouterData PreprocessModel Inference 0510152025 Response Time (s) QCATS w/o pipeline QCATS w/o all opt MoEBERT MoEfication BERT-base (a) Amazon (BERT) 020406080 Response Time (s) (b) OSHA (BERT) 050100150 Response Time (s) QCATS w/o pipeline QCATS w/o all opt Qwen-base (c) Amazon (Qwen) 0100200300400 Response Time (s) (d) OSHA (Qwen) Fig. 10: Latency Breakdown. padding through bucketing, thereby effectively compressing the runtime memory footprint. C. System Effectiveness (RQ2) To answer RQ2, we evaluate the prediction robustness of QCATS under the effectiveness evaluation setting (§V-A4) using accuracy and weighted F1-score. Table I presents the comparative performance of all models across four datasets. It demonstrates that QCATS preserves task accuracy while activating only a subset of FFN parameters. QCATS consistently rivals or outperforms dense baselines and matches the performance of MoEBERT, despite using a much simpler routing strategy. Our results show that for database-oriented tasks, understanding the global context of the entire query is sufficient to select the right experts. This implies that the heavy computational cost of routing every token is redundant, as QCATS delivers the same level of accuracy with a much more efficient, query-level approach. D. Ablation & Breakdown (RQ3) This section answers RQ3 by evaluating the specific impact of key components. We perform a latency breakdown to evaluate the impact of the sparsity mechanism of QCATS and the hierarchical bucketing. Then, we conduct an ablation study to isolate the roles of the routing mechanism and knowledge distillation in maintaining model accuracy. 1) Latency Breakdown and Bottleneck Analysis: To analyze the sources of QCATS’s performance gains in database scenar- ios, we executed a query that retrieves 100,000 rows and de- composed the total response time into four main components: database overhead, router computation (specific to QCATS), data preprocessing, and model inference. As shown in Fig. 10, the QCATS router (green) introduces only negligible overhead. During model inference, QCATS not only reduces latency by lowering FLOPs through sparsification, but also avoids the runtime overhead of fine-grained dynamic routing in MoE baselines through query-level global routing. Meanwhile, by TABLE I: End-to-end execution time (s) comparison. Dataset BERT-base BackboneQwen-base Backbone BERT-baseMoEficationMoEBERTQCATS w/o pipelineQCATSQwen-baseQCATS w/o pipelineQCATS Amazon279.05299.32 (0.93×)236.16 (1.18×)164.02 (1.70×)98.69 (2.83×)1499.22517.49 (2.90×)463.30 (3.24×) Kickstarter352.94376.14 (0.94×)295.33 (1.20×)172.89 (2.04×)129.13 (2.73×)1739.19874.38 (1.99×)780.19 (2.23×) OSHA785.40847.95 (0.93×)636.40 (1.23×)307.46 (2.55×)230.08 (3.41×)5241.142231.12 (2.35×)1981.82 (2.64×) CVE798.29862.21 (0.93×)653.82 (1.22×)231.51 (3.45×)180.51 (4.42×)4521.261564.17 (2.89×)1418.08 (3.19×) TABLE I: Comparison of Accuracy and Weighted F1-Score. Boldface indicates the best performance. ModelAmazonKickstarterOSHACVE (Acc / F1)(Acc / F1)(Acc / F1)(Acc / F1) BERT-base89.7 / 88.681.4 / 81.093.0 / 92.977.1 / 77.0 MoEfication89.5 / 88.380.8 / 80.492.7 / 92.476.7 / 76.4 MoEBERT89.9 / 88.781.6 / 81.393.0 / 92.978.1 / 78.0 QCATS (BERT)89.6 / 88.581.8 / 81.493.0 / 92.977.9 / 77.8 Qwen-base89.2 / 87.580.1 / 80.092.2 / 91.974.1 / 73.7 QCATS (Qwen)89.5 / 87.581.2 / 81.092.8 / 92.676.6 / 76.2 Base Model QCATS QCATS-w/o distill QCATS-w/o router QCATS-w/o both CVEKickstarter 70.0 72.5 75.0 77.5 80.0 82.5 Weighted F1(%) (a) BERT Arch CVEKickstarter 70.0 72.5 75.0 77.5 80.0 82.5 Weighted F1(%) (b) Qwen Arch Fig. 11: Ablation study. reducing padding through hierarchical bucketing, QCATS w/o pipeline outperforms QCATS w/o all opt in both preprocessing and inference. Finally, the latency breakdown further confirms the necessity of pipelining for lightweight models such as BERT: once GPU inference latency is significantly reduced, CPU preprocessing remains a non-negligible bottleneck. 2) Context-Aware Router and Knowledge Distillation: To quantify the independent contributions of knowledge distilla- tion (KD) and dynamic routing, we compare four ablation vari- ants under the effectiveness evaluation setting (Fig. 11). The results show that QCATS variants with routing (red/orange) consistently achieve performance comparable to the dense baseline, while the w/o distill variant even surpasses the teacher model on the Qwen-CVE task, indicating that KD is not the decisive factor behind the performance gain. In contrast, removing routing leads to a clear performance drop, showing that the routing mechanism is critical for expert specialization. E. Sensitivity & Robustness (RQ4) To answer RQ4, we study sensitivity to expert granularity and context columns, evaluate diverse SQL workloads and test transferability to a similar domain. 1) Expert granularity: Following the effectiveness evalu- ation setting, we investigated expert granularity (N vs. K) N4K1N8K2N16K4 75 78 81 84 Weighted F1 (%) Weighted F1 Latency 105 120 135 Latency (s) (a) Impact of expert granularity (N andK). 012345 M 75 78 81 84 Weighted F1 (%) Weighted F1 Routing Latency 0 100 200 300 Routing Latency (ms) (b) Impact of context column. Fig. 12: Hyperparameter experiments. TABLE IV: Transferability to Similar Domains. Boldface indicates the best performance. Model Source Domain (Amazon)Target Domain (Flipkart) (Acc / F1)(Acc / F1) BERT-base89.7 / 88.688.7 / 86.5 QCATS (BERT)89.6 / 88.588.7 / 86.6 Qwen-base89.2 / 87.588.6 / 85.6 QCATS (Qwen)89.5 / 87.588.8 / 85.6 on the Kickstarter dataset while maintaining a constant 25% activation ratio. Fig. 12a reveals high performance stability, indicating that accuracy depends primarily on parameter reten- tion rather than slicing size. However, the latency for a query retrieving 100,000 rows, highlights the cost of finer granu- larity. Therefore, we adopt N4K1 as the default to maximize computational density and throughput without compromising predictive performance. 2) Context Columns: Under the effectiveness evaluation setting, we analyze the sensitivity of the retained context column count M on the Kickstarter dataset. Fig. 12b illustrates this trade-off between weighted F1-score and average routing latency across varying M . As shown, a small M fails to capture the global data distribution, weakening context align- ment and routing accuracy. Conversely, a large M introduces redundant attributes; this increases preprocessing overhead. 3) QCATS under diverse SQL workloads: To evaluate the adaptability of QCATS across diverse SQL workloads, we analyze four representative queries on the Amazon dataset using the Qwen backbone (Fig. 13). As shown in Fig. 14, QCATS consistently reduces end-to-end latency across all scenarios while exerting only minimal impact on prediction accuracy. Notably, even for Q4, where the UDF appears in the filter predicate, QCATS still achieves substantial latency reduction by significantly decreasing the amount of data fed into the model through query rewriting. SELECT Sentiment(R.reviewText) FROM Reviews R JOIN Metadata M ON R.asin = M.asin WHERE M.main_category = 'Grocery & Gourmet Food' AND R.reviewName IS NOT NULL SELECT M.sub_category, AVG(Sentiment(R.reviewText)) FROM Reviews R JOIN Metadata M ON R.asin = M.asin WHERE M.price > 25.72 AND M.price <= 45 AND (R.unixReviewTime<= 1521158400 OR M.brand LIKE 'Ave%'); GROUP BY M.main_category SELECT Sentiment(R.reviewText) FROM Reviews R WHERE R.verified != 'True' AND R.vote=0 SELECT M.brand FROM Reviews R JOIN Metadata M ON R.asin = M.asin WHERE M.sub_category='Outdoor Recreation' AND R.verified='True' AND Sentiment(R.reviewText) = 2 (a) Q1: Normal (b) Q2: Aggregation (c) Q3: No Context(d) Q4: UDF Predicate Fig. 13: Predictive queries in the experiment. Q1Q2Q3Q4 70 75 80 85 90 95 100 Weighted F1(%) Base QCATS (a) Weighted F1-score. 1800 1900 Base QCATS Q1Q2Q3Q4 0 100 200 300 Latency(s) (b) End-to-end latency. Fig. 14: Performance evaluation of QCATS across diverse SQL workloads. 4) Case Study: Transferability to Similar Domains: To assess generalization, we directly transfer models trained on Amazon to the Flipkart dataset [42]–[44] in a zero-shot manner, using Product Category and Product Price as routing contexts (following effectiveness evaluation set- ting). As shown in Table IV, QCATS achieves performance comparable to dense baselines in this similar-domain setting, indicating that its query-based slicing mechanism preserves both transferability and core model capabilities. VI. RELATED WORK In-database ML and predictive query processing have long focused on integrating model inference into SQL ex- ecution pipelines. Early approaches [16], [17] and modern Cloud APIs [3], [4] predominantly rely on UDFs to invoke models, but they treat inference as a black-box task, discard- ing rich query-time context. To enable deeper optimization, some frameworks [7], [45] decompose models into basic re- lational or tensor operators to perform fine-grained execution. However, these white-box optimizations are largely confined to traditional ML or neural networks and do not scale to the massive parameter space of Transformers. In contrast, our work bridges this gap: we move beyond external data movement or scheduling by coupling database analytics with model internals. By leveraging query context, we enable the deterministic customization of the model inference path itself for Transformer-based models. Database-aware model serving has been explored to improve inference flexibility, but existing approaches differ fundamen- tally from QCATS in optimization granularity and system objectives. ARM-Net [46] adapts feature interactions at the instance level to improve prediction accuracy, without target- ing inference efficiency or execution-path optimization. Model Slicing [47] supports elastic inference cost via width-wise slicing driven by resource budgets, but does not exploit query semantics or workload locality. LEADS [48] also exploits database context, but selects among replicated experts or multiple conventional models primarily to improve prediction quality for structured-data analytics. Like the preceding ap- proaches, it does not specifically address Transformer-based text inference. QCATS instead uses query context to deter- ministically select FFN neuron slices within a single Trans- former backbone. This targets text prediction while reducing inference cost and GPU memory usage and preserving dense- model accuracy. Furthermore, all the aforementioned methods do not specifically address the inference characteristics of Transformer-based text prediction, while QCATS uses query context to deterministically select model parameters, thereby improving inference efficiency while preserving accuracy. Efficient Transformer inference and MoE architecture [12], [13] scales model capacity while maintaining manageable inference costs. Recent efforts in MoE-fication [6], [23], [34], [49] have extended this concept by refactoring pre-trained dense models into expert-based structures. However, standard MoEs rely on token-level, layer-wise routing, which treats input data as independent instances and ignores high-level semantic contexts. While recent works attempt to optimize routing via decoupling [6], [50] or speculation [51], [52], they remain constrained by micro-level token dependencies, limiting optimization in data-intensive database environments. In contrast, QCATS is tailored for in-database analytics, where data is accessed via structured queries rather than random batches. We exploit the query context to implement query-level global routing. By leveraging these database-specific hints to deterministically pre-select expert slices, we eliminate run- time dispatching overhead and enable efficient, homogeneous batching that is unattainable in context-agnostic systems. VII. CONCLUSION We presented QCATS, a query context-aware transformer slicing framework that enables efficient sparse inference for in- database predictive queries. By shifting conditional execution from token level to query level, QCATS aligns sparse neural execution with database semantics. Extensive experimental results demonstrate that QCATS significantly improves in- ference throughput and reduces memory consumption while preserving predictive accuracy compared to dense baselines. This suggests that query-aware execution provides a practical foundation for bridging Transformer-based model inference with high-performance database systems. We believe this work opens new opportunities for designing database-native inference engines that treat neural execution as a first-class query operator rather than an external black- box service. Building on this direction, we will evaluate QCATS on larger Transformer backbones and more complex, realistic workloads under distribution shifts and heterogeneous domains. We will also investigate adaptation and cross-domain transferability of expert libraries and routing policies, while interactions with generative-model serving systems remain separate future work. VIII. ACKNOWLEDGMENT This work was supported by the Fundamental and Inter- disciplinary Disciplines Breakthrough Plan of the Ministry of Education of China (JYB2025XDXM103), the National Natural Science Foundation of China (Grant No. 62602571), and the Leading Talent of Technological Innovation Program (No. 2023R5214) of Zhejiang Province. We disclose that GPT was used only to assist with En- glish polishing, grammar improvement, and the writing of some utility code. The research questions, overall design, implementation process, experiments, analysis of results, and conclusions were developed entirely by the authors. REFERENCES [1] Q. Lin, S. Wu, J. Zhao, J. Dai, F. Li, and G. Chen, “A comparative study of in-database inference approaches,” in 2022 IEEE 38th International Conference on Data Engineering (ICDE). IEEE, 2022, p. 1794–1807. [2] Y. Peng, Z. Xie, K. Chen, G. Chen, and L. Shou, “Towards automatic and efficient prediction query processing in analytical database,” in 2025 IEEE 41st International Conference on Data Engineering (ICDE). IEEE, 2025, p. 2253–2266. [3] Databricks, “Model serving with Databricks,” https://docs.databricks. com/en/machine-learning/model-serving/index.html, 2024, accessed: 2026-01-20. [4] Google Cloud, “Model inference overview,” https://cloud.google.com/ bigquery/docs/inference-overview, 2024, accessed: 2026-01-20. [5] Alibaba Cloud, “PolarDB for AI,” https://w.alibabacloud.com/ help/en/polardb/polardb-for-mysql/user-guide/polardb-for-ai/, 2023, ac- cessed: 2026-01-20. [6] R. Cai, Y. Ro, G.-W. Kim, P. Wang, B. Ehteshami Bejnordi, A. Akella, Z. Wang et al., “Read-me: Refactorizing llms as router-decoupled mix- ture of experts with system co-design,” Advances in Neural Information Processing Systems, vol. 37, p. 116 126–116 148, 2024. [7] K. Park, K. Saur, D. Banda, R. Sen, M. Interlandi, and K. Karanasos, “End-to-end optimization of machine learning prediction queries,” in Proceedings of the 2022 International Conference on Management of Data, 2022, p. 587–601. [8] C. Yan, Y. Lin, and Y. He, “Predicate pushdown for data science pipelines,” Proceedings of the ACM on Management of Data, vol. 1, no. 2, p. 1–28, 2023. [9] Y. Guo, G. Li, R. Hu, and Y. Wang, “In-database query optimization on sql with ml predicates,” The VLDB Journal, vol. 34, no. 1, p. 12, 2025. [10] D. J. Abadi, S. R. Madden, and N. Hachem, “Column-stores vs. row- stores: how different are they really?” in Proceedings of the 2008 ACM SIGMOD international conference on Management of data, 2008, p. 967–980. [11] M. A. Jose and F. G. Cozman, “A multilingual translator to sql with database schema pruning to improve self-attention,” International Journal of Information Technology, vol. 15, no. 6, p. 3015–3023, 2023. [12] D. Lepikhin, H. Lee, Y. Xu, D. Chen, O. Firat, Y. Huang, M. Krikun, N. Shazeer, and Z. Chen, “Gshard: Scaling giant models with conditional computation and automatic sharding,” arXiv preprint arXiv:2006.16668, 2020. [13] W. Fedus, B. Zoph, and N. Shazeer, “Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity,” Journal of Machine Learning Research, vol. 23, no. 120, p. 1–39, 2022. [14] J. He, J. Qiu, A. Zeng, Z. Yang, J. Zhai, and J. Tang, “Fastmoe: A fast mixture-of-expert training system,” arXiv preprint arXiv:2103.13262, 2021. [15] T. Gale, D. Narayanan, C. Young, and M. Zaharia, “Megablocks: Effi- cient sparse training with mixture-of-experts,” Proceedings of Machine Learning and Systems, vol. 5, p. 288–304, 2023. [16] J. M. Hellerstein, C. R ́ e, F. Schoppmann, D. Z. Wang, E. Fratkin, A. Gorajek, K. S. Ng, C. Welton, X. Feng, K. Li et al., “The madlib analytics library: or mad skills, the sql,” Proceedings of the VLDB Endowment, vol. 5, no. 12, p. 1700–1711, 2012. [17] M. Boehm, I. Antonov, S. Baunsgaard, M. Dokter, R. E. Ginthoer, K. Innerebner, F. Klezin, S. Lindstaedt, A. Phani, B. Rath et al., “Systemds: A declarative machine learning system for the end-to-end data science lifecycle,” in 10th Conference on Innovative Data Systems Research, 2020. [18] M. Jasny, T. Ziegler, T. Kraska, U. Roehm, and C. Binnig, “Db4ml-an in- memory database kernel with machine learning support,” in Proceedings of the 2020 ACM SIGMOD International Conference on Management of Data, 2020, p. 159–173. [19] W. Zheng, M. Xu, S. Song, and K. Ye, “Bucketserve: Bucket-based dynamic batching for smart and efficient llm inference serving,” in 2025 IEEE International Conferences on Internet of Things (iThings) IEEE Green Computing & Communications (GreenCom) IEEE Cyber, Physi- cal & Social Computing (CPSCom) and IEEE Smart Data (SmartData) and IEEE Congress on Cybermatics (Cybermatics).IEEE, 2025, p. 103–111. [20] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” Advances in neural information processing systems, vol. 30, 2017. [21] Z. Li, C. You, S. Bhojanapalli, D. Li, A. S. Rawat, S. J. Reddi, K. Ye, F. Chern, F. Yu, R. Guo et al., “The lazy neuron phenomenon: On emergence of activation sparsity in transformers,” in The Eleventh International Conference on Learning Representations. [22] Z. Liu, J. Wang, T. Dao, T. Zhou, B. Yuan, Z. Song, A. Shrivastava, C. Zhang, Y. Tian, C. Re et al., “Deja vu: Contextual sparsity for efficient llms at inference time,” in International Conference on Machine Learning. PMLR, 2023, p. 22 137–22 176. [23] Z. Zhang, Y. Lin, Z. Liu, P. Li, M. Sun, and J. Zhou, “Moefication: Transformer feed-forward layers are mixtures of experts,” in Findings of the Association for Computational Linguistics: ACL 2022, 2022, p. 877–890. [24] T. Mikolov, E. Grave, P. Bojanowski, C. Puhrsch, and A. Joulin, “Ad- vances in pre-training distributed word representations,” in Proceedings of the eleventh international conference on language resources and evaluation (LREC 2018), 2018. [25] L. Breiman, “Random forests,” Machine learning, vol. 45, no. 1, p. 5–32, 2001. [26] T. Cover and P. Hart, “Nearest neighbor pattern classification,” IEEE transactions on information theory, vol. 13, no. 1, p. 21–27, 1967. [27] P. Li, W. Wei, R. Zhu, B. Ding, J. Zhou, and H. Lu, “Alece: An attention-based learned cardinality estimator for spj queries on dynamic workloads,” Proceedings of the VLDB Endowment, vol. 17, no. 2, p. 197–210, 2023. [28] W. Kwon, Z. Li, S. Zhuang, Y. Sheng, L. Zheng, C. H. Yu, J. Gonzalez, H. Zhang, and I. Stoica, “Efficient memory management for large language model serving with pagedattention,” in Proceedings of the 29th symposium on operating systems principles, 2023, p. 611–626. [29] L. Zheng, L. Yin, Z. Xie, C. Sun, J. Huang, C. H. Yu, S. Cao, C. Kozyrakis, I. Stoica, J. E. Gonzalez et al., “Sglang: Efficient execution of structured language model programs,” Advances in neural information processing systems, vol. 37, p. 62 557–62 583, 2024. [30] T. Dao, D. Fu, S. Ermon, A. Rudra, and C. R ́ e, “Flashattention: Fast and memory-efficient exact attention with io-awareness,” Advances in neural information processing systems, vol. 35, p. 16 344–16 359, 2022. [31] E. Frantar, S. Ashkboos, T. Hoefler, and D.-A. Alistarh, “Optq: Accurate post-training quantization for generative pre-trained transformers,” in 11th International Conference on Learning Representations, 2023. [32] J. Lin, J. Tang, H. Tang, S. Yang, W.-M. Chen, W.-C. Wang, G. Xiao, X. Dang, C. Gan, and S. Han, “Awq: Activation-aware weight quanti- zation for on-device llm compression and acceleration,” Proceedings of machine learning and systems, vol. 6, p. 87–100, 2024. [33] G. Xiao, J. Lin, M. Seznec, H. Wu, J. Demouth, and S. Han, “Smoothquant: Accurate and efficient post-training quantization for large language models,” in International conference on machine learning. PMLR, 2023, p. 38 087–38 099. [34] S. Zuo, Q. Zhang, C. Liang, P. He, T. Zhao, and W. Chen, “Moebert: from bert to mixture-of-experts via importance-guided adaptation,” in Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 2022, p. 1610–1623. [35] Y. Hou, J. Li, Z. He, A. Yan, X. Chen, and J. McAuley, “Bridging language and items for retrieval and recommendation,” arXiv preprint arXiv:2403.03952, 2024. [36] Y. Kantharia, “Kickstarter campaigns dataset 2.0,” https://w.kaggle. com/datasets/yashkantharia/kickstarter-campaigns-dataset-20,2020, data originally scraped from webrobots.io. Accessed: 2026-01-26. [37] Z. Ou, D. Li, Z. Tan, W. Li, H. Liu, and S. Song, “Building safer sites: A large-scale multi-level dataset for construction safety benchmark,” in Proceedings of the 34th ACM International Conference on Information and Knowledge Management, 2025, p. 6508–6512. [38] F. Manzoni, “Vulnerability management datasets,” https://w.kaggle. com/datasets/francescomanzoni/vulnerability-management-datasets, 2025, accessed: 2026-01-26. [39] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,” in Pro- ceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume 1 (long and short papers), 2019, p. 4171–4186. [40] A. Yang, A. Li, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Gao, C. Huang, C. Lv et al., “Qwen3 technical report,” arXiv preprint arXiv:2505.09388, 2025. [41] The Apache Software Foundation, “Apache Spark,” https://spark.apache. org, 2024, accessed: 2026-01-20. [42] T. MSS, “Water bottle review dataset (flipkart),” https://w.kaggle. com/datasets/tharunmss/water-bottle-dataset-flipkart, 2025, accessed: 2026-01-26. [43] Guhan,“Speakerscrappeddatawithsentiments (flipkart),”https://w.kaggle.com/datasets/guhan0912/ speaker-scrapped-data-with-sentiments-flipkart,2025,accessed: 2026-01-26. [44] sublimE, “Laptop reviews dataset (flipkart),” https://w.kaggle.com/ datasets/gitadityamaddali/flipkart-laptop-reviews, 2024, accessed: 2026- 01-26. [45] Q. Lin, S. Wu, J. Zhao, J. Dai, M. Shi, G. Chen, and F. Li, “Smartlite: A dbms-based serving system for dnn inference in resource-constrained environments,” Proceedings of the VLDB Endowment, vol. 17, no. 3, p. 278–291, 2023. [46] S. Cai, K. Zheng, G. Chen, H. Jagadish, B. C. Ooi, and M. Zhang, “Arm-net: Adaptive relation modeling network for structured data,” in Proceedings of the 2021 International Conference on Management of Data, 2021, p. 207–220. [47] S. Cai, G. Chen, B. C. Ooi, and J. Gao, “Model slicing for supporting complex analytics with elastic inference cost and resource constraints,” Proceedings of the VLDB Endowment, vol. 13, no. 2, p. 86–99, 2019. [48] L. Zeng, N. Xing, S. Cai, G. Chen, B. C. Ooi, J. Pei, and Y. Wu, “Pow- ering in-database dynamic model slicing for structured data analytics,” Proceedings of the VLDB Endowment, vol. 17, no. 13, p. 4813–4826, 2024. [49] Z. Zhang, C. Xiao, Q. Qin, Y. Lin, Z. Zeng, X. Han, Z. Liu, R. Xie, M. Sun, and J. Zhou, “Exploring the benefit of activation sparsity in pre-training,” in Proceedings of the 41st International Conference on Machine Learning, 2024, p. 60 040–60 056. [50] R. Hwang, J. Wei, S. Cao, C. Hwang, X. Tang, T. Cao, and M. Yang, “Pre-gated moe: An algorithm-system co-design for fast and scalable mixture-of-expert inference,” in 2024 ACM/IEEE 51st Annual Interna- tional Symposium on Computer Architecture (ISCA).IEEE, 2024, p. 1018–1031. [51] Z. Du, S. Li, Y. Wu, X. Jiang, J. Sun, Q. Zheng, Y. Wu, A. Li, H. H. Li, and Y. Chen, “Sida: Sparsity-inspired data-aware serving for efficient and scalable large mixture-of-experts models,” Proceedings of Machine Learning and Systems, vol. 6, p. 224–238, 2024. [52] X. He, S. Zhang, Y. Wang, H. Yin, Z. Zeng, S. Shi, Z. Tang, X. Chu, I. Tsang, and O. Y. Soon, “Expertflow: Optimized expert activation and token allocation for efficient mixture-of-experts inference,” arXiv preprint arXiv:2410.17954, 2024. APPENDIX We attach the analysis and implementation details of QCATS in this appendix. QUERY CONTEXT ANALYSIS To evaluate the neuron activation across diverse query contexts, we constructed a benchmark of 16 predictive queries based on the Kickstarter dataset. We first selected 8 seed queries (Q1-Q8) from the workload and applied minor per- turbations—such as adding, deleting, or modifying filtering conditions—to generate a corresponding set of variants (Q9- Q16). TABLE V: Filter Predicates for the Queries. IDFilter Predicates P Q1 main_category = ‘technology’ AND sub_category = ‘Gadgets’ Q2 sub_category = ‘Video Games’ Q3 main_category = ‘music’ AND 1000 < goal < 2000 Q4 main_category = ‘publishing’ AND country = ‘GB’ AND 20 < duration < 30 Q5 main_category = ‘film & video’ AND sub_category = ‘Drama’ Q6 sub_category = ‘Drinks’ Q7 main_category = ‘fashion’ AND goal < 2000 Q8 main_category = ‘art’ Q9 main_category = ‘technology’ AND sub_category = ‘Gadgets’ AND city = ‘Hong Kong’ Q10 sub_category = ‘Video Games’ AND country = ‘FR’ Q11 main_category = ‘music’ AND goal < 2000 AND duration > 30 Q12 main_category = ‘publishing’ AND country = ‘CA’ Q13 sub_category = ‘Drama’ AND city = ‘London’ Q14 sub_category = ‘Vegan’ Q15 main_category = ‘fashion’ AND goal > 2000 Q16 sub_category = ‘Sculpture’ AND currency = ‘GBP’ All predictive queries follow a unified structure, formally defined as: SELECT MODEL(name_blurb) FROM Kickstarter WHERE P; (17) where P represents the set of variable predicates detailed in Table V. Given that Q9-Q16 are derivatives of Q1-Q8, their result sets often exhibit inclusion relationships (e.g., a modified query’s results being a subset of the original). To decouple the analysis of neuron activation patterns from inherent data overlap, we randomly sampled 200 distinct tuples from the result set of each query rather than using the full retrieval sets. We then monitored the activation states of a fine-tuned dense model on these samples. Aligned with our experimental sparsity setting—which maintains a 25% activation rate—we focused our analysis exclusively on the top 768 neurons (the top 25% of the post-activation hidden layer) ranked by average magnitude. Q1Q2Q3Q4Q5Q6Q7Q8Q9Q10Q11Q12Q13Q14Q15Q16 Query Context (B) Q1 Q2 Q3 Q4 Q5 Q6 Q7 Q8 Q9 Q10 Q11 Q12 Q13 Q14 Q15 Q16 Query Context (A) 100565254505750568856535553566159 561005264535060575589526456465154 525210061605948654952936360525558 546461100646054695165608568545862 505360641005545644654606988525259 575059605510045615351596256746061 506048544545100565161485346445252 565765696461561005259657465566277 885549514653515210055495149525855 568952655451615955100526556475256 535293606059486549521006259545658 556463856962537451656210071586065 535660688856466549565971100515458 564652545274445652475458511006361 615155585260526258525660546310065 595458625961527755565865586165100 50 60 70 80 90 100 Overlap Percentage (%) Fig. 15: Cross-Query Top-768 Neuron Overlap Matrix. This heatmap visualizes the pairwise overlap percentage of the top- 768 most active neurons across 16 distinct query contexts (Q1- Q16). Darker blue indicates higher overlap (similarity), while lighter colors indicate distinct activation patterns. Diagonal elements are 100% by definition. The pairwise overlap rates of these top-ranked neurons are visualized in Fig. 15. We observe strong context locality: there is significant divergence between unrelated groups (e.g., between the Q1-Q8 block and unrelated queries in Q9-Q16), whereas semantically related pairs (e.g., Q1 & Q9, Q2 & Q10) exhibit high similarity. This empirical evidence supports our core hypothesis: different query contexts induce distinct acti- vation patterns in Transformer feed-forward networks (FFNs). HYPERPARAMETER SETTINGS In Stage I: Expert Construction, we first employ the KNN algorithm to partition the dataset into N = 4 semantic groups. Subsequently, we sample D n = 500 representative instances from each group to perform activation profiling. This sample size is chosen to mitigate statistical noise while ensuring robust neuron distinction, thereby maintaining high profiling accuracy with minimal offline computational overhead. Based on this grouping, we adopt an N 4K1 configuration, where the feed-forward network is partitioned into N = 4 experts, and K = 1 expert is dynamically activated during inference. This results in a fixed activation rate of 25%, consistent with settings established in prior work [1], [2]. We prioritize the N = 4,K = 1 configuration because finer- grained splitting (e.g., N = 8,K = 2 or N = 16,K = 4), while potentially allowing better collaboration among semantic experts, incurs significant inference latency and introduces load balancing challenges while yielding only limited accuracy gains. Furthermore, we designate 50% of the neurons as shared neurons U shared and the remaining 50% as context-specific neurons. This split is motivated by the observation in Fig. 15, which indicates that the overlap of highly activated neurons across varying query contexts is approximately 50%. For One-time Feature Selection in Stage I, we train a Random Forest classifier on a subset of 30,000 samples to evaluate metadata importance, selecting the top M = 3 columns as effective context attributes. Empirically, we find that selecting the top three features is sufficient for the datasets used in this work, although the optimal M may vary across different datasets. In Context-Aware Expert Routing, the router is implemented as a Transformer block with a hidden dimension of 512 and 4 attention heads, followed by a 3-layer MLP with a dimension of 256. During Phase I: Router Semantic Alignment of Decoupled Training Strategy, we generate ground-truth labels using a calibration coefficient η = 0.02 and a softmax temperature τ = 1.0. We selected η = 0.02 based on observations from the training set: this value ensures that the distribution of selected expert indices aligns well with the ranking of expert scores, while simultaneously preventing load imbalance caused by the under-utilization of lower-scoring experts. Finally, during Phase I: Context-Driven Joint Expert Train- ing of Decoupled Training Strategy, we set the distillation loss weight to α = 5.0, consistent with [1]. REFERENCES [1] S. Zuo, Q. Zhang, C. Liang, P. He, T. Zhao, and W. Chen, “Moebert: from bert to mixture-of-experts via importance-guided adaptation,” in Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 2022, p. 1610–1623. [2] Z. Zhang, Y. Lin, Z. Liu, P. Li, M. Sun, and J. Zhou, “Moefication: Transformer feed-forward layers are mixtures of experts,” in Findings of the Association for Computational Linguistics: ACL 2022, 2022, p. 877–890.