Paper deep dive
When Machines Speak: A Unified Generative Framework for Integrating Machine-Native Symbols into Pretrained Large Language Models
Su Yan, Rakesh Iyer
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 92%
Last extracted: 8/21/2026, 3:15:48 AM
Summary
The paper introduces UniLang, a unified generative framework that extends pretrained Large Language Models (LLMs) to treat machine-native symbols (discrete codes from vector quantization) as first-class generative units alongside natural-language tokens. By expanding the LLM's vocabulary and using contrastive grounding to align machine tokens with textual descriptions, UniLang enables joint autoregressive modeling of heterogeneous data. The framework is evaluated on sequential recommendation and legal precedent prediction, consistently outperforming strong baselines without requiring task-specific architectures.
Entities (10)
Relation Signals (9)
UniLang ā evaluatedon ā Sequential Recommendation
confidence 95% Ā· We evaluate UniLang on two structurally distinct tasks, sequential recommendation and legal precedent prediction.
UniLang ā evaluatedon ā Legal Precedent Prediction
confidence 95% Ā· We evaluate UniLang on two structurally distinct tasks, sequential recommendation and legal precedent prediction.
UniLang ā extends ā Large Language Models
confidence 95% Ā· We introduce UniLang, a unified generative framework that bridges this divide by extending pretrained LLMs to treat machine-native symbols as first-class generative units
Machine-Native Symbols ā generatedby ā Residual Quantized Variational Autoencoder
confidence 90% Ā· More recent work employs hierarchical residual quantization models, such as Residual Quantized Variational Autoencoders (RQ-VAE), to produce multi-level discrete codes
UniLang ā outperforms ā strong baselines
confidence 90% Ā· Across both tasks, UniLang consistently outperforms strong baselines
Llama-3.2-1B-Instruct ā usedasbasefor ā UniLang
confidence 90% Ā· We useLlama-3.2-1B-Instruct 3 as the base model and extend its vocabulary with 1,024 machine- native tokens.
Amazon Product Reviews ā usedfor ā Sequential Recommendation
confidence 90% Ā· For sequential recommendation, we use the "Beauty" subset of the Amazon Product Reviews dataset
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Many real-world AI systems represent entities, behaviors, and structured information using discrete machine-native symbols rather than natural language. While these representations are compact and preserve task-relevant structure, they lie outside the linguistic token space of pretrained large language models (LLMs), creating a fundamental divide between language modeling and structured prediction. We introduce UniLang, a unified generative framework that bridges this divide by extending pretrained LLMs to treat machine-native symbols as first-class generative units alongside natural-language tokens. UniLang expands the LLM's vocabulary and embedding space with grounded machine-native representations, enabling textual and symbolic tokens to be jointly modeled and generated under a single autoregressive objective. This unified interface allows pretrained LLMs to directly operate on machine-native representations without requiring them to be verbalized as natural language or relying on task-specific architectures. We evaluate UniLang on two structurally distinct tasks, sequential recommendation and legal precedent prediction, spanning different domains and types of structured prediction. Across both tasks, UniLang consistently outperforms strong baselines, demonstrating a path toward extending pretrained LLMs beyond language and using them as a common generative modeling backbone for heterogeneous machine-native representations.
Tags
Links
- Source: https://arxiv.org/abs/2608.19529v1
- Canonical: https://arxiv.org/abs/2608.19529v1
Trouble viewing inline? Open PDF directly ā
Full Text
51,363 characters extracted from source content.
Expand or collapse full text
When Machines Speak: A Unified Generative Framework for Integrating Machine-Native Symbols into Pretrained Large Language Models Su Yan Google Inc. sueyan@google.com Rakesh Iyer Google Inc. rni@google.com Abstract Many real-world AI systems represent entities, behaviors, and structured informa- tion using discrete machine-native symbols rather than natural language. While these representations are compact and preserve task-relevant structure, they lie outside the linguistic token space of pretrained large language models (LLMs), creating a fundamental divide between language modeling and structured predic- tion. We introduce UniLang, a unified generative framework that bridges this divide by extending pretrained LLMs to treat machine-native symbols as first-class generative units alongside natural-language tokens. UniLang expands the LLMās vocabulary and embedding space with grounded machine-native representations, enabling textual and symbolic tokens to be jointly modeled and generated under a single autoregressive objective. This unified interface allows pretrained LLMs to directly operate on machine-native representations without requiring them to be verbalized as natural language or relying on task-specific architectures. We evaluate UniLang on two structurally distinct tasks, sequential recommendation and legal precedent prediction, spanning different domains and types of structured prediction. Across both tasks, UniLang consistently outperforms strong baselines, demonstrat- ing a path toward extending pretrained LLMs beyond language and using them as a common generative modeling backbone for heterogeneous machine-native representations. 1 Introduction Large language models (LLMs) have emerged as a general modeling framework across a wide range of applications. Their success stems from a unified autoregressive formulation in which diverse natural- language tasks are expressed through a common vocabulary and generation objective. However, this unified interface is fundamentally linguistic: pretrained LLMs operate over natural-language tokens. Many modern AI systems, in contrast, represent information using machine-native symbolic repre- sentations rather than natural language. These representations increasingly take the form of discrete codes produced by vector quantization or related techniques, representing complex entities such as compressed audio signals [37], semantic identifiers in recommendation systems [9,26], and struc- tured relational information in graph learning [20]. Such representations preserve explicit discrete structure and offer computational efficiency that natural language cannot easily replicate. However, they remain outside the linguistic token space of pretrained LLMs, making them difficult for LLMs to natively model and generate. This mismatch creates a methodological divide. Existing approaches largely follow one of two paradigms. The first treats natural language as a universal interface, converting structured informa- tion into textual descriptions so that pretrained LLMs can process it [5,13,16,19,38,39]. While effective, verbalization may obscure native structure, introduce representational ambiguity, and arXiv:2608.19529v1 [cs.CL] 20 Aug 2026 sacrifice machine-level precision. The second operates directly on machine-native representations using task-specific models [14,26,36,40]. Although these approaches preserve structural fidelity, they do not directly leverage the linguistic and world knowledge encoded in a pretrained LLM within the same generative space. At a deeper level, the limitation is representational. Pretrained LLMs lack a unified interface that allows machine-native symbols to function as first-class generative units alongside natural-language tokens. Without such an interface, structured symbolic prediction and language modeling remain separate paradigms. This observation motivates a simple question: Can pretrained LLMs be extended to directly model and generate machine-native symbols, rather than forcing symbolic information to become natural language or abandoning pretrained language models altogether? We answer this question with UniLang 1 , a unified framework that extends pretrained LLMs to jointly model natural-language tokens and machine-native symbols within a single autoregressive vocabulary and generation objective. Rather than treating machine-native representations as external objects or auxiliary embeddings, UniLang explicitly grounds them into the pretrained LLM representation space, allowing machine-native symbols and natural-language tokens to function as first-class generative units within the same autoregressive framework. This unified representational interface enables structurally different prediction problems to be ex- pressed under the same modeling framework without task-specific architectures. To demonstrate its generality, we evaluate UniLang on two structurally distinct tasks: sequential recommendation and legal precedent prediction. These tasks were intentionally selected because they represent heteroge- neous structured prediction problems rather than multiple datasets from the same application domain. Across both tasks, UniLang consistently outperforms strong baselines while using the same modeling framework. We make the following contributions: ā¢ A unified representational interface. We formulate structured prediction as typed autore- gressive generation over heterogeneous token types, allowing natural-language tokens and machine-native symbols to function as first-class generative units within the same pretrained LLM. ā¢ Grounded integration of machine-native symbols. We introduce a vocabulary-extension and contrastive grounding procedure that maps machine-native symbols into the pretrained LLMās representation space, enabling them to be processed and generated alongside natural- language tokens while retaining their discrete structure. ā¢ Generality across heterogeneous structured prediction tasks. We demonstrate that the same UniLang framework applies to structurally distinct tasks, including sequential recom- mendation and legal precedent prediction, without task-specific architectural modifications, and consistently outperforms strong baselines. 2 Background and related work 2.1 Machine-native representation. Machine-native representations convert natural-language content into compact, structured sequences of discrete symbols optimized for computation. The central idea is to encode rich semantic informa- tion in a concise symbolic form that facilitates efficient storage, retrieval, and reasoning. A well-known example is the use of ICD codes in the medical domain [8], where detailed diagnoses are represented as standardized symbolic identifiers. These codes capture complex, structured information that would otherwise require lengthy textual descriptions. Nowadays, machine-native representations follow the same principle but often are learned automatically from data rather than manually defined. 1 Code is available at https://github.com/Stella-S-Yan/llm-internalization 2 A variety of approaches have been proposed to generate such representations. Early methods rely on clustering-based discretization, such as hierarchical k-means over embedding spaces [33]. Hashing-based techniques map high-dimensional continuous vectors into compact binary codes [18]. Quantization-based approaches, including subspace quantization via k-means, learn discrete code- books for vector compression [9,12]. More recent work employs hierarchical residual quantization models, such as Residual Quantized Variational Autoencoders (RQ-VAE), to produce multi-level discrete codes that preserve semantic structure [20,26,36]. Figure 1 illustrates an example where each item (a movie) is associated with a discrete, hierarchical machine-native code generated by RQ-VAE. Details of the code construction procedure are provided in Section 3.1. 2.2 Sequential recommendation. Sequential recommendation aims to predict the next item a user will interact with given their historical interaction sequence. A wide range of neural architectures have been explored for this task, including convolutional, recurrent, graph-based, and Transformer-based models [3, 7, 15, 17, 30, 32, 40]. Figure 1: Example of the sequential prediction task. Each item is collectively represented bymachine-native codes and various metadata innaturallanguage. Recent work has explored reformulating recom- mendation as a language modeling problem by converting user, item, and interaction informa- tion into unified natural-language sequences and training encoder-decoder models [5]. While this verbalization strategy enables the use of LLMs, flattening structured interactions into text can obscure type information and introduce repre- sentational ambiguity. Other approaches encode items as discrete sym- bols derived from vector quantization of dense embeddings [9]. Extending this idea, [26] gener- ates hierarchical discrete representations using Residual Quantized VAEs (RQ-VAE), referring to the resulting identifiers as Semantic IDs, and integrates them within generative retrieval models. Later work notes that purely discrete codes may lose fine-grained semantic information, and proposes augmenting them with continuous embeddings to recover these signals [36]. While this hybrid ap- proach mitigates information loss, it still treats discrete symbols indirectly within learned embedding spaces rather than as first-class generative units in a unified language modeling framework. In contrast, UniLang models machine-native symbols and natural-language tokens together as generative units, enabling joint autoregressive prediction of structured and textual content. 2.3 Legal precedent prediction. Figure 2: Example of the legal precedent prediction task. Givencontext from the citing opinion (Garcia-Giraldo v. United States), predict thequotation sentence(s) or paragraph(s) from thecitedopinion (United States v. Broce), which is unknown at inference time. Given the context of a legal argument, legal precedent prediction seeks to identify the spe- cific sentence(s) or paragraph(s) from preceden- tial court decisions that supports the claim [22]. This task is substantially more challenging than case-level citation prediction or document re- trieval, where the objective is to identify a rel- evant case as a whole [2,11]. Legal cases of- ten span tens or hundreds of pages, requiring passage-level methods to perform fine-grained semantic alignment and more involved legal rea- soning. Figure 2 illustrates an example citation context and its corresponding target passage from the legal datasetāLePaRD [21]. 2.4 Contrastive alignment and token grounding. To integrate non-linguistic data into language models, the primary challenge is bridging the semantic gap between human vocabulary and specialized machine symbols. Contrastive alignment has been 3 widely used to ground heterogeneous representations into shared embedding spaces, most prominently in multimodal models such as CLIP [25]. Related techniques have also been explored for grounding new tokens or symbols into pretrained models [4,31]. Our work differs fundamentally in scope and purpose: rather than maintaining separate representational pathways or learning auxiliary embeddings, we use contrastive grounding as a mechanism to elevate machine-native discrete representations to first-class language tokens. This allows them to be generated, composed, and jointly modeled over within a unified sequence modeling framework. 3 UniLang framework The core philosophy of UniLang is that natural language and machine-native representations are complementary representations of the same underlying entities. Natural language provides rich, human-interpretable context, while machine-native representations encode compact semantic struc- tures optimized for computation. Unlike existing approaches that force a choice between these modalities, UniLang treats them as heterogeneous fields within a unified representation and generation space. By enabling joint computation over both types of information, UniLang combines the pretrained linguistic knowledge of LLMs with the compact semantic structure encoded by machine-native symbols. 3.1 Constructing machine-native representations UniLang is agnostic to the specific mechanism used to generate machine-native representations, requiring only that they consist of discrete, identifiable tokens that can be incorporated into the model vocabulary. In this work, following prior literature on Semantic IDs and residual vector quantization [26,36], we adopt an RQ-VAEābased discretization pipeline as a concrete instantiation. Many items of interest admit partial textual descriptions. For example, a movie can be characterized by its title, release year, and genres, while a legal case is naturally described by its textual context, potentially augmented with structured metadata such as court name or case date. A pretrained text encoder maps these descriptions into high-dimensional embeddings, which an RQ-VAE discretizes into a sequence of codes acrosslquantization levels. Denoting the codebook size at levelibyc i , the resulting representation isq = (q (1) ,...,q (l) ), q (i) ā 0,...,c i ā 1.Following common practice, we setl = 3andc i = 256for all levels. To mitigate potential code collisions, we append an additional disambiguation levelq (l+1) , yielding(q (1) ,...,q (l) ,q (l+1) ). For example,(11, 43, 204, 0) and(11, 43, 204, 1)denote two distinct items sharing the same base quantized code. To integrate these discrete codes into the LLM vocabulary, we prepend level-specific prefixes to ensure token- level distinguishability, forming Semantic IDs (SIDs) such as (A11, B43, C204, D0). The resulting machine-native vocabulary contains4Ć 256 = 1,024tokens, which we call machine tokens, capable of representing up to256 4 ā 4.3billion distinct items. Figure 1 illustrates how items are jointly represented using SIDs and natural-language tokens. 3.2 Vocabulary expansion and semantic grounding To facilitate seamless modeling of non-linguistic symbols, UniLang establishes a shared vocabulary across modalities. Although pretrained LLMs are optimized for natural language, we extend their latent space with a dedicated set of machine tokens. Machine tokens are grounded by aligning their embeddings with the textual description embeddings of corresponding items using contrastive learning. Given a pretrained LLM with natural-language vocabularyV NL , we extend its vocabulary with a fixed set of machine tokensV ML (1,024 tokens in our instantiation), each assigned a learnable embedding initialized from a zero-mean normal distribution with standard deviation equal to the modelās initializer range. Letm i = (m 1 i ,m 2 i ,...,m L i )āV L ML denote the machine token sequence representing itemi, where Lis fixed (e.g.L = 4in our setting). Lett i = (w 1 i ,w 2 i ,...,w T i i )āV T i NL denote the corresponding textual description. Both sequences are encoded by the same pretrained LLM f Īø : z ML i = f Īø (m i ),z NL i = f Īø (t i ) 4 The grounding objective encourages the machine token sequence embeddingz ML i to align with its corresponding textual description embeddingz NL i , while remaining distinct from embeddings of other items. Given a batch of N items, we employ an InfoNCE-style contrastive loss [34]: L i =ā log exp sim(z ML i ,z NL i )/Ļ P N j=1 exp sim(z ML i ,z NL j )/Ļ (1) wheresim(u,v) = u T v ā„uā„vā„ denotes cosine similarity andĻis a temperature hyperparameter. To promote bidirectional alignment, we also compute the symmetric loss: L sym i =ā log exp sim(z NL i ,z ML i )/Ļ P N j=1 exp sim(z NL i ,z ML j )/Ļ (2) The final grounding objective is given byL = 1 2N P N i=1 (L i +L sym i ). During grounding, only machine token embeddings are updated; all LLM parameters and natural- language token embeddings remain fixed. Because textual description embeddings are constant,z NL j can be precomputed and reused across training batches, reducing computation. Textual description embeddings are represented using the final hidden state of the sequence, while machine token embeddings are formed via mean pooling over token-level hidden states to ensure equal contribution from each quantization level. 3.3 Unified autoregressive modeling After grounding machine tokens, we formulate all downstream tasks as sequence-to-sequence genera- tion over heterogeneous token types. Natural-language and machine tokens share a single extended vo- cabularyV =V NL āŖV ML and are modeled autoregressively: for an input sequencex = (x 1 ,...,x T ) with x t āV , the model defines p Īø (y|x) = ā„yā„ Y t=1 p Īø (y t |x,y <t ), y t āV No distinction is made between token classes at the modeling level. All tokens share the same embedding table, self-attention layers, and output head, allowing natural-language and machine tokens to interact directly. The framework is task-agnostic. Different downstream tasks are defined by inputāoutput sequence constructions, rather than by changes to the model architecture or training objective. 3.4 Task formulations To operationalize this framework, we represent tasks as structured hybrid sequences that interleave natural-language tokens and machine-native symbols under a consistent template. Inputs follow a fixed formatting schema that organizes heterogeneous fields in a stable order, while outputs are divided into typed segments using delimiter tokens (e.g.,<year>,<genre>,<sid>). These output tags define semantic prediction fields and guide autoregressive generation. Sequential recommendation. We formulate next-item prediction as typed autoregressive generation over heterogeneous fields. A userās interaction history is encoded as a structured sequence of SID, year, genre, where the SID is a machine-native identifier and the remaining attributes are natural- language tokens. Rather than directly predicting a single item ID, the model generates the next item in a field-wise manner: it first produces<year>and<genre>segments, followed by<sid>, the machine-native identifier (Figure 3a). This structured design discourages degenerate solutions that rely solely on machine-token pattern continuation. Instead, it requires the model to align symbolic sequence prediction with natural- language semantics, coupling structural precision with semantic grounding. Legal precedent prediction. We apply the same typed generative formulation to legal citation modeling. Citation contexts and quoted passages, together with associated metadata, are represented as structured hybrid sequences. Each text segment is encoded and quantized into a machine-native SID, while metadata fields remain in natural-language form. The input prompt interleaves the context 5 (a) Sequential prediction (b) Legal precedent prediction Figure 3: Examples of unified structured prompting across structurally distinct tasks. Each task uses task- specific metadata and a tailored combination of natural-language and machine tokens, yet all follow the same autoregressive SFT training procedure. SID with its metadata, and the target sequence consists of the cited passageās metadata followed by its SID (Figure 3b). This formulation casts citation prediction as structured autoregressive generation over heterogeneous token types. It mirrors the recommendation setting, though it operates over a structurally distinct prediction problem. 4 Experiment 4.1 Datasets We evaluate UniLang on six real-world benchmarks covering two structurally distinct domains: sequential recommendation and legal precedent prediction. For sequential recommendation, we use the "Beauty" subset of the Amazon Product Reviews dataset [23], along with the MovieLens [6] 1M (ML-1m) and 20M (ML-20m) datasets, which differ substantially in size. For legal precedent prediction, we use LePaRD [21], the largest public dataset for legal case retrieval and prediction, which provides three dataset splits of different sizes. Detailed dataset statistics are provided in Appendix A. 4.2 Evaluation metrics and data splits We evaluate model performance using Recall@kand NDCG@k. For sequential recommendation, we report results atk ā 5, 10, while for legal precedent prediction we usek ā 1, 10to emphasize the importance of top-1 accuracy in legal retrieval tasks. For sequential recommendation, we follow the standard leave-one-out protocol, where the last interaction of each user is used for testing, the second-to-last for validation, and the remaining interactions for training. In legal precedent prediction, we use a 90%/5%/5% split for training, validation, and testing, respectively. To enable efficient hyperparameter tuning, we perform model selection on a random subset of the validation set, while all reported test results are computed on the full test set (see Appendix B for details). 4.3 Machine-native representation generation details For the Amazon Beauty dataset, we concatenate the raw text from the "title", "categories", "descrip- tion", "brand", and "price" fields to form a single textual description. For the MovieLens datasets, we combine the "title" with the "genre" field. For the LePaRD dataset, the case context and quoted 6 precedent passages are directly used as provided. Each resulting text blob is encoded using the open-sourcesentence-t5 2 [24] model to obtain a 768-dimensional dense embedding. We then train an RQ-VAE to quantize these embeddings into discrete machine-native codes (SIDs). Detailed training hyperparameters are provided in Appendix C. 4.4 Machine token grounding implementation details We useLlama-3.2-1B-Instruct 3 as the base model and extend its vocabulary with 1,024 machine- native tokens. The embeddings of these tokens are trained using the contrastive objective described in Section 3.2 (see Table 9 in Appendix for hyperparameters). Grounding quality is monitored via an alignment metric, computed as the cosine similarity between a machine-token sequence and its corresponding textual embedding. Higher values indicate stronger semantic correspondence. Early stopping is applied based on this metric. 4.5 LLM fine-tuning details After grounding the machine-native tokens and integrating them into the pretrainedLlama-3.2- 1B-Instructmodel, we perform downstream adaptation. During this stage, all original model parametersāincluding the pretrained token embeddings and the grounded machine-token embeddingsā are frozen. We train task-specific Low-Rank Adaptation (LoRA) adapters [10] using supervised fine-tuning (SFT). For sequential recommendation, the maximum user history length is set to 30 for MovieLens and 50 for Amazon Beauty. For all tasks, LoRA is applied to the projection layers in each transformer block, namelyq_proj, k_proj,v_proj,o_proj,gate_proj,up_proj, anddown_proj. The inputāoutput formatting used for SFT, along with an example prompt and target sequence for MovieLens, is illustrated in Figure 3a. The corresponding formats for LePaRD and Amazon Beauty are shown in Figures 3b and 5, respectively. Complete fine-tuning hyperparameters can be found in Table 10 in the Appendix. Table 1: Performance comparison on sequential recommendation task. DatasetMetric DiscriminativeGenerative Improvement SASRecBERT4RecS 3 -RecP5 TIGER (re-impl.) UniLang (ours) Beauty Recall@50.0387 0.02030.03870.01630.03610.04198.27% NDCG@50.02490.01240.02440.01070.02410.029920.08% Recall@100.06050.03470.06470.02540.05950.0603-6.80% NDCG@100.03180.01700.03270.01360.03180.03589.48% ML-1m Recall@50.12730.10110.12580.10980.06340.166130.48% NDCG@50.08430.06490.08240.07340.04120.114535.82% Recall@100.20130.15330.20230.15750.10550.237417.35% NDCG@100.1083 0.08100.10700.08880.05470.137627.05% ML-20m Recall@50.08890.06160.0884N/A0.07630.1911114.96% NDCG@50.0549 0.03450.0531N/A0.05230.1382151.73% Recall@100.14530.09480.1493N/A0.11370.259773.95% NDCG@100.07280.04760.0802N/A0.06430.160399.88% Note: The best results are shown in bold, and the second-best results are underlined. 4.6 Performance on sequential recommendation We first evaluate UniLang on the sequential recommendation task, comparing it against strong task-specific baselines. BERT4Rec [30], SASRec [14], and S 3 -Rec [40] are discriminative models specifically designed for sequential recommendation. P5 [5] is a generative approach that verbal- izes user, item, and interaction information into natural language, whereas TIGER [26] is also generative but operates purely on machine-native representations (SIDs), predicting the next SID autoregressively. 2 https://huggingface.co/sentence-transformers/sentence-t5-base 3 https://huggingface.co/meta-llama/Llama-3.2-1B-Instruct 7 Table 2: Run-to-run variability on MovieLens-20M (meanĀ± SE). metricmeanĀ± standard error Recall@50.1908Ā± 0.00016 NDCG@50.1378Ā± 0.00017 Recall@100.2596Ā± 0.00014 NDCG@100.1600Ā± 0.00014 For all baselines except TIGER and P5, we report results from the original papers or produce them using the original authorsā implementations. For TIGER, we follow the paper and implement the model accordingly. For P5, we adopt corrected and extended results from subsequent works where applicable. Details are provided in Appendix D. As shown in Table 1, UniLang achieves remarkable performance, outperforming nearly all baselines across benchmarks with up to 151% improvement in NDCG@5 and 115% in Recall@5 on MovieLens- 20M. These results demonstrate the effectiveness of extending pretrained LLMs with machine-native symbols: UniLang can directly model and generate structured representations, enabling it to surpass specialized sequential recommendation architectures. To assess statistical stability, we perform three independent runs with different random seeds on the MovieLens-20M dataset and report mean Ā± standard error in Table 2. Table 3: Performance comparison on legal precedent prediction task. DatasetMetric RetrievalClassificationGenerative Improvement BM25 fine-tuned SBERT LEGAL-BERTDistilBERT UniLang (ours) 10K Recall@10.05010.08990.16660.19670.293849.36% Recall@100.19520.44790.47650.59120.682315.41% NDCG@100.11370.26270.30750.37730.477526.56% 20K Recall@10.04130.07530.12800.16740.248648.51% Recall@100.16670.38500.36840.52160.629020.59% NDCG@100.09560.20720.23770.3291 0.426429.57% 50K Recall@10.03410.05000.08770.12310.167636.15% Recall@100.13530.25900.25220.39340.488224.10% NDCG@100.07790.13780.16250.24570.313127.43% Note: The best results are shown in bold, and the second-best results are underlined. 4.7 Performance on legal precedent prediction To further demonstrate its generality, we apply the UniLang frameworkāwithout any architectural modificationāto the structurally different task of legal precedent prediction. Unlike sequential recommendation, which involves modeling chronological sequences of user interactions, legal precedent prediction requires identifying the relevant legal passage from a single citation context, providing a stringent test of UniLangās ability to generalize across fundamentally different problem structures. We compare UniLang against established retrieval and discriminative baselines. BM25 and fine-tuned SBERT are retrieval-based methods, whereas LEGAL-BERT and DistilBERT approach passage retrieval as a text classification task. All baseline results are taken from the LePaRD paper [21]. Further details on these baselines are provided in Appendix E. Comparison results are provided in Table 3. Despite the differences in task structure between se- quential recommendation and legal passage retrieval, UniLang consistently outperforms all baselines by a substantial margin. In particular, it achieves a 49% improvement in Recall@1 on the 10k dataset, significantly surpassing even specialized discriminative models. These results underscore UniLangās capability to provide a unified modeling framework across tasks with fundamentally different structures. 8 Figure 4: Ablation test on MovieLens-20m. 4.8 Ablation study We conduct ablation studies on MovieLens-20M to isolate the contributions of semantic grounding, natural-language fields, and structural delimiters. For efficiency, all ablations are evaluated on the fixed validation subset. We consider three variants: (1) NLRemoved, which trains exclusively on machine-native tokens; (2) NoTypeDelim, which removes typed output delimiters (e.g.,<year>, </dname>); and (3) NoWarmup, which uses randomly initialized symbolic embeddings instead of those aligned via contrastive learning. As shown in Figure 4, NLRemoved exhibits rapid initial improvement followed by performance degradation after 40k steps, suggesting early overfitting when semantic signals from natural language are absent. NoTypeDelim remains stable but consistently underperforms the full UniLang model, indicating that explicit input and output structuring facilitates optimization and improves prediction quality. Notably, NoWarmup fails to achieve non-zero metrics (NDCG and Recall), demonstrating that pre-aligning machine-native symbols to the LLMās latent space is essential for effective generative learning in our setting. We omit this variant from Figure 4 as it remains at the zero-baseline throughout training. Overall, these results suggest that natural language provides regularizing context, while grounded embeddings and typed structures are critical for stability and prediction quality. Figure 5: Prompt template and example for the Amazon Beauty dataset. 4.9 Qualitative comparison So far, our results demonstrate the quantitative benefits of jointly representing natural language and machine-native tokens. We now provide a qualitative comparison with approaches that verbalize all user and item information into natural language, such as P5 [35]. P5 typically requires multiple handcrafted prompts per dataset per task (e.g., 13 prompts for Beauty sequential recommendation). One example prompt is shown below: Input template: I find the purchase history list of user user_id: history item list of item_id. I wonder which is the next item to recommend to the user? Target template: item [item_id] In contrast, UniLang uses a single structured prompt that seamlessly combines natural language with machine tokens, as shown in Figure 5. This unified representation reduces the need for extensive prompt engineering while maintaining or improving performance. By treating language and struc- 9 tured symbols as complementary generative units, UniLang provides a scalable and generalizable framework for heterogeneous tasks, enabling models to jointly model both token types. 4.10 Emergent representational capabilities Beyond task performance, UniLang exhibits an important representational advantage. Existing generative recommender systems struggle with user identity modeling. For example, TIGER [26] represents users by hashing raw user IDs into a fixed set of 2,000 ID tokens. As a result, these user ID tokens occupy a large portion of the modelās extended vocabulary (66%), which can dominate the token space and limit scalability as the number of users grows. In contrast, UniLang requires no dedicated user tokens or hashing schemes. User identifiers (e.g., "b_1009") can be directly represented in natural-language form and processed by the pretrained tokenizer. Although not the primary objective of our method, this property highlights UniLangās representational scalability and reinforces its role as a unified modeling framework rather than a task-engineered solution. 5 Conclusion and future work We introduced UniLang, a unified framework that extends pretrained LLMs to model natural lan- guage and machine-native symbols within a single autoregressive objective. By treating structured symbols as first-class generative units, UniLang enables structurally diverse tasksāsuch as sequential recommendation and legal precedent predictionāto be addressed without task-specific architectures. Empirically, UniLang consistently outperforms strong baselines, while ablations demonstrate the importance of natural-language context and typed structure. Overall, UniLang provides a general interface for extending pretrained LLMs beyond language to model heterogeneous machine-native representations. While UniLang demonstrates strong performance on structured prediction tasks, we did not evaluate the impact of symbolic fine-tuning on the modelās natural language generation quality. Our focus in this work is on accuracy in predictive tasks, and studying potential effects on language fluency and coherence is left for future work. 10 Table 4: Statistics of the sequential recommendation datasets. Dataset#users#items#actionsAvg. lengthDensity Beauty40,22654,5420.35m8.80.02% ML-1m6,0403,4161m163.54.79% ML-20m138,49326,74420m144.40.54% Table 5: Statistics of the legal precedent prediction datasets. Dataset (# cited passages) # Citing Passage Surface Forms # Cited Passage Surface Forms 10k1,294,730798,558 20k1,586,6631,140,058 50k2,079,7041,821,751 A Dataset ā¢ Amazon Beauty 4 [23]: This dataset consists of product reviews collected from Amazon.com. The full collection is organized by top-level product categories, and in our experiments, we use the subset corresponding to the "Beauty" category. ā¢ MovieLens: A widely used benchmark for evaluating recommendation systems. In this study, we use two standard versions: MovieLens 1M (ML-1m) 5 and MovieLens 20M (ML-20m) 6 . ā¢The LePaRD dataset 7 [21] is a large-scale collection of 4.3 million U.S. federal judicial citations. It leverages judgesā quotation behavior as supervision, pairing precedential quotations with their corresponding citation contexts. The dataset provides three subsets that differ in the number of top-nmost frequently cited passages included. Due to unavoidable preprocessing noiseāsuch as OCR errors, sentence segmentation inconsistencies, and metadata formatting variationsāthe same citing context or cited quotation may appear in multiple surface forms. Table 4 presents the statistics of the sequential recommendation datasets. Table 5 provides an overview of the legal precedent prediction dataset. Table 6 summarizes the statistics of the textual features in the legal dataset. B Validation set sampling We use Recall@kon the validation set for early stopping during training. Since our framework is generative, we perform beam search to deterministically generate the top-kresults and compute Recall@kaccordingly. Running beam search over the full validation set is computationally expensive, so to expedite model selection, we instead use a fixed, randomly sampled subset of the validation set generated with a fixed random seed. For all datasets, reported results are based on models selected using this sampled subset. Table 7 lists the sample sizes for each dataset. For all experiments, we use a beam size of 20. C Model training hyperparameters All experiments were conducted on 8 NVIDIA H100 GPUs (80GB each). Training can also be performed on a single GPU; multiple GPUs are used primarily to accelerate training via data parallelism. Training time varies depending on batch composition. On the MovieLens-20M dataset, 4 https://cseweb.ucsd.edu/~jmcauley/datasets/amazon/links.html 5 https://grouplens.org/datasets/movielens/1m/ 6 https://grouplens.org/datasets/movielens/20m/ 7 https://huggingface.co/datasets/rmahari/LePaRD 11 Table 6: Summary statistics of legal precedent dataset text features FeatureMeanStdMinMax Length of cited text (chars)3062252418,342 Length of citing context (chars)562216514,062 Table 7: Validation subset sizes for model selection DatasetValidation sizeSample size Beauty40,2265,000 MovieLens-1m6,0401,000 MovieLens-20m138,4931,000 10k103,8121,000 20k134,7371,000 50k190,0511,000 under our configuration, training proceeds at approximately 0.15 seconds per optimization step (batch size 128), with total runtime scaling linearly with the number of training steps. (a) Beauty(b) LePaRD 20k Figure 6: RQ-VAE training progress on different datasets. C.1 RQ-VAE training parameters The same RQ-VAE architecture is used across all datasets. The model takes 768-dimensional text embeddings as input. The encoder comprises three fully connected layers with output dimensions 512, 256, and 128, each followed by ReLU activation, and outputs a 16-dimensional latent representation. The decoder mirrors the encoder with layers of dimensions 128, 256, and 512. At each quantization level, we maintain a codebook of size 256. Table 8 summarizes the detailed hyperparameter settings, and Figure 6 illustrates the training dynamics of the individual loss components across datasets. C.2 Machine token grounding parameters Table 9 summarizes the hyperparameters used for training the embeddings of the machine-native tokens. C.3 Supervised fine tuning parameters Table 10 lists the hyperparameters used for supervised fine-tuning ofLlama-3.2-1B-Instruct on structured inputāoutput sequences containing interleaved natural-language and machine-native tokens. 12 Table 8: RQ-VAE hyperparameters. HyperparameterBeauty ML-1M ML-20M LePaRD 10k / 20k / 50k Total training steps30k20k20k Warm up3k2k3k Peak learning rate1e-31e-31e-3 End learning rate1e-55e-51e-5 Ema decay0.990.990.99 Commitment cost1.51.50.1 Note: All experiments use a codebook embedding size of 16 and a batch size of 2048, and are optimized with AdamW (weight decay 0.055, Ī² 1 = 0.9, Ī² 2 = 0.98) under a cosine learning rate schedule. Table 9: Machine token alignment hyperparameters. DatasetBeauty ML-1M ML-20M LePaRD 10k / 20k / 50k Batch size1024512128 Learning rate1eā31eā36eā3 Total steps4k4k8k Note: All experiments use the AdamW optimizer (weight decay 10 ā2 , Ī² 1 = 0.9, Ī² 2 = 0.98) with a cosine learning rate schedule and 400 warmup steps. The temperature for the contrastive loss is set to 0.2. D Sequential recommendation baseline methods ā¢SASRec [14] is a self-attention-based sequential recommendation model that efficiently captures long-term user behavior. ā¢BERT4Rec [30] trains a bidirectional Transformer with a masked item prediction objective to learn contextualized representations of user behavior sequences. ā¢S 3 -Rec [40] adopts a self-supervised pretraining strategy to capture correlations among items, attributes, and interaction sequences, alleviating data sparsity. ā¢P5 [5] formulates recommendation as a language modeling problem, representing userāitem interactions, user profiles, item metadata, and reviews in natural language. ā¢TIGER [26] represents items using semantic IDs (SIDs), converting user interaction histories into SID sequences. A generative model is trained from scratch to autoregressively predict the next SID for sequential recommendation. In evaluating recommendation systems, two protocols are commonly adopted: (1) sampled evaluation, where the ground-truth item is ranked against a fixed set of negative samples (e.g., 99 items), and (2) full-ranking, where the ground-truth item is ranked against all items in the candidate set. While sampled evaluation is computationally efficient, it can lead to overly optimistic performance estimates. In this work, we adopt full-ranking evaluation throughout to provide a more rigorous and realistic assessment of model performance. For the Amazon Beauty dataset, results for SASRec, BERT4Rec, and S 3 -Rec are taken from the publicly released benchmarks 8 provided by the S 3 -Rec authors. The P5 results are obtained from the TIGER paper [26], where the authors introduced a preprocessing modification to ensure a fair comparison. For the MovieLens 1M and 20M datasets, results for SASRec, BERT4Rec, and S 3 -Rec are obtained using the authorsā released code. All models are trained with their default hyperparameter settings. For SASRec and BERT4Rec, we additionally implement full-ranking evaluation to obtain the reported results. Results for P5 on the MovieLens 1M dataset are taken from the OpenP5 [35] paper. OpenP5 incorporates the original P5 backbone model, downstream tasks, and item indexing method. Since P5 8 https://github.com/RUCAIBox/CIKM2020-S3Rec 13 Table 10: SFT hyperparameters. DatasetBeauty ML-1M ML-20M LePaRD 10k / 20k / 50k Batch size832/128512 Learning rate1eā42eā42eā4 LoRA dropout0.250.050.25 Total training steps30k20k30k Warm up2k2k2k Note: All experiments use LoRA with rank 32 and Ī± = 2. Optimization is performed using AdamW (weight decay 0.005, Ī² 1 = 0.9, Ī² 2 = 0.98) with a cosine learning rate schedule. requires substantial manual prompt engineering when adapted to new datasets, we consider using OpenP5ās reported results to provide a fairer comparison. Due to the prohibitively high storage and computational cost of applying P5 to the MovieLens 20M dataset, we do not report its results on this dataset (denoted as N/A in Table 1). E Legal precedent prediction baseline methods We consider four baseline methods, two retrieval-based and two classification-based. BM25 represents a sparse lexical retrieval approach [28]. The "fine-tuned SBERT" baseline is a dense embedding-based retrieval method that uses a fine-tuned SBERT [27] model to generate embeddings, followed by maximum dot-product similarity for retrieval. Passage retrieval can also be formulated as a text classification task, where each target passage is assigned a unique label that serves as the prediction target for its preceding context. Two classification- based baselines are considered: DistilBERT [29] and LEGAL-BERT [1], the latter being a domain- adapted BERT model trained on a large corpus of legal documents. All baseline results are taken from the LePaRD paper [21]. 14 References [1] Ilias Chalkidis, Manos Fergadiotis, Prodromos Malakasiotis, Nikolaos Aletras, and Ion Androut- sopoulos. Legal-bert: The muppets straight out of law school. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 2898ā2904, Online, 2020. Association for Computational Linguistics. [2]Faraz Dadgostari, Mauricio Guim, Peter A. Beling, Michael A. Livermore, and Daniel N. Rockmore. Modeling law search as prediction. Artificial Intelligence and Law, 29(1):3ā34, 2021. [3]Gabriel de Souza Pereira Moreira, Sara Rabhi, Jeong Min Lee, Ronay Ak, and Even Oldridge. Transformers4rec: Bridging the gap between nlp and sequential/session-based recommendation. In Proceedings of the Fifteenth ACM Conference on Recommender Systems (RecSys 2021), pages 143ā153, 2021. [4]Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit H. Bermano, Gal Chechik, and Daniel Cohen-Or. An image is worth one word: Personalizing text-to-image generation using textual inversion, 2022. [5]Shijie Geng, Shuchang Liu, Zuohui Fu, Yingqiang Ge, and Yongfeng Zhang. Recommendation as language processing (rlp): A unified pretrain, personalized prompt & predict paradigm (p5). In RecSys 2022 - Proceedings of the 16th ACM Conference on Recommender Systems, RecSys 2022 - Proceedings of the 16th ACM Conference on Recommender Systems, pages 299ā315. Association for Computing Machinery, Inc, September 2022. Publisher Copyright: Ā© 2022 ACM.; 16th ACM Conference on Recommender Systems, RecSys 2022 ; Conference date: 18-09-2022 Through 23-09-2022. [6] F. Maxwell Harper and Joseph A. Konstan. The movielens datasets: History and context. ACM Transactions on Interactive Intelligent Systems, 5(4):19:1ā19:19, Dec 2015. [7] BalĆ”zs Hidasi, Alexandros Karatzoglou, Linas Baltrunas, and Domonkos Tikk. Session-based recommendations with recurrent neural networks, 2015. cite arxiv:1511.06939Comment: Camera ready version (17th February, 2016) Affiliation update (29th March, 2016). [8] J. A. Hirsch, G. Nicola, G. McGinty, R. W. Liu, R. M. Barr, M. D. Chittle, and L. Manchikanti. Icd-10: History and context. American Journal of Neuroradiology, 37(4):596ā599, 2016. [9] Yupeng Hou, Zhankui He, Julian McAuley, and Wayne Xin Zhao. Learning vector-quantized item representation for transferable sequential recommenders. In Proceedings of The Web Conference 2023, pages 2808ā2818, 2023. [10]Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models, 2021. [11]Zihan Huang, Charles Low, Mengqiu Teng, Hongyi Zhang, Daniel E. Ho, Mark S. Krass, and Matthias Grabmair. Context-aware legal citation recommendation using deep learning. In Proceedings of the Eighteenth International Conference on Artificial Intelligence and Law, ICAIL ā21, pages 79ā88, New York, NY, USA, 2021. Association for Computing Machinery. [12]HervĆ© JĆ©gou, Matthijs Douze, and Cordelia Schmid. Product quantization for nearest neighbor search. IEEE Transactions on Pattern Analysis and Machine Intelligence, 33(1):117ā128, 2011. [13]Liangyi Jiang, Xiaocong Liu, Nejatian Nasir-Moin, Hongyi Wang, Abdullah Abidin, Kai Eaton, . . . , and [additional authors]. Health system-scale language models are all-purpose prediction engines. Nature, 619:357ā362, 2023. [14] Wang-Cheng Kang and Julian McAuley. Self-attentive sequential recommendation, 2018. [15] Wang-Cheng Kang and Julian McAuley. Self-attentive sequential recommendation. In Pro- ceedings of the 2018 IEEE International Conference on Data Mining (ICDM), pages 197ā206, 2018. 15 [16]Xiaoyu Kong, Junguang Jiang, Bin Liu, Ziru Xu, Han Zhu, Jian Xu, Bo Zheng, Jiancan Wu, and Xiang Wang. Think before recommendation: Autonomous reasoning-enhanced recommender. In NeurIPS 2025 Poster Session, San Diego, 2025. [17] Jing Li, Pengjie Ren, Zhumin Chen, Zhaochun Ren, Tao Lian, and Jun Ma. Neural attentive session-based recommendation. In Proceedings of the 2017 ACM on Conference on Information and Knowledge Management (CIKM 2017), pages 1419ā1428, 2017. [18]Fangyuan Luo, Yankai Chen, Jun Wu, Tong Li, Philip S. Yu, and Xue Liu. Learning to hash for recommendation: A survey, 2025. [19]Yanchen Luo, Junfeng Fang, Sihang Li, Zhiyuan Liu, Jiancan Wu, An Zhang, Wenjie Du, and Xiang Wang. Text-guided small molecule generation via diffusion model. iScience, 27(11):110992, 2024. [20]Yuankai Luo, Hongkang Li, Qijiong Liu, Lei Shi, and Xiao-Ming Wu. Node identifiers: Compact, discrete representations for efficient graph learning. In The Thirteenth International Conference on Learning Representations, 2025. [21] Robert Mahari, Dominik Stammbach, Elliott Ash, and Alex Pentland. Lepard: A large-scale dataset of judicial citations to precedent. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 9863ā9877, Bangkok, Thailand, 2024. Association for Computational Linguistics. [22]Robert Zev Mahari. Autolaw: Augmented legal reasoning through legal precedent prediction, 2021. [23]Julian McAuley, Christopher Targett, Qinfeng Shi, and Anton van den Hengel. Image-based recommendations on styles and substitutes. In Proceedings of the 38th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 43ā52, New York, NY, USA, 2015. ACM. [24]Jianmo Ni, Gustavo Hernandez Abrego, Noah Constant, Ji Ma, Keith Hall, Daniel Cer, and Yinfei Yang. Sentence-t5: Scalable sentence encoders from pre-trained text-to-text models. In Findings of the Association for Computational Linguistics: ACL 2022, pages 1864ā1874, Dublin, Ireland, 2022. Association for Computational Linguistics. [25] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. CoRR, abs/2103.00020, 2021. [26] Shashank Rajput, Nikhil Mehta, Anima Singh, Raghunandan Hulikal Keshavan, Trung Vu, Lukasz Heldt, Lichan Hong, Yi Tay, Vinh Tran, Jonah Samost, Maciej Kula, Ed Chi, and Maheswaran Sathiamoorthy. Recommender systems with generative retrieval. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Information Processing Systems, volume 36, pages 10299ā10315. Curran Associates, Inc., 2023. [27]Nils Reimers and Iryna Gurevych. Sentencebert: Sentence embeddings using siamese bert networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing (EMNLP). Association for Computational Linguistics, 2019. [28]Stephen E. Robertson, Steve Walker, Susan Jones, Micheline Hancock-Beaulieu, and Mike Gatford. Okapi at TREC-3. In Proceedings of the Third Text REtrieval Conference (TREC 1994), Gaithersburg, USA, November 1994. [29]Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, a distilled version of bert: Smaller, faster, cheaper and lighter. In Proceedings of the 5th Workshop on Energy Efficient Machine Learning and Cognitive Computing (EMNLP 2020 Workshop), 2020. 16 [30]Fei Sun, Jun Liu, Jian Wu, Changhua Pei, Xiao Lin, Wenwu Ou, and Peng Jiang. Bert4rec: Sequential recommendation with bidirectional encoder representations from transformer. In Proceedings of the 28th ACM International Conference on Information and Knowledge Man- agement (CIKM 2019), pages 1441ā1450, 2019. [31] Hao Tan and Mohit Bansal. Vokenization: Improving language understanding with contextual- ized, visual-grounded supervision. CoRR, abs/2010.06775, 2020. [32]Jiaxi Tang and Ke Wang. Personalized top-n sequential recommendation via convolutional sequence embedding. In Proceedings of the Eleventh ACM International Conference on Web Search and Data Mining (WSDM 2018), pages 565ā573, 2018. [33] Yi Tay, Vinh Q. Tran, Mostafa Dehghani, Jianmo Ni, Dara Bahri, Harsh Mehta, Zhen Qin, Kai Hui, Zhe Zhao, Jai Gupta, Tal Schuster, William W. Cohen, and Donald Metzler. Transformer memory as a differentiable search index, 2022. [34] AƤron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. CoRR, abs/1807.03748, 2018. [35]Shuyuan Xu, Wenyue Hua, and Yongfeng Zhang. Openp5: An open-source platform for developing, training, and evaluating llm-based recommender systems. SIGIR, 2024. [36]Liu Yang, Fabian Paischer, Kaveh Hassani, Jiacheng Li, Shuai Shao, Zhang Gabriel Li, Yun He, Xue Feng, Nima Noorshams, Sem Park, Bo Long, Robert D. Nowak, Xiaoli Gao, and Hamid Eghbalzadeh. Unifying generative and dense retrieval for sequential recommendation. Trans. Mach. Learn. Res., 2025, 2025. [37]Neil Zeghidour, Alejandro Luebs, Ahmed Omran, Jan Skoglund, and Marco Tagliasacchi. Soundstream: An end-to-end neural audio codec. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 30:495ā507, 2021. [38]Kepu Zhang, Weijie Yu, Sunhao Dai, and Jun Xu. Citalaw: Enhancing llm with citations in legal domain, 2024. [39]Jianan Zhao, Le Zhuo, Yikang Shen, Meng Qu, Kai Liu, Michael Bronstein, Zhaocheng Zhu, and Jian Tang. Graphtext: Graph reasoning in text space. In Proceedings of the Thirty-Eighth Conference on Neural Information Processing Systems (NeurIPS 2024), 2024. Poster in the workshop *Adaptive Foundation Models: Evolving AI for Personalized and Efficient Learning*. [40]Kun Zhou, Hui Wang, Wayne Xin Zhao, Yutao Zhu, Sirui Wang, Fuzheng Zhang, Zhongyuan Wang, and Ji-Rong Wen. S3-rec: Self-supervised learning for sequential recommendation with mutual information maximization. In Proceedings of the 29th ACM International Conference on Information and Knowledge Management (CIKM 2020), pages 1893ā1902, 2020. 17