Paper deep dive

Self Knowledge Re-expression: A Fully Local Method for Adapting LLMs to Tasks Using Intrinsic Knowledge

Mengyu Wang, Xiaoying Zhi, Zhiyi Li, Robin Schmucker, Shay B. Cohen, Tiejun Ma, Fran Silavong

Year: 2026Venue: arXiv preprintArea: cs.CLType: PreprintEmbeddings: 88

Intelligence

Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 96%

Last extracted: 6/21/2026, 6:02:28 AM

Summary

The paper introduces Self-Knowledge Re-expression (SKR), a novel, task-agnostic adaptation method for Large Language Models (LLMs). The authors argue that the performance bottleneck in non-generative tasks (like information retrieval, object detection, and anomaly detection) stems from the Next-Token Prediction (NTP) paradigm rather than a lack of knowledge. SKR is a two-step, fully local process: first, 'Self-Annotation' uses the LLM's existing generative capability to create training labels from unannotated data; second, 'Knowledge Re-expression' fine-tunes the model to transition from generic token generation to a task-specific output mechanism (e.g., embeddings, regression heads, or classification logits). Experiments on financial datasets and the MMDocRAG dataset show significant improvements in Recall@1, latency reduction, and AUPRC compared to standard LLM outputs and leading retrieval models.

Entities (7)

Anomaly Detection · task · 100%Information Retrieval · task · 100%LoRA · technique · 100%MMDocRAG · dataset · 100%Next-Token Prediction · paradigm · 100%Object Detection · task · 100%Self-Knowledge Re-expression · method · 100%

Relation Signals (5)

Self-Knowledge Re-expression → relatedto → LoRA

confidence 100% · We implement this process using LoRA (Hu et al., 2022).

Self-Knowledge Re-expression → addressesbottleneckof → Next-Token Prediction

confidence 95% · We propose Self-Knowledge Re-expression (SKR)... to address this, we propose Self-Knowledge Re-expression (SKR), a novel, task-agnostic adaptation method.

Self-Knowledge Re-expression → improvesperformanceon → Information Retrieval

confidence 90% · over 40% in Recall@1 for information retrieval tasks

Self-Knowledge Re-expression → improvesperformanceon → Object Detection

confidence 90% · over 76% reduction in object detection latency

Self-Knowledge Re-expression → improvesperformanceon → Anomaly Detection

confidence 90% · over 33% increase in anomaly detection AUPRC

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:While the next-token prediction (NTP) paradigm enables large language models (LLMs) to express their intrinsic knowledge, its sequential nature constrains performance on specialized, non-generative tasks. We attribute this performance bottleneck to the LLMs' knowledge expression mechanism, rather than to deficiencies in knowledge acquisition. To address this, we propose Self-Knowledge Re-expression (SKR), a novel, task-agnostic adaptation method. SKR transforms the LLM's output from generic token generation to highly efficient, task-specific expression. SKR is a fully local method that uses only unannotated data, requiring neither human supervision nor model distillation. Experiments on a large financial document dataset demonstrate substantial improvements: over 40% in Recall@1 for information retrieval tasks, over 76% reduction in object detection latency, and over 33% increase in anomaly detection AUPRC. Our results on the MMDocRAG dataset surpass those of leading retrieval models by at least 12.6%.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

87,516 characters extracted from source content.

Expand or collapse full text

Self Knowledge Re-expression: A Fully Local Method for Adapting LLMs to Tasks Using Intrinsic Knowledge Mengyu Wang * 1 2 Xiaoying Zhi 2 Zhiyi Li 2 Robin Schmucker 2 Shay B. Cohen 1 Tiejun Ma 1 Fran Silavong 2 Abstract While the next-token prediction (NTP) paradigm enables large language models (LLMs) to ex- press their intrinsic knowledge, its sequential nature constrains performance on specialized, non-generative tasks. We attribute this perfor- mance bottleneck to the LLMs’ knowledge ex- pression mechanism, rather than to deficiencies in knowledge acquisition. To address this, we propose Self-Knowledge Re-expression (SKR), a novel, task-agnostic adaptation method. SKR transforms the LLM’s output from generic token generation to highly efficient, task-specific expres- sion. SKR is a fully local method that uses only unannotated data, requiring neither human super- vision nor model distillation. Experiments on a large financial document dataset demonstrate sub- stantial improvements: over 40% in Recall@1 for information retrieval tasks, over 76% reduction in object detection latency, and over 33% increase in anomaly detection AUPRC. Our results on the MMDocRAG dataset surpass those of leading re- trieval models by at least 12.6%. 1. Introduction Large language models (LLMs) have fundamentally trans- formed natural language processing (NLP), evolving from simple token generators into vast knowledge reposito- ries (Radford et al., 2018; 2019; Brown et al., 2020). Tradi- tionally, LLM performance is viewed as being directly re- lated to the volume and quality of knowledge encapsulated within their parameters, acquired during the pre-training phase (Kaplan et al., 2020). This perspective has driven continuous efforts to scale up model size and pre-training corpora, underscoring the belief that knowledge acquisition is the primary driver of LLMs’ capability. 1 The University of Edinburgh 2 JPMorgan Chase & Co. Corre- spondence to: Mengyu Wang <mengyu.wang@ed.ac.uk>. Preprint. * Work completed during an internship at the Machine Learn- ing Center of Excellence (MLCOE), JPMorgan Chase & Co. LLM Summarization Conversation Spoken Language Knowledge Input Encoding Non-generative Tasks: Information Retrieval Object Detection Embeddings Generative Tasks: ... just doable ............ Coordinates Values Next Token Prediction Regression LLMs Knowledge Expression is Limited ... Figure 1.The inherent limitation of LLMs’ knowledge expression: Next-token prediction serves as a universal paradigm but is a suboptimal output mechanism for many non-generative tasks. However, knowledge acquisition is not the only performance bottleneck. Recent studies suggest that the intrinsic knowl- edge within LLM parameters is not merely a collection of statistical co-occurrences but is structured and highly reusable (Rothenfusser & Blando, 2025). Therefore, we argue that knowledge expression, the process of organizing latent parametric knowledge into coherent and effective task outputs, is a critical, yet under-explored, factor in model performance. The richness of the knowledge existing within current LLMs implies that a promising direction for per- formance enhancement lies in optimizing their knowledge expression rather than knowledge acquisition. Currently, the prevailing mechanism for LLMs’ knowledge expression is the next-token prediction (NTP) paradigm. While NTP facilitates the generation of fluent text, discrete, token-based strings are often ill-suited for tasks requiring non-textual outputs, such as high-quality embeddings or continuous values. In these contexts, the reliance on NTP restricts a model’s ability to articulate its internal knowledge, necessitating cumbersome post-processing to bridge the gap between linguistic outputs and task requirements (Ma et al., 2024; Lukasik et al., 2025). Consequently, as illustrated in Figure 1, while NTP suits most generative tasks, it imposes constraints on many non-generative tasks. The limitations of the NTP paradigm are manifold, primar- ily stemming from a fundamental objective mismatch. In information retrieval tasks, for instance, the ideal expression of knowledge consists of dense, high-dimensional vectors optimized for semantic proximity (Karpukhin et al., 2020). However, embeddings derived from NTP-optimized models often fail to meet these requirements (Ma et al., 2024). Fur- thermore, the sequential nature of NTP struggles with global 1 arXiv:2604.22939v1 [cs.CL] 24 Apr 2026 Self Knowledge Re-expression: A Fully Local Method for Adapting LLMs to Tasks Using Intrinsic Knowledge planning and ”lookahead” tasks (Bachmann & Nagarajan, 2024). In tasks such as object detection or numerical reason- ing, a numerical value (e.g., a coordinate) is split into mul- tiple tokens. Errors in early token predictions accumulate, compromising both numerical accuracy and structural in- tegrity. Additionally, NTP imposes further constraints, such as the absence of task-oriented probability distributions for classification and the inefficiency of post-processing purely linguistic outputs. These limitations underscore a funda- mental problem: although NTP offers a universal interface by mapping all information to a textual format, it is rarely the optimal expression for specific tasks. We propose addressing these challenges by transitioning the expression mechanism (E) from the genericE ntp to a task- specific expressionE T . Unlike standard supervised fine- tuning (SFT), which aims to refine model knowledge or in- still domain nuances through extensive annotated data (Han et al.; Zheng et al., 2024), our method uses only unannotated data and the model’s intrinsic knowledge. Therefore, this ap- proach enables highly efficient task adaptation by bypassing the requirement of high-quality external supervision. We term our method Self Knowledge Re-expression (SKR), a fully local process that unlocks the latent potential of LLMs by using their intrinsic parametric knowledge without external human or model-based supervision. This method consists of two steps, Self Annotation and Knowledge Re- expression. First, the LLM uses its generative capability (E ntp ) to create its own training labels from unannotated data. The model is then fine-tuned to map its internal states di- rectly to a task-specific output expressionE T . This process is significantly more cost-effective than SFT and mitigates data exposure risks, which is vital for sensitive sectors like finance and healthcare. In summary, our contributions are as follows: 1. Formalization of Knowledge Expression Bottleneck: We are the first to rigorously decouple knowledge expression from LLMs’ parametric knowledge. We define the expres- sion mechanismEas a trainable component and formalize the adaptation process as the transition E ntp → E T . 2. The SKR Method: We propose a novel, task-agnostic method that enables an LLM to adapt to specific tasks by fine-tuning itself to a task-suitable expressionE T without using external supervision. As a fully local process, SKR offers an economical and secure route for task adaptation. 3. Validation Across Diverse Task Types: We demonstrate the versatility of SKR across three distinct task types that reveal NTP’s limitations: information retrieval, object de- tection, and anomaly detection. Our results show that SKR- adapted models’ consistently and significantly improve per- formance, demonstrating that optimizing expression can extract substantial latent potential from LLMs. 2. Related Work The evolution of LLMs has been largely defined by the scaling hypothesis, which posits that model performance improves predictably with increases in parameter count, training data and computation (Hestness et al., 2017; Ka- plan et al., 2020). Subsequent research has emphasized that optimizing the volume and quality of training tokens to enhance knowledge acquisition is as critical as scaling model size (Brown et al., 2020; Hoffmann et al., 2022). To further augment these capacities, retrieval-augmented gener- ation (RAG) has emerged to externalize knowledge into non- parametric memory, improving performance on knowledge- intensive tasks (Lewis et al., 2020). However, despite these advancements in knowledge acquisition, LLMs still exhibit significant performance gaps in non-generative tasks (Grabu- loski et al., 2025; Song et al., 2025). The prevailing strategy to bridge this gap remains fine-tuning models using task- specific datasets and reward signals (Kaufmann et al., 2023; Koukounas et al., 2024; Xie et al., 2023). With increasing training data volume, model size, and risks associated with external data exposure, research has shifted toward unlocking the latent potential within pre-trained mod- els without relying on extensive external supervision (Chen et al., 2024). Current post-training paradigms have explored methods such as self-reflection (Qiu & Yang, 2024; Yuan et al., 2024), weak supervision (Yu et al., 2021), and self- generated reward signals (Shao et al., 2025) to refine spe- cific task performance. While these methods use a model’s internal parameters to enhance performance, they remain confined to the standard generative paradigm, focusing on optimizing the quality of next-token prediction (NTP). However, a growing body of evidence suggests that the NTP paradigm is not a panacea for all tasks (Bachmann & Na- garajan, 2024). For non-generative applications, such as information retrieval or structured regression, task-oriented output formats have been found to outperform linguistic generation (Wang & Ma, 2024; Ma et al., 2024; Wang et al., 2024a). Our work, SKR, is motivated by these insights and extends beyond mere annotation-free fine-tuning. Unlike ex- isting self-improvement methods that refine what the model generates within the NTP paradigm, SKR fundamentally alters how the model expresses its knowledge. Using the model’s native generative capability to extract its intrinsic knowledge and subsequently refining its expression mecha- nism to a task-optimized structure, SKR enables LLMs to achieve superior task performance and inference efficiency. 3. Problem Setup 3.1. LLMs’ Knowledge Expression Completing an NLP task involves providing inputs to a model to elicit desired outputs. A model’s ability to gen- 2 Self Knowledge Re-expression: A Fully Local Method for Adapting LLMs to Tasks Using Intrinsic Knowledge erate desired outputs stems from its parametric knowledge acquired during pre-training and stored within its parame- ters (Wang et al., 2024c). In current LLMs, this knowledge is universally expressed through the next-token prediction (NTP) paradigm (Achiam et al., 2023; Dubey et al., 2024). Formally, given an LLMMwith parametersθ K storing knowledgeK. The standard NTP expression paradigm, E ntp , uses these parameters to generate tokens in sequence. This process is formalized as modeling the probability of the next tokenw n within a vocabularyV, conditioned on the sequence of preceding tokens (w n−1 0 ): E ntp (w n−1 0 ;θ K ) : = P (w n = w | w n−1 0 ;K) w ∈V. (1) For a given taskT ∈ T, whereTdenotes the set of all possible tasks, the execution ofTby LLMMbegins with a prompt with task-specific instructions and raw input data. Subsequently, the modelMapplies theE ntp paradigm to generate tokens as the task output. Within this pipeline, all task-specific characteristics and structural requirements are integrated into and inherently constrained by the se- quential token stream. Current research often treatsE ntp as a versatile, all-purpose expression mechanism, using it for both generative and non-generative tasks by encoding non-textual information (e.g., numerical values) into textual formats. Therefore, prior work has primarily focused on optimizing the parametric knowledgeKvia fine-tuning or post-training to enhance LLM performance (Han et al.). 3.2. Limitations of Next-Token Prediction However, task performance is not solely determined by para- metric knowledgeK. The expression mechanism,E ntp , is a vital component of the task completion process. The NTP paradigm can constrain the use ofK(Gekhman et al., 2025). Even when internal knowledgeKis sufficient for a task, the LLM is forced to manifest its capacity by generating textual outputs based on a probability distribution over the vocabulary. Consequently, the expressive power ofKis bottlenecked by the NTP expression mechanism (E ntp ). To illustrate this expression bottleneck, we examine three representative examples of tasks whereE ntp is sub-optimal: 1. Information Retrieval (T IR ). Standard NTP outputs lack the dense, high-dimensional information required for semantic proximity matching, and the generation process is time-consuming. An expression mechanism that encodes inputs directly into high-quality embeddings is significantly more effective thanE ntp . This example represents scenarios where the required output cannot be adequately represented by sequential tokens. 2. Object Detection (T OD ). Expressing coordinates of an object as tokenized strings introduces sequential dependency and potential syntax errors. Post-processing is necessary to make outputs usable. A more efficient expression is the direct regression of numerical values from the model’s latent representations. This example represents scenarios where E ntp may be functionally capable, but with low efficiency and cumbersome output formats. 3. Anomaly Detection (T AD ). LLMs typically perform clas- sification tasks like anomaly detection by generating label tokens (e.g., yes or no). However, the resulting token proba- bilities are often poor proxies for task-oriented confidence scores. Because the model operates over a vast vocabulary, probability mass frequently leaks to semantically similar tokens. An expression that uses task-oriented classification logits, provides explicit, normalized class probabilities, that are more robust than the scattered distribution ofE ntp . This task represents scenarios whereE ntp fails to provide the necessary information for rigorous task evaluation. TheseE ntp constraints are critical, as such tasks are founda- tional to practical, domain-specific applications, including retrieval-augmented generation (RAG) (Wei et al., 2024; Li et al., 2025b), table localization (Sui et al., 2024), and ab- normal figure detection (Xu et al., 2025; Deng et al., 2025). 3.3. Definition of Self Knowledge Re-expression Adapting LLMs to specialized tasks typically requires extensive, high-quality annotated dataD = ⟨D x ,D y ⟩, whether derived from human (Chen et al., 2021) or model- assisted (Wu et al., 2025) sources. However, we hypothesize that the LLM already possesses sufficient intrinsic knowl- edgeKto address these tasks, and that the performance gap is a failure of expression rather than a lack of knowledge. We therefore formalize the Self Knowledge Re-expression (SKR) process as follows. Given a taskTand only unanno- tated dataD x , SKR transforms LLMM’s expression from the genericE ntp to a task-adapted mechanismE T . This pro- cess relies primarily onM’s intrinsic knowledgeK, without external supervision from humans or other LLMs. For rigor, we introduceK T to represent the task priors required for adaptation, such as expected output layer structure, loss function, and task prompts. WhileK T may include knowl- edge external toK, its informational content is negligible in scale (K T ≪ K). Unlike the extensive effort required for annotatingD x , definingK T consists of only lightweight, high-level design choices. Therefore,K T does not under- mine the premise that SKR primarily relies on theM’s intrinsic knowledge. LetM E ∗ denote the modelMusing expression mechanism E ∗ . SKR is formalized as: M E ntp SKR −→ K,K T ,D x M E T .(2) SKR differs from established techniques like knowledge distillation or teacher-student paradigm in two aspects: (1) SKR relies on the LLMM’s intrinsic knowledge to achieve 3 Self Knowledge Re-expression: A Fully Local Method for Adapting LLMs to Tasks Using Intrinsic Knowledge Transformer Layers (Parametric Knowledge) Output Layers (Next Token Prediction) Transformer Layers (Parametric Knowledge) Output Layers (Task-adapted Format) Post-processing Unannotated Task Data Task Prompt Task-specific Annotations Task-specific Annotations Unannotated Task Data Task Loss Fine-tune Test Data Direct Task-Specific Outputs LoRA Adapter Preparation Required for SKR Unannotated Task Data Task Prompt Output Layers (Task-adapted Format) Post-processing Task Loss Information Retrieval (IR) Task Remove the token generation layer and retain the attention layers. Obtain the last layer of the last token from the inputs. Contrastive Language–Image Pretraining (CLIP) loss. Image to be retrieved. High-quality Multi- modal Embeddings SKR LLM-based Embedding Model SKR Object Detection (OD) Task Document Page Images Prompt: “... identify the bounding box of the chart/table/figure ...” A regression layer producing the coordinates of bounding boxes. Convert the string containing bounding box coordinates into numerical values. Mean Squared Error (MSE) loss Object Bounding Box Coordinates LLM-based Object Detector SKR Images and Queries LLM-based Classifier A binary classification head generating class probability outputs. Cross-entropy (CE) loss Convert the yes/no judgment into classification results. For a given task, selecting the appropriate preparation factors enables SKR to produce a task-adapted LLM. ............ Probability Distribution Over Classes Task-Specific LLM (푀 (휃 퐾 , Δ휃 푇 ), 퐸 푇 ) 퐾 푇 Collected task-related raw data (practical inputs). A concise prompt to guide task annotation. Output layers with task- specific output formats. Processing to convert strings into task-required formats. Task-specific fine-tuning loss. 퐷 푥 Base LLM (푀 휃 퐾 , 퐸 푛푡푝 ) Prompt: “Based on the input image, generate one question that can be answered using the provided information.” Prompt: “... whether the following question can be answered using the information provided in the image...” Anomaly Detection (AD) Task Self Knowledge Re-expression (SKR) Method Step 1: Self-Annotation Step 2: Knowledge Re-expression Figure 2.The Self Knowledge Re-expression (SKR) method implemented across three different tasks (i.e., information retrieval, objection detection, anomaly detection). The process executes using only unannotated raw data and task-related configurations (e.g., prompts, task loss, output formats, lightweight post-processing), without requiring human annotation or external supervision. task adaptation, without the support of a superior “teacher model” to inject external knowledge. (2) SKR aims at opti- mizing the expression mechanism (E ntp → E T ) to unlock latent potential, rather than introducing new knowledge to improveK. These characteristics make SKR a highly eco- nomical method by significantly reducing adaptation costs. Furthermore, as a fully local process, it mitigates the privacy risks of external data exposure, making it particularly suit- able for sensitive domains such as finance and healthcare. 4. Methods To implement SKR, we propose a method enabling pre- trained LLMs to adapt to specialized tasks by optimizing their knowledge expression mechanism. Our method uses the LLM itself to generate task-specific annotations, guided by task-specific priors (K T ), to transition from generic to- ken generation to task-optimized outputs (Figure 2). 4.1. SKR Method The SKR method executes in two sequential steps: 1. Self-Annotation: Knowledge Extraction viaE ntp . The process begins with unannotated task dataD x . The original LLM is queried to generate target outputs in a sequential token format (e.g., image-related text forT IR , or coordinate strings forT OD ). Here,E ntp serves not as the final output method, but as a mechanism for knowledge extraction. Post- processing converts outputs into the task-specific format, producing self-generated annotations ˆ D y forD x : (D x ,T ) M E ntp −→ Task Query ˆ D y .(3) 2. Knowledge Re-expression: Transitioning toE T . Using the self-annotated data(D x , ˆ D y ), the model is fine-tuned with a task-specific objective (e.g., contrastive loss for em- beddings, or mean squared error loss for regression). This step transforms the model’s output mechanism fromE ntp to a task-suitable expressionE T . We implement this process using LoRA (Hu et al., 2022). With original parametersθ K and trainable parameters∆θ T (comprising LoRA adapters and a task-specific head), the adaptation is formalized as: M θ,E ntp Fine-tune −→ (D x , ˆ D y ), Loss of E T M (θ,∆θ T ),E T .(4) This step results in a task-adapted model M (θ,∆θ T ),E T . 4 Self Knowledge Re-expression: A Fully Local Method for Adapting LLMs to Tasks Using Intrinsic Knowledge 4.2. Case Studies and Task Implementations We implement our SKR method on the three representative task types (Figure 2). While LLMs and multi-modal LLMs (MLLMs) share the same expression mechanism, we priori- tize MLLMs experiments due to their broader application potential. Our evaluation covers both single- and multi- modal scenarios:T IR includes a text-only variant to validate SKR on pure LLMs. All three task types are implemented with multi-modal inputs to demonstrate MLLM adaptation. 4.2.1. SKR FOR EXTENDED CAPABILITY (T IR ) The first case study uses the IR tasks to highlight capabilities that standard NTP does not support. We cover both text-to- text (paragraph) and text-to-image (chart/figure) retrieval, where the objective is to identify elements containing the information to answer a user query. The genericE ntp expres- sion lacks the ability to produce vectors required for efficient semantic search, especially in multi-modal contexts (Jiang et al., 2024). Standard solutions rely on fine-tuning using text-answer pairs created by humans or superior teacher models (Zhou et al., 2024; Koukounas et al., 2024). Our SKR method only needs raw paragraphs or images. We prompt the model to generate queries based on these inputs. The queries and inputs then form the training pairs for con- trastive learning (Radford et al., 2021). In the Knowledge Re-expression phase, we bypass NTP layers and use the final token’s last layer as the representation. Fine-tuning with a contrastive loss transitions the model fromM θ,E ntp to M (θ,∆θ IR ),E IR . This transition resolves theE ntp bottleneck, making SKR-adapted models surpass advanced baselines. 4.2.2. SKR FOR EFFICIENCY (T OD ) The study ofT OD demonstrates how SKR streamlines multi- step tasks into efficient end-to-end operations. Practical ob- ject detection often requires flexible granularity. For exam- ple, extracting a chart requires including the title and source for a full context. Traditional detectors, like YOLO (Red- mon et al., 2016), lack zero-shot instruction flexibility. Al- though MLLMs can adjust detection objectives by adding instructions, they often struggle to detect such “compos- ite” areas (figure + title + source). Current workarounds split detection into sub-tasks (e.g., separate title and fig- ure detection) and merge results via post-processing, which dramatically increases latency and risk of parsing failures. Through SKR, we implement Self-Annotation by running multi-step detection on a small data subset to generate coor- dinates. We then replace the NTP layers with a regression headE OD that directly outputs bounding box values. Fine- tuned with an MSE loss, the adapted modelM (θ,∆θ OD ),E OD enables end-to-end detection. This process eliminates multi- turn detection and post-processing, significantly reducing latency. Crucially, for models lacking inherent grounding abilities (e.g., LLaVA), SKR effectively evolves latent im- age understanding into explicit grounding capability. 4.2.3. SKR FOR MORE INFORMATION (T AD ) This case study transforms the expression mechanism to provide task-specific information. Our anomaly detection task is an image-text matching problem: determining if an image contains the information relevant to a specific query. Since valid matches are rare, the data is highly imbalanced. E ntp restricts output to next-token probabilities, explicit probability distribution over target classes is missing. In the Self-Annotation phase, we prompt the model to output an explicit yes or no for query-image pairs. In the Knowl- edge Re-expression phase, we replace the generation head with a binary classification head (E AD ) that outputs normal- ized class probabilities. This transitions the model from NTP to providing class distributions, allowing for the cal- culation of robust metrics like AUROC and AUPRC. These metrics are important for imbalanced datasets (Hu et al., 2020; Wang et al., 2022) and impossible to derive accu- rately from simple token generation. 5. Experimental Setting 5.1. Dataset We evaluate SKR using a large-scale industrial-scale finan- cial dataset and the open-source MMDocRAG dataset (Li et al., 2025a), DocVQA-2020 dataset (Mathew et al., 2021), SciMMIR dataset (Wu et al., 2024), and CUB2002011 dataset (Wah et al., 2011). For all datasets, SKR uses only unannotated raw data. Annotations are strictly reserved for performance evaluation. More detailed descriptions are provided in Appendix A. Financial Document Dataset: This industrial-scale dataset comprises over 100,000 professional financial reports, in- cluding millions of analysis paragraphs, tables, and charts. These documents provide a rigorous environment forT IR , T OD , andT AD , reflecting the complexities of real-world fi- nancial analysis. While the total corpus is vast, the SKR method requires only a small, randomly sampled subset of raw data (≈1,000 samples) to serve as the unannotated training setD x . To ensure robust evaluation, we constructed specialized test sets for each task, all subject to meticulous manual verification. ForT IR , the model retrieves relevant images from a candidate pool in response to specific tex- tual queries. ForT OD , the model identifies and localizes charts and tables within a document page, outputting pre- cise bounding boxes. ForT AD , the model performs a binary verification task, determining whether a given query and image form a valid question-answer pair. 5 Self Knowledge Re-expression: A Fully Local Method for Adapting LLMs to Tasks Using Intrinsic Knowledge MMDocRAG Dataset: We evaluate the multi-modal IR task on the open-source MMDocRAG dataset. We strictly follow their evaluation protocols but use only training data without labels. This allows for a direct comparison with state-of-the-art models in a standardized RAG environment. DocVQA-2020 Dataset and SciMMIR Dataset: To assess the generalization of SKR, we test models, which are fine- tuned only on MMDocRAG, directly on DocVQA-2020 and SciMMIR. DocVQA-2020 includes documents spanning multiple decades and diverse industries. SciMMIR is a specialized scientific domain dataset. This allows us to evaluate if SKR models can generalize to unseen datasets for similar tasks. CUB2002011 dataset: For the OD and AD tasks, we fur- ther evaluate SKR on this open-source dataset. It contains 200 bird species with fine-grained bounding box annota- tions, providing a standard environment to assess SKR’s ability to localize objects and identify anomalies. 5.2. Evaluation Metrics We evaluate each task using a comprehensive set of met- rics, with definitions and equations detailed in Appendix B. ForT IR , we report mean reciprocal rank (MRR) and Re- call@k (R@k). ForT OD , we report intersection over union (IoU) and inference time (Time). ForT AD , we use accuracy (Acc), area under the ROC curve (AUROC), area under the precision-recall curve (AUPRC), and inference time (Time). 5.3. Models and Baselines We evaluate four popular LLMs for the text-only task and three MLLMs for the multi-modal tasks. LLMs include Mistral-7B-Instruct-v0.2 (Mistral-0.2; Jiang et al. 2023), Phi-3-mini-128k-instruct (Phi-3; Abdin et al. 2024), Qwen2- 7B-Instruct (Qwen2; Team et al. 2024), and Llama-3.1-8B (Llama-3.1; Dubey et al. 2024). MLLMs include LLaVA- v1.6-Mistral-7B (LLaVA-1.6; Liu et al. 2023), Phi-3-vision- 128k-instruct (Phi-3-VL; Abdin et al. 2024), and Qwen2- VL-7B-Instruct (Qwen2-VL; Wang et al. 2024b). These models serve as the base modelsM, where the genericE ntp is adapted to a task-specific E T via SKR. Our primary baseline is the base model before SKR, which isolates the gains strictly attributable to our method. Since SKR operates in a zero-annotation regime, few established adaptation methods are directly comparable. We include E5-V (Jiang et al., 2024) (a training-free representation method) and several state-of-the-art embedding models (Nomic (Team, 2025), OpenAI, and top-tier models from the MMDocRAG benchmark) that are specifically fine-tuned on massive datasets for IR, to demonstrate SKR’s benefits. We also compare with models fine-tuned on external super- vision (e.g., GPT-4o) and implement an SFT baseline using the same self-annotations to facilitate the ablation study. Although our study does not use the latest open-source mod- els due to resource constraints, our results already surpass many specialized systems, suggesting the possibility of fur- ther improvement on better LLMs. However, our primary goal is not to explore the upper bounds of SOTA perfor- mance, but to provide a controlled proof of the existence of the expression bottleneck. By comparing the same model underE ntp andE T , we show that diverse LLMs possess vast unexploited capabilities that can be unlocked simply by opti- mizing their expression mechanisms, allowing even modest models to outperform highly specialized counterparts. 5.4. SKR Implementation Details The Self-Annotation phase is tailored to the base model’s inherent capabilities. ForT IR andT AD , we directly prompt the model to generate input-related queries or assess image- text relevance. ForT OD , the strategy diverges by model capability. We directly prompt Qwen2-VL model to output coordinates since it possesses native grounding capabilities. In contrast, models with semantic understanding but no native coordinate generation (LLaVA-1.6 and Phi-3-VL) use a “fuzzy localization” strategy (i.e., recursively cropping inputs and querying object presence) to get bounding boxes. This distinction allows us to prove that SKR can not only optimize existing capabilities but also awaken latent skills in models that lack native support. Additionally, we also experiment with using lightweight, rule-based OCR as a post-processing step to assist in coordinate refinement. Further details on self-annotation, prompts, task head struc- tures, and hyper-parameters are provided in Appendix C. 6. Results and Discussion We analyze the performance gains of SKR-adapted models relative to their original base versions and established in- dustrial baselines acrossT IR ,T OD , andT AD in Section 6.1. The comparative analysis between SKR and standard SFT is presented in Section 6.2 to demonstrate the necessity of transitioning from NTP to a task-specific expression head. 6.1. Task Improvement of SKR 6.1.1. SKR FOR EXTENDED CAPABILITY (T IR ) Table 1 demonstrates that SKR yields a transformative im- provement in the semantic representation capabilities of both LLMs and MLLMs. Under the standardE ntp paradigm, base models perform poorly on retrieval tasks because they are not optimized to generate high-quality representations. While the prompt-based method E5-V mitigates this issue to an extent, its performance remains insufficient. 6 Self Knowledge Re-expression: A Fully Local Method for Adapting LLMs to Tasks Using Intrinsic Knowledge Table 1.Performance comparison onT IR . SKR’s results are high- lighted in bold. Across all models and modalities, SKR consis- tently achieves the best performance. Qwen2 models adapted via SKR outperform specialized industrial baselines. ModelMRRR@1R@3R@5R@10 Text-to-text Retrieval OpenAI ada-0020.4200.3220.4600.5180.598 Nomic-text-v1.50.4950.3920.5580.6240.702 Nomic-text-v20.5490.4540.6000.6480.728 Mistral-0.2 Base0.1080.0520.1160.1540.204 E5-V0.2230.1440.2500.2920.386 SFT0.0760.0360.0780.1040.152 SKR0.6220.5100.7020.7540.822 Phi-3 Base0.0980.0680.0960.1120.146 E5-V0.2090.1420.2300.2700.338 SFT0.0820.0540.0840.1000.124 SKR0.6320.5180.7180.7780.826 Qwen2 Base0.1150.0620.1300.1640.230 E5-V0.2700.1960.2940.3360.416 SFT0.0630.0320.0560.0900.136 SKR0.6740.5660.7560.8140.860 Llama3.1 Base0.1960.1360.2140.2580.330 E5-V0.3060.2400.3180.3600.426 SFT0.1970.1400.2100.2560.328 SKR0.6690.5660.7420.7920.848 Text-to-image Retrieval Nomic-m-7b0.6690.5600.7450.7950.860 LLaVA-1.6 Base0.0140.0000.0100.0300.035 E5-V0.1970.1500.2050.2300.305 SFT0.0310.0200.0250.0300.065 SKR0.5520.4450.6250.6650.730 Phi-3-VL Base0.0710.0500.0650.0800.125 E5-V0.2030.1500.2100.2350.300 SFT0.0270.0200.0300.0300.035 SKR0.5560.4500.6300.6800.750 Qwen2-VL Base0.0440.0300.0350.0400.090 E5-V0.2910.2100.3400.3950.450 SFT0.0550.0350.0550.0650.085 SKR0.6910.5950.7550.7950.875 Our SKR method directly adapts base models into high- quality encoding models. Without any external annotations, SKR-adapted Qwen2 models consistently outperform indus- trial models trained on massive annotated datasets. More evaluations on open-source benchmarks further validate these findings (Appendix D.1). Results on the open-source MMDocRAG dataset demonstrate that SKR consistently outperforms various SOTA retrieval models. Results on the DocVQA and SciMMIR datasets underscore the robust cross-dataset generalization ability of SKR-adapted models across similar tasks. All these results proves that the bottle- neck in current retrieval systems is often not a scarcity of knowledge, but rather the inability of the expression mecha- nism to properly use knowledge. Table 2. Performance comparison onT OD . Results for the SKR method are highlighted in bold. The best results within each model are underlined. Runtime is reported in minutes per 100 test cases. ModelIoUTime LLaVA-1.6 Base (Fuzzy Location)0.14241.0 SFT (Fuzzy Location)0.0656.4 SKR (Fuzzy Location)0.2661.7 Base (OCR)0.59910.3 SKR (OCR)0.614 1.6 Phi-3-VL Base (Fuzzy Location)0.16435.0 SFT (Fuzzy Location)0.0567.1 SKR (Fuzzy Location)0.2320.8 Base (OCR)0.33630.7 SKR (OCR)0.3440.9 Qwen2-VL Base (One-turn Prompt)0.6286.8 Base (Multi-turn Prompt)0.73920.4 SFT (Multi-turn Prompt)0.7416.5 SKR (Multi-turn Prompt)0.7264.8 6.1.2. SKR FOR EFFICIENCY (T OD ) The results in Table 2 highlight SKR’s ability to elicit latent capabilities and streamline multi-step pipelines. Models such as LLaVA-1.6 and Phi-3-VL, which cannot localize ob- jects, acquire object detection capability through SKR. Ba- sic fuzzy location post-processing enables models to output more reasonable coordinates. Adding simple OCR yields substantial improvements. Both models exhibit dramatic IoU improvements of 0.472 and 0.180, respectively, com- pared to their base states. For Qwen2-VL, which inherently possesses grounding ca- pabilities, SKR compresses multi-turn actions into a sin- gle turn. The adapted model achieves an IoU of 0.726, significantly outperforming the original model’s one-turn result and remaining comparable to the multi-turn perfor- mance. Across all models, SKR reduces inference time by at least 76%. Evaluations on the open-source CUB2002011 dataset demonstrate a consistent trend (see Appendix D.2). ReplacingE ntp withE reg eliminates the computational over- head of sequential token generation. 6.1.3. SKR FOR MORE INFORMATION (T AD ) In anomaly detection, where datasets are highly imbalanced, standard accuracy metrics can be deceptive. Table 3 reveals that SKR provides a significantly more robust output mech- anism than binary token prediction. To ensure a rigorous baseline, we tried two probability extraction methods for the base models: assigning 100% probabilities to generated classes and getting softmax probabilities for specific tokens. We report the superior of these two as base model results. While accuracy gains appear modest due to the rarity of outliers, AUROC and AUPRC scores, which are more in- formative for anomaly detection, show massive gains. All 7 Self Knowledge Re-expression: A Fully Local Method for Adapting LLMs to Tasks Using Intrinsic Knowledge Table 3.Performance comparison onT AD . SKR results are high- lighted in bold. SKR-adapted models consistently achieve the highest performance with lowest runtime. Runtime is reported in minutes per 100 test cases. ModelAccAUROCAUPRCTime LLaVA-1.6 Base0.9300.8940.5313.8 SFT0.8350.8920.3811.4 SKR0.9750.9910.8791.3 Phi-3-VL Base0.7700.8500.2874.7 SFT0.9450.9120.8507.1 SKR0.9350.9810.9330.7 Qwen2-VL Base0.9350.8750.5321.1 SFT0.9600.9390.7460.9 SKR0.9600.9840.8680.7 models achieve at least a 0.1 improvement in AUROC and a 0.3 increase in AUPRC; Phi-3-VL’s AUPRC jumps from 0.287 to 0.933. The performance benefit is further corrob- orated by evaluations on the open-source CUB2002011 benchmark (see Appendix D.2). This confirms thatE ntp is ill-suited for providing the calibrated confidence scores nec- essary for such tasks. Additionally, SKR-adapted models demonstrate enhanced efficiency, reducing runtime by at least 30%. 6.2. Ablation Studies To investigate the relationship between an LLM’s latent knowledge and its functional capabilities, we conduct abla- tion studies on the two SKR steps. Results reveals that an LLM’s internal knowledge sufficiency is contingent upon: (i) model’s intrinsic knowledge depth, (i) model’s pre- existing task-specific abilities, and (i) task complexity. 6.2.1. IMPACT OF ANNOTATION KNOWLEDGE: INTRINSIC VS. EXTERNAL We investigate the sufficiency of intrinsic knowledge by comparing it with external supervision. We use GPT-4o, which is stronger than our evaluated models on multi-modal benchmarks (Zhu et al., 2025), to generate annotations for T IR andT OD . We use the same re-expression architectures (E T ) of SKR, changing only annotations. As illustrated in Figure 3, this comparison clarifies where internal knowledge suffices and where external knowledge remains beneficial. ForT IR , self-annotation yields performance gains compa- rable to, and occasionally surpassing, external supervision. Since image understanding is a foundational capability of modern MLLMs, the gap between our evaluated models and GPT-4o in generating image-related text is marginal. This confirms that the retrieval bottleneck stems primarily from the expression mechanism rather than knowledge deficiency. In contrast,T OD results vary by model. External supervision works better for LLaVA-1.6 and Phi-3-VL, as these models LLaVA-1.6Phi-3-VLQwen2-VL 0.0 0.2 0.4 0.6 0.8 Recall@5 Text-to-image Retrieval 0.665 0.680 0.795 0.685 0.670 0.780 LLaVA-1.6Phi-3-VLQwen2-VL 0.0 0.2 0.4 0.6 IoU Object Detection 0.266 0.232 0.720 0.392 0.397 0.401 Self-AnnotationAnnotations from Superior Models Figure 3.Performance comparison of LLMs fine-tuned using self- annotations versus external annotations across T IR and T OD . lack native grounding capabilities. However, Qwen2-VL’s self-annotation outperforms GPT-4o’s labels. Despite GPT- 4o’s general dominance across benchmarks, its grounding ability (IoU of 0.414 on our test set) is inferior to Qwen2- VL’s specialized capability. Therefore, while GPT-4o pro- vides useful knowledge for models with clear gaps, it can- not match the performance of Qwen2-VL when SKR re- expresses Qwen2-VL’s grounding knowledge. This obser- vation is reinforced by our analysis of the Gemma-3 family (Appendix E), which indicates that larger models do not always possess more task potential than smaller models. In summary, external supervision is redundant in many sce- narios, where poor task performance stems from a failure of expression rather than a lack of knowledge. However, SKR is not a total substitute for knowledge acquisition. For models with clear knowledge gaps, external supervision re- mains essential. Since pre-training cannot encompass all tasks, these findings suggest that even leading LLMs likely possess untapped abilities, waiting to be correctly expressed. 6.2.2. IMPACT OF EXPRESSION MECHANISM: NTP VS. TASK-SPECIFIC HEAD To isolate the impact of the expression mechanism, we com- pare SKR’s re-expression step against standard SFT. The SFT baseline uses the same self-annotations as SKR but fine- tunes the models via their native NTP objective. Results are summarized in the SFT rows of Tables 1, 2, 3. InT IR , SFT yields poor results, with marginal gains in text- to-image retrieval and degradation in text-to-text retrieval. These results underscore a fundamental objective mismatch: SFT forces the model to express knowledge through se- quential token generation, whereasT IR requires high-quality representation vectors. SKR’s task-specific head bypasses this bottleneck by directly using latent representations. The effectiveness of SFT inT OD depends heavily on pre- existing abilities. While SFT improves Qwen2-VL, it severely degrades LLaVA-1.6 and Phi-3-VL. Lacking na- tive grounding capabilities, these two models struggle to produce structured coordinate strings via NTP without the constraints of a task-specific head. Furthermore, even for Qwen2-VL, SFT remains 35% slower than SKR. Although SFT also compressed multi-turn detection into a single-turn running, the NTP mechanism still requires multiple decod- 8 Self Knowledge Re-expression: A Fully Local Method for Adapting LLMs to Tasks Using Intrinsic Knowledge ing steps for token generation, whereas SKR produces the entire bounding box in a single forward pass. ForT AD , performance remains substantially inferior to SKR. In terms of AUPRC, the most critical metric for imbalanced classification, SKR consistently outperforms SFT by at least 8%, proving that direct class projections are more effective than mapping classifications to vocabulary tokens. These findings demonstrate that a dedicated re-expression mechanism is vital for model adaptation. SKR’s ability to extract latent knowledge and optimize inference efficiency provides a decisive advantage that standard NTP-based fine- tuning cannot replicate. Further investigations into the ne- cessity of LoRA adapters for facilitating the re-expression process, as well as an analysis of how SKR reshapes la- tent representations, are provided in Appendix F and Ap- pendix G, respectively. 7. Conclusions and Future Work This paper formalizes the knowledge expression bottleneck, demonstrating that NTP often constrains LLMs’ latent po- tential. We propose the SKR method, which is task-agnostic and fully local, to elicit this potential by transitioning from generic token generation to task-specific expressions. SKR significantly reduces adaptation costs and mitigates data ex- posure risks. Experiments across three task types show that SKR achieves substantial performance and latency gains without external supervision, even surpassing leading indus- trial models. Ablation studies confirm that external super- vision is often redundant when the necessary knowledge resides within the model’s parameters, proving that the ex- pression bottleneck is a primary factor limiting LLM utility. By decoupling knowledge from linguistic outputs, this work establishes knowledge re-expression as a novel adaptation paradigm. This shift opens promising research directions, such as the full automation of SKR, where models au- tonomously design their outputs. Furthermore, our find- ings suggest a path for evolving advanced LLMs without continuous scaling of data or parameters. These insights offer a practical, efficient, and secure route for deploying state-of-the-art models in sensitive, real-world applications. Impact Statement This paper presents work whose goal is to advance the field of machine learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here. Disclaimer This paper was prepared for informational purposes in part by the Machine Learning Center of Excellence group of JP- Morgan Chase & Co. and its affiliates (”JP Morgan”) and is not a product of the Research Department of JP Morgan. JP Morgan makes no representation and warranty whatsoever and disclaims all liability, for the completeness, accuracy or reliability of the information contained herein. This docu- ment is not intended as investment research or investment advice, or a recommendation, offer or solicitation for the purchase or sale of any security, financial instrument, fi- nancial product or service, or to be used in any way for evaluating the merits of participating in any transaction, and shall not constitute a solicitation under any jurisdiction or to any person, if such solicitation under such jurisdiction or to such person would be unlawful. References Abdin, M., Aneja, J., Behl, H., Bubeck, S., Eldan, R., Gunasekar, S., Harrison, M., Hewett, R. J., Javaheripi, M., Kauffmann, P., et al. Phi-4 technical report. arXiv preprint arXiv:2412.08905, 2024. Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023. Bachmann, G. and Nagarajan, V. The pitfalls of next-token prediction. In Proceedings of the 41st International Con- ference on Machine Learning, p. 2296–2318, 2024. Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. Language models are few-shot learners. Advances in neural information processing systems, 33: 1877–1901, 2020. Chen, Z., Chen, W., Smiley, C., Shah, S., Borova, I., Lang- don, D., Moussa, R., Beane, M., Huang, T.-H., Routledge, B. R., et al. Finqa: A dataset of numerical reasoning over financial data. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, p. 3697–3711, 2021. Chen, Z., Deng, Y., Yuan, H., Ji, K., and Gu, Q. Self-play fine-tuning converts weak language models to strong lan- guage models. In International Conference on Machine Learning, p. 6621–6642. PMLR, 2024. Deng, H., Luo, H., Zhai, W., Guo, Y., Cao, Y., and Kang, Y. Vmad: Visual-enhanced multimodal large language model for zero-shot anomaly detection. IEEE Transac- tions on Automation Science and Engineering, 2025. 9 Self Knowledge Re-expression: A Fully Local Method for Adapting LLMs to Tasks Using Intrinsic Knowledge Dong, K., Chang, Y., Huang, S., Wang, Y., Tang, R., and Liu, Y. Benchmarking retrieval-augmented multimomal gen- eration for document question answering. arXiv preprint arXiv:2505.16470, 2025. Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., et al. The llama 3 herd of models. arXiv e-prints, p. arXiv–2407, 2024. Faysse, M., Sibille, H., Wu, T., Omrani, B., Viaud, G., Hudelot, C., and Colombo, P. Colpali: Efficient document retrieval with vision language models. In ICLR, 2025. Gekhman, Z., David, E. B., Orgad, H., Ofek, E., Belinkov, Y., Szpektor, I., Herzig, J., and Reichart, R. Inside- out: Hidden factual knowledge in llms. arXiv preprint arXiv:2503.15299, 2025. Grabuloski, M., Karadimce, A., and Sefidanoski, A. En- hancing language models with retrieval-augmented gen- eration a comparative study on performance. WSEAS Transactions on Information Science and Applications, 22:272–297, 2025. Han, Z., Gao, C., Liu, J., Zhang, J., and Zhang, S. Q. Parameter-efficient fine-tuning for large models: A com- prehensive survey. Transactions on Machine Learning Research. Hestness, J., Narang, S., Ardalani, N., Diamos, G., Jun, H., Kianinejad, H., Patwary, M. M. A., Yang, Y., and Zhou, Y. Deep learning scaling is predictable, empirically. arXiv preprint arXiv:1712.00409, 2017. Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E., Cai, T., Rutherford, E., de Las Casas, D., Hendricks, L. A., Welbl, J., Clark, A., et al. Training compute- optimal large language models. In Proceedings of the 36th International Conference on Neural Information Processing Systems, p. 30016–30030, 2022. Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., Chen, W., et al. Lora: Low-rank adaptation of large language models. ICLR, 1(2):3, 2022. Hu, W., Wang, M., Qin, Q., Ma, J., and Liu, B. Hrn: A holistic approach to one class learning. Advances in neural information processing systems, 33:19111–19124, 2020. Jiang, D., Liu, Y., Liu, S., Zhao, J., Zhang, H., Gao, Z., Zhang, X., Li, J., and Xiong, H. From clip to dino: Visual encoders shout in multi-modal large language models. arXiv preprint arXiv:2310.08825, 2023. Jiang, T., Song, M., Zhang, Z., Huang, H., Deng, W., Sun, F., Zhang, Q., Wang, D., and Zhuang, F. E5-v: Univer- sal embeddings with multimodal large language models. arXiv preprint arXiv:2407.12580, 2024. Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., and Amodei, D. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361, 2020. Karpukhin, V., Oguz, B., Min, S., Lewis, P., Wu, L., Edunov, S., Chen, D., and Yih, W.-t. Dense passage retrieval for open-domain question answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), p. 6769–6781, 2020. Kaufmann, T., Weng, P., Bengs, V., and H ̈ ullermeier, E. A survey of reinforcement learning from human feedback. arXiv preprint arXiv:2312.14925, 2023. Kingma, D. P. and Ba, J. L. Adam: A method for stochastic gradient descent. In ICLR: international conference on learning representations, p. 1–15, 2015. Kornblith, S., Norouzi, M., Lee, H., and Hinton, G. Sim- ilarity of neural network representations revisited. In International conference on machine learning, p. 3519– 3529. PMlR, 2019. Koukounas, A., Mastrapas, G., G ̈ unther, M., Wang, B., Martens, S., Mohr, I., Sturua, S., Akram, M. K., Mart ́ ınez, J. F., Ognawala, S., et al. Jina clip: Your clip model is also your text retriever. arXiv preprint arXiv:2405.20204, 2024. Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., K ̈ uttler, H., Lewis, M., Yih, W.-t., Rockt ̈ aschel, T., et al. Retrieval-augmented generation for knowledge- intensive nlp tasks. Advances in neural information pro- cessing systems, 33:9459–9474, 2020. Li, B., Zhang, Y., Wang, X., Liang, W., Schmidt, L., and Yeung-Levy, S.Closing the modality gap for mixed modality search. arXiv preprint arXiv:2507.19054, 2025a. Li, Y., Wang, M., de Carvalho, M., Sabanis, S., and Ma, T. FinGEAR: Financial mapping-guided enhanced an- swer retrieval. In Christodoulopoulos, C., Chakraborty, T., Rose, C., and Peng, V. (eds.), Findings of the Asso- ciation for Computational Linguistics: EMNLP 2025, p. 7239–7255, Suzhou, China, November 2025b. As- sociation for Computational Linguistics. ISBN 979-8- 89176-335-7. doi: 10.18653/v1/2025.findings-emnlp. 382. URLhttps://aclanthology.org/2025. findings-emnlp.382/. 10 Self Knowledge Re-expression: A Fully Local Method for Adapting LLMs to Tasks Using Intrinsic Knowledge Liu, H., Li, C., Wu, Q., and Lee, Y. J. Visual instruction tun- ing. Advances in neural information processing systems, 36:34892–34916, 2023. Lukasik, M., Meng, Z., Narasimhan, H., Chang, Y.-W., Menon, A. K., Yu, F., and Kumar, S. Better autoregressive regression with llms via regression-aware fine-tuning. In ICLR, 2025. Ma, X., Lin, S.-C., Li, M., Chen, W., and Lin, J. Unifying multimodal retrieval via document screenshot embed- ding. In Proceedings of the 2024 Conference on Em- pirical Methods in Natural Language Processing, p. 6492–6505, 2024. Mathew, M., Karatzas, D., and Jawahar, C. Docvqa: A dataset for vqa on document images. In Proceedings of the IEEE/CVF winter conference on applications of computer vision, p. 2200–2209, 2021. Qiu, J. and Yang, Y. Training large language models to follow system prompt with self-supervised fine-tuning. In 2024 International Joint Conference on Neural Networks (IJCNN), p. 1–8. IEEE, 2024. Radford, A., Narasimhan, K., Salimans, T., Sutskever, I., et al. Improving language understanding by generative pre-training. 2018. Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I., et al. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9, 2019. Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, p. 8748–8763. PmLR, 2021. Redmon, J., Divvala, S., Girshick, R., and Farhadi, A. You only look once: Unified, real-time object detection. In Proceedings of the IEEE conference on computer vision and pattern recognition, p. 779–788, 2016. Rothenfusser, K. and Blando, B. Vector ontologies as an llm world view extraction method. arXiv preprint arXiv:2506.13252, 2025. Shao, R., Li, S. S., Xin, R., Geng, S., Wang, Y., Oh, S., Du, S. S., Lambert, N., Min, S., Krishna, R., et al. Spu- rious rewards: Rethinking training signals in rlvr. arXiv preprint arXiv:2506.10947, 2025. Song, Z., Yan, B., Liu, Y., Fang, M., Li, M., Yan, R., and Chen, X. Injecting domain-specific knowledge into large language models: a comprehensive survey. arXiv preprint arXiv:2502.10708, 2025. Sui, Y., Zhou, M., Zhou, M., Han, S., and Zhang, D. Table meets llm: Can large language models understand struc- tured table data? a benchmark and empirical study. In Proceedings of the 17th ACM International Conference on Web Search and Data Mining, p. 645–654, 2024. Team, G., Kamath, A., Ferret, J., Pathak, S., Vieillard, N., Merhej, R., Perrin, S., Matejovicova, T., Ram ́ e, A., Rivi ` ere, M., et al. Gemma 3 technical report. arXiv preprint arXiv:2503.19786, 2025. Team, N. Nomic embed multimodal: Interleaved text, image, and screenshots for visual document retrieval, 2025. URLhttps://nomic.ai/blog/posts/ nomic-embed-multimodal. Team, Q. et al. Qwen2 technical report. arXiv preprint arXiv:2407.10671, 2(3), 2024. Wah, C., Branson, S., Welinder, P., Perona, P., and Belongie, S. The caltech-ucsd birds-200-2011 dataset. 2011. Wang, M. and Ma, T. Mana-net: Mitigating aggregated sen- timent homogenization with news weighting for enhanced market prediction. In Proceedings of the 33rd ACM In- ternational Conference on Information and Knowledge Management, p. 2379–2389, 2024. Wang, M., Shao, Y., Lin, H., Hu, W., and Liu, B. Cmg: A class-mixed generation approach to out-of-distribution detection. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, p. 502–518. Springer, 2022. Wang, M., Cohen, S. B., and Ma, T. Modeling news inter- actions and influence for financial market prediction. In Findings of the Association for Computational Linguis- tics: EMNLP 2024, p. 3302–3314, 2024a. Wang, P., Bai, S., Tan, S., Wang, S., Fan, Z., Bai, J., Chen, K., Liu, X., Wang, J., Ge, W., et al. Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution. arXiv preprint arXiv:2409.12191, 2024b. Wang, S., Zhu, Y., Liu, H., Zheng, Z., Chen, C., and Li, J. Knowledge editing for large language models: A survey. ACM Computing Surveys, 57(3):1–37, 2024c. Wei, C., Chen, Y., Chen, H., Hu, H., Zhang, G., Fu, J., Ritter, A., and Chen, W. Uniir: Training and benchmarking universal multimodal information retrievers. In European Conference on Computer Vision, p. 387–404. Springer, 2024. Wu, S., Li, Y., Zhu, K., Zhang, G., Liang, Y., Ma, K., Xiao, C., Zhang, H., Yang, B., Chen, W., et al. Scimmir: Bench- marking scientific multi-modal information retrieval. In Findings of the Association for Computational Linguis- tics: ACL 2024, p. 12560–12574, 2024. 11 Self Knowledge Re-expression: A Fully Local Method for Adapting LLMs to Tasks Using Intrinsic Knowledge Wu, Y., Yan, L., Zhu, Y., Mei, Y., Wang, J., Tang, N., and Luo, Y. Boosting text-to-chart retrieval through train- ing with synthesized semantic insights. arXiv preprint arXiv:2505.10043, 2025. Xie, Q., Han, W., Zhang, X., Lai, Y., Peng, M., Lopez-Lira, A., and Huang, J. Pixiu: a large language model, in- struction data and evaluation benchmark for finance. In Proceedings of the 37th International Conference on Neu- ral Information Processing Systems, p. 33469–33484, 2023. Xu, J., Lo, S.-Y., Safaei, B., Patel, V. M., and Dwivedi, I. Towards zero-shot anomaly detection and reasoning with multimodal large language models. In Proceedings of the Computer Vision and Pattern Recognition Conference, p. 20370–20382, 2025. Yu, Y., Zuo, S., Jiang, H., Ren, W., Zhao, T., and Zhang, C. Fine-tuning pre-trained language model with weak supervision: A contrastive-regularized self-training ap- proach. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Compu- tational Linguistics: Human Language Technologies, p. 1063–1077, 2021. Yuan, H., Chen, Z., Ji, K., and Gu, Q. Self-play fine-tuning of diffusion models for text-to-image generation. Ad- vances in Neural Information Processing Systems, 37: 73366–73398, 2024. Zhao, Z., Ziser, Y., and Cohen, S. B. Layer by layer: Uncov- ering where multi-task learning happens in instruction- tuned large language models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, p. 15195–15214, 2024. Zheng, J., Hong, H., Wang, X., Su, J., Liang, Y., and Wu, S. Fine-tuning large language models for domain-specific machine translation. CoRR, 2024. Zhou, J., Liu, Z., Xiao, S., Zhao, B., and Xiong, Y. Vista: Visualized text embedding for universal multi-modal re- trieval. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 3185–3200, 2024. Zhu, Z., Jia, M., Zhang, Z., Li, L., and Jiang, M. Multi- chartqa: Benchmarking vision-language models on multi- chart problems. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Associa- tion for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), p. 11341–11359, 2025. Appendix Overview This Appendix provides comprehen- sive supplementary material to support the methodology, experimental setup, and extended analysis. We first present technical details in Appendix A, B, and C, which cover dataset specifications, formal definitions, and implementa- tion details of the evaluation metrics, respectively. Subse- quently, we present extensive additional experimental re- sults to offer deeper insights into the SKR method. These include: •Performance evaluations across the three core tasks using the open-source MMDocRAG, DocVQA-2020, SciMMIR, and CUB2002011 datasets (Appendix D). •An investigation into the impact of model scaling on SKR performance (Appendix E). •Additional ablation studies regarding the necessity of LoRA adapters (Appendix F). • A comparative study of latent representation shifts us- ing CKA analysis (Appendix G). A. Dataset Details We evaluate the SKR method using two distinct data sources: a large-scale proprietary document dataset and the open- source MMDocRAG benchmark. This dual evaluation demonstrates SKR’s effectiveness in both professional, prac- tical applications and standardized academic benchmarks. A.1. Financial Document Dataset All three task types (T IR ,T OD ,T AD ) are implemented using our proprietary financial document dataset. As highlighted in Section 5.1, we use only a small, randomly sampled subset of raw, unannotated data to activate the SKR process. To ensure rigorous evaluation, all ground-truth labels in the test sets were established through the consensus of at least two human experts. Information Retrieval (T IR ): For text-to-text retrieval, we sample 1,000 paragraphs as unannotated training data for self-annotation. For testing, we construct a candidate pool of 5,000 distinct paragraphs, within which we identify 500 ground-truth query-paragraph pairs verified by humans. For text-to-image retrieval, we sample 1,100 images (including charts, tables, and figures) as the unannotated task data. The test set consists of a candidate pool of 5,000 images, with 200 human-verified query-image pairs. Object Detection (T OD ): We sample 800 document pages, each containing a chart, table, or figure, as the unannotated task dataD x . For the test set, we separately sample 200 pages and manually annotate the bounding boxes of the target objects. 12 Self Knowledge Re-expression: A Fully Local Method for Adapting LLMs to Tasks Using Intrinsic Knowledge Anomaly Detection (T AD ): We reuse the 1,100-image set fromT IR as the unannotated training data. The test set is derived from theT IR test set, constructed to simulate a class- imbalanced anomaly detection scenario. It comprises 200 samples: 20 positive cases (images matching the query) and 180 negative cases (images irrelevant to the query), all manually verified. A.2. MMDocRAG MMDocRAG is one of the latest multi-modal benchmark designed to evaluate the retrieval and reasoning capabilities of MLLMs (Dong et al., 2025). It encompasses 14,826 images across 222 documents and includes 4,055 expert- annotated question-answer pairs. The original benchmark evaluates query-image retrieval on several state-of-the-art retrievers, including DSE models (Ma et al., 2024) and Col- Pali models (Faysse et al., 2025). These baselines represent the current industrial standard for retrieval-augmented gen- eration (RAG), as they are LLMs specifically fine-tuned on massive multi-modal datasets. To validate SKR’s competitiveness, we adhered strictly to the official MMDocRAG test protocols. However, our train- ing regime is significantly more rigorous: SKR starts from base LLMs and uses only the raw images from the devel- opment set to fine-tune models. We deliberately ignore all expert-annotated questions to maintain a strictly annotation- free adaptation process. As shown in Appendix D.1, SKR- adapted models outperform all advanced models evaluated in the original benchmark. A.3. DocVQA-2020 and SciMMIR DocVQA-2020 (Mathew et al., 2021) includes documents spanning multiple decades and diverse industries. It consists of 50,000 questions defined on 12,000+ document images. SciMMIR (Wu et al., 2024) is a specialized scientific do- main dataset that presents a significant retrieval challenge, requiring the model to identify one relevant figure from a large candidate pool of 16,263 items. To evaluate cross-dataset generalization, we use only the test sets of these two benchmarks. The evaluation is conducted using the SKR model exclusively fine-tuned on the unanno- tated training split of MMDocRAG, without any exposure to DocVQA or SciMMIR data during training. As detailed in the experimental results (see Appendix D.1), SKR models exhibit robust generalization, significantly outperforming advanced baseline RAG models. A.4. CUB 2002011 The CUB2002011 dataset (Wah et al., 2011) is a fine- grained image recognition benchmark containing 200 bird species, with a total of 5,994 training and 5,794 test sam- ples. Each sample includes an image and its corresponding bounding box annotation. To format the AD task, we se- lected hummingbirds (4 species, covering 2% of the data) as the anomalous class. Consistent with our core methodology, we strictly adhere to an annotation-free regime during the training phase, using only the raw images from the training set without access- ing any ground-truth labels or bounding boxes. Detailed performance results on this benchmark are presented in Appendix D.2. B. Evaluation Metrics This section provides the formal definitions and calculation formulas for the metrics used to evaluate the performance of SKR across the three task types. B.1. Information Retrieval (T IR ) For retrieval tasks, we focus on the model’s ability to rank the ground-truth item as high as possible within a candidate pool. Inference latency is not reported for these tasks, as encoding times remain comparable between base and SKR- adapted models. LetQbe the set of test queries, and for each queryq ∈ Q, letrank q denote the position of the most relevant document/image in the retrieved list. 1. Mean Reciprocal Rank (MRR): MRR calculates the av- erage of the reciprocal ranks of the most relevant document for all queries: MRR = 1 |Q| |Q| X i=1 1 rank i (5) 2. Recall@k (R@k): This metric measures the proportion of queries for which at least one relevant document/image is found within the topkretrieved results.Given our as- sumption that a single ”golden truth” exists for each query in our dataset, this metric reflects the probability that the ground-truth item is captured within the top k results: Recall@k = 1 |Q| |Q| X i=1 I(rank i ≤ k)(6) whereI(·)is the indicator function. We report results us- ingk ∈ 1, 3, 5, 10for our financial document datasets but usingk ∈ 10, 15, 20for MMDocRAG to follow its metrics. B.2. Object Detection (T OD ) The performance of object detection is measured by the spatial overlap between the predicted bounding box and the ground truth, as well as the inference efficiency. 13 Self Knowledge Re-expression: A Fully Local Method for Adapting LLMs to Tasks Using Intrinsic Knowledge 1. Intersection over Union (IoU): IoU quantifies the over- lap between the predicted bounding boxB p and the ground- truth bounding boxB gt . It is calculated as the area of their intersection divided by the area of their union: IoU = Area(B p ∩ B gt ) Area(B p ∪ B gt ) (7) We report the mean IoU across all test samples. 2. Inference Time (Time): We record the total running time required to process the test set and report it as minutes per 100 test cases. This metric highlights the efficiency gains of the task-specific head (E T ) compared to sequential token generation (E ntp ). B.3. Anomaly Detection (T AD ) For the binary classification-based anomaly detection task, we evaluate both threshold-dependent and threshold- independent metrics to account for the class imbalance. 1. Accuracy (Acc): The ratio of correctly predicted samples (both normal and anomalous) to the total number of samples: Acc = TP + TN TP + TN + FP + FN (8) whereTP, TN, FP, FNrepresent true positives, true nega- tives, false positives, and false negatives, respectively. 2. AUROC: The Area Under the Receiver Operating Characteristic curve.It plots the True Positive Rate (TPR = TP TP + FN ) against the False Positive Rate (FPR = FP FP + TN ) at various threshold settings. It measures the probability that a randomly chosen positive instance is ranked higher than a randomly chosen negative one. 3. AUPRC: The Area Under the Precision-Recall Curve. Precision (P = TP TP + FP ) is plotted against Recall (R = TP TP + FN ). AUPRC is a more robust metric for the highly imbalanced datasets used in our anomaly detection scenar- ios, as it does not reward high numbers of True Negatives. C. Implementation Details We introduce details of the self-annotation process, the fine-tuning settings, baseline settings, and various hyper- parameters for different tasks and different LLMs in this section. C.1. Self-Annotation for Information Retrieval (T IR ) The self-annotation process includes using the LLM to gen- erate text relevant to the input document or image. In this framework, we prompt the LLM to generate a specific query that acts as a semantic anchor. The prompt for text-to-text retrieval is: Prompt for Text-to-text Retrieval Self-Annotation Document: <doc> Based on the above document, generate one question that can be answered using the information provided. The prompt for text-to-image retrieval is: Prompt for Text-to-image Retrieval Self-Annotation <image title information> Based on the above image, generate one question that can be answered using the information provided. Since the required annotations are direct, input-related tex- tual queries, no additional post-processing is required for this task. C.2. Self-Annotation for Object Detection (T OD ) Due to the significant variance in the native capabilities of different models regardingT OD , we use three distinct self- annotation strategies: Direct Ask, Fuzzy Location, and the OCR-supported method. C.2.1. DIRECT ASK This method is applied to Qwen2-VL, as the base model possesses inherent grounding capabilities and can output formalized coordinates. We evaluate this model’s potential using both one-turn and multi-turn prompting strategies. The one-turn prompt is: Prompt for One-turn Object Detection You are given a screenshot of a document page containing a target object: either a chart, table, or figure. Your task is to identify the bounding box of the whole object, including its title, and the text underlying the source of this object. Output normalized coordinates of the bounding box relative to the image size. The multi-turn prompts decompose the task into identifying the main object, its caption, and its source individually: 14 Self Knowledge Re-expression: A Fully Local Method for Adapting LLMs to Tasks Using Intrinsic Knowledge Prompts for Multi-turn Object Detection You are given a screenshot of a document page containing a target object: either a chart, table, or figure. Your task is to identify the bounding box of the whole object. Output normalized coordinates of the bounding box relative to the image size. You are given a screenshot of a document page containing a target object: either a chart, table, or figure. Your task is to identify the bounding box of this object’s caption Output normalized coordinates of the bounding box relative to the image size. You are given a screenshot of a document page containing a target object: either a chart, table, or figure. Your task is to identify the bounding box of the text underlying the source of this object. Output normalized coordinates of the bounding box relative to the image size. All outputs will be processed as four numerical numbers to represent the bounding boxes. For one-turn prompts, the output is used directly as the self-annotation. For multi-turn prompts, we calculate the minimum bounding rectangle that encompasses all three predicted boxes to form the final annotation. C.2.2. FUZZY LOCATION Since LLaVA-1.6 and Phi-3-VL lack native grounding abili- ties, we implement a bisection-based post-processing strat- egy to convert semantic understanding into spatial coordi- nates. We first prompt the model to describe the object: Prompt for Object Description You are given a screenshot of a document page which may contain a target object: either a chart, table, or figure. Briefly describe this object in no more than five words. We apply this query to the original image and store the description. We then iteratively crop the image (e.g., moving the bottom boundary to half the image height) and re-query the model. If the semantic similarity between the new and original description exceeds 60%, we assume the object remains largely within the crop and continue the bisection. If similarity drops, we backtrack. This process repeats for all four boundaries until the boundary shift is less than 30 pixels, resulting in a ”fuzzy” bounding box. C.2.3. OCR-SUPPORTED METHOD To enhance the accuracy of annotations for weaker models, we utilize pytesseract, a classic non-deep-learning OCR package. This method provides coordinate support while remaining computationally much lighter than model-based approaches. We prompt the model to extract surrounding text like: Prompt for Surrounding Text Extraction You are given a screenshot of a document page containing a target object: either a chart, table, or figure. Output the title of this object. The title of this object is: We then use the OCR engine to locate the output text on the page. By combining the bounding boxes of the extracted text anchors, we derive the smallest rectangle that covers the target area as the self-annotation. C.3. Self-Annotation for Anomaly Detection (T AD ) For anomaly detection, we provide the model with an image- query pair and use the following prompt: Prompt for Surrounding Text Extraction Based on the image above, determine whether the following question can be answered using the information provided in the image. Respond with ’yes’ or ’no’. Question: <question>. The textual responses are subsequently mapped to binary class labels (0 or 1) to form the training set. C.4. Fine-tuning Details The fine-tuning process uses task-specific output layers and specialized objective functions. ForT IR , we bypass the base model’s token prediction layer and use the hidden state of the final token from the last layer as the dense representation. The model is fine-tuned using a standard contrastive learning objective. ForT OD , we attach a regression head, a three- layer MLP, to the final hidden state of LLMs. The hidden size of the regression head is 1,024, and the final layer outputs 4 dimensions followed by a Sigmoid function to normalize coordinates between 0 and 1. We useL 1 loss for fine-tuning. ForT AD , we add a classification head (three- layer MLP, hidden size 1,024). The output is a single logit used for binary classification, optimized via binary cross- entropy loss. All training sets are split into training and validation data at a 9:1 ratio. We use LoRA to adapt the models, adding 15 Self Knowledge Re-expression: A Fully Local Method for Adapting LLMs to Tasks Using Intrinsic Knowledge Table 4.Performance comparison on MMDocRAG’s information retrieval task. SKR’s results are highlighted in bold. Across all models and modalities, SKR consistently achieves the best perfor- mance. Qwen2 models adapted via SKR outperform specialized industrial baselines. ModelR@10R@15R@20 DSE wiki−s 0.6710.7530.799 DSE docmatix 0.6570.7500.782 ColPali0.6820.7750.812 ColQwen0.7080.7920.843 LLaVA-1.6 Base0.5270.6340.705 SKR0.8340.8860.920 Phi-3-VL Base0.6070.7030.768 SKR0.8420.8910.923 Qwen2-VL Base0.5780.6870.749 SKR0.8770.9160.944 trainable parameters to the query, key, value, and output projections (W q ,W k ,W v ,W o ) of all attention layers. The rank (r) is set to 8 and lora alpha (α) to 16. The total trainable parameters account for less than 0.5% of the total model size. The training configurations are tailored to the specific requirements of each task. ForT IR , the model is trained for 5 epochs with a batch size of 4, whileT OD utilizes a batch size of 4 over 20 epochs. In the case ofT AD , the training duration is extended to 50 epochs with a batch size of 4. Across all three tasks, the models are optimized using the Adam optimizer (Kingma & Ba, 2015) with a consistent learning rate of 1× 10 −4 . D. Results on Open-source datasets D.1. IR results on MMDocRAG, DocVQA-2020, and SciMMIR Table 4 presents the performance of SKR on the open-source MMDocRAG benchmark. To ensure a fair comparison, we use the reported results of the four top-performing models in text-to-image retrieval from their original publications. Our experimental setup strictly adheres to their established evaluation protocol and metrics. These baselines are leading retrieval models, including DSE (Ma et al., 2024), built on the Phi-3-Vision architecture, and ColQwen (Faysse et al., 2025), which uses the Qwen2-VL backbone. The results demonstrate that while specialized fine-tuning (as seen in DSE and ColQwen) provides a performance boost over base MLLMs, SKR consistently outperforms all industrial baselines across the evaluated architectures. Notably, SKR-adapted models achieve at least a 12.6% abso- lute improvement in Recall@10 compared to the strongest industrial baseline, ColQwen. Furthermore, Table 5 highlights the results of Qwen-SKR Table 5.Performance comparison on the DocVQA-2020 and SciM- MIR datasets. SKR’s results are highlighted in bold. ModelMRRR@1R@3R@5R@10 DocVQA-2020 Nomic0.8480.754 0.931 0.9700.994 Qwen-SKR0.9310.882 0.976 0.9930.999 SciMMIR Nomic0.3260.283 0.440 0.4910.531 Qwen-SKR0.5620.467 0.622 0.6750.732 (fine-tuned exclusively on MMDocRAG) when evaluated on the DocVQA-2020 and SciMMIR test sets. Despite having no exposure to these datasets during the SKR process, our model significantly outperforms the Nomic baseline. This demonstrates SKR’s exceptional cross-dataset generaliza- tion for similar tasks. The fact that SKR, using only unannotated images, surpasses models trained on massive, annotated retrieval datasets proves that the expression bottleneck is the primary limiting factor in current multi-modal RAG systems. By transition- ing to a task-optimized expression mechanism, SKR allows the model to use its internal knowledge far more effectively in specific tasks. D.2. OD and AD results on CUB 2002011 Tables 6 and 7 present the performance of SKR on the open-source CUB2002011 benchmark for OD and AD tasks, respectively. ForT OD , the IoU improved across all models, with the runtime being reduced by at least 88%. This performance trend on publicly available data directly mirrors the efficiency gains and accuracy improvements observed on our specialized financial dataset. ForT AD , consistent with the findings on our financial dataset, accuracy improvements are marginal due to the extreme class imbalance inherent in the task. However, the AUROC and AUPRC scores, which are more diagnostic of anomaly Table 6.Performance comparison onT OD . Results for the SKR method are highlighted in bold. Runtime is reported in minutes per 100 test cases. ModelIoUTime Qwen Base0.6872.7 Qwen SKR0.6920.3 LLaVA Base0.27520.2 LLaVA SKR0.3110.4 Phi3 Base0.23719.3 Phi3 SKR0.2660.4 16 Self Knowledge Re-expression: A Fully Local Method for Adapting LLMs to Tasks Using Intrinsic Knowledge Table 7.Performance comparison onT AD . SKR results are high- lighted in bold. Runtime is reported in minutes per 100 test cases. ModelAcc↑AUROCAUPRCTime Qwen Base0.9850.8570.4710.8 Qwen SKR0.9860.8700.5450.3 LLaVA Base0.9650.8400.2462.2 LLaVA SKR0.9660.9180.4420.3 Phi3 Base0.9710.8630.3002.7 Phi3 SKR0.9800.8820.4760.3 detection performance, show significant gains across all models. These results on publicly available data further confirm that SKR is highly effective in both tasks, significantly improv- ing performance while reducing runtime. E. Impact of Model Scaling To investigate the relationship between model scale and the effectiveness of SKR, we evaluate the Gemma-3 model fam- ily (Team et al., 2025) across three parameter scales: 4b, 12b, and 27b. We specifically use the Gemma-3 family for this study rather than the models evaluated in our primary experiments to ensure a controlled comparison. While the previous evaluated models (e.g., Qwen, Phi, LLaVA) exhibit architectural and generational variances that could confound results, the Gemma-3 family provides a homogeneous en- vironment where model size is the primary variable. This allows us to more rigorously isolate the impact of parameter scale on the SKR process. The results, summarized in Table 8, provide insights into how model size influences both the quality of intrinsic knowledge and the ultimate ceiling for task adaptation. In addition to the standard SKR pipeline, we include two cross- model supervision variants: (1) 27b-ft, where the 4b model is fine-tuned using annotations from the 27b model, and (2) 4b-ft, where the 27b model is fine-tuned using annotations from the 4b model. Our results indicate that model size positively correlates with both the base model knowledge level and the perfor- mance of SKR-adapted models. The 12B and 27B versions outperform the 4B model, suggesting that larger architec- tures are more adept at internalizing and re-expressing infor- mation. However, this benefit exhibits asymptotic behavior; the performance gap between the 12B and 27B models is marginal, indicating a potential upper limit for the SKR process on this specific T IR task. The performance gains observed in larger models can be attributed to two distinct factors: annotation quality (the Table 8.Performance comparison onT IR for Gemma-3 models. SKR’s results are highlighted in bold. The lines “27b-ft” and “4b- ft” denote models fine-tuned using annotations generated by the Gemma-3-27b-it and Gemma-3-4b-it models, respectively. ModelMRRR@1R@3R@5R@10 Gemma-3 4b-it Base0.0030.0000.0000.0000.000 E5-V0.2220.1650.2500.2750.330 SKR0.4150.3450.4150.4850.590 27b-ft0.5010.4100.5400.5950.660 Gemma-3 12b-it Base0.0250.0100.0200.0250.045 E5-V0.2930.2300.3050.3550.425 SKR0.5260.4200.5800.6350.736 Gemma-3 27b-it Base0.0510.0350.0500.0700.080 E5-V0.0470.0250.0600.0650.080 SKR0.5310.4400.5800.6250.715 4b-ft0.4720.3750.5350.5800.635 quality of self-generated annotations) and expressive ca- pacity (the model’s inherent ability to be adapted). The cross-model supervision experiments provide a clear decou- pling of these factors: The 4b model’s performance increases significantly when fine-tuned on 27b-generated labels (27b-ft) compared to its own self-annotations. Conversely, the 27b model’s perfor- mance drops when using 4b-generated labels (4b-ft). This confirms that larger models possess superior intrinsic knowl- edge, leading to higher-quality self-annotation. Even when provided with high-quality 27b annotations, the 4b model still fails to match the performance of the 12b or 27b SKR-adapted models. Furthermore, the 27b model fine-tuned on 4b annotations (4b-ft) still outperforms the 4b model’s best SKR results. This suggests that larger models have a higher architectural ceiling for re-expression, a more potential latent space that responds better to task-specific adaptation. In summary, model size affects both the model’s intrinsic knowledge and its potential for re-expression. Larger mod- els not only generate superior self-annotations but also offer more potential for task-specific improvement. However, these gains do not scale indefinitely. Our findings suggest that we should calibrate model size to task difficulty, select- ing the most efficient architecture that meets the required performance threshold. F. Ablation Study on LoRA Adapters The SKR framework is designed as a knowledge re- expression process rather than a mere formatting alignment for specific tasks. The LoRA adapters are essential to har- monize the model’s internal parametric knowledge with the newly adapted output mechanism. To validate this design, we conducted ablation experiments comparing the full SKR 17 Self Knowledge Re-expression: A Fully Local Method for Adapting LLMs to Tasks Using Intrinsic Knowledge 0510152025 Layer 0.4 0.6 0.8 1.0 Similarity QwenVL Base vs. QwenVL E5V QwenVL Base vs. QwenVL SKR QwenVL E5V vs. QwenVL SKR (a)Layer-wise CKA similarity across methods for Qwen2-VL. 051015202530 Layer 0.4 0.6 0.8 1.0 Similarity Llava Base vs. Llava E5V Llava Base vs. Llava SKR Llava E5V vs. Llava SKR (b)Layer-wise CKA similarity across methods for LLaVA-1.6. 05101520 Layer 0.4 0.6 0.8 1.0 Similarity QwenVL Base vs. Llava Base QwenVL E5V vs. Llava E5V QwenVL SKR vs. Llava SKR (c)Layer-wise CKA similarity be- tween Qwen2-VL and LLaVA-1.6 under the same method. Figure 4. Layer-wise representation similarity (CKA) under different settings. Table 9. Performance comparison onT IR on the MMDocRAG dataset. ModelR@10R@15R@20 Qwen Base0.5780.6870.749 Qwen Head-only0.6520.7350.797 Qwen SKR0.8770.9160.944 framework against a ”Head-only” baseline. In the Head- only configuration, we implement the task-specific head and loss functions but keep the backbone parameters frozen, bypassing the parameter adaptation phase. Tables 9, 10, and 11 summarize these comparative results across retrieval, detection, and anomaly detection tasks. The results clearly demonstrate that modifying the output head alone is insufficient for effective task adaptation. In T IR , the R@10 gain achieved by SKR (0.299) is over 400% greater than that of the Head-only baseline (0.074). InT OD , the Head-only approach significantly degrades performance, with IoU dropping to 0.491, whereas SKR maintains and slightly improves upon the base model’s accuracy while dramatically increasing efficiency. InT AD , while the Head- only model shows a marginal increase in AUPRC (0.021), it remains substantially lower than the improvement provided by SKR (0.074). Collectively, these findings confirm that our performance gains are not a trivial byproduct of using task heads. Instead, they stem from the SKR’s unique ability to effectively re- align latent representations with a new expression paradigm. This proves that for successful zero-shot adaptation, internal knowledge re-expression is a fundamental necessity. Table 10. Performance comparison on T OD on the CUB dataset. ModelIoU Qwen Base0.687 Qwen Head-only0.491 Qwen SKR0.692 Table 11. Performance comparison on T AD on the CUB dataset. ModelAccAUROCAUPRC Qwen Base0.9850.8570.471 Qwen Head-only0.9800.8560.492 Qwen SKR0.9860.8700.545 G. CKA Analysis We analyze representation similarity across layers, meth- ods, and architectures to gain deeper insights into how task- specific capabilities emerge within the model’s latent space. We use Center Kernel Alignment (CKA (Kornblith et al., 2019)) to quantify these similarities. CKA measures the alignment between representations in a kernel space, provid- ing a robust metric that is invariant to isotropic scaling and orthogonal transformations. This method is widely adopted for studying the internal representational characteristics of LLMs (Zhao et al., 2024). Figures 4(a) and 4(b) illustrate the layer-wise CKA simi- larity for Qwen2-VL and LLaVA-1.6 when comparing dif- ferent adaptation methods. Several key patterns emerge. In both architectures, representations remain highly simi- lar in the early layers but diverge significantly as the data progresses into deeper layers. The degree of divergence is closely tied to the performance gap between methods. Because the SKR-adapted models achieve the highest per- formance, their similarity to the original base models is the lowest, dropping sharply after the midpoint layers. In- stead, the similarity between E5-V and SKR (green lines) remains relatively high compared to their similarity to the Base model. This suggests that for a given model, differ- ent task-adaptation methods shift internal representations in a consistent direction toward task-specific utility. The dramatic drop in similarity in the deepest layers, those im- mediately preceding the output head, confirms that SKR fun- damentally reconfigures the model’s outputs to overcome the default NTP bottleneck. 18 Self Knowledge Re-expression: A Fully Local Method for Adapting LLMs to Tasks Using Intrinsic Knowledge Then Figure 4(c) provides a joint comparison between the two different base architectures (Qwen2-VL vs. LLaVA- 1.6). As the two models differ in depth, we align them by selecting the last 20 layers for comparison. Even when both models are adapted for the same task using the same SKR method, their representational similarity consistently remains below 0.8. While both E5-V and SKR adaptation drive the models toward a shared task-oriented objective, they do not erase the fundamental difference of the base model’s parametric knowledge. This underscores the core premise of SKR: it optimizes the expression of a model’s existing intrinsic knowledge rather than forcing diverse mod- els into a single, homogenized state. The models reach the same high-performance goal while maintaining their unique internal representational knowledge. 19

Instant research discovery