Paper deep dive
Find Before You Fine-Tune: A Diagnostic Study of Small LLMs for Cybersecurity QA
Shaswata Mitra, Subash Neupane, Trisha Chakraborty, Himanshu Tripathi, Sudip Mittal, Aritran Piplai, Shahram Rahimi
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 91%
Last extracted: 7/22/2026, 3:03:00 AM
Summary
The paper introduces FiT (Find before Fine-Tune), a diagnostic framework for evaluating small Large Language Models (LLMs) for cybersecurity Question-Answering (QA). It assesses three capabilities: vocabulary recognition, parametric knowledge, and contextualization of retrieved information. The study empirically evaluates five 7-billion-parameter open-weight models under knowledge-focused and instruction-focused fine-tuning regimes. Results indicate that fine-tuning does not uniformly improve performance; it often degrades vocabulary and parametric knowledge. Specifically, knowledge-focused tuning causes moderate degradation, while instruction-focused tuning leads to induced abstention, collapsing knowledge rankings. Pre-fine-tuning FiT scores are shown to anticipate post-tuning behavioral changes, suggesting the framework can help screen unsuitable models and avoid unnecessary fine-tuning.
Entities (16)
Relation Signals (15)
FIT → assesses → Parametric Knowledge
confidence 95% · FiT decomposes suitability into three aspects (Vocabulary, Knowledge, Contextualization)
FIT → assesses → Contextualization
confidence 95% · FiT decomposes suitability into three aspects (Vocabulary, Knowledge, Contextualization)
FIT → assesses → Vocabulary Recognition
confidence 95% · FiT decomposes suitability into three aspects (Vocabulary, Knowledge, Contextualization)
Fine-tuning → degrades → Vocabulary Recognition
confidence 90% · fine-tuning does not uniformly help: it consistently degrades vocabulary and parametric knowledge in small models
Fine-tuning → degrades → Parametric Knowledge
confidence 90% · fine-tuning does not uniformly help: it consistently degrades vocabulary and parametric knowledge in small models
Llama-2-7b → evaluatedby → FIT
confidence 90% · We evaluate five open-weight 7-billion-parameter... models: Llama-2-7b
WestSeverus-7b → evaluatedby → FIT
confidence 90% · WestSeverus-7b
Mistral-7B → evaluatedby → FIT
confidence 90% · Mistral-7b
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Large Language Models (LLMs) are increasingly fine-tuned for critical-domain Question-Answering (QA), yet choosing which small model to adapt, before paying the cost of adaptation, remains difficult. Fine-tuning can improve domain alignment, but it may also erode prior knowledge, weaken instruction-following, or increase hallucination, especially when labeled data are scarce or rapidly evolving as in cybersecurity. We present FiT (Find before Fine-Tune), a task-oriented diagnostic framework that characterizes small LLMs along three capabilities required for cybersecurity QA: vocabulary recognition, parametric knowledge, and contextualization of retrieved information. Using FiT, we conduct an empirical study of five open-weight 7-billion-parameter models under two fine-tuning regimes. We find that fine-tuning does not uniformly help: it consistently degrades vocabulary and parametric knowledge in small models, and the two regimes trade off differently. Knowledge-focused tuning causes moderate, rank-preserving degradation, whereas instruction-focused tuning collapses measured knowledge through induced abstention, inverting the knowledge ranking while leaving retrieval-grounded contextualization essentially intact. We quantify these regime-specific patterns with rank-correlation analysis and show that pre-fine-tuning FiT scores anticipate the direction of post-tuning change. Our results suggest that task-oriented diagnosis can screen out unsuitable models, avoid unnecessary fine-tuning, and support safer deployment of small LLMs in cybersecurity QA pipelines.
Tags
Links
- Source: https://arxiv.org/abs/2607.18725v1
- Canonical: https://arxiv.org/abs/2607.18725v1
Trouble viewing inline? Open PDF directly →
Full Text
38,361 characters extracted from source content.
Expand or collapse full text
Find Before You Fine-Tune: A Diagnostic Study of Small LLMs for Cybersecurity QA Shaswata Mitra1, Subash Neupane2, Trisha Chakraborty3, Himanshu Tripathi4, Sudip Mittal5, Aritran Piplai7, Shahram Rahimi6 1456The University of Alabama – 1smitra3, 4htripathi, 5sudip.mittal, 6shahram.rahimi@ua.edu 2Meharry Medical College – 2subash.neupane@mmc.edu 3Mississippi State University – 3tc2006@msstate.edu 7The University of Texas at El Paso – 7apiplai@utep.edu Abstract Large Language Models (LLMs) are increasingly fine-tuned for critical-domain Question-Answering (QA), yet choosing which small model to adapt, before paying the cost of adaptation, remains difficult. Fine-tuning can improve domain alignment, but it may also erode prior knowledge, weaken instruction-following, or increase hallucination, especially when labeled data are scarce or rapidly evolving as in cybersecurity. We present FiT (Find before Fine-Tune), a task-oriented diagnostic framework that characterizes small LLMs along three capabilities required for cybersecurity QA: vocabulary recognition, parametric knowledge, and contextualization of retrieved information. Using FiT, we conduct an empirical study of five open-weight 7-billion-parameter models under two fine-tuning regimes. We find that fine-tuning does not uniformly help: it consistently degrades vocabulary and parametric knowledge in small models, and the two regimes trade off differently. Knowledge-focused tuning causes moderate, rank-preserving degradation, whereas instruction-focused tuning collapses measured knowledge through induced abstention, inverting the knowledge ranking while leaving retrieval-grounded contextualization essentially intact. We quantify these regime-specific patterns with rank-correlation analysis and show that pre-fine-tuning FiT scores anticipate the direction of post-tuning change. Our results suggest that task-oriented diagnosis can screen out unsuitable models, avoid unnecessary fine-tuning, and support safer deployment of small LLMs in cybersecurity QA pipelines. I Introduction Large Language Models (LLMs) have produced remarkable advances in Natural Language Processing (NLP), demonstrating unprecedented language understanding and generation capabilities. Organizations across many industries now deploy LLMs as domain-specific Question Answering (QA) systems. In cybersecurity, a field tied to critical infrastructure and national security, incorrect model responses can trigger attacks, facilitate breaches, or expose sensitive information, making reliable deployment especially consequential [5]. Effective cybersecurity QA requires a model to possess domain vocabulary, specialized knowledge, and the ability to synthesize heterogeneous retrieved information. Consider the query: “Can the LangChain vulnerability affect my system?” Answering correctly demands both general knowledge of LangChain vulnerability classes and context-specific awareness of the user’s deployed version. These two information types are qualitatively different, yet both are necessary for a correct, non-leaking response. Because cybersecurity knowledge evolves continuously and labeled data are scarce, practitioners frequently couple a small, deployable LLM with Retrieval-Augmented Generation (RAG) [13] and, where data permit, fine-tune the model to the deployment scope. This raises a practical question that precedes deployment: which small model should be selected, and whether fine-tuning will help at all. Answering empirically by fine-tuning every candidate is expensive in both compute and time. We therefore ask whether a lightweight, pre-adaptation diagnosis can guide this choice. Figure 1: Implementation scope of FiT in the generative-AI life-cycle (colored area implies applicability) with an example of our three evaluation tasks. By aligning the tasks in a complete process, we visualize the propagation of the domain suitability required to generate Completion (C) for a given Prompt (P). We introduce FiT, a task-oriented diagnostic framework that characterizes a small LLM’s domain understanding and contextualization ability for knowledge-intensive QA. Whereas existing cybersecurity benchmarks such as CYBERBENCH [14] and SecureBERT [1] report static, pre-trained performance, to the best of our knowledge FiT is the first framework to diagnose a small LLM’s cybersecurity QA suitability before adaptation and to characterize how fine-tuning reshapes that suitability. Rather than positing FiT as a general predictive benchmark, we use it as a lens to study how small models behave before and after two distinct fine-tuning regimes, so that organizations can screen candidates and anticipate the direction of post-tuning change. FiT decomposes suitability into three aspects (Vocabulary, Knowledge, Contextualization), and we instantiate the study in cybersecurity111Code and data: github.com/shaswata09/FiT. Concretely, we investigate three research questions: • RQ1. Can a lightweight, pre-adaptation diagnosis characterize a small LLM’s suitability for cybersecurity QA along vocabulary, knowledge, and contextualization, without the cost of fine-tuning each candidate? • RQ2. How do knowledge-focused and instruction-focused fine-tuning regimes alter these capabilities in small LLMs, and do they help uniformly? • RQ3. Do pre-fine-tuning FiT scores anticipate post-fine-tuning behavior closely enough to guide model selection and avoid unnecessary tuning? In addressing the above research questions (RQs), we make the following contributions: • Addressing RQ1, we propose FiT, a diagnostic decomposition of cybersecurity QA suitability into vocabulary, knowledge, and contextualization, with a metric for each. • Addressing RQ2, we present an empirical study of how knowledge-focused and instruction-focused fine-tuning trade off these capabilities in small (7B) LLMs, including an abstention-driven inversion of knowledge rankings that we quantify with rank-correlation analysis. • Addressing RQ3, we show that retrieval-grounded contextualization is robust to fine-tuning, and derive practical guidance for model selection in dynamic, low-data domains such as cybersecurity. The rest of the paper is organized as follows. Section I formulates the problem. Section I reviews background and related work. Section IV details the FiT tasks. Sections V and VI present the experiment, findings, and limitations, followed by concluding remarks. I Problem Formulation In this section we define the problem and its foundations. We first describe the implementation scope, then the evaluation tasks; Table I summarizes the notation used throughout. Fig. 1 provides a visual reference for the scope and tasks. TABLE I: Description of Notations. Notation Description P User Input Prompt d∈\N^d \ Domain-specific Vocabulary d∈\K^d \ Domain-specific Knowledge id∣id∈d\G_i^d _i^d ^d\ Domain Information for P id∣id∈d\S_i^d _i^d ^d\ Specific Information for id∪G_i^d C Completion for P given (id∪id)∣d(G_i^d _i^d) ^d id¯,i¯ N_i^d,\ C_i Ground-truth (expected) outputs ℒL LLM under evaluation Φ(⋅∣) (· ) Task suitability score over P In a typical knowledge-intensive, critical-domain QA task using an LLM with RAG, the objective is to generate a relevant completion (C) for a given prompt (P) without disclosing sensitive information. Irrespective of the deployment domain, two types of information are primarily required to generate C. One is domain-specific information (idG_i^d) relevant to P; the other is contextual or specific information (idS_i^d) needed to contextualize idG_i^d for P. The LLM then combines its domain vocabulary (dN^d) and knowledge (dK^d) to produce C. To assess an LLM’s contextualization ability within this scope, we adopt a process-oriented decomposition into three tasks, each addressing a distinct capability: 1. Vocabulary. We assess familiarity with domain vocabulary via a keyword-recognition task that instructs the LLM to identify important keywords (idN_i^d) in P. 2. Knowledge. We assess domain knowledge via a multiple-choice QA task probing the LLM’s domain understanding (dK^d). 3. Contextualization. We assess whether the LLM can comprehend and tailor idG_i^d in light of idS_i^d to generate C for P, without leaking unnecessary information. This decomposition lets us characterize an LLM’s suitability for critical-domain QA from a relevancy and reliability standpoint. We further analyze how each capability changes after fine-tuning, in order to identify systematic patterns of behavioral change. As we show, these patterns are regime-dependent, and understanding them helps practitioners decide whether and how to fine-tune a given model and curate data accordingly. I Preliminaries The application of pre-trained LLMs in specialized domains has been an active research area [20]. We briefly review the prerequisite background and related developments. I-A LLM, RAG, and Fine-tuning LLMs have advanced NLP through transformer architectures [24] that offer remarkable parallelization [17]. Pre-trained on massive Internet text with large parameter counts, these models exhibit strong learning capabilities, yet they can produce plausible-but-inaccurate predictions and struggle on problems requiring specialized domain knowledge. Reported reasons [25] for the failure of general-purpose LLMs in closed domains include a deficit in domain knowledge (lack of exposure to a specialized domain), outdated information (a training cutoff that omits post-training developments), and forgetting (catastrophic forgetting [11] during additional training, where prior knowledge is lost). To mitigate knowledge deficiency for domain-specific tasks, an additional knowledge-ingestion step is required. The two most common approaches are Retrieval-Augmented Generation (RAG) and fine-tuning. RAG, introduced by Lewis et al. [13], leverages an external knowledge base (a document corpus, a structured database, or any source of domain information) to overcome the knowledge limitations of pre-trained LLMs. Given an input query, the RAG architecture retrieves the most relevant passages and integrates them into the input, supplying the LLM with additional context. Figure 2: Flowchart of the FiT evaluation process. The three tasks align with the deployment scope: Task 1 assesses vocabulary, Task 2 assesses domain knowledge, and Task 3 assesses contextualization for relevant, reliable generation. As models grow in size, updating all parameters becomes computationally demanding and cost-prohibitive, particularly under limited hardware. This motivates parameter-efficient tuning methods that target strong task performance while minimizing the number of tunable parameters. Representative efforts include adapter-based [10], prompt-based [12], LoRA [23], QLoRA [6], and hybrid [15] approaches. I-B LLM Benchmarking Numerous frameworks evaluate general and domain-specific language models across NLP tasks. Some, such as GLUE-X [27] and PromptBench [29], assess general capabilities including robustness to out-of-distribution and adversarial inputs, while KOLA [28] evaluates knowledge and reasoning. Domain-specific benchmarks also exist: MultiMedQA targets medical QA, and MATH [22] evaluates mathematical reasoning. In cybersecurity, multi-task benchmarks include CyberBench [14] and SecureBERT [1] for sentiment analysis and NER. These efforts largely report static, pre-trained performance. FiT is complementary: rather than ranking models on a fixed leaderboard, it decomposes suitability into process-aligned capabilities and uses them to study how those capabilities shift under fine-tuning, an aspect existing cybersecurity benchmarks do not address. IV FiT Framework We now describe the three FiT tasks used to characterize a small LLM’s suitability for domain-specific QA with RAG, exemplified in Fig. 2. IV-A Task 1: Vocabulary Assessment Ensuring a model understands domain vocabulary is a prerequisite for domain-specific QA: it lets the model comprehend the input and respond in compatible terms. We assess this via a Keyword Recognition (KR) task, an NLP task that identifies important entities in unstructured text. The LLM is instructed to identify the keywords in the input prompt (P). This probes two aspects: the number of correct keyword identifications (a proxy for domain-oriented linguistic understanding, since vocabulary differs sharply across domains, e.g., medical versus cybersecurity), and recognition of instruction-relevant terms (a proxy for understanding the task itself). Formally, let =ii=1nP=\P_i\_i=1^n be the set of prompts, d the domain, and dN^d the domain vocabulary. For each iP_i there exists a gold keyword set id¯ N_i^d. The vocabulary score Φ(d∣) (N^d ) is Φ(d∣)=1n∑i=1nΔ(ℒ(i),id¯) (N^d )= 1n _i=1^n \! (L(P_i),\, N_i^d ) (1) where Δ(⋅,⋅) (·,·) is the F1F_1 score between the predicted and gold keyword sets, ℒL is the model, and n>0n>0. Task-1: Vocabulary Assessment (dN^d) Instruction: Print the keywords from the following … iP_i: Can langchain vulnerability affect my system? id¯ N_i^d: langchain, vulnerability, affect, system IV-B Task 2: Knowledge Analysis Precisely defining “knowledge” is a philosophical question beyond our scope; we instead quantify domain knowledge through a multiple-choice question-answering (MCQ) task. If a model comprehends the question, it can select the correct option, providing a measurable proxy for parametric knowledge. Formally, let =ii=1nP=\P_i\_i=1^n be a set of MCQ problems, each with m options, and let γi _i be the correct answer for iP_i. Writing a^i=ℒ(i) a_i=L(P_i) for the model’s selected option and [⋅]1[·] for the indicator function, the knowledge score is the accuracy Φ(d∣)=1n∑i=1n[a^i=γi], (K^d )= 1n _i=1^n1\! [ a_i= _i ], (2) where a^i a_i is the model’s selected option, γi _i the correct option, m the number of choices per item, and n>0n>0. We note in advance that this score conflates two distinct behaviors when a model declines to answer: a wrong selection and an abstention both reduce accuracy. We exploit this distinction in Section VI. Task-2: Knowledge Analysis (dK^d) Instruction: Answer the correct choice for the question … iP_i: A hash function guarantees the integrity of a message. It guarantees that the message has not been 1: Replaced 2: Overviewed 3: Changed 4: Violated γi _i: (3) Changed IV-C Task 3: Contextualization Analysis Contextualization is the ability to understand and generate text based on the relationships among multi-faceted information. For example, “What potential impact could CVE-2023-3894 have on the integrity of our TOML configuration data?” requires jointly reasoning over the CVE and the referenced TOML server. In knowledge-intensive QA, RAG bridges the model’s knowledge deficiency; contextualization lets the model produce relevant, reliable answers from retrieved multi-faceted information while following instructions. This ability is critical: in scenarios involving recommendation or privacy, domain information must be tailored to the specifics of the request, and without accurate contextualization a model may provide misinformation or surface inappropriate detail. We therefore adopt contextualized RAG [9] as our final task. Comparing the generated response with a Subject-Matter-Expert (SME) reference, we report two RAGAS metrics, answer correctness and semantic similarity, to assess, respectively, how factually and contextually correct the answer is and how closely it tracks the expert reference. Formally, let =ii=1nP=\P_i\_i=1^n be domain questions, idG_i^d the domain information and idS_i^d the specific information for iP_i, and i¯ C_i the expert answer. The contextualization score is Φ(∣)=1n∑i=1nΩ(ℒ(i∣id∪id),i¯) (C )= 1n _i=1^n \! (L(P_i _i^d _i^d),\, C_i ) (3) where Ω(⋅,⋅) (·,·) is the RAGAS scoring function (answer correctness or semantic similarity), ℒ(i∣id∪id)L(P_i _i^d _i^d) is the model’s completion conditioned on the retrieved context, and n>0n>0. We are careful not to interpret semantic similarity as a direct measure of information leakage; it captures fidelity to the expert reference, and we treat it as a reliability signal rather than a security guarantee (see Section VI). Task-3: Contextualization (C) Instructions: Given the following retrieved knowledge, answer … idG_i^d: CVE ID: CVE-2023-36189 Description: SQL injection vulnerability in langchain before v0.0.247 allows a remote attacker to obtain … CVE ID: CVE-2023-36188 Description: An issue in langchain v.0.0.64 allows a remote attacker to execute arbitrary code via the PALChain … idS_i^d: ‘langchain==0.0.270‘: Used for generation of document embeddings for text, enhancing information retrieval capabilities … iP_i: Can langchain vulnerability affect my system? i¯ C_i: Langchain has multiple reported vulnerabilities … your system is running langchain version 0.0.270, which is not mentioned to be vulnerable in the provided information… V Experiment & Evaluation We describe the dataset, infrastructure, and evaluation protocol. Cybersecurity, an information-critical domain, serves as our case study. TABLE I: FiT results for pre-trained, knowledge-focused (Finetuned-1), and instruction-focused (Finetuned-2) models. Task 1: keyword-recognition F1F_1; Task 2: MCQ accuracy; Task 3: RAGAS semantic similarity (Sim) and answer correctness (Cor). Cell shading indicates change relative to the pre-trained value (red: decrease, green: increase). Values are from a single fine-tuning run per regime and should be read as indicative (see Section VI). Model Pretrained Finetuned-1 Finetuned-2 Task 1 Task 2 Task 3 Task 1 Task 2 Task 3 Task 1 Task 2 Task 3 F1 Acc Sim Cor F1 Acc Sim Cor F1 Acc Sim Cor gpt-3.5-turbo 0.85 0.76 0.92 0.77 – – – – – – – – llama2-7b 0.62 0.51 0.91 0.78 0.35 0.32 0.87 0.75 0.48 0.31 0.92 0.79 mistral-7b 0.47 0.59 0.90 0.72 0.27 0.39 0.86 0.74 0.43 0.18 0.86 0.76 prometheus-7b 0.76 0.75 0.92 0.73 0.65 0.61 0.85 0.73 0.55 0.16 0.86 0.75 westlake-7b 0.77 0.71 0.92 0.74 0.59 0.65 0.85 0.73 0.73 0.08 0.93 0.79 westseverus-7b 0.74 0.72 0.89 0.69 0.69 0.66 0.89 0.76 0.73 0.12 0.91 0.78 V-A Data Description and Preparation We construct one dataset per task. Task 1 uses 50 cybersecurity questions paired with expert-annotated gold keywords. Task 2 uses the computer-security subset of MMLU [3] (100 MCQ items). Task 3 draws on two repositories: a domain information repository (dG^d) built from NIST [18], and a QA-specific information repository (dS^d) curated from an organization-specific infrastructure wiki; since such infrastructure detail is sensitive, we substitute synthetic data of the same form. We then author 50 questions that require both repositories to answer, each with an SME ground-truth completion. For fine-tuning we prepare two datasets: a knowledge-focused set (Finetuned-1), built by generating QA pairs from the Cisco Talos corpus [4], and an instruction-focused set (Finetuned-2), drawn from a training split of the Task-3 evaluation data with explicit instructions to abstain when uncertain. All curated datasets and fine-tuned checkpoints will be released. We report dataset sizes explicitly since, given the modest n, individual numeric differences should be read as indicative rather than significant (Section VI). V-B Experiment Infrastructure We evaluate five open-weight 7-billion-parameter, 4-bit-quantized QA models: Llama-2-7b222huggingface.co/meta-llama/Llama-2-7b-chat-hf, Mistral-7b333huggingface.co/mistralai/Mistral-7B-Instruct-v0.2, Prometheus-7b444huggingface.co/AiMavenAi/AiMaven-Prometheus, WestLake-7b555huggingface.co/senseable/WestLake-7B-v2, and WestSeverus-7b666huggingface.co/FelixChao/WestSeverus-7B-DPO-v2. We also include GPT-3.5-Turbo777platform.openai.com/docs/models/gpt-3-5-turbo as a strong reference point (pre-trained only; we do not fine-tune the API model). These models were chosen deliberately: their pre-training predates much of the evolving threat intelligence (recent CVEs, advisories, infrastructure detail) used here for fine-tuning and evaluation, which reduces data-contamination risk and lets us attribute gains to retrieval and fine-tuning rather than prior exposure. For retrieval we use ChromaDB888trychroma.com as the vector store. Experiments ran on an Intel i9-12900 with an NVIDIA GeForce RTX™ 3090 Ti and 128 GB RAM. All models were decoded greedily (temperature 0) for deterministic, comparable outputs; the 4-bit quantization is held fixed across models so comparisons are like-for-like, though quantization may shift absolute scores. V-C Evaluation Protocol We combine quantitative and qualitative evaluation. The quantitative evaluation uses the RAGAS [8] framework for Task 3; the qualitative evaluation uses two cybersecurity SMEs to judge generated responses. Results appear in Tables I–I. V-C1 Quantitative Evaluation For Task 1 (KR) we report F1F_1 between predicted and gold keywords. Task 2 (MCQ) is scored by accuracy. For Task 3 we report RAGAS answer correctness and semantic similarity. We use RAGAS rather than BLEU [19] or ROUGE [21] because the latter are tailored to machine translation and summarization and correlate poorly with answer correctness in QA. V-C2 Qualitative Evaluation TABLE I: Inter-rater agreement (Fleiss Kappa) between the two cybersecurity SMEs across models. Model Kappa (K) Standard Error gpt-3.5-turbo 0.861 0.080 llama2-7b 0.845 0.084 mistral-7b 0.782 0.082 prometheus-7b 0.864 0.077 westlake-7b 0.944 0.081 westseverus-7b 0.868 0.078 Two SMEs assessed FiT’s contextual responses on a 5-point Likert scale [2], from 1 (“factually incorrect and contextually irrelevant”) to 5 (“factually accurate and contextually relevant”). Inter-rater agreement, measured by Fleiss’ κ [16] (Table I), was strong for most models (gpt-3.5-turbo 0.861, llama2-7b 0.845, prometheus-7b 0.864, westlake-7b 0.944, westseverus-7b 0.868) and moderate for mistral-7b (0.782). This agreement establishes the reliability of the SME judgments used to ground the Task-3 references. V-D Fine-tuning We fine-tune with QLoRA [6], a PEFT [7] method, under both the knowledge-focused and instruction-focused regimes. Hyper-parameters were held constant across regimes (rank 64, batch size 4, 5 epochs). Each regime was run once per model; we therefore frame the resulting numbers as indicative and analyze patterns of change rather than individual cell differences. VI Findings and Limitations VI-A Fine-tuning does not uniformly help small models Across Table I, both fine-tuning regimes degrade vocabulary (Task 1) and parametric knowledge (Task 2) for every model relative to its pre-trained baseline. In a domain like cybersecurity, where prompts routinely contain newly disclosed terms and CVEs, knowledge-focused tuning (Finetuned-1) cannot keep pace: it lowers keyword F1F_1 for all models (e.g., Llama-2 0.62→0.350.62\!→\!0.35) and reduces MCQ accuracy in parallel. The practical implication is that, absent abundant and current labeled data, a well-chosen pre-trained model paired with RAG is often the safer choice. VI-B The two regimes trade off differently but predictably in direction TABLE IV: Spearman rank correlation (ρ) between pre-trained and post-fine-tuning scores across the five open-weight models. High positive ρ indicates preserved rankings; negative ρ indicates inversion. Task Pre → Finetuned-1 Pre → Finetuned-2 Task 1 (vocabulary) 0.600.60 0.820.82 Task 2 (knowledge) 0.700.70 −0.60-0.60 Task 3 (correctness) not interpretable (range 0.690.69–0.790.79) To quantify how rankings move, we compute Spearman rank correlations between pre-trained and post-tuned scores (Table IV). Vocabulary rankings are well preserved under both regimes (ρ=0.60ρ=0.60 and 0.820.82), and knowledge rankings are preserved under knowledge-focused tuning (ρ=0.70ρ=0.70). Strikingly, knowledge rankings invert under instruction-focused tuning (ρ=−0.60ρ=-0.60): the strongest pre-trained knowledge models (WestLake, WestSeverus) become the weakest on Task 2 (accuracy 0.080.08 and 0.120.12). This inversion is the key reason a naive “best-before-equals-best-after” heuristic fails, and why FiT must be read as a regime-aware diagnostic rather than a monotonic predictor. VI-C The instruction-tuning collapse is abstention, not knowledge loss The Task-2 collapse under Finetuned-2 is an artifact of the abstention instruction, not erasure of knowledge. Two pieces of evidence support this. First, the instruction-focused data explicitly directed the model to withhold an answer when uncertain, making conservatism the trained behavior. Second, and more tellingly, the same instruction-tuned models retain or slightly improve contextual answer correctness (Task 3, e.g., WestLake 0.74→0.790.74\!→\!0.79) even as their standalone MCQ accuracy falls to near zero. A model that has truly lost knowledge could not answer correctly when that knowledge is retrieved; these models can. Instruction tuning thus trades parametric recall for caution, a property that is harmful for closed-book MCQ but potentially desirable for retrieval-grounded deployment, where unsupported answers are a liability. This aligns with the abstention behavior described in [26]. VI-D Retrieval-grounded contextualization is robust Task-3 correctness is remarkably stable across all models and conditions (range 0.690.69–0.790.79), and semantic similarity is uniformly high (∼0.85 \!0.85–0.930.93). When relevant context is retrieved and supplied, model choice and fine-tuning have little effect on contextual correctness. We therefore caution against interpreting similarity as an information-leakage metric: it is saturated and non-discriminative here, and reflects fidelity to the expert reference rather than the presence or absence of sensitive disclosure. The practical takeaway is that, for contextual correctness in this setting, investment in retrieval quality yields more than investment in fine-tuning. VI-E Selection guidance Taken together, these patterns make FiT useful as a pre-adaptation screen: weak pre-trained models (e.g., Mistral on vocabulary) remain weak after tuning, and the strongest knowledge models survive knowledge-focused tuning. Where instruction-focused tuning is planned, practitioners should expect, and can pre-empt, an abstention-driven collapse in closed-book knowledge, and should evaluate such models in their retrieval-grounded configuration rather than on standalone knowledge probes. VI-F Threats to Validity Construct. Our metrics are proxies. Task 2 accuracy conflates a wrong answer with a deliberate abstention; we cross-check against retrieval-grounded correctness (Task 3), but the proxy is imperfect. RAGAS similarity captures fidelity to the expert reference, not information leakage, so we read it only as a reliability signal. Internal. Each regime was run once with fixed hyper-parameters and greedy decoding, so regime effects are not separated from run-to-run variance; Finetuned-1 uses Cisco Talos while Task 2 uses MMLU, so part of the Task 2 drop may be distribution shift rather than a general fine-tuning effect; fixed 4-bit quantization keeps comparisons like-for-like but may shift absolute scores; and using models whose pre-training predates the evaluation content reduces, but does not eliminate, contamination risk. External. We cover five open-weight models at one 7B scale plus an API reference in a single domain, so patterns may not transfer across scales, architectures, or domains [14]; Task-3 specific data are synthetic for confidentiality and the threat-intelligence snapshot is fixed in time. Conclusion. Datasets are modest (n=50/100/50n=50/100/50) and we report no significance tests, so numeric differences are indicative; in particular, the Spearman values in Table IV are computed over five models and are descriptive of direction rather than statistically significant. A larger model pool and repeated runs would be needed to test these trends formally. VII Conclusion Cybersecurity is tied to critical infrastructure, making the reliable deployment of LLMs in this domain consequential. We presented FiT, a task-oriented diagnostic that decomposes cybersecurity QA suitability into vocabulary, knowledge, and contextualization, and used it to study how five small LLMs behave under knowledge-focused and instruction-focused fine-tuning. Fine-tuning did not uniformly help: it degraded parametric capabilities in every model, knowledge-focused tuning preserved relative rankings, and instruction-focused tuning collapsed measured knowledge through abstention while leaving retrieval-grounded contextualization intact, a regime-dependent trade-off we quantified with rank-correlation analysis. These results indicate that, in dynamic, low-data domains, a well-chosen pre-trained model paired with strong retrieval often strikes a better balance than fine-tuning, and that pre-adaptation diagnosis can guide model selection and reduce unnecessary tuning cost. Ethics Statement Our study uses datasets that contain no sensitive information. To obtain cyber-threat intelligence, specifically Common Vulnerabilities and Exposures (CVEs), we used web crawlers issuing API calls within the limits set by authorized sources. We anonymized our human evaluators and ensured no personally identifiable information was disclosed. We confirm that our research aligns, to the best of our knowledge, with the IEEE Code of Ethics. Acknowledgment This work was supported by the National Science Foundation under Grant No. 2611682. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of their institution or the National Science Foundation. Figure 3: Performance of pre-trained models on Tasks 1 and 3. (a) F1F_1 on vocabulary assessment; (b) answer correctness on contextual completion; (c) semantic similarity of completions. Figure 4: Performance of Finetuned-1 (knowledge-focused) models on Tasks 1 and 3. (a) F1F_1 on vocabulary assessment; (b) answer correctness on contextual completion; (c) semantic similarity of completions. Figure 5: Performance of Finetuned-2 (instruction-focused) models on Tasks 1 and 3. (a) F1F_1 on vocabulary assessment; (b) answer correctness on contextual completion; (c) semantic similarity of completions. References [1] E. Aghaei, X. Niu, W. Shadid, and E. Al-Shaer (2022) Securebert: a domain-specific language model for cybersecurity. In International Conference on Security and Privacy in Communication Systems, p. 39–56. Cited by: §I, §I-B. [2] I. E. Allen and C. A. Seaman (2007) Likert scales and data analyses. Quality progress 40 (7), p. 64–65. Cited by: §V-C2. [3] CAIS (2024) Measuring massive multitask language understanding. Note: huggingface.co/datasets/cais/mmlu Cited by: §V-A. [4] Cisco (2024) NATIONAL vulnerability database. Note: talosintelligence.com Cited by: §V-A. [5] M. Dahl, V. Magesh, M. Suzgun, and D. E. Ho (2024) Hallucinating law: legal mistakes with large language models are pervasive. Note: https://hai.stanford.edu/news/hallucinating-law-legal-mistakes-large-language-models-are-pervasive Cited by: §I. [6] T. Dettmers, A. Pagnoni, A. Holtzman, and L. Zettlemoyer (2024) Qlora: efficient finetuning of quantized llms. Advances in Neural Information Processing Systems 36. Cited by: §I-A, §V-D. [7] N. Ding, Y. Qin, G. Yang, F. Wei, Z. Yang, Y. Su, S. Hu, Y. Chen, C. Chan, W. Chen, et al. (2023) Parameter-efficient fine-tuning of large-scale pre-trained language models. Nature Machine Intelligence 5 (3), p. 220–235. Cited by: §V-D. [8] S. Es, J. James, L. Espinosa-Anke, and S. Schockaert (2023) RAGAS: automated evaluation of retrieval augmented generation. arXiv preprint arXiv:2309.15217. Cited by: §V-C. [9] K. Greshake, S. Abdelnabi, S. Mishra, C. Endres, T. Holz, and M. Fritz (2023) Not what you’ve signed up for: compromising real-world llm-integrated applications with indirect prompt injection. In Proceedings of the 16th ACM Workshop on Artificial Intelligence and Security, p. 79–90. Cited by: §IV-C. [10] N. Houlsby, A. Giurgiu, S. Jastrzebski, B. Morrone, Q. De Laroussilhe, A. Gesmundo, M. Attariyan, and S. Gelly (2019) Parameter-efficient transfer learning for nlp. In International conference on machine learning, p. 2790–2799. Cited by: §I-A. [11] J. Kirkpatrick, R. Pascanu, N. Rabinowitz, J. Veness, G. Desjardins, A. A. Rusu, K. Milan, J. Quan, T. Ramalho, A. Grabska-Barwinska, et al. (2017) Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences 114 (13), p. 3521–3526. Cited by: §I-A. [12] B. Lester, R. Al-Rfou, and N. Constant (2021) The power of scale for parameter-efficient prompt tuning. arXiv preprint arXiv:2104.08691. Cited by: §I-A. [13] P. Lewis, E. Perez, A. Piktus, F. Petroni, V. Karpukhin, N. Goyal, H. Küttler, M. Lewis, W. Yih, T. Rocktäschel, et al. (2020) Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in Neural Information Processing Systems 33, p. 9459–9474. Cited by: §I, §I-A. [14] Z. Liu, J. Shi, and J. F. Buford (2024) CyberBench: a multi-task benchmark for evaluating large language models in cybersecurity. Cited by: §I, §I-B, §VI-F. [15] Y. Mao, L. Mathias, R. Hou, A. Almahairi, H. Ma, J. Han, W. Yih, and M. Khabsa (2021) Unipelt: a unified framework for parameter-efficient language model tuning. arXiv preprint arXiv:2110.07577. Cited by: §I-A. [16] M. L. McHugh (2012) Interrater reliability: the kappa statistic. Biochemia medica 22 (3), p. 276–282. Cited by: §V-C2. [17] B. Min, H. Ross, E. Sulem, A. P. B. Veyseh, T. H. Nguyen, O. Sainz, E. Agirre, I. Heintz, and D. Roth (2023) Recent advances in natural language processing via large pre-trained language models: a survey. ACM Computing Surveys 56 (2), p. 1–40. Cited by: §I-A. [18] NIST (2024) National vulnerability database. Note: nist.gov Cited by: §V-A. [19] K. Papineni, S. Roukos, T. Ward, and W. Zhu (2002) Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, p. 311–318. Cited by: §V-C1. [20] P. Ranade, A. Piplai, S. Mittal, A. Joshi, and T. Finin (2021) Generating fake cyber threat intelligence using transformer-based models. In 2021 International Joint Conference on Neural Networks (IJCNN), p. 1–9. Cited by: §I. [21] L. C. Rouge (2004) A package for automatic evaluation of summaries. In Proceedings of Workshop on Text Summarization of ACL, Spain, Vol. 5. Cited by: §V-C1. [22] K. Singhal, S. Azizi, T. Tu, S. S. Mahdavi, J. Wei, H. W. Chung, N. Scales, A. Tanwani, H. Cole-Lewis, S. Pfohl, et al. (2023) Large language models encode clinical knowledge. Nature 620 (7972), p. 172–180. Cited by: §I-B. [23] M. Valipour, M. Rezagholizadeh, I. Kobyzev, and A. Ghodsi (2022) Dylora: parameter efficient tuning of pre-trained models using dynamic search-free low-rank adaptation. arXiv preprint arXiv:2210.07558. Cited by: §I-A. [24] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin (2017) Attention is all you need. Advances in neural information processing systems 30. Cited by: §I-A. [25] C. Wang, X. Liu, Y. Yue, X. Tang, T. Zhang, C. Jiayang, Y. Yao, W. Gao, X. Hu, Z. Qi, et al. (2023) Survey on factuality in large language models: knowledge, retrieval and domain-specificity. arXiv preprint arXiv:2310.07521. Cited by: §I-A. [26] J. Xin, R. Tang, Y. Yu, and J. Lin (2021) The art of abstention: selective prediction and error regularization for natural language processing. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), p. 1040–1051. Cited by: §VI-C. [27] L. Yang, S. Zhang, L. Qin, Y. Li, Y. Wang, H. Liu, J. Wang, X. Xie, and Y. Zhang (2022) Glue-x: evaluating natural language understanding models from an out-of-distribution generalization perspective. arXiv preprint arXiv:2211.08073. Cited by: §I-B. [28] J. Yu, X. Wang, S. Tu, S. Cao, D. Zhang-Li, X. Lv, H. Peng, Z. Yao, X. Zhang, H. Li, et al. (2023) Kola: carefully benchmarking world knowledge of large language models. arXiv preprint arXiv:2306.09296. Cited by: §I-B. [29] K. Zhu, J. Wang, J. Zhou, Z. Wang, H. Chen, Y. Wang, L. Yang, W. Ye, N. Z. Gong, Y. Zhang, et al. (2023) Promptbench: towards evaluating the robustness of large language models on adversarial prompts. arXiv preprint arXiv:2306.04528. Cited by: §I-B.