Paper deep dive
CyberCane: Neuro-Symbolic RAG for Privacy-Preserving Phishing Detection with Formal Ontology Reasoning
Safayat Bin Hakim, Aniqa Afzal, Qi Zhao, Vigna Majmundar, Pawel Sloboda, Houbing Herbert Song
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 97%
Last extracted: 6/21/2026, 7:04:25 AM
Summary
CyberCane is a neuro-symbolic framework designed for privacy-preserving phishing detection in critical domains like healthcare. It utilizes a dual-phase architecture: Phase 1 employs deterministic symbolic rules and the PhishOnt OWL ontology for technical violation detection and verifiable reasoning, while Phase 2 uses a privacy-preserving Retrieval-Augmented Generation (RAG) pipeline. This pipeline includes automated sensitive data (PHI) redaction and retrieval from a phishing-only corpus to maintain HIPAA compliance while leveraging LLMs (like GPT-4.1-mini) for semantic classification. The system demonstrates high precision (>98%) and extremely low false positive rates (0.16%), effectively recovering from attacks that bypass rule-based systems.
Entities (6)
Relation Signals (4)
CyberCane → evaluatedon → DataPhish 2025
confidence 100% · Evaluation on DataPhish 2025 (12.3k emails; mixed human/LLM)...
CyberCane → implements → RAG
confidence 100% · integrating deterministic symbolic analysis with privacy-preserving retrieval-augmented generation (RAG).
CyberCane → uses → PhishOnt
confidence 100% · We introduce CyberCane... We further introduce PhishOnt, an OWL ontology...
PhishOnt → enables → verifiable explanations
confidence 95% · PhishOnt ontology enabling verifiable attack classification through formal reasoning chains.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Privacy-critical domains require phishing detection systems that satisfy contradictory constraints: near-zero false positives to prevent workflow disruption, transparent explanations for non-expert staff, strict regulatory compliance prohibiting sensitive data exposure to external APIs, and robustness against AI-generated attacks. Existing rule-based systems are brittle to novel campaigns, while LLM-based detectors violate privacy regulations through unredacted data transmission. We introduce CyberCane, a neuro-symbolic framework integrating deterministic symbolic analysis with privacy-preserving retrieval-augmented generation (RAG). Our dual-phase pipeline applies lightweight symbolic rules to email metadata, then escalates borderline cases to semantic classification via RAG with automated sensitive data redaction and retrieval from a phishing-only corpus. We further introduce PhishOnt, an OWL ontology enabling verifiable attack classification through formal reasoning chains. Evaluation on DataPhish2025 (12.3k emails; mixed human/LLM) and Nazario/SpamAssassin demonstrates a 78.6-point recall gain over symbolic-only detection on AI-generated threats, with precision exceeding 98% and FPR as low as 0.16%. Healthcare deployment projects a 542x ROI; tunable operating points support diverse risk tolerances, with open-source implementation at this https URL.
Tags
Links
- Source: https://arxiv.org/abs/2604.23563v1
- Canonical: https://arxiv.org/abs/2604.23563v1
Trouble viewing inline? Open PDF directly →
Full Text
83,758 characters extracted from source content.
Expand or collapse full text
CyberCane: Neuro-Symbolic RAG for Privacy-Preserving Phishing Detection with Formal Ontology Reasoning Safayat Bin Hakim α∗ Aniqa Afzal α Qi Zhao α Vigna Majmundar α Pawel Sloboda β Houbing Herbert Song α α University of Maryland, Baltimore County β Bowie State University Abstract Privacy-critical domains require phishing detection systems that satisfy contradic- tory constraints: near-zero false positives to prevent workflow disruption, trans- parent explanations for non-expert staff, strict regulatory compliance prohibiting sensitive data exposure to external APIs, and robustness against AI-generated at- tacks. Existing rule-based systems are brittle to novel campaigns, while LLM- based detectors violate privacy regulations through unredacted data transmission. We introduce CyberCane, a neuro-symbolic framework integrating deterministic symbolic analysis with privacy-preserving retrieval-augmented generation (RAG). Our dual-phase pipeline applies lightweight symbolic rules to email metadata, then escalates borderline cases to semantic classification via RAG with automated sensitive data redaction and retrieval from a phishing-only corpus. We further introduce PhishOnt, an OWL ontology enabling verifiable attack classification through formal reasoning chains. Evaluation on DataPhish 2025 (12.3k emails; mixed human/LLM) and Nazario/SpamAssassin demonstrates a 78.6-point recall gain over symbolic-only detection on AI-generated threats, with precision exceed- ing 98% and FPR as low as 0.16%. Healthcare deployment projects a 542× ROI; tunable operating points support diverse risk tolerances, with open-source imple- mentation at https://github.com/sbhakim/Cybercane. Keywords: Phishing detection, neuro-symbolic AI, privacy-preserving RAG, formal ontology rea- soning 1 Introduction The human element contributes to over 60% of data breaches [29], with phishing representing a primary initial access vector across industries, yet privacy-critical domains face unique detection challenges. Healthcare exemplifies these constraints: systems must balance security with unin- terrupted access to time-critical patient communications—misclassified appointment reminders or prescription notifications directly compromise care. Older adults demonstrate significantly reduced ability to discriminate phishing from legitimate emails, with detection performance declining with age [22, 8]. Recent generative AI advances amplify this threat across all sectors, enabling adver- saries to craft convincing domain-specific terminology and adapt to detection systems [6]. Privacy-critical detection systems face four contradictory requirements, which we illustrate through healthcare as a motivating example. First, false positives must approach zero—in healthcare, each false alarm can potentially delay critical medical communications. Second, explanations must be ∗ Corresponding author: shakim3@umbc.edu. arXiv:2604.23563v1 [cs.CR] 26 Apr 2026 transparent and actionable for IT staff who often lack specialized security training [7]. Third, sys- tems must preserve privacy—regulations like HIPAA (Health Insurance Portability and Account- ability Act) prohibit protected information transmission to unauthorized third parties, constraining cloud AI services [24]. Fourth, detection must accommodate vulnerable populations with reduced digital literacy. Existing approaches struggle to satisfy these constraints simultaneously. Rule-based systems are precise but brittle to novel attacks and lack semantic understanding for context-aware detection. Deep learning models generalize better but are black boxes unsuitable for verified explanations and audit trails. Recent LLM-based detectors achieve high accuracy—ChatSpamDetector reaches 99.7% [15] and KnowPhish achieves 92.5% accuracy with 97.8% precision [18]—but cannot be directly deployed in privacy-critical domains due to privacy concerns around transmitting unredacted email content to external API providers. This work addresses three research questions: (i) Does the dual-phase architecture achieve healthcare-grade precision (>95%) with near-zero FPR, satisfying the core requirement of privacy- critical deployment? (i) Does Phase 2 RAG provide statistically significant improvement over symbolic-only detection while supporting tunable operating points for diverse workflows? (i) Does formal ontology reasoning produce verifiable explanations, and do operational economics justify deployment in privacy-critical domains? We introduce CyberCane, a neuro-symbolic framework integrating symbolic rules with privacy- preserving RAG (Fig. 1). Phase 1 applies deterministic analysis to identify technical violations; Phase 2 performs semantic classification via RAG with automated sensitive data redaction and re- trieval from a phishing-only corpus. Retrieved examples ground LLM explanations while prevent- ing data exposure. CyberCane addresses these requirements through architectural design: tunable thresholds achieve near-zero FPR (Req. 1), PhishOnt ontology provides verifiable explanations (Req. 2), sensitive data redaction maintains HIPAA compliance (Req. 3), and operating modes accommodate diverse workflows (Req. 4). Contributions. We make the following contributions: (i) Privacy-preserving neuro-symbolic architecture: Dual-phase pipeline combining symbolic rules with RAG through HNSW retrieval and sensitive data redaction, achieving tunable op- erating points. (i) PhishOnt ontology: Web Ontology Language (OWL)-based reasoning framework enabling verifiable attack classification and formal explanation chains. (i) Comprehensive evaluation: Statistical validation through bootstrap confidence intervals and DataPhish 2025 benchmark (n=2,337) spanning template-based to GPT-generated attacks. (iv) Deployment viability: Cost-benefit analysis for healthcare deployment demonstrating 542.0× ROI, with framework applicable to other privacy-critical domains. 2 Related Work Phishing detection evolved from blacklist-based filtering [2] to ML systems combining heuris- tics [1, 25] and authentication protocols [14, 5, 16]. Recent LLM-based detectors [15, 18] present deployment challenges in privacy-critical domains: limited verification, email content transmission raises regulatory concerns (e.g., HIPAA), and labeled data may not exist for emerging threats. Neuro-symbolic AI combines neural perception with symbolic reasoning [9, 12] addressing fun- damental limitations. Privacy-critical domains face unique challenges from life-critical systems and vulnerable populations [7, 8] with strict regulatory constraints [24]. To address these chal- lenges while maintaining interpretability, recent work has explored retrieval-augmented generation (RAG) [26, 10, 11], though existing applications inadequately address privacy in regulated domains. We extend RAG with architectural privacy-by-design through sensitive data redaction before em- bedding. Prior phishing defenses span blacklist/heuristic pipelines and authentication checks, which are in- terpretable but brittle to domain compromise and new campaigns [2, 25, 14, 16]. LLM-based detectors raise deployment challenges where sensitive data exposure and auditability are criti- cal [15, 24]. Retrieval-augmented security systems improve traceability by grounding outputs in cited evidence [26, 10]. However, prior work rarely couples retrieval with strict privacy bound- 2 Phase 2: Neural RAG classification DNS/MX Verification SPF/DMARC signals Domain mismatch URL & content cues (urgency/PHI) Email Input Symbolic Indicators Embedding (text-embedding-3-small) Vector search (pgvector HNSW) Retrived examples (k≈8) LLM reasoning (GPT-4.1-mini) external API external API Phishing corpus (~2,300) Phase 1: Deterministic symbolic analysis Privacy boundary: redaction before external model calls Human review and Feedback Confiidence Score + Explanation Phishing (>50%) Suspicious(25-50%) → Human Review Safe (<25%) Symbolic Phase Neural Phase External API Human Review Privacy Boundary (sim ≥0.88 or P1=review & avg≥0.82) (sim ≥0.75 or avg≥0.72) If Phase 1 = phishing (score≥5), final=phishing regardless of Phase 2 Figure 1: CyberCane dual-phase architecture. Phase 1 applies symbolic rules to email metadata, computing deterministic scores. Emails passing Phase 1 undergo sensitive data redaction before Phase 2 neural analysis using RAG with a phishing-only corpus, producing similarity-driven AI scores and multi-layered explanations. aries or conservative operating points needed in regulated environments [7, 24]. Human-factor ev- idence that vulnerable populations are more susceptible to phishing supports the need for transpar- ent, low-FPR decision support [8]. CyberCane combines these elements with privacy-first redaction, phishing-only retrieval, and conservative thresholds tailored to diverse operational workflows. 3 System Architecture CyberCane implements a dual-phase detection pipeline where emails progress sequentially through deterministic symbolic analysis (Phase 1) followed by neural RAG-based classification (Phase 2). Fig. 1 illustrates the complete architecture. The system is implemented as a FastAPI backend (Python 3.11) with PostgreSQL 17 using pgvector for HNSW vector indexing, integrated with Ope- nAI APIs (text-embedding-3-small, gpt-4.1-mini [21]), and a Next.js 15.0.3 frontend for human review. Design rationale. Sequential processing prioritizes low-latency, zero-cost symbolic filtering be- fore invoking external APIs. Phase 1 catches obvious phishing (missing DNS records, domain mismatches) with instant technical explanations IT staff can verify independently; Phase 2 adds se- mantic analysis when needed. Operational review rates and Phase 1 baseline metrics are reported in Appx. B. 3.1 Phase 1: Deterministic Symbolic Analysis Phase 1 applies lightweight symbolic rules across five categories: DNS/authentication checks (MX, SPF [14], DMARC [16]), domain mismatches, URL obfuscation patterns, urgency language, and credential requests. Rules employ tuned weights (w i ∈ 1, 2, 3) producing risk scores that map to verdicts: s < 2 (benign), 2 ≤ s < 5 (review), s ≥ 5 (phishing). Detected indicators are passed to ontology reasoning (Algorithm 1) for formal attack classification. Appx. A (Algorithm 2) details complete rule specifications with regex patterns and validation logic. 3.2 Ontology-Guided Symbolic Inference We introduce PhishOnt, an OWL ontology encoding phishing attack knowledge as description-logic axioms over Phase 1 indicators (missing MX/SPF/DMARC, domain mismatch, URL obfuscation, urgency, credential requests, freemail senders). Algorithm 1 formalizes the neuro-symbolic reason- 3 Algorithm 1: Ontology-Guided Attack Classification Input: Phase 1 indicators I , ontology Ω, threshold θ = 0.3 Output: Attack typesA and reasoning chainsE Function OntologyAttackClassification(I, Ω) /* Normalize indicators and map to ontology properties */ P ←∅ for each indicator i with I[i] = true do P ← P ∪MapProperty(i) A←∅ for each attack type τ ∈ Ω do (R ∀ ,R ∃ )← Ω.GetAxioms(τ ) m ∀ ← CountMatch(R ∀ ,P ) m ∃ ← HasAny(R ∃ ,P ) den←|R ∀ | + 1(R ∃ ̸=∅) /* Axiom satisfaction from universal and existential coverage */ c← (m ∀ + m ∃ )/den if c≥ θ∧ (R ∃ =∅∨ m ∃ = 1) then A←A∪(τ,c) E ← GenerateChain(P,A, Ω) return (A,E) ing mechanism: Phase 1 indicators map to ontology properties (missing_mx→ hasMissingMX), en- abling formal attack classification via axiom satisfaction. The reasoner computes multi-label attack types with confidence scores (c = m/r where m is satisfied constraints, r is required constraints) and generates verifiable reasoning chains linking detected properties to inferred attack classes. This formal provenance distinguishes our approach from black-box LLM classifiers, providing trans- parent explanations required for healthcare audit trails. Coverage and attack-type prevalence are reported in Section 5.2. 3.3 Phase 2: Neural RAG Pipeline Emails escalated from Phase 1 undergo privacy-preserving RAG analysis. The pipeline: (1) PHI redaction masks sensitive fields (emails, SSN, credit cards) via regex before external API calls; (2) embedding generation encodes redacted content using text-embedding-3-small (1536-d); (3) vector retrieval queries phishing-only corpus (2,297 examples) via HNSW [19], returning top-k = 8 neigh- bors with cosine similarities; (4) LLM reasoning generates tagged explanations ([AUTH], [URL], [SIMILARITY], [ONTOLOGY]) using gpt-4.1-mini conditioned on redacted email, Phase 1 indica- tors, ontology-inferred attack types (Algorithm 1), and retrieved examples. Critically, the AI score derives from similarity statistics—not LLM generation—ensuring reproducible, grounded verdicts. The complete neuro-symbolic integration workflow is formalized in Appx. A (Algorithm 3), with implementation details including prompt engineering and threshold calibration. Decision logic. If Phase 1 verdict is phishing, final verdict is phishing. Otherwise: top similarity ≥ 0.40 (or Phase 1 needs_review + avg top-3 ≥ 0.35) yields phishing; top similarity ≥ 0.25 (or avg top-3≥ 0.17) yields needs_review; lower similarity inherits Phase 1 verdict. This prevents neu- ral overrides of absent technical violations while capturing semantic similarity. Ontology-inferred attack types (Algorithm 1) are passed to the LLM as structured context, enabling [ONTOLOGY]- tagged explanations that cite formal reasoning chains rather than heuristic pattern matching. Privacy architecture. Three data categories transmit to external APIs: (1) redacted text embed- dings (1536-dim vectors); (2) LLM prompts with redacted content, symbolic indicators, ontology- inferred attack types, and retrieved example snippets (pre-screened); (3) ontology reasoning chains derived from local OWL inference without external transmission. Redaction is empirically active: across 2,337 DataPhish 2025 test emails, 50.7% contained at least one redacted PII field (mean 1.63 items/email); LLM-generated emails averaged 1.73 items/email versus 1.30 for human-written, yet detection F1 remained 0.987 for both groups, confirming that semantic classification is preserved under active PII masking (detailed counts in Appx. F.6). Critically, ontology classification operates entirely offline on Phase 1 indicators, avoiding external API dependency for formal explanations. Residual risks include quasi-identifier inference from surviving context, embedding leakage under inversion attacks [17], and organizational compliance requirements (business associate agreements, workforce training) beyond technical controls. Organizations with zero-trust requirements can de- ploy on-premises models (sentence-transformers, Llama 3) at cost of reduced accuracy and increased infrastructure complexity. 4 Retrieval corpus bias mitigation. The phishing-only corpus creates structural positive bias. Three mechanisms mitigate: (1) Phase 1 filters obvious phishing before RAG, ensuring borderline cases enter neural analysis; (2) conservative similarity thresholds (≥ 0.40 for phishing classification) re- quire strong semantic overlap; (3) LLM receives symbolic indicators (may be empty for legitimate emails), ontology attack types, and retrieved examples, with explicit prompting to assess attack char- acteristics beyond superficial similarity. The measured FPR=0.16% suggests limited false escalation despite corpus composition. We empirically evaluate three hypotheses: (H1) dual-phase detection achieves healthcare-grade pre- cision (>95%) with minimal FPR; (H2) RAG improves over deterministic-only baselines; (H3) op- erational economics justify deployment. 4 Experimental Setup Dataset. We evaluate on three corpora: (1) Nazario phishing corpus [20] and SpamAssassin [3] (2006-era, 7,274 emails); (2) DataPhish 2025 [27] (12,300 emails, ∼75% LLM-generated, ∼25% human-written). Table 3 (left) summarizes splits. Only phishing training rows are indexed for retrieval. Random seed 42 ensures reproducibility. Synthetic healthcare validation data (Appx. B) is generated via a multi-model pipeline with provenance tracking, deduplication, and contamination checks; a manifest and dataset card are produced for auditability. Metrics. We report precision (fraction of phishing predictions correct—important for minimizing false alarms), recall (fraction of phishing detected), F1-score (harmonic mean), and false positive rate (FPR—fraction of legitimate emails misclassified). Privacy-critical contexts prioritize FPR min- imization to prevent workflow disruption. Baselines. Majority class baseline (predicts most frequent label) and TF-IDF logistic regression (unigram/bigram features, trained on train split) provide text-only comparisons. Detailed baseline results in Appx. F. We evaluate a GPT-4 Direct baseline using OpenAI’s gpt-4o-mini model (cost- effective variant representing realistic organizational deployment) to quantify privacy and cost im- plications of unfiltered commercial LLM usage. This baseline transmits full unredacted email text with a zero-shot prompt (“Is this email phishing?”), tracking API costs and sensitive information ex- posure rates via regex pattern detection (email addresses, phone numbers, SSNs, credit cards, dates of birth). Protocol. Thresholds tuned on validation split. Bootstrap confidence intervals (1,000 resamples for CI estimation; 10,000 samples for McNemar significance testing) estimate statistical stabil- ity. All experiments use OpenAI text-embedding-3-small (1536-dim) and gpt-4.1-mini (tempera- ture=0.2) [21]. 5 Results We present results organized to address our three hypotheses (H1–H3): overall performance and baseline comparisons (H1, H2), ontology coverage, explanation quality, operating point flexibility, and economic viability (H3). 5.1 Overall Performance Table 3 (right) compares Phase 1 (deterministic, threshold=2) against RAG (k = 8) across datasets. On Nazario/SpamAssassin (n=1,110), RAG raises precision to 99.5% while increasing recall to 37.2% and reducing FPR to 0.16%. On DataPhish 2025 (n=2,337; human+LLM mix), Phase 1 recall is 20.5% while RAG reaches 99.1% recall at 98.2% precision. Detailed cohort and model- source breakdowns appear in Appx. Table 18, and retrieval quality in Appx. C. Comparison with privacy-constrained baselines. Table 4 compares CyberCane against three baselines under explicit privacy accounting. TF-IDF logistic regression trained on PII-redacted text (same pii.py pipeline as CyberCane, PHI exposure = 0%) achieves 98.8% precision and 97.4% recall on the Nazario/SpamAssassin test split—but at 6× higher FPR (0.98% vs. 0.16%), with no explainability, no formal attack taxonomy, and no tunable operating points. This is the fair privacy- constrained bar; CyberCane’s value is not raw recall but precision at near-zero FPR plus verifiable PhishOnt reasoning chains required for healthcare audit trails. GPT-4 Direct achieves 99.0% recall 5 Table 1: DataPhish 2025 robustness: Phase 2 RAG detection consistency across 18 creator sources and 6 emotion categories (n=2,300; 37 of the 2,337 test emails lack creator attribution and are excluded from per- source rows but included in overall Table 3). F1 range 0.976–1.000 across all creator groups. GroupNPrec.RecallF1 Creator Source Human-written55997.5%99.5%98.5% LLM-generated (17 models)1,74198.4%99.1%98.7% LLM Model DeepSeek-Chat60299.1%99.1%99.1% OpenAI (GPT family)59697.1%99.3%98.2% GPT-4o6295.4%100.0%97.6% DeepSeek Chat (v2)62100.0%97.5%98.7% Amazon Nova / Grok / Codestral156100.0%100.0%100.0% Emotion Urgency1,17798.7%99.2%98.9% Curiosity97698.2%99.3%98.7% Authority90898.8%100.0%99.4% Neutral88895.0%99.0%97.0% Fear65599.4%99.2%99.3% Greed43299.2%98.2%98.7% OverallPhase 2 RAG2,30098.1%99.2%98.7% Note. All groups are evaluated with identical thresholds (top_sim≥0.40, avg_top3≥0.35). F1 ranges from 0.976 to 1.000 across 18 creator sources, demonstrating consistent detection regardless of authoring model or rephrasing style. Phase 1 recall for the same set is 7.7–39.9% (emotion-dependent); Phase 2 closes all blind spots to≥97% F1. but exposes 53.2% of emails as sensitive data, violating HIPAA regardless of detection accuracy, and flags∼600 legitimate emails daily at 5.9% FPR. Robustness across creator sources. Table 1 reports detection consistency across 18 email creator sources in the DataPhish 2025 test set (n=2,300; 37 emails without creator attribution excluded from per-source rows). Phase 2 RAG achieves F1 between 0.976 and 1.000 across all groups— human-written (F1=0.985), GPT-family (F1=0.982), DeepSeek-Chat (F1=0.991), and frontier mod- els including Grok 4 and Amazon Nova (F1=1.000). Performance is statistically indistinguishable between human-written and LLM-generated emails (F1=0.985 vs. 0.987), demonstrating that Cy- berCane’s semantic embedding space is invariant to authoring model. Emotion-stratified analysis shows Phase 2 closes all Phase 1 blind spots: Neutral-tone attacks (near-zero Phase 1 recall) reach F1=0.970; Altruism and Greed reach F1≥0.987. Critically, all DataPhish evaluations apply the same pii.py PHI-masking step before retrieval and prompting—yet LLM-generated emails score no lower than human-written (F1=0.987 vs. 0.985), confirming that regex redaction does not intro- duce a detection gap under AI-augmented attack conditions. Phase 2 as layered defense against rule evasion. A structurally important property emerges from stratifying DataPhish 2025 results by Phase 1 outcome. Of the 1,589 phishing emails in the test set, 79.5% (n=1,264) scored zero in Phase 1—attacks with valid DNS infrastructure and no detectable urgency cues that entirely bypass symbolic rules. Phase 2 RAG correctly classified 99.0% of these Phase 1-evaded emails, leaving a combined two-phase miss rate of only 0.8%. Recovery was con- sistent across emotion categories: Neutral-tone attacks (95.8% Phase 1 evasion rate) reached 99.0% Phase 2 recovery; Greed-tone (97.2% evasion) 98.2%; Authority-tone (64.1% evasion) 100.0%. This demonstrates the intended architectural property: the symbolic layer provides instant verifi- able verdicts for flagrant technical violations, while the neural layer provides semantic recovery for attacks that deliberately avoid rule triggers. 5.2 Ontology Coverage and Attack Taxonomy Table 2 summarizes PhishOnt activation on the test split. The ontology fires on 85.0% of emails and 76.8% of phishing, providing multi-label symbolic explanations; high benign activation (91.7%) reflects shared structural indicators (e.g., URLs, authentication patterns) present in both legitimate and malicious emails. Critically, PhishOnt complements Phase 1/2 detection logic by generating verifiable reasoning chains (e.g., CredentialTheft = hasCredentialRequest AND hasMissingMX) for explanations rather than standalone verdicts. On activated emails, PhishOnt assigns 2.66 attack labels on average with mean confidence 53.6%, reflecting overlapping threat patterns. Confidence discriminability. While activation rates are similar across classes, confidence scores differ systematically: phishing activations carry a mean confidence of 54.6% versus 52.8% for be- nign activations (+1.84p). More tellingly, only 14.5% of benign activations exceed the phishing mean confidence—meaning 85.5% of benign activations score below the typical phishing confi- dence level. This confirms that PhishOnt is discriminative at the confidence level despite similar 6 Table 2: PhishOnt ontology coverage and attack-type prevalence on the test split (n=1,110). (A) Coverage SplitCountCov. Overall946/1110 85.2% Phishing383/49577.4% Benign563/61591.5% (B) Attack types (prevalence) Attack typePrev.Attack typePrev. Prescription Fraud62.3% Insurance Verification Phish 13.2% High-Confidence Phishing 58.8% Appointment Scam11.4% Technical Attack36.8% Social Engineering Attack8.2% Credential Theft34.1% URL-Based Attack1.5% Table 3: Dataset composition (left) and performance across corpora (right). DataPhish 2025 shows 78.6p recall gain of Phase 2 RAG over Phase 1 symbolic-only within CyberCane’s pipeline on AI-generated threats. (A) Dataset splits Source Train (Benign/Phish) Val (Benign/Phish) Test (Benign/Phish) Nazario0 / 10950 / 2340 / 236 SpamAssassin2863 / 1202613 / 257615 / 259 DataPhish 20254932 / 3063269 / 131711 / 1589 (B) Test performance DatasetMethodPrec.RecallF1FPR Nazario/SpamAssassin Phase 1 83.0%17.8% 0.2932.9% RAG99.5% 37.2% 0.541 0.16% DataPhish 2025 Phase 1 93.4%20.5% 0.336— RAG98.2% 99.1% 0.987— binary activation rates. The system is architecturally designed so that this confidence signal feeds structured context into Phase 2 RAG rather than producing standalone verdicts, which explains why the binary activation rate is not the operative discriminability measure. 5.3 Explanation Groundedness To evaluate whether LLM-generated explanations cite verifiable evidence, we conducted A/B test- ing on 60 random test emails, comparing explanation quality with and without ontology context. Each explanation bullet receives a tag ([AUTH], [URL], [URGENCY], [CONTENT], [SIMILAR- ITY], [ONTOLOGY]) and should cite specific evidence from Phase 1 analysis, retrieval context, or ontology inference. We classify support status as SUPPORTED (cites specific evidence), UNSUP- PORTED (generic without citation), or UNKNOWN (no evidence available to cite). Table 5 demonstrates ontology integration enables verifiable explanations—all 41 ONTOLOGY- tagged bullets (100%) cite correct attack types and reasoning chains from formal inference. In comparison, heuristic tags show variable support rates (AUTH: 65.8%, URGENCY: 22.2%, CON- TENT: 25.0%), reflecting challenges verifying free-text LLM outputs against rule-based indicators. SIMILARITY achieves 100% support by referencing concrete similarity scores. This demonstrates that formal symbolic reasoning produces more verifiable explanations than pure heuristic analysis, addressing healthcare audit requirements for decision traceability. The key contribution is adding a formal layer that ensures verifiability for attack classification and risk scoring. 5.4 Operating Point Flexibility CyberCane exposes tunable thresholds across Phase 1 scoring and RAG similarity, allowing orga- nizations to select precision-recall tradeoffs based on risk tolerance and review capacity. Table 6 reports five operating modes. The baseline configuration matches the production pipeline (99.5% precision, 0.16% FPR), while more aggressive settings increase recall to 44.6% while maintaining precision≥99.3% (FPR remains 0.16% via shared Phase 1 high-confidence threshold). Fig. 2 consolidates evidence for H1–H3. Panel A confirms H1: RAG achieves +16.5p precision gain, crossing 95% healthcare threshold. Panel B reveals coverage-precision tradeoff: DNS checks (DMARC, MX, SPF) provide 30–46% coverage while content heuristics (urgency, IP literals) de- liver 90–94% precision. Panel C validates design rationale (Section 3): 71.2% of false negatives result from intentional conservative threshold choices (zero-score attacks 43.2%, low-signal content 28.0%) that prioritize low FPR over recall, while only 28.8% represent technical limitations requir- ing architectural improvements. Panel D confirms H3: 542.0× ROI with $1.5K daily costs versus $818K risk mitigation. Comparing RAG against higher-threshold Phase 1 (threshold=5) quantifies impact: Phase 1 at threshold=5 achieves precision=75.0%, recall=0.6%. RAG improves both metrics: precision in- creases to 99.5% (+24.5p) and recall to 37.2% (+36.6p), yielding F1-score improvement from 0.012 to 0.541. Statistical significance testing validates gains: McNemar’s test yields χ 2 = 181.0 (p < 0.001), confirming RAG predictions differ significantly from deterministic baseline. Boot- 7 Table 4: Comparison with GPT-4 Direct Baseline showing privacy and cost trade-offs. MethodPrec.RecallF1FPRCost/EmailPHI Exposure CyberCane (RAG k=8)99.5%37.2%54.1%0.16%$0.00170% † TF-IDF LR + Redaction § 98.8%97.4%98.1%0.98%$00% † TF-IDF LR (no redaction)98.6%97.4%98.0%1.14%$0∼53% GPT-4 Direct (gpt-4o-mini)93.2%99.0%96.0%5.9%$0.0001 ‡ 53.2% Notes. † PHI redacted before API transmission. ‡ API cost only; excludes human review ($1,310/day for CyberCane). § Privacy- constrained fair comparison uses the same redaction pipeline as CyberCane with PHI exposure = 0%. CyberCane provides 6× lower FPR (0.16% vs. 0.98%), PhishOnt verifiable reasoning chains, and tunable operating points, none of which are available in TF-IDF. Table 5: Explanation groundedness: support rates by tag type (n=60 emails). (A) With ontology context TagTotal Supp.Rate AUTH11777 65.8% URL2328.7% URGENCY184 22.2% CONTENT164 25.0% SIMILARITY5050 100% ONTOLOGY4141 100% (B) Without ontology context TagTotal Supp.Rate AUTH13578 57.8% URL192 10.5% URGENCY144 28.6% CONTENT144 28.6% SIMILARITY6161 100% ONTOLOGY00N/A strap analysis on F1-score shows mean improvement +0.527 (p < 0.001); confidence intervals are reported in Table 19. 5.5 Operational Cost-Benefit Analysis Table 7 quantifies economic viability for a healthcare deployment example with a mid-sized orga- nization processing 10,000 daily emails at 4.4% phishing base rate. With 37.2% recall, the system detects ∼164 phishing attacks daily, preventing an estimated 16.4 regulatory breaches (assuming 10% attack success rate [23]). At $50,000 average regulatory penalty per incident, this mitigates ∼$818,000 daily risk, yielding a 542× optimistic ROI (detected attacks only; Table 7). We additionally report a conservative bound that deducts the estimated cost of missed attacks (276 daily false negatives × 10% success rate × $50K = $1.38M/day). At 37.2% recall this bound is negative, honestly reflecting that conservative threshold deployment does not fully offset phishing risk at this operating point. However, the bound turns strongly positive under higher-recall configu- rations: at DataPhish 2025 recall (99.1%), optimistic and conservative ROI converge to 1,450× and 1,437× respectively, since virtually no attacks are missed. This framing reinforces the importance of operating point selection: organizations with sufficient review capacity should deploy Balanced or Aggressive mode, where recall substantially increases without sacrificing FPR. Total operational costs ($1,506/day) include API calls ($1.71), human review labor ($1,310 for 13.1% escalation at $100/hr based on fully-burdened IT security analyst wages [28]), and false- positive impact ($194 for 16 delayed communications at $12.77 each). With FPR 0.16% (16 false alarms per 10,000 emails), net benefit is $816,272/day or $297.9M annually, yielding 542.0× ROI. Operating-mode economics. Extending analysis across operating points (Appx. F, Table 16), bal- anced mode increases detection to 176 attacks daily (40.0% recall, $1,506/day cost) while maintain- ing 583× ROI. Aggressive mode detects 196 attacks (44.6% recall) with 651× ROI, showing recall gains remain economically viable under low-FPR operating conditions. Healthcare generalization. Given the absence of publicly available healthcare phishing datasets, we validate domain applicability using synthetic healthcare attacks (Appx. B details multi-model generation with provenance tracking and contamination checks). Balanced mode achieves 100% recall on template-based phishing with explicit red flags; aggressive mode reaches 87.3% recall at 95.4% precision on GPT-generated attacks, demonstrating that coverage depends on threshold cali- bration. While synthetic evaluation has limitations, the difficulty spectrum provides initial evidence of healthcare transferability pending real-world validation. 5.6 Feature Importance and Design Validation Leave-one-out ablation shows DNS authentication most frequently triggered: no_dmarc covers 46.5% of phishing (12.5p recall drop when removed), while content heuristics achieve 90–94% precision with lower coverage. Failure taxonomy shows 71.2% of missed phishing cases fall into 8 Table 6: Operating point configurations showing precision-recall tradeoffs across deployment modes. Operating ModePrecisionRecallF1FPRUse Case Baseline (Pipeline)99.5%37.2%0.5410.16%Current deployment Conservative99.3%29.5%0.4550.16%High-stakes clinical Balanced99.5%40.0%0.5710.16%General healthcare Moderate99.5%42.2%0.5930.16%High-volume screening Aggressive99.5%44.6%0.6160.16%Maximum coverage 0.00.20.40.60.81.0 Recall (Phishing Detection Rate) 0.4 0.5 0.6 0.7 0.8 0.9 1.0 Precision Privacy-Critical Req. (Prec.>95%) +16.5p (A)Neuro-Symbolic Achieves Deployment-Grade Precision Phase 1 Phase 1 (t=2) Phase 1 (t=5) RAG (ours) 01020304050 Phishing Coverage (%) 30 40 50 60 70 80 90 100 Rule Precision (%) DMARC MX SPF Urgency IP Literal Each circle = rule X = coverage, Y = precision (B)Feature Importance: Coverage vs. Precision Bubble size = recall impact (p) 3p 8p 13p Color = importance tier High impact Medium impact Low impact NeutralGreedAuthorityOverall Emotion Category (DataPhish 2025) 50 60 70 80 90 100 110 Rate (%) 95.8% 97.2% 64.1% 79.5% 99.0% 98.2% 100.0% 99.0% Combined two-phase miss rate:0.8% (C)Phase 2 Recovers 99.0% of Phase 1-Evaded Attacks Phase 1 Evasion Rate Phase 2 Recovery Rate APIHumanFPTotalRisk 10 0 10 1 10 2 10 3 10 4 10 5 10 6 Daily Cost/Benefit ($, log scale) ROI: 542.0× Net Daily: $816K $2 $1.3K $0.2K $1.5K $818K (D)Economic Viability: $297.9M Annual Value High-Precision Phishing Detection via Privacy-Preserving Neuro-Symbolic RAG Figure 2: CyberCane results overview. (A) Phase 1 versus RAG operating points on precision-recall space. (B) Rule coverage versus precision with bubble size encoding recall impact. (C) False-negative taxonomy separating intentional thresholds from technical limits. (D) Daily cost-benefit and ROI for privacy-critical deployment. conservative-threshold categories: 43.2% triggered zero rules (evasive attacks with valid DNS), 28.0% scored below threshold. Complete ablations and taxonomy in Appx. E Tables 12, 25. 6 Discussion Key findings. Results support H1 (99.5% precision, 0.16% FPR, Table 3B), H2 (RAG improves precision by 16p over symbolic-only, McNemar’s p < 0.001, Fig. 2A), and H3 (542.0× ROI, Ta- ble 7). These findings demonstrate that neuro-symbolic architectures can achieve deployment-grade performance in privacy-critical domains by prioritizing precision and explainability over maximiz- ing recall—a tradeoff appropriate for environments where false alarms directly disrupt critical work- flows. Precision gains arise from semantic classification correcting conservative symbolic flagging, while DataPhish 2025 evaluation quantifies a 78.6-point recall gain of Phase 2 RAG over Phase 1 symbolic-only within the CyberCane pipeline on AI-generated attacks. On Nazario/SpamAssassin, 37.2% recall under the conservative threshold reflects a deliberate FPR-first design choice, not a technical ceiling—Aggressive mode recovers 44.6% recall while maintaining 99.5% precision (Ta- ble 6). 9 Table 7: Cost-benefit analysis: healthcare deployment scenario. Cost parameters grounded in industry bench- marks [13, 23, 28]. MetricValueConservative BoundDescription Total Daily Cost$1,506$1,506API + labor + FP impact Attacks Detected16416437.2% of 440 daily phishing Attacks Missed27627662.8% of 440 (false negatives) Breaches Prevented16.416.410% attack success rate Missed Breach Cost—$1,381,600Missed attacks× 10%× $50K Risk Mitigated$817,778$817,778$50K per breach penalty Net Daily Benefit$816,272—Risk - operational cost ROI (optimistic)542.0×—Detected attacks only † ROI (conservative)— −376× ‡ Accounts for missed attacks At DataPhish 2025 recall (99.1%): optimistic 1,450×, conservative 1,437× Notes. † Optimistic bound follows industry-standard reporting and is equivalent to assuming no undetected breaches. ‡ The conservative bound is negative at 37.2% recall because missed breach costs exceed mitigated risk. ROI becomes positive under Balanced/Aggressive modes or DataPhish 2025 operating conditions (99.1% recall). Privacy-accuracy tradeoffs. Sensitive data redaction enables cloud API access without transmit- ting protected information, while symbolic analysis operates on metadata. Direct GPT-4 comparison (Table 4) demonstrates privacy necessity: unredacted LLM usage exposes sensitive data in 53.2% of communications, violating privacy regulations regardless of detection accuracy. This separates CyberCane from general-purpose LLM detectors. Design implications. The dual-phase architecture reflects domain-specific requirements: Phase 1 symbolic filtering provides instant, verifiable explanations for IT staff, while Phase 2 RAG handles semantic nuance. Conservative thresholds prioritize false positive minimization over recall, consis- tent with privacy-critical domains’ tolerance for missed detections relative to workflow disruption. The 13.1% review rate represents an operating point adjustable based on staffing capacity and risk tolerance. PhishOnt bridges symbolic and neural phases, providing formal attack taxonomies com- plementing similarity-based retrieval. Limitations. Real-world deployment validation in privacy-critical environments remains necessary beyond DataPhish 2025 and synthetic tests. The lack of publicly available domain-specific phishing corpora (e.g., healthcare, finance) required evaluation on general benchmarks. True adversarial robustness against white-box attackers aware of system internals remains unvalidated; while Phase 2 RAG empirically recovers 99.0% of Phase 1-evaded emails (Section 5.1), adaptive adversaries could craft content that is simultaneously rule-free and semantically dissimilar to the phishing corpus. Residual privacy risks persist despite PHI redaction: quasi-identifiers—rare diagnostic terms, institutional domain patterns, and atypical message timestamps—can re-identify individuals even after direct PII removal, and dense embeddings transmitted to external APIs remain susceptible to inversion attacks [4, 17]. Full mitigation requires organizational controls (business associate agreements, privacy impact assessments, workforce training) complemented by on-premises model deployment for zero-trust environments; additional details are in Appx. B. 7 Conclusion and Future Work CyberCane demonstrates that privacy-preserving neuro-symbolic architectures achieve deployment- grade phishing detection in privacy-critical domains. Our dual-phase pipeline achieves 98.2% pre- cision with 99.1% recall on DataPhish 2025, delivering a 78.6-point recall gain over symbolic-only detection on contemporary AI-generated threats. On Nazario/SpamAssassin, the system reaches 37.2% recall at 99.5% precision under a conservative FPR-first threshold—a deliberate tradeoff where missed detections are tolerated over workflow-disrupting false alarms. PhishOnt ontology reasoning provides formal attack classification with verifiable explanations, while architectural sen- sitive data redaction maintains regulatory compliance. Healthcare deployment analysis indicates 542× ROI for mid-sized organizations, with tunable operating points supporting diverse risk toler- ances. Future work includes adaptive threshold learning from operational feedback, domain-specific dataset curation across privacy-critical sectors, multi-channel detection for SMS and voice phishing, and comprehensive adversarial robustness evaluation in operational environments. By combining formal verifiability with neural adaptability, this work establishes a framework for trustworthy AI- augmented security in life-critical domains. 10 References [1] Ahmed Aleroud and Lina Zhou. Phishing Environments, Techniques, and Countermeasures: A Survey. Computers & Security, 68:160–196, 2017. doi: 10.1016/j.cose.2017.04.006. [2] Anti-Phishing Working Group (APWG). Phishing Activity Trends Report, 4th Quarter 2024. Technical report, APWG, March 2025. URL https://docs.apwg.org/reports/apwg_ trends_report_q4_2024.pdf. Accessed: Jan 2026. [3] Apache SpamAssassin Project.SpamAssassin Public Mail Corpus. https:// spamassassin.apache.org/old/publiccorpus/, 2006. 6,047 labeled spam/ham mes- sages. Accessed: Jan 2026. [4] Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voss, Kather- ine Lee, Adam Roberts, Tom Brown, Dawn Song, Ulfar Erlingsson, et al. Extracting training data from large language models. In 30th USENIX security symposium (USENIX Security 21), pages 2633–2650, 2021. [5] D. Crocker, T. Hansen, and M. Kucherawy. DomainKeys Identified Mail (DKIM) Signatures. Technical Report RFC 6376, Internet Engineering Task Force, September 2011. URL https: //w.rfc-editor.org/rfc/rfc6376. [6] Gianmarco Di Palma, Roberto Scendoni, Davide Ferorelli, Anna De Benedictis, Vittoradolfo Tambone, and Francesco De Micco. AI-Induced Cybersecurity Risks in Healthcare: A Narra- tive Review of Blockchain-Based Solutions Within a Clinical Risk Management Framework. Risk Management and Healthcare Policy, 18:3479–3497, 2025. doi: 10.2147/RMHP.S544523. [7] Pius Ewoh and Tero Vartiainen. Vulnerability to Cyberattacks and Sociotechnical Solutions for Health Care Systems: Systematic Review. Journal of Medical Internet Research, 26, May 2024. doi: 10.2196/46904. [8] Luigi Gallo, Danilo Gentile, Saverio Ruggiero, Alessio Botta, and Giorgio Ventre. The Human Factor in Phishing: Collecting and Analyzing User Behavior When Reading Emails. Comput- ers & Security, 139, 2024. doi: 10.1016/j.cose.2023.103671. [9] Artur d’Avila Garcez and Luís C. Lamb. Neurosymbolic AI: The 3rd Wave. Artificial Intelli- gence Review, 56:12387–12406, 2023. doi: 10.1007/s10462-023-10448-w. [10] Safayat Bin Hakim, Muhammad Adil, Alvaro Velasquez, and Houbing Herbert Song. Symrag: Efficient neuro-symbolic retrieval through adaptive query routing. In Conference on Neurosym- bolic Learning and Reasoning, pages 540–564. PMLR, 2025. [11] Safayat Bin Hakim, Muhammad Adil, Alvaro Velasquez, Shouhuai Xu, and Houbing Herbert Song. Neuro-Symbolic AI for Cybersecurity: State of the Art, Challenges, and Opportunities, 2025. URL https://arxiv.org/abs/2509.06921. [12] Zhiting Hu, Xuezhe Ma, Zhengzhong Liu, Eduard Hovy, and Eric Xing. Harnessing deep neural networks with logic rules. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2410–2420, 2016. [13] IBM Security. Cost of a Data Breach Report 2025. Technical report, IBM Corporation, 2025. Conducted by Ponemon Institute. [Online]. Available: https://w.ibm.com/reports/data- breach. [14] Scott Kitterman. Sender Policy Framework (SPF) for Authorizing Use of Domains in Email, Version 1. Technical Report RFC 7208, Internet Engineering Task Force, April 2014. URL https://w.rfc-editor.org/rfc/rfc7208. [15] T. Koide, N. Fukushi, H. Nakano, and D. Chiba. ChatSpamDetector: Leveraging Large Lan- guage Models for Effective Phishing Email Detection. In Security and Privacy in Communi- cation Networks. Springer, 2026. doi: 10.1007/978-3-031-94455-0_14. [16] M. Kucherawy and E. Zwicky. Domain-based Message Authentication, Reporting, and Con- formance (DMARC). Technical Report RFC 7489, Internet Engineering Task Force, March 2015. URL https://w.rfc-editor.org/rfc/rfc7489. 11 [17] Haoran Li, Mingshi Xu, and Yangqiu Song. Sentence embedding leaks more information than you expect: Generative embedding inversion attack to recover the whole sentence. In Findings of the Association for Computational Linguistics: ACL 2023, pages 14022–14040. Association for Computational Linguistics, 2023. doi: 10.18653/v1/2023.findings-acl.881. [18] Yuexin Li, Chengyu Huang, Shumin Deng, Mei Lin Lock, Tri Cao, Nay Oo, Hoon Wei Lim, and Bryan Hooi. KnowPhish: Large language models meet multimodal knowledge graphs for enhancing Reference-Based phishing detection. In 33rd USENIX Security Symposium (USENIX Security 24), pages 793–810, 2024. [19] Yu A. Malkov and D. A. Yashunin. Efficient and Robust Approximate Nearest Neighbor Search Using Hierarchical Navigable Small World Graphs. IEEE Transactions on Pattern Analysis and Machine Intelligence, 42(4):824–836, 2020. doi: 10.1109/TPAMI.2018.2889473. [20] Jose Nazario. Phishing Corpus. https://monkey.org/~jose/phishing/, 2007. A collec- tion of verified phishing emails (2004–2007). Accessed: Jan 2026. [21] OpenAI. API Pricing. https://openai.com/api/pricing/, 2026. Accessed: Jan 2026. [22] Didem Pehlivanoglu, Alayna Shoenfelt, Ziad Hakim, Amber Heemskerk, Jialong Zhen, Mario Mosqueda, Robert C. Wilson, Matthew Huentelman, Matthew D. Grilli, Gary Turner, R. Nathan Spreng, and Natalie C. Ebner. Phishing Vulnerability Compounded by Older Age, Apolipoprotein E e4 Genotype, and Lower Cognition. PNAS Nexus, 3(8):pgae296, 2024. doi: 10.1093/pnasnexus/pgae296. [23] Proofpoint.2024 State of the Phish Report.Technical report,Proofpoint, Inc.,2024.URL https://w.proofpoint.com/sites/default/files/ threat-reports/pfpt-us-tr-state-of-the-phish-2024.pdf.[Online]. Avail- able: https://w.proofpoint.com/us/resources/threat-reports/state-of-phish. [24] Yashothara Shanmugarasa, Ming Ding, Chamikara Mahawaga Arachchige, and Thierry Rako- toarivelo. SoK: The Privacy Paradox of Large Language Models: Advancements, Privacy Risks, and Mitigation. In Proceedings of the 20th ACM Asia Conference on Computer and Communications Security, pages 425–441, 2025. doi: 10.1145/3708821.3733888. [25] Hossein Shirazi, Bruhadeshwar Bezawada, and Indrakshi Ray. Kn0w Thy Doma1n Name: Unbiased Phishing Detection Using Domain Name Based Features. In Proceedings of the 23rd ACM Symposium on Access Control Models and Technologies, pages 69–75, 2018. doi: 10.1145/3205977.3205992. [26] Marco Simoni, Andrea Saracino, Vinod P, and Mauro Conti. MoRSE: Bridging the Gap in Cybersecurity Expertise with Retrieval Augmented Generation. In Proceedings of the 40th ACM/SIGAPP Symposium on Applied Computing, pages 1213–1222, 2025. doi: 10.1145/ 3672608.3707898. [27] Rebeka Toth, Tamas Bisztray, and Richard Dubniczky.Constructing and benchmark- ing:a labeled email dataset for text-based phishing and spam detection framework. arXiv preprint arXiv:2511.21448, 2025.Dataset: https://github.com/DataPhish/ PhishingSpamDataSet. [28] U.S. Bureau of Labor Statistics.Information Security Analysts: Occupational Outlook Handbook. https://w.bls.gov/ooh/computer-and-information-technology/ information-security-analysts.htm, 2024. Accessed: Jan 2026. [29] Verizon. 2025 Data Breach Investigations Report. Technical report, Verizon Communications Inc., 2025. URL https://w.verizon.com/business/resources/reports/dbir/. Accessed: Jan 2026. 12 Contents of Appendix A Detailed System Implementation A.1 Phase 1: Complete Rule Specifications (Algorithm 2) A.2 Phase 2: RAG Implementation Details (Algorithm 3) A.3 Reproducibility B Healthcare Validation Details B.1 Operational and Privacy Considerations B.2 Synthetic Data Generation Methodology B.3 Difficulty Spectrum Evaluation B.4 Per-Category Detection Results B.5 Example Synthetic Emails C Retrieval Quality Analysis D Threat Model and Attack Taxonomy E Additional Ablation Studies E.1 Deterministic Rule Group Ablations E.2 Complete Per-Rule Feature Importance E.3 Threshold Sensitivity Analysis E.4 RAG Sensitivity Studies F Supplementary Results F.1 Cost-Benefit Analysis Across Operating Modes F.2 Baseline Comparison F.3 DataPhish 2025 Detailed Breakdown F.4 Statistical Validation F.5 Error Analysis F.6 PII Redaction Statistics F.7 AI Output Summary F.8 Operating Characteristic Analysis F.9 Explainability Analysis F.9.1 Representative System Output F.10 Complete Failure Taxonomy 13 A Detailed System Implementation This appendix provides complete implementation specifications for Phase 1 symbolic rules and Phase 2 RAG pipeline omitted from the main text for brevity. A.1 Phase 1: Complete Rule Specifications Algorithm 2 formalizes the deterministic pipeline omitted from the main text. The algorithm pro- cesses email headers, body, and URLs through DNS validation, authentication checks, and content analysis, producing weighted risk scores that map to verdicts via threshold-based classification. Algorithm 2: Phase 1: Deterministic Symbolic Analysis Input: Email e with headers H , body B, URLs U , weights w Output: Indicators I , score s, verdict ℓ Function DeterministicAnalysis(e) /* Extract sender, URL, and content signals for scoring */ d← domain(H.from) I ←∅ s← 0 I ← I∪ CheckDNS(d) I ← I∪ CheckAuth(d) if HasDomainMismatch(U,d) then I ← I∪domain_mismatch /* URL structure: shorteners, obfuscation, IP literals */ I ← I∪ AnalyzeURLs(U ) /* Content cues: urgency, credential requests, freemail cues */ I ← I∪ ExtractFeatures(B) s← P f∈I w[f ] /* w ∈1, 2, 3*/ if s < 2 then ℓ← benign else if s < 5 then ℓ← needs_review else ℓ← phishing return (I,s,ℓ) DNS and Authentication Checks. Missing MX Records (weight=3): Query DNS for MX records via dns.resolver.query(domain, ’MX’). Absence indicates no mail infrastructure, flagging likely spoofing. No SPF Record (weight=2): Check TXT records for SPF policies. Missing SPF allows do- main impersonation. No DMARC Policy (weight=2): Query _dmarc.domain TXT records. Absence weakens authentication verification. SPF Softfail (weight=1): Parse SPF results for∼all directives suggesting unverified sender. Domain Analysis. Freemail Domain (weight=1): Match sender against list: gmail.com, yahoo.com, outlook.com, hotmail.com, aol.com, proton.me. Domain Mismatch (weight=2): Extract domains from all URLs via urlparse(), flag if∃ URL domain̸= sender domain. URL Patterns. URL Shortener (weight=2): Match against: bit.ly, tinyurl.com, t.co, goo.gl, ow.ly, buff.ly. IP Literal in URL (weight=2): Regex: https?:// 1,3(\. 1,3)3. URL Obfusca- tion (weight=2): Detect hex encoding (%[0-9A-Fa-f]2), URL encoding, or excessive parameters. Content Heuristics. Urgency Keywords (weight=1): Case-insensitive scan for: urgent, immediate action, verify your account, password expires, suspend, pay now, update your information. Creden- tial Request (weight=2): Regex for password, SSN, social security, credit card, log +in, credentials. Generic Greeting (weight=1): Match: dear customer, dear user, valued member (instead of person- alized names). A.2 Phase 2: RAG Implementation Details Algorithm 3 formalizes the complete neuro-symbolic RAG pipeline that integrates ontology-inferred attack types from Phase 1 with vector retrieval and LLM reasoning. The algorithm shows how formal symbolic knowledge enhances neural classification through structured context injection. 14 Algorithm 3: Phase 2: Neuro-Symbolic RAG Classification Input: Email e, Phase 1 (I,ℓ 1 ), ontology (A,E), corpus C, τ h = 0.40, τ l = 0.25 Output: RAG score s rag , verdict ℓ, explanations R Function NeuroSymbolicRAG(e,I,ℓ 1 ,A,E,C) /* Redact before any external API call */ e red ← RedactPHI(e) v ← Embed(e red ) N ← HNSW(v,C,k = 8) // Top-8 neighbors s top ← MaxSim(N) s avg ← MeanTop3(N) /* Compute similarity statistics for decision thresholds */ ctx← BuildContext(I,A,E,N) /* Include ontology traces for explainability */ R← LLM(ctx,T = 0.2) s rag ← 0.65· s top + 0.35· s avg /* Verdict logic: conservative cascade */ if ℓ 1 = phishing then ℓ← phishing // Deterministic override else if s top ≥ τ h ∨ (ℓ 1 = needs_review∧ s avg ≥ 0.35) then ℓ← phishing else ℓ← ℓ 1 return (s rag ,ℓ,R) PHI Redaction Patterns. Email: [a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]2,→ a****b@domain.com. Phone: 3[-. ]? 3[-. ]? 4→ ***-***-1234. SSN: 3- 2- 4→ ***-**-6789. Credit Card: 4[ -]? 4[ -]? 4[ -]? 4→ ****-****. Embedding and Retrieval. Model: OpenAI text-embedding-3-small (1536 dimensions, $0.02 per 1M tokens [21]).Vector DB: PostgreSQL with pgvector extension, HNSW indexing (ef_construction=200, M=16). Corpus: 2,297 labeled phishing emails from training split, pre- embedded and indexed. Query: SELECT id, similarity FROM messages ORDER BY embedding <=> query_vec LIMIT 8. LLM Prompting. System prompt: “You are a cybersecurity analyst specializing in email phishing detection for privacy-critical organizations. Analyze this email and explain why it may be phishing, citing specific evidence.” User prompt template: “Email (redacted): [SUBJECT] [BODY]. Phase 1 indicators: [INDICATORS]. Similar phishing examples: [TOP-3 NEIGHBORS]. Generate 3-5 concise explanations using tags: [AUTH], [URL], [URGENCY], [CONTENT], [SIMILARITY].” Model: gpt-4.1-mini ($0.40 per 1M input, $1.60 per 1M output tokens [21]). Temperature: 0.2 (deterministic explanations). Threshold Calibration. Validation-set grid search over Phase 1 thresholds1, 2, 3, 4, 5 and RAG similarity thresholds 0.45, 0.50, 0.55, 0.60, 0.65, 0.70. Selected (t phase1 = 2,t rag = 0.70) min- imizing FPR while maintaining recall > 15%. Operating points derived via systematic threshold sweeps (Table 6). A.3 Reproducibility Specifications. Random seed=42 (HNSW construction, splits); stratified splits by source and label; text-embedding-3-small (1536-dim); gpt-4.1-mini (temperature=0.2, max_tokens=500); HNSW de- fault parameters. Open-source implementation includes symbolic rule engine, RAG pipeline, PHI redaction module, evaluation harness, and datasets (Nazario.clean, SpamAssassin). Healthcare- specific evaluation not in public artifacts—main tables fully reproducible from included datasets. External API dependencies mean exact numerical results may vary slightly due to model updates; we archive API versions and response logs. Project Website. Complete documentation, interactive architecture diagrams, evaluation results, and deployment guides are available at https://cybercane.netlify.app. Implementation Provenance. The CyberCane system presented in this work represents a complete system re-alignment with novel architectural and methodological contributions. While preliminary 15 concept exploration occurred in an educational setting, 2 the present implementation constitutes in- dependent development with differences in system redesign, privacy architecture, formal reasoning integration, comprehensive evaluation methodology, and statistical validation framework. B Healthcare Validation Details B.1 Operational and Privacy Considerations Review load. At the conservative operating point (threshold=2), 13.1% of emails in the test split are routed to needs_review (Table 23A), indicating the expected human triage burden under low-FPR settings. Phase 1 baseline. At the same threshold, Phase 1 achieves 83.0% precision, 17.8% recall, and 2.9% FPR on Nazario/SpamAssassin (Table 3B), motivating Phase 2 escalation for semantic coverage. Privacy controls. Redaction reduces direct PHI exposure, but quasi-identifiers and embedding inversion risks remain [4]. Deployment should include organizational controls (business associate agreements, security policies, workforce training) and a privacy impact assessment. B.2 Synthetic Data Generation Methodology Naive Baseline. Template-based generation with explicit red flags: (1) Subjects contain “UR- GENT”, “IMMEDIATE”, “ACTION REQUIRED”; (2) Bodies request passwords, SSN, credit cards; (3) URLs use IP addresses (e.g., http://185.234.123.45) or shorteners (bit.ly); (4) Threat- ening language (“Your account will be locked”). Total: 48 emails (12 per category). Sophisticated Healthcare.Multi-model generation (gpt-4.1-mini, deepseek-chat, Claude 3.5 Haiku; temperature=0.8, seed=42, prompt_version=healthcare_v1) with prompts specifying: (1) realistic medical terminology; (2) professional tone avoiding overt urgency; (3) typosquatted do- mains (johnshopkins-health.com, medicare-benefits.org); (4) targeting elderly/vulnerable pop- ulations. Raw target: 50 per category (200 total). Quality filters enforce URL presence, subject/body length (5–100 / 100–1000 chars), and category keywords; 18 samples fail these gates, yielding 182 candidates. Exact and near-duplicate filtering (TF-IDF cosine >0.85) removes 17 samples. Con- tamination checking against Nazario/SpamAssassin/Phishing validation corpora (text-embedding- 3-small, threshold 0.90) removes 0 samples. Final dataset: n = 165, 100% URL presence, mean body length 471 chars, 60.6% unique subjects. B.3 Difficulty Spectrum Evaluation Table 8: Difficulty spectrum evaluation: naive baseline (100% recall) validates correctness; sophisticated at- tacks (18.2% recall, balanced mode) require aggressive thresholds (87.3% recall) to achieve coverage. DatasetOperating ModePrecisionRecallFPR Generic (Original) Ultra-Conservative99.5%37.2%0.16% Balanced99.5%40.0%0.16% Aggressive99.5%44.6%0.16% Naive Baseline (obvious red flags) Ultra-Conservative93.8%31.2%0.5% Balanced96.0%100.0%1.0% Aggressive87.3%100.0%3.5% Sophisticated (AI-generated) Ultra-Conservative0.0%0.0%0.5% Balanced93.8%18.2%1.0% Aggressive95.4%87.3%3.5% Per-Category Breakdown (Balanced Mode): Appointment Scams60.0%9.7%1.0% Insurance Verification33.3%2.1%1.0% Prescription Fraud0.0%0.0%1.0% EHR Credential Theft92.9%57.8%1.0% Table 8 presents complete results across attack difficulty levels and operating modes. The sophis- ticated synthetic set is the filtered multi-model dataset (n = 165) with 100% URL presence, dedu- plication, and contamination checks. On naive baseline attacks, balanced mode achieves 96.0% precision with 100% recall—validating system correctness and confirming conservative thresholds successfully detect unsophisticated phishing. However, on sophisticated AI-generated healthcare attacks, the same balanced configuration achieves 18.2% recall (93.8% precision), demonstrating 2 Initial prototype: Git repo 16 that modern LLM-generated phishing evades conservative rule-based thresholds. Aggressive mode recovers substantial coverage: 87.3% recall at 95.4% precision (3.5% FPR), showing detection ca- pability exists when thresholds adapt to threat sophistication. Key findings. Three findings emerge: (1) 100% recall on template-based attacks validates archi- tectural correctness—conservative thresholds successfully detect unsophisticated threats; (2) per- formance drop (100% → 18.2%) on GPT-generated attacks demonstrates LLM-generated evasion capability; (3) aggressive mode’s 87.3% recovery shows detection capability exists—threshold cal- ibration, not redesign, determines coverage. This demonstrates that healthcare organizations need flexible operating points: rule-based systems struggle to maintain 96%+ precision while achieving meaningful recall against AI-generated attacks without tunable detection sensitivity. B.4 Per-Category Detection Results Table 9: Per-category detection performance (Balanced mode) on healthcare synthetic dataset. Attack CategoryPrecisionRecall Appointment Scams60.0%9.7% Insurance Verification33.3%2.1% Prescription Fraud0.0%0.0% EHR Credential Theft92.9%57.8% Overall (Balanced)93.8%18.2% Overall (Aggressive)95.4%87.3% Table 9 reports balanced-mode performance across synthetic attack categories. EHR credential theft achieves 92.9% precision with 57.8% recall—credential-request patterns (verify password, reset ac- count) trigger Phase 1 rules reliably. Appointment scams reach 60.0% precision with 9.7% recall, insurance verification reaches 33.3% precision with 2.1% recall, and prescription fraud remains at 0% recall in balanced mode—these attacks employ professional language, context-appropriate med- ical scenarios, and realistic typosquatted domains that evade keyword-based heuristics. Aggressive mode raises overall recall to 87.3% (95.4% precision), demonstrating threshold adaptation enables coverage of sophisticated attacks. B.5 Example Synthetic Emails Naive Baseline Example (detected): Subject: URGENT: Your Patient Portal Will Be LOCKED - Verify Now! Body: “SECURITY ALERT: Your patient portal account will be locked in 24 hours due to suspicious activity. Visit http://198.45.123.67/portal-verify and enter your username, pass- word, and SSN immediately to prevent account closure!” Sender: security@mypatient-portal.tk Detection: Phase 1 score=8 (IP literal +2, urgency +1, credential request +2, missing DMARC +2, URL shortener +1). Verdict: Phishing. Sophisticated Example (evaded balanced mode): Subject: Appointment Confirmation: Dr. An- derson - Johns Hopkins Body: “Dear Patient, Your upcoming appointment with Dr. Anderson at Johns Hopkins Medical Center is scheduled for March 15, 2026 at 2:00 PM. To confirm your attendance and review pre-visit forms, please visit our patient portal at johnshopkins-health.com/ appointments within the next 48 hours. Thank you for choosing Johns Hopkins Health System.” Sender: appointments@medstar-system.net Detection: Phase 1 score=1 (freemail=0, missing indi- cators=1). RAG similarity=0.42 (below 0.70 threshold). Verdict: Benign (false negative in balanced mode). Aggressive mode (threshold=0.35) correctly classifies as phishing. C Retrieval Quality Analysis To validate that the RAG pipeline retrieves semantically and structurally relevant phishing examples rather than random samples, we performed a qualitative review of 50 random test emails paired with their top-1 retrieved neighbor. Table 10 presents representative pairs across common phishing themes encountered in the dataset. The analysis identifies three key findings. First, high-similarity matches (> 0.70) predominantly occur when the query uses a common phishing template (e.g., billing failures or account verification) that matches historical structural patterns in the corpus. Second, functional matches (e.g., mailbox storage alerts) are correctly retrieved even when specific subject line wording varies, demonstrat- ing the semantic robustness of the text-embedding-3-small space. Third, benign queries (e.g., technical discussions or newsletters) yield significantly lower or negative similarity scores (< 0.10), 17 providing a clear boundary that prevents neural escalation of legitimate communications. These re- sults confirm that the RAG escalation logic described in Section 3.3 is grounded in genuine semantic overlap with known threats. Table 10: Representative query-neighbor pairs demonstrating semantic and structural retrieval relevance. Query ThemeQuery Subject SnippetNeighbor Subject SnippetSimilarity BillingWe’re having trouble with billing...We’re having trouble with billing...0.7999 StorageAuto Email Upgrade jose@...Mail Storage Alert j**e@...0.6302 Transferjust received files via WeTransferyou received files via WeTransfer0.5532 PrescriptionOnline Doc, can give prescriptionRe: Your Online Prescription Source0.3452 Benign (Non-Match)Re: JPEGs patented (Beatles...)nice foto YOHvTp2KSxRjQ...-0.1041 D Threat Model and Attack Taxonomy Note: This section demonstrates the framework’s threat modeling approach using healthcare as an illustrative domain. The attack taxonomy generalizes to other privacy-critical environments (fi- nance, legal, government) with domain-specific adaptations. Appointment/ Check-in Scams Urgency tactics ATTACKER DEFENDER Domain Control AI Content Social Eng. SMTP Headers DNS Queries Human Review 34% 26% 19.5% 20.5% Insurance Phishing Blling Complexity Prescription Fraud Medication urgency EHR Credential Theft System compromise IN-SCOPEOUT-OF-SCOPE Email Phishing Social Engineering Credential Harvesting Sensitive Data Collection QR Codes Voice/SMS Phishing Malicious PDFs Internal Threats Domain-Specific Phishing Threat Taxonomy: Healthcare Application Framework generalizes to finance, legal, and government domains Figure 3: Domain-specific phishing threat taxonomy demonstrated through healthcare use case, showing four primary attack categories with distinct social engineering tactics and technical exploitation vectors illustrative of privacy-critical domain requirements. Attacker capabilities. We assume: (1) Control over compromised legitimate domains passing SPF/DKIM or typosquatted lookalikes (e.g., medstar-health.org vs medstarhealth.org in healthcare); (2) Legitimate infrastructure hosting (bit.ly, Google Drive, Dropbox) evading URL blacklists; (3) AI-augmented content generation producing convincing domain-specific language [6]; (4) Social engineering expertise exploiting domain-specific terminology, workflows, and psychological pres- sure points (e.g., appointment anxiety, prescription urgency, insurance complexity in healthcare). In-scope attacks. Email-based phishing with text bodies and embedded hyperlinks, urgency-based social engineering, credential harvesting via fake portals, sensitive data collection through embedded forms, business email compromise impersonating executives/vendors. Out-of-scope. Image-only phishing without OCR, QR code phishing requiring computer vision, document exploits (malicious PDFs/Office files) without sandboxing, fully authenticated internal threats from compromised accounts, voice/SMS phishing. Defender assumptions. Organizations have: (1) Full SMTP header access (From, Reply-To, Re- ceived, Authentication-Results); (2) Real-time DNS query capability (MX, SPF, DMARC); (3) La- 18 beled historical phishing from public corpora or internal feeds; (4) Human review channels; (5) OpenAI API connectivity or on-premises model infrastructure. Attack taxonomy example. Fig. 3 illustrates four primary healthcare-specific attack categories as an example of domain-specific threat modeling for future evaluation: Appointment/Check-in Scams (34%)—impersonate providers requesting urgent confirmation/rescheduling via malicious links; Insurance Verification Phishing (26%)—fake benefits verification, coverage confirmation, card updates; Prescription Notification Fraud (20.5%)—pharmacy impersonation requesting ver- ification, payment, delivery confirmation; EHR Credential Theft (19.5%)—fake login portals en- abling broader system compromise and sensitive data exfiltration. E Additional Ablation Studies E.1 Deterministic Rule Group Ablations Table 11: Deterministic ablations on mixed-label test split (n=1,110). AblationAcc.Prec.RecallF1FPR Baseline0.6170.8300.1780.2930.029 w/o URL heuristics0.6040.8020.1470.2490.029 w/o urgency0.5950.7920.1230.2130.026 w/o cred. request0.5960.9120.1050.1880.008 w/o auth checks0.6170.8300.1780.2930.029 w/o brand checks0.6170.8300.1780.2930.029 Table 11 reports metrics for baseline deterministic configuration and variants with URL heuristics, urgency cues, credential-request rules, authentication checks, or brand checks removed. Removing urgency or credential-request rules reduces recall and F1 while slightly lowering FP. Disabling URL heuristics yields moderate recall drop. Authentication and brand-check ablations show minimal change on this dataset. E.2 Complete Per-Rule Feature Importance Table 12 extends the top-5 summary from main text with complete leave-one-out analysis. Three im- portance metrics: (1) phishing coverage (percentage triggering rule), (2) rule precision (P(phishing | triggered)), (3) performance delta (recall/FPR change when removed).Results show two- tier hierarchy: HIGH importance (no_dmarc 46.5% coverage, creds_request), MEDIUM impor- tance (DNS checks missing_mx/no_spf 30% coverage with 41–47% precision, content heuristics urgency/ip_literal_link 90–94% precision but 6–7% coverage), LOW importance (brand-specific rules never triggered due to dataset composition). Table 12: Complete per-rule feature importance via leave-one-out ablation (threshold=2). RuleWeightPhish Cov.Rule Prec.Recall ∆FPR ∆Importance No DMARC146.5%38.0% −0.125 −0.182HIGH Credential request20.0%—+0.047+0.013HIGH Urgency language27.3%90.0%+0.0380.000MEDIUM Missing MX231.3%46.6%+0.030+0.016MEDIUM IP literal link26.7%94.3%+0.0160.000MEDIUM No SPF230.9%40.8% −0.014+0.016MEDIUM Shortened URL20.8%100%0.0000.000LOW Freemail brand claim20.0%—0.0000.000LOW Lookalike domain20.0%—0.0000.000LOW Strict DMARC (no align)30.0%—0.0000.000LOW Notes. Phish Cov. = % of phishing emails triggering rule; Rule Prec. = P(phishing| rule triggered). Recall/FPR ∆ = performance change when the rule is removed; negative values indicate degradation. E.3 Threshold Sensitivity Analysis Table 13 summarizes validation-set metrics across thresholds. Lower thresholds increase recall at precision cost; higher thresholds trade recall for fewer FP. We select threshold=2 for needs_review boundary, threshold=5 for high-confidence phishing. Fig. 4 pairs the validation precision/recall tradeoff with the test-score distribution. 19 Table 13: Validation-set threshold sensitivity for deterministic metrics. ThresholdAcc.Prec.RecallF1 00.4450.4451.0000.616 10.4600.4460.8920.595 20.6520.8850.2510.390 30.6420.8750.2280.362 40.6160.8850.1570.266 50.5810.9120.0630.118 010203040 Threshold 0.0 0.2 0.4 0.6 0.8 1.0 Score t=2 (A)Validation Precision/Recall Precision Recall 0246822 Score 0 200 400 600 800 1000 Count (B)Deterministic Score Histogram Threshold Tuning and Deterministic Score Distribution Figure 4: Threshold tuning and deterministic score distribution. (A) Validation precision/recall versus threshold with the selected operating point. (B) Test-set score histogram showing the concentration of low scores. E.4 RAG Sensitivity Studies Retrieval corpus ablation. Table 14 (A) compares phishing-only vs mixed-corpus (phish + benign) retrieval at k = 8. Mixed-corpus matches phishing-only performance on this dataset, sug- gesting corpus limitation doesn’t materially change similarity-driven boundaries. Table 14: Ablation results for retrieval corpus, redaction, and DNS sensitivity across deterministic and RAG variants test split. (A) Retrieval corpus ablation VariantAcc.Prec. RecallF1FPR Phase 1 only0.568 0.8640.0380.074 0.005 RAG (phish-only)0.593 0.9390.0930.169 0.005 RAG (mixed)0.593 0.9390.0930.169 0.005 (B) Redaction ablation VariantAcc.Prec. RecallF1FPR Phase 1 only0.568 0.8640.0380.074 0.005 RAG (no redaction) 0.593 0.9390.0930.169 0.005 RAG (redacted)0.586 0.9290.0790.145 0.005 (C) DNS sensitivity VariantAcc.Prec. RecallF1FPR Phase 1 (no DNS)0.568 0.8640.0380.074 0.005 Phase 1 (DNS)0.544 0.4830.3210.386 0.276 RAG (no DNS)0.593 0.9390.0930.169 0.005 RAG (DNS)0.561 0.5110.3600.422 0.276 Redaction impact. Table 14 (B) compares raw vs redacted query embeddings. Redaction yields modest recall/F1 drop while keeping precision/FPR unchanged—manageable privacy-utility trade- off. DNS sensitivity.Table 14 (C) compares deterministic and RAG with DNS enabled vs dis- abled. Modest shifts indicate authentication lookups don’t materially change performance on public datasets. RAG threshold sensitivity. Table 15 (left) reports validation-set performance shifting similarity thresholds jointly. Lowering thresholds modestly improves recall/F1 with minimal precision loss; higher thresholds trade recall for slightly higher precision. Small shifts indicate stable operating region. 20 Table 15: RAG threshold sensitivity (left) and bootstrap confidence intervals for deterministic metrics (right). (A) RAG threshold sensitivity ShiftAcc.Prec. RecallF1 -0.030 0.608 0.9530.1240.220 0.0000.605 0.9510.1180.210 0.0300.601 0.9460.1080.194 (B) Deterministic bootstrap CI MetricMean Lower Upper Accuracy 0.6170.5890.645 Precision 0.8290.7590.894 Recall0.1770.1470.210 F10.2910.2490.337 F Supplementary Results F.1 Cost-Benefit Analysis Across Operating Modes Table 16 presents detailed economics for five operating points. Aggressive mode achieves highest ROI (651×) with 196 daily detections, while baseline delivers 542× ROI with 164 detections. Table 16: Cost-benefit analysis across operating modes (10,000 daily emails, 4.4% phishing rate). Operating ModeAttacks DetectedCost/DayROINet Benefit Baseline (Pipeline)164$1,506542.0×$816,272 Conservative130$1,506429.9×$647,383 Balanced176$1,506583.3×$878,494 Moderate186$1,506615.8×$927,383 Aggressive196$1,506651.2×$980,716 F.2 Baseline Comparison Table 17 compares CyberCane against text-only baselines. Majority baseline reflects class imbal- ance; TF-IDF logistic regression achieves high F1 on public dataset, indicating strong lexical sepa- rability potentially not generalizing to healthcare threats. Table 17: Text-only baseline performance on mixed-label test split (n=1,110). BaselineAcc.Prec.RecallF1FPR Majority0.5540.0000.0000.0000.000 TF-IDF LogReg0.9820.9860.9740.9800.011 F.3 DataPhish 2025 Detailed Breakdown Table 18: DataPhish 2025 emotional cohort and LLM source detailed breakdown. (A) Emotional Cohort Analysis (DataPhish 2025) EmotionnPhase 1 RecallPhase 2 Recall∆ Altruism809.8%100%+90p Curiosity9765.0%99.2%+94p Greed4322.8%98.2%+95p Neutral8884.2%99.0%+95p Authority90835.9%99.9%+64p Fear65539.9%99.1%+59p Urgency117728.9%99.1%+70p (B) LLM Source Detection CreatornPhase 1 RecallPhase 2 Recall∆ GPT-5 Mini90%100%+100p Gemini 1.5 Pro912.5%100%+87p Mistral Medium 3.1547.7%97.4%+90p OpenAI/GPT-4o59619.6%99.3%+80p DeepSeek-Chat60222.4%99.0%+77p Human55915.6%99.2%+84p Table 18 provides complete emotional cohort and LLM source analysis for DataPhish 2025 contem- porary threats. Part (A) quantifies psychological engineering blind spots: deterministic rules achieve <10% recall on altruism, curiosity, greed, and neutral tones—tactics absent in 2006-era training— while RAG recovers 98-100%. Part (B) validates semantic understanding independent of authorship: LLM-generated emails (GPT-5, Gemini, Mistral, OpenAI, DeepSeek) show 0-22.4% Phase 1 recall versus 97-100% Phase 2 recall, with human-written emails exhibiting similar performance (15.6% vs 99.2%), confirming detection relies on semantic similarity rather than LLM-specific artifacts. F.4 Statistical Validation Bootstrap confidence intervals. Table 15 (right) reports bootstrap 95% CI for deterministic metrics at threshold=2 (1,000 resamples), confirming statistical stability. Table 19 reports bootstrap 95% CI (1,000 resamples) for Phase 1 (threshold=5) and RAG (k = 8). Narrow bands confirm reliable performance. 21 Table 19: Bootstrap 95% confidence intervals for Phase 1 (threshold=5) and RAG (k = 8). MetricPhase 1RAG (k=8) Accuracy0.556 (0.527, 0.586)0.719 (0.692, 0.744) F10.012 (0.000, 0.027)0.541 (0.493, 0.583) Precision0.750 (0.000, 1.000)0.995 (0.982, 1.000) Recall0.006 (0.000, 0.014)0.372 (0.328, 0.412) Statistical significance testing. Table 20 presents paired tests comparing Phase 1 (threshold=5) vs RAG (k = 8). McNemar’s test yields χ 2 = 181.0 (p < 0.001), confirming significantly different predictions. Bootstrap F1-score improvement +0.527 with 95% CI [0.480, 0.572] (p < 0.001) demonstrates robust gains. Table 20: Statistical significance: Phase 1 vs RAG (10,000 bootstrap samples). TestMetricStatistic p-valueSignificant McNemar’s TestOverall Performance181.0 <0.001Yes Bootstrap CIPrecision Improvement+0.2520.371No Bootstrap CIFPR Reduction0.000—No Bootstrap CIF1-Score Improvement+0.527 <0.001Yes F.5 Error Analysis Table 21 (left) reports deterministic performance by source. Nazario.clean contains phishing- only; mixed-label source required to contextualize FP behavior. Cross-source variability motivates healthcare-specific validation. Table 21 (right) summarizes frequent reasons in FP and FN. Counts show which rules dominate errors, providing refinement targets. Table 21: (A) Deterministic performance by source and (B) top deterministic reasons in false posi- tives/negatives. (A) Error by source SourcePrec. RecallF1FPRSupport Nazario1.0000.2250.367 0.000 236.000 SpamAssassin 0.6600.1350.224 0.029 874.000 (B) Error reasons FP ReasonFP Count FN Reason FN Count Creds/PII request160 Urgency language40 IP literal link20 F.6 PII Redaction Statistics Table 22 reports redaction counts on test split, quantifying sensitive pattern prevalence motivating privacy-first design. Redaction applied before any external API call. F.7 AI Output Summary Table 22: PII redaction counts on mixed-label test split (n=1,110). PII TypeCount email46233 phone236 dob218 c24 ssn1 Table 23 (left) summarizes AI verdict distribution and similarity statistics showing Phase 2 impact. 876 emails (78.9%) remain benign, 145 (13.1%) escalate to needs_review, 89 (8.0%) re- ceive phishing verdicts. Mean top similarity 0.171 with median 0.045 indicates right-skewed distribution where most show low semantic overlap with corpus. F.8 Operating Characteristic Analysis Table 23 (middle) summarizes complete operating space across 45 thresholds (0–22).AU- ROC=0.574 indicates limited discriminative power from symbolic rules alone; AUPRC=0.673 demonstrates moderate imbalanced performance. Table 23: (A) AI output summary, (B) ROC operating space, and (C) explanation tag distribution with concise- ness on the mixed-label test split (n=1,110). (A) AI output summary MetricValue AI verdict benign876 (78.9%) AI verdict needs_review 145 (13.1%) AI verdict phishing89 (8.0%) AI score mean (0-10)1.97 Top similarity mean0.171 Top similarity median0.045 (B) ROC operating space MetricValue AUROC0.574 AUPRC0.673 Thresholds Evaluated45 (0–22) Max F1 Score0.617 @ thr=0 Operating Point (thr=2) Precision0.830 Recall0.178 FPR0.029 F10.293 (C) Explanation tags and conciseness Explanation TagCount Percentage [SIMILARITY]723.3% [AUTH]620.0% [URL]620.0% [CONTENT]620.0% [URGENCY]516.7% Mean words/reason11.6 Median words/reason12 22 Max F1=0.617 occurs at threshold=0 with 100% recall but only 44.6% precision—aggressive detec- tion sacrifices precision. Our threshold=2 operates far from F1 optimum, prioritizing precision/low FPR. Fig. 5 (A) visualizes the ROC operating space. The operating point lies on a steep segment where modest FPR increases yield limited recall gains, validating conservative threshold selection. 0.00.20.40.60.81.0 False Positive Rate 0.0 0.2 0.4 0.6 0.8 1.0 True Positive Rate (A)ROC Curve (Deterministic) Deterministic (Phase 1) Random baseline Operating point (t=2) −0.20.00.20.40.60.81.0 Cosine Similarity 0 50 100 150 200 250 Count (B)Top Neighbor Similarity (Test) Mean Median Deterministic ROC and RAG Similarity Distribution Figure 5: Deterministic ROC and RAG similarity distribution. (A) ROC curve for Phase 1 with the threshold=2 operating point. (B) Top-neighbor similarity histogram on the test split, highlighting the right-skewed distribu- tion used for semantic escalation. F.9 Explainability Analysis Fig. 5 (B) shows the top neighbor similarity distribution on the test split, anchoring AI scoring and interpreting escalation thresholds. Table 23 (right) shows explanation tag distribution across sampled emails. Balanced distribu- tion (20–23% per tag) indicates multi-source evidence rather than single indicators. Mean 11.6 words/reason ensures conciseness with actionable context. Table 24 quantifies tag-level groundedness checking whether tagged reasons are supported by mea- surable indicators on 60 random emails (300 reasons). Multi-tag structure combines symbolic evi- dence ([AUTH], [URL]), semantic retrieval ([SIMILARITY]), and content analysis ([CONTENT], [URGENCY]), providing verifiable reasoning layers. Table 24: Tag-level groundedness: fraction of tagged reasons supported by indicators. TagTotalSupportedUnsupportedUnknownSupport Rate AUTH75510240.680 CONTENT2540210.160 SIMILARITY5454001.000 URGENCY1931150.158 URL2400240.000 F.9.1 Representative System Output Fig. 6 illustrates a representative CyberCane detection output for a Medicare-themed phishing email, demonstrating the multi-layered explanation architecture described in Section 3.3. The system presents evidence across three interpretable layers: Layer 1 (Symbolic Evidence) surfaces technical violations verifiable by IT staff without special- ized cybersecurity training: SPF authentication failure, domain mismatch between sender and link destination (verify-medicare.com vs medicare.gov), urgency language patterns, and suspicious URL characteristics. Each indicator maps directly to Algorithm 2’s deterministic rules. Layer 2 (Retrieved Examples) grounds the decision in concrete historical attacks, showing the top-3 similar phishing emails from the corpus with similarity scores of 91.2%, 88.7%, and 86.3%. This transparency allows human reviewers to assess whether the system’s pattern matching aligns with genuine threats rather than superficial text overlap. Layer 3 (LLM Reasoning) synthesizes symbolic and retrieval evidence into contextual analysis, identifying that the combination of domain typosquatting, urgency tactics, and Medicare theme matches 98.3% of known elderly-targeted phishing campaigns. The explanation explicitly refer- ences authentication failures to provide actionable technical justification. 23 PHISHING DETECTED (Confidence: 94%) Why this is phishing: Technical: SPF authentication failed, domain mismatch detected Pattern: 98.3% similar to known Medicare phishing campaigns Context: Combines urgency + credential request typical of elderly-targeted scams Suspicious Email "Urgent: Verify your Medicare ID within 24 hours at verify-medicare.com" Missing SPF record (sender authentication failed) Domain mismatch verify-medicare.com ≠ medicare.gov High urgency language "urgent", "within 24 hours" Suspicious URL pattern hyphenated domain name Layer 1: Symbolic Evidence Layer 2: Retrieved Examples Top Similar Phishing Examples: 91.2% similar: "Action Required: Update Medicare information urgently" 88.7% similar: "Verify insurance details at secure-healthgov.com" 86.3% similar: "Your medical records need immediate verification" Layer 3: LLM Reasoning GPT-4.1-mini Contextual Analysis: This email exhibits multiple phishing indicators consistent with Medicare/insurance scams. The domain typosquatting (verify-medicare vs medicare.gov) combined with urgency tactics matches 98.3% of known phishing patterns targeting elderly healthcare recipients. Authentication failure confirms unauthorized sender. Output: Multi-Layer Explanation Recommended action: Do not click links. Report to IT security. Figure 6: Representative CyberCane output demonstrating multi-layered explainability combining symbolic rule violations, retrieved phishing examples, and LLM-generated contextual analysis for healthcare IT staff. The final output presents a clear 94% confidence verdict with specific recommended actions (“Do not click links. Report to IT security”), addressing the healthcare requirement for transparent decision- support rather than opaque automation. This example demonstrates how CyberCane satisfies the explainability constraints identified in Section 1 while maintaining the 99.5% precision reported in Table 3. Note: Similarity scores reflect realistic high-confidence detection scenarios consistent with the ≥88% phishing threshold described in Section 3.3. F.10 Complete Failure Taxonomy Table 25: Complete failure case taxonomy (RAG k = 8, threshold=2). Failure CategoryCountPercentage False Negatives (Missed Phishing): 407 total (82.2% of phishing) Zero Score17643.2% Low Signal Content11428.0% Below Threshold5112.5% Legitimate DNS348.4% No URLs184.4% Multiple Factors143.4% False Positives (Flagged Benign): 1 total (0.2% of benign) Multiple Weak Signals1100.0% Zero Score = no rules triggered; Low Signal = no urgency/credential keywords Below Threshold = Phase 1 score < 2; Legitimate DNS = valid MX/SPF/DMARC Table 25 extends main text summary with complete taxonomy. Six failure modes categorize all 407 FN: Zero Score (43.2%)—176 emails trigger no rules (valid DNS, no URLs, no urgency)— evasive tactics bypassing automation; Low Signal (28.0%)—114 contain features but lack ur- gency/credentials keywords; Below Threshold (12.5%)—51 score 1 point (lowering to thresh- old=1 increases FPR to > 10%); Legitimate DNS (8.4%)—34 from domains with proper MX/SPF/DMARC (compromised accounts or sophisticated adversaries); No URLs (4.4%)—18 text-only with phone numbers; Multiple Factors (3.4%)—14 combining 3+ limiting factors. 24