Paper deep dive
Redakto - The Incognito Tab for LLMs
Saurav Kumar Saha, Tom Röhr, Felix BieĂmann
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 92%
Last extracted: 8/20/2026, 4:11:18 AM
Summary
The paper introduces Redakto, an open-source tool for anonymizing text by detecting and redacting Personally Identifiable Information (PII) before it is processed by Large Language Models (LLMs). It provides a web application, REST APIs, and Model Context Protocol (MCP) hooks. The authors evaluate the tool's privacy performance (PII detection) and utility preservation (downstream task performance) across legal, medical, and email domains, demonstrating that anonymization strategies like semantic label masking maintain high utility scores comparable to original texts.
Entities (11)
Relation Signals (9)
Redakto â performstask â PII detection
confidence 95% · state-of-the-art functionalities for both redaction of PII
Redakto â performstask â Text Anonymization
confidence 95% · a tool that can be used for anonymizing text
Redakto â providesinterface â REST APIs
confidence 95% · These functionalities are exposed such that they can easily be used by end-users... and by developers and researchers, via REST APIs
Redakto â providesinterface â Model Context Protocol
confidence 95% · via REST APIs and model context protocol (MCP) hooks.
Redakto â evaluatedon â LER
confidence 90% · and LER (legal documents)
Redakto â evaluatedon â CodE Alltag
confidence 90% · we performed experiments on three datasets from distinct domains: CodE Alltag
Redakto â evaluatedon â GraSCCo
confidence 90% · GraSCCo (clinical texts)
Semantic Label Masking â achieveshighutility â Medical Intent Classification
confidence 85% · The least effect of redaction is observed with semantic label masking.
xlm-roberta-large â â
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Large Language Models (LLMs) are being increasingly used in everyday applications. A major challenge in the context of LLMs or Artificial Intelligence (AI) in general is to ensure privacy when using them, meaning that personally identifiable information (PII) is removed from any text that enters an LLM. These challenges have become more urgent with novel EU legislation. Uncertainty around LLM usage with respect to privacy concerns in EU countries can be a major blocker for the speed of innovation and transfer from research to applications. Here we present \textbf{Redakto}, a tool that can be used for anonymizing text prior to feeding it to an LLM or other downstream text processing. We provide state-of-the-art functionalities for both redaction of PII but also when used for pseudonymization. These functionalities are exposed such that they can easily be used by end-users, through the Redakto web application, and by developers and researchers, via REST APIs and model context protocol (MCP) hooks. The implementation is fully open source, requires modest compute resources, and can be readily deployed on local hardware. In contrast to prior work and in order to better assess the quality of the anonymized texts, we conduct extensive empirical evaluations on textual data from legal and medical domain with respect to both privacy and utility of the redacted texts. Our empirical results demonstrate that the texts anonymized with different redaction strategies achieve utility scores on par with the original texts, suggesting that anonymization with Redakto can be used for LLM tasks without substantial negative impact for the tasks we explored.
Tags
Links
- Source: https://arxiv.org/abs/2608.18260v1
- Canonical: https://arxiv.org/abs/2608.18260v1
Trouble viewing inline? Open PDF directly â
Full Text
27,271 characters extracted from source content.
Expand or collapse full text
Redakto â The Incognito Tab for LLMs Saurav Kumar Saha 1[0009â0002â9812â4150] , Tom Röhr 1[0009â0007â7448â8941] , and Felix BieĂmann 1,2[0000â0002â3422â1026] 1 Berlin University of Applied Sciences, Luxemburger Str. 10, 13353 Berlin, Germany sauravkumar.saha,tom.roehr,felix.biessmann@bht-berlin.de https://w.bht-berlin.de 2 Einstein Center Digital Future, WilhelmstraĂe 67, 10117 Berlin, Germany https://w.digital-future.berlin Abstract. Large Language Models (LLMs) are being increasingly used in everyday applications. A major challenge in the context of LLMs or Ar- tificial Intelligence (AI) in general is to ensure privacy when using them, meaning that personally identifiable information (PII) is removed from any text that enters an LLM. These challenges have become more urgent with novel EU legislation. Uncertainty around LLM usage with respect to privacy concerns in EU countries can be a major blocker for the speed of innovation and transfer from research to applications. Here we present Redakto, a tool that can be used for anonymizing text prior to feeding it to an LLM or other downstream text processing. We provide state-of- the-art functionalities for both redaction of PII but also when used for pseudonymization. These functionalities are exposed such that they can easily be used by end-users, through the Redakto web application, and by developers and researchers, via REST APIs and model context pro- tocol (MCP) hooks. The implementation is fully open source, requires modest compute resources, and can be readily deployed on local hard- ware. In contrast to prior work and in order to better assess the quality of the anonymized texts, we conduct extensive empirical evaluations on textual data from legal and medical domain with respect to both privacy and utility of the redacted texts. Our empirical results demonstrate that the texts anonymized with different redaction strategies achieve utility scores on par with the original texts, suggesting that anonymization with Redakto can be used for LLM tasks without substantial negative impact for the tasks we explored. Keywords: Privacy-Preserving NLP· PII Detection· Text Anonymiza- tion· PrivacyâUtility Trade-off· GDPR Compliance 1 Introduction In an era where large language models (LLMs) are transforming how we pro- cess and analyze text, the need to safeguard sensitive information has never been more critical. While these models bear potential for research and appli- cations, the risk of inadvertently exposing personally identifiable information (PII) remains a significant barrier to their safe and ethical use. This challenge arXiv:2608.18260v1 [cs.AI] 18 Aug 2026 2S. K. Saha et al. is particularly acute in fields such as public administration, healthcare, or legal studies, where the analysis of confidential documents is routine. To foster more responsible usage of LLMs in these domains and beyond, we present a novel soft- ware demonstrator designed to automatically detect and redact PII from text, enabling users to confidently share sanitized content with LLM-based chatbots or other downstream applications where privacy is important. Extending prior work on practical implementations of anonymizations, we develop evaluation protocols to evaluate the effectiveness of different redaction strategies such as semantic label masking, random masking etc. with respect to utility. Our exper- imental validations on a variety of textual tasks demonstrate that our solution not only preserves privacy but can also be used to maintain the integrity and utility of the original text, paving the way for secure and responsible AI-assisted analysis. In summary, this study makes the following contributions: â Redakto web application and APIs (REST & MCP) â Privacy evaluation of Redakto models â Utility evaluation of redacted texts in downstream tasks 2 Redakto web application and APIs The Redakto models are exposed in three different ways to make them as easy to use as possible for end users, developers, and researchers. More concretely, we provide a readily usable web application as well as REST APIs and a Model Con- text Protocol (MCP) server which are hosted on institutional computing infras- tructure. All components are containerized and can be self-hosted (e.g., on Ku- bernetes clusters) and the MCP server can be integrated within MCP-compatible LLM assistants and agent frameworks, enabling privacy-preserving redaction as a native tool inside automated LLM workflows instead of just prompting. A brief overview of the system design and implementation architecture of Redakto is provided in Appendix B. 2.1 Web Application A publicly accessible instance of the Redakto web application is available at https://redakto.demo.calgo-lab.de. For ease of installation and deployment, we also make Redakto available as a Docker container image, accompanied by de- ployment documentation 3 . 2.2 REST APIs The REST API endpoints are documented and exposed with Swagger UI within the Redakto web application 4 . The first main endpoint performs named entity recognition over one or more input texts while the second one extends this func- tionality by additionally generating pseudonymized versions of the input texts. Both endpoints support coarse- and fine-grained entity labeling. 3 https://hub.docker.com/r/sksdotsauravs/redakto-app 4 https://redakto.demo.calgo-lab.de/api/docs Redakto â The Incognito Tab for LLMs3 2.3 MCP Server We additionally provide an MCP (Model Context Protocol) server, distributed as an npm package 5 , that exposes Redaktoâs functionalities as structured tools for LLM assistants and agent frameworks. Through this interface, LLM assis- tants such as Claude can invoke schema-defined tool calls for entity detection and pseudonymization, instead of relying on prompt-only processing. This en- ables privacy-aware text transformation to be programmatically integrated into agentic workflows, developer tooling, and interactive AI systems. 3 Privacy Evaluation In this section, we evaluate the anonymization performance quantitatively. To demonstrate the robustness and domain generalization of Redaktoâs anonymiza- tion capabilities, we performed experiments on three datasets from distinct do- mains: CodE Alltag (emails) 6 , GraSCCo (clinical texts) 7 and LER (legal docu- ments) 8 . In summary the transformer models optimized for the Redakto demon- strator reach macro F1â 0.95 for PII detection when trained on domains with abundant annotated data (e.g., CodE Alltag, LER). For the low-resource clinical GraSCCo corpus performance is lower, in line with prior work such as [2], which reports up to 0.95 macro F1 using thousands of in-domain clinical documents from University Hospital Essen (with neither data nor models publicly avail- able), highlighting that data availability, annotation quality, and entity coverage - rather than model architecture - are the main bottlenecks for privacy-aware named entity recognition (NER). 3.1 CodE Alltag - German Email Text Building upon the prior work of PII detection and pseudonymization of German- language text using the CodE Alltag corpus [4â7,14], we extend our experiments with a substantially larger sample size and a broader range of transformer-based model architectures for personally identifiable information (PII) detection. The full CodE Alltag corpus comprises approximately 1.5 million email messages, representing diverse instances of naturally occurring PII in informal and semi- formal communication. To obtain robust and reliable performance estimates, we adopt a 5-fold cross-validation setup over a subset of 175 thousand email texts using a 60/20/20 trainâdevelopmentâtest split. Further details on the training setup, model configurations, evaluation protocol, and complete experimental re- sults are provided in the project repository. 9 5 https://w.npmjs.com/package/@sksdotsauravs/redakto-app-mcp-server 6 https://github.com/codealltag/CodEAlltag 7 https://zenodo.org/records/15747389 8 https://huggingface.co/datasets/elenanereiss/german-ler 9 https://github.com/calgo-lab/redakt-codealltag 4S. K. Saha et al. Table 1. PII-detection performance on CodE Alltag ModelPrecRecF1 xlm-roberta-large0.9442±0.00.9358±0.00.9399±0.0 gelectra-large0.9433±0.00.9371±0.00.9401±0.0 bert-base-german-cased0.9251±0.00.9160±0.00.9204±0.0 PrecRecF1 0.78 0.81 0.84 0.87 0.90 0.93 Scores bert-base-german-cased codealltag-bert-base-german-cased xlm-roberta-large codealltag-xlm-roberta-large gelectra-large codealltag-gelectra-large Fig. 1. PII-detection performance on GraSCCo Table 2. PII-detection performance on LER ModelPrecRecF1 xlm-roberta-large0.940.950.94 gelectra-large0.950.960.95 bert-base-german-cased0.950.940.95 We fine-tune and compare three transformer-based models, evaluating their ef- fectiveness in detecting 14 categories of PII in German email text. Table 1 shows comparisons of macro averaged performance on the test sets with mean and standard deviation across the five cross-validation folds. 3.2 GraSCCo - German Clinical Text To study PII detection in German clinical documents, we make use of the Graz Synthetic Clinical Corpus (GraSCCo) [10,11]. Given the small size of the GraSCCo corpus (1,439 annotated private entities across 19 PII label types), we explored transfer learning using PII detection model checkpoints previously fine-tuned on the CodE Alltag corpus. The task-specific classification heads were replaced to predict GraSCCo PII labels, and the models were further fine-tuned on the clinical corpus. Figure 1 summarizes the micro-averaged evaluation results for all models on the GraSCCo test splits of five cross-validation folds. We also list here the GitHub repository 10 and other resources 11 for more information. 10 https://github.com/calgo-lab/redakt-grascco 11 https://grascco.demo.calgo-lab.de Redakto â The Incognito Tab for LLMs5 3.3 LER - German Legal Text To broaden the privacy-oriented evaluation beyond email and clinical text, we additionally fine-tuned the same three model architectures on the German Legal Entity Recognition (LER) dataset [8, 9]. Although LER is not a dedicated PII corpus, it contains a substantial proportion of entities that overlap with privacy- relevant categories, namely persons, locations, and organizations, which together account for 25.66% of all annotated entities. The remaining 74.34% correspond to types of entities specific to the legal-domain, such as legal norms, regula- tions, court decisions etc. This mixture allows us to assess how robustly mod- els can identify classical personal identifiers when they appear alongside dense domain-specific terminology. We adopt the original trainâdevelopmentâtest split provided with the dataset and do not introduce additional resampling or cross- validation. Table 2 depicts the re-calculated macro averaged performance of the models for detecting entities of 12 PII labels present on test samples of LER dataset. 4 Utility Evaluation We study the utility of anonymized texts by training models on original texts and evaluating classification performance on redacted texts as an extension of a prior work by [12] with two new tasks, medical intent classification and legal violation prediction. Across these domains, we compare utility by replacing PII with semantic placeholders, random and generic masks to analyze how different anonymization strategies influence loss of task specific linguistic information, model robustness, and downstream utility. For additional experimental details and to facilitate reproducibility we refer readers to the project GitHub reposi- tory 12 . 4.1 Medical Intent Classification For our first experiment, we use the Medical Intent Classification (MIC) dataset 13 introduced by [13] where the researchers study on physician intent trajectories in doctorâpatient dialogues. The dataset is derived from the Ambient Clinical Intelligence Benchmark (ACI-Bench) [17] corpus and contains 5,541 physician turns annotated with 20 fine-grained intent taxonomy. Each sample consists of a single physician utterance, and the objective is to predict one or more asso- ciated medical intents, making this a multi-label classification problem. We use a model 14 finetuned with Flair [1, 15] on OntoNotes [16], an English NER cor- pus 15 , to identify and redact private entities (PERSON, DATE, GPE, ORG), with additional filtering to remove spurious detections. 12 https://github.com/calgo-lab/redacted-text-utility 13 https://huggingface.co/datasets/DATEXIS/med_intent_classification 14 https://huggingface.co/flair/ner-english-ontonotes-large 15 https://catalog.ldc.upenn.edu/LDC2013T19 6S. K. Saha et al. Table 3. Examples of redaction strategies applied to a clinical text VersionText Originalmiss edwards is here for evaluation of facial pain this is a 54-year-old female Semantic Label Maskingmiss [PERSON] is here for evaluation of facial pain this is a [DATE] female Random Maskingmiss lhyZXSX is here for evaluation of facial pain this is a vejE4fPRUxkG female Generic Maskingmiss X is here for evaluation of facial pain this is a X female xlm-roberta-largebert-large-casedpubmedbert-base-uncased 0.75 0.80 0.85 Micro F1-score Redaction Strategy No Redaction Semantic Label Masking Random Masking Generic Masking Redaction Strategy No Redaction Semantic Label Masking Random Masking Generic Masking Fig. 2. Medical Intent classification on anonymized texts Table 4. Binary Violation prediction on anonymized texts Redaction Strategy xlm- roberta- large bert- large- cased electra- large- discri. No Redac- tion 0.86 ±0.01 0.85 ±0.01 0.84 ±0.01 Semantic Label Masking 0.86 ±0.01 0.84 ±0.01 0.82 ±0.02 Random Masking 0.85 ±0.01 0.82 ±0.02 0.82 ±0.01 Generic Masking 0.84 ±0.02 0.85 ±0.01 0.82 ±0.02 We fine-tune three different transformer models for the multi-label intent clas- sification downstream task and report performance metrics for each redaction (Table 3) condition in Figure 2 where error bars denote performance variability across the five cross-validation folds. 4.2 Binary Violation Prediction We conduct a second utility evaluation on the European Court of Human Rights (ECHR) dataset 16 introduced by [3]. In line with the original work, we restrict 16 https://huggingface.co/datasets/glnmario/ECHR Redakto â The Incognito Tab for LLMs7 our experiments to the Binary Violation Prediction (BVP) task, i.e., determining whether any human rights article of the European Convention on Human Rights was violated. We utilize an experimental setup similar to that used for the MIC task (same redaction model and strategies). Three transformer-based document classifiers are fine-tuned using the Flair framework with long-sequence support. Table 4 shows macro averaged performance of these models for different redaction strategies evaluated on the test set to measure the utility impact of privacy- driven sanitization. Our results demonstrate that redaction does not impact utility substantially. The least effect of redaction is observed with semantic label masking. We present a more in-depth analysis, controlling for the number of PII entities in a text for this task in Appendix A. In the majority of the experimental conditions investigated, redaction appears to have limited impact on the utility of the anonymized texts. 5 Conclusion Our Redakto demonstrator provides a web application, REST APIs and MCP hooks for direct usage inside an LLM. In contrast to other implementations of redaction tools for LLM usage we combine the implementation with com- prehensive evaluations on redaction performance and utility of redacted texts. Empirical evaluations demonstrate not only that Redakto reliably anonymizes texts across a variety of domains, our results also show that the utility of the anonymized texts is not impacted substantially in the majority of cases, even when controlling for the number of PII entities in a given text. These findings highlight the potential of Redakto for researchers, practitioners and every-day usage. Acknowledgements This research was supported by the German Federal Ministry of Research, Tech- nology and Space grant numbers 16SV8857, by the Einstein Center Digital Fu- ture, Berlin, and by the German Research Foundation (DFG) - Project number: 528483508 - FIP 12. References 1. Akbik, A., Bergmann, T., Blythe, D., Rasul, K., Schweter, S., Vollgraf, R.: FLAIR: An easy-to-use framework for state-of-the-art NLP. In: Ammar, W., Louis, A., Mostafazadeh, N. (eds.) Proceedings of NACL. p. 54â59 (2019). https://doi.org/ 10.18653/v1/N19-4010 2. Arzideh, K., Baldini, G., Winnekens, P., Friedrich, C.M., Nensa, F., Idrissi-Yaghir, A., Hosch, R.: A Transformer-Based Pipeline for German Clinical Document De- Identification. Applied Clinical Informatics p. 31â43 (2025). https://doi.org/10. 1055/a-2424-1989 8S. K. Saha et al. 3. Chalkidis, I., Androutsopoulos, I., Aletras, N.: Neural Legal Judgment Prediction in English. In: Proceedings of ACL 2019. p. 4317â4323 (2019). https://doi.org/ 10.18653/v1/P19-1424 4. Eder, E., Krieg-Holz, U., Hahn, U.: De-Identification of Emails: Pseudonymizing Privacy-Sensitive Data in a German Email Corpus. In: Proceedings of RANLP. p. 259â269. INCOMA Ltd., Varna, Bulgaria (Sep 2019). https://doi.org/10.26615/ 978-954-452-056-4_030, https://aclanthology.org/R19-1030/ 5. Eder, E., Krieg-Holz, U., Hahn, U.: CodE Alltag 2.0: A Pseudonymized German- Language Email Corpus. In: LREC 2020 â Proceedings of the 12th International Conference on Language Resources and Evaluation. p. 4466â4477. European Lan- guage Resources Association (ELRA), Marseille, France (2020) 6. Eder, E., Wiegand, M., Krieg-Holz, U., Hahn, U.: âBeste GrĂŒĂe, Maria Meyerâ â Pseudonymization of Privacy-Sensitive Information in Emails. In: LREC 2022 - Proceedings of the 13th Language Resources and Evaluation Conference. p. 741â752. European Language Resources Association, Marseille, France (Jun 2022), https://aclanthology.org/2022.lrec-1.79/ 7. Krieg-Holz, U., Schuschnig, C., Matthies, F., Redling, B., Hahn, U.: CodE All- tag: A German-Language E-Mail Corpus. In: LREC 2016 - Proceedings of the 10th International Conference on Language Resources and Evaluation. p. 2543â 2550. European Language Resources Association (ELRA), PortoroĆŸ, Slovenia (may 2016), https://aclanthology.org/L16-1404/ 8. Leitner, E., Rehm, G., Moreno-Schneider, J.: Fine-Grained Named Entity Recog- nition in Legal Documents. In: SEMANTiCS 2019 - Semantic Systems: The Power of AI and Knowledge Graphs. Lecture Notes in Computer Science, vol. 11702, p. 272â287. Springer International Publishing, Cham (2019). https://doi.org/10. 1007/978-3-030-33220-4_20 9. Leitner, E., Rehm, G., Moreno-Schneider, J.: A Dataset of German Legal Docu- ments for Named Entity Recognition. In: LREC 2020 â Proceedings of the 12th International Conference on Language Resources and Evaluation. p. 4478â4485. European Language Resources Association (ELRA), Marseille, France (2020), https://aclanthology.org/2020.lrec-1.551/ 10. Lohr, C., Matthies, F., Faller, J., Modersohn, L., Riedel, A., Hahn, U., Kiser, R., Boeker, M., Meineke, F.: De-Identifying GRASCCO â A Pilot Study for the De-Identification of the German Medical Text Project (GeMTeX) Corpus. In: Ger- man Medical Data Sciences 2024. Studies in Health Technology and Informatics, vol. 317, p. 171â179. IOS Press (2024). https://doi.org/10.3233/SHTI240853 11. Modersohn, L., Schulz, S., Lohr, C., Hahn, U.: GRASCCO â The First Publicly Shareable, Multiply-Alienated German Clinical Text Corpus. In: Studies in Health Technology and Informatics. vol. 296, p. 66â72. IOS Press (2022). https://doi. org/10.3233/SHTI220805 12. Pal, A., Bhargava, R., Hinsz, K., Esterhuizen, J., Bhattacharya, S.: The Empirical Impact of Data Sanitization on Language Models. arXiv preprint arXiv:2411.05978 (2024) 13. Röhr, T., Roy, S., Al Mohamad, F., Papaioannou, J.M., Nejdl, W., Gers, F., Löser, A.: âWhere does it hurt?â â Dataset and Study on Physician Intent Trajectories in DoctorâPatient Dialogues (2025), https://arxiv.org/abs/2508.19077 14. Saha, S.K., Biessmann, F.: End-to-end Pseudonymization of German Texts with Deep Learning â An Empirical Comparison of Classical and Modern Approaches. In: Yurish, S.Y. (ed.) Proceedings of AiMH). p. 98â104 (2025). https://doi.org/ 10.13140/RG.2.2.20387.77606 Redakto â The Incognito Tab for LLMs9 15. Schweter, S., Akbik, A.: FLERT: Document-Level Features for Named Entity Recognition (2020) 16. Weischedel, R., Hovy, E., Marcus, M., Palmer, M., Belvin, R., Pradhan, S., Ramshaw, L., Xue, N.: OntoNotes: A Large Training Corpus for Enhanced Pro- cessing. In: Handbook of Natural Language Processing and Machine Translation: DARPA Global Autonomous Language Exploitation. Springer (2011) 17. Yim, W.w., Fu, Y., Ben Abacha, A., Snider, N., Lin, T., Yetisgen, M.: ACI- BENCH: A Novel Ambient Clinical Intelligence Dataset for Benchmarking Au- tomatic Visit Note Generation. arXiv preprint arXiv:2306.02022 (2023). https: //doi.org/10.48550/arXiv.2306.02022, https://arxiv.org/abs/2306.02022 10S. K. Saha et al. A Additional Experimental Results The ECHR dataset used for one of our utility evaluation tasks (4.2) represents a particularly interesting setting for some additional experiments - after pre- processing and sample selection across all the test folds, documents contain on average approximately 2000 tokens and 86 private entities, with most entity- dense documents containing several hundred PII entities. This makes the dataset well suited for evaluating whether and by how much extensive redaction affects downstream model performance. 0-1000-2525-5050-7575-100 0.70 0.75 0.80 0.85 0.90 0.95 Macro F1-score unsegmentedtest samples segmented by entity count percentile xlm-roberta-large: No Redaction vs Semantic Label Masking No Redaction Semantic Label Masking 0-1000-2525-5050-7575-100 0.70 0.75 0.80 0.85 0.90 0.95 Macro F1-score unsegmentedtest samples segmented by entity count percentile xlm-roberta-large: No Redaction vs Random Masking No Redaction Random Masking 0-1000-2525-5050-7575-100 0.70 0.75 0.80 0.85 0.90 0.95 Macro F1-score unsegmentedtest samples segmented by entity count percentile xlm-roberta-large: No Redaction vs Generic Masking No Redaction Generic Masking Fig. 3. Entity density wise performance (Macro-F1) comparison for different redaction strategies (xlm-roberta-large) Redakto â The Incognito Tab for LLMs11 To investigate the robustness of different redaction strategies under increasing amounts of sensitive information, we perform an additional analysis based on en- tity density. For each cross-validation fold, test documents are ranked according to their number of detected entities and partitioned into four percentile ranges (0â25, 25â50, 50â75, and 75â100). Classification performance is then evaluated separately for each subset for all fine-tuned binary violation prediction models and for each redaction strategy. 0-1000-2525-5050-7575-100 0.70 0.75 0.80 0.85 0.90 0.95 Macro F1-score unsegmentedtest samples segmented by entity count percentile bert-large-cased: No Redaction vs Semantic Label Masking No Redaction Semantic Label Masking 0-1000-2525-5050-7575-100 0.70 0.75 0.80 0.85 0.90 0.95 Macro F1-score unsegmentedtest samples segmented by entity count percentile bert-large-cased: No Redaction vs Random Masking No Redaction Random Masking 0-1000-2525-5050-7575-100 0.70 0.75 0.80 0.85 0.90 0.95 Macro F1-score unsegmentedtest samples segmented by entity count percentile bert-large-cased: No Redaction vs Generic Masking No Redaction Generic Masking Fig. 4. Entity density wise performance (Macro-F1) comparison for different redaction strategies (bert-large-cased) 12S. K. Saha et al. This experiment enables a fine-grained assessment of utility preservation under progressively more intense redaction scenarios. Since higher entity-count per- centiles require substantially larger portions of a document to be transformed, performance trends across percentile ranges provide insight into whether redac- tion introduces additional degradation beyond the inherent difficulty of process- ing longer documents. 0-1000-2525-5050-7575-100 0.70 0.75 0.80 0.85 0.90 0.95 Macro F1-score unsegmentedtest samples segmented by entity count percentile electra-large-discriminator: No Redaction vs Semantic Label Masking No Redaction Semantic Label Masking 0-1000-2525-5050-7575-100 0.70 0.75 0.80 0.85 0.90 0.95 Macro F1-score unsegmentedtest samples segmented by entity count percentile electra-large-discriminator: No Redaction vs Random Masking No Redaction Random Masking 0-1000-2525-5050-7575-100 0.70 0.75 0.80 0.85 0.90 0.95 Macro F1-score unsegmentedtest samples segmented by entity count percentile electra-large-discriminator: No Redaction vs Generic Masking No Redaction Generic Masking Fig. 5. Entity density wise performance (Macro-F1) comparison for different redaction strategies (electra-large-discriminator) Redakto â The Incognito Tab for LLMs13 We present the results of this experiment with Figure 3, Figure 4 and Figure 5. As entity density increases, documents also become substantially longer and more complex, resulting in a gradual performance decline across all models and redaction strategies. However, the relative performance gap remains largely sta- ble across density levels in comparison to the unredacted counterpart, suggesting that document complexity â not redaction â is the primary driver of degradation, with electra-large-discriminator showing slightly greater sensitivity that may be related to its replaced-token-detection pre-training objective. B Redakto System Design Redakto is organized as a layered text-redaction system consisting of a Streamlit- based interaction layer and a FastAPI backend. Through the user interface, users select an entity set, label granularity, and model. The interface then submits structured JSON requests to backend prediction endpoints for either named entity recognition or pseudonym generation. The backend validates incoming requests using Pydantic schemas and dele- gates inference to the application-level prediction service. This service dispatches requests according to the configured model type. NER models return detected entity spans, whereas NER-PG models return entity spans together with gener- ated pseudonyms and reconstructed pseudonymized text variants. Fig. 6. System design of Redakto 14S. K. Saha et al. Model loading and inference are handled by a separate infrastructure layer. At startup, the application builds a model registry from YAML configuration files that define the available entity sets, labels, model metadata, loading strategies, and framework paths. Cached loaders support Flair sequence-tagging models and Transformer-based mT5 models. The inference adapters perform SoMaJo tokenization, sentence boundary handling, and chunking for longer inputs. After inference, Redakto applies task-specific post-processing steps, including the merging of adjacent date entities, optional mapping from fine-grained to coarse-grained labels, and reconstruction of pseudonymized text. The system is containerized with Docker and includes Kubernetes manifests for deploying both the user interface and the API services.