Paper deep dive

Do It Right! A Methodology for Successful NLP System Development

Olga V. Patterson, Brett South, T. Elizabeth Workman, Scott L DuVall

Year: 2026Venue: arXiv preprintArea: cs.CLType: PreprintEmbeddings: 66

Intelligence

Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 91%

Last extracted: 7/8/2026, 2:34:07 AM

Summary

This paper proposes a structured methodology for developing clinical Natural Language Processing (NLP) systems by applying the Systems Development Life Cycle (SDLC). It emphasizes that algorithmic knowledge alone is insufficient for successful projects and highlights critical phases such as planning, feasibility analysis, and annotation. The authors address key challenges including semantic and contextual ambiguity, the risks of Large Language Model hallucination, and the necessity of concept sheets and annotation guidelines to ensure reliable information extraction from electronic medical records for clinical research.

Entities (8)

Systems Development Life Cycle · framework · 98%Natural Language Processing · technology · 95%Large Language Models · technology · 92%Feasibility Analysis · process · 90%Concept Sheets · tool · 88%Annotation Guidelines · document · 87%Contextual Ambiguity · challenge · 85%Semantic Ambiguity · challenge · 85%

Relation Signals (6)

Systems Development Life Cycle → appliesto → Clinical NLP Projects

confidence 95% · Applying the SDLC deliberately gives clinical researchers a structured means of reducing the likelihood of failure.

Natural Language Processing → extractsfrom → Electronic Medical Records

confidence 92% · NLP is a common method for supplying data to clinical research and decision making by extracting information from electronic medical records.

Feasibility Analysis → assesses → Information Existence and Extraction Accuracy

confidence 91% · Feasibility hinges on two questions, applicable regardless of the extraction approach: (1) Does the information exist in the available clinical text? (2) Can it be extracted with sufficient accuracy?

Large Language Models → exhibits → Hallucination

confidence 90% · These models fabricate plausible content not present in the source, omit information that is there, and produce inconsistent results from prompts that say the same thing differently.

Concept Sheets → manages → Variable Definitions

confidence 88% · A concept sheet records the detailed definition and prevents unplanned drift.

Annotation Guidelines → specifies → Extraction Targets and Relationships

confidence 87% · an annotation guideline is written with the study investigators to specify the targets, define concepts and attributes, and describe relationships between concepts.

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:Natural language processing (NLP) is a common method for supplying data to clinical research and decision making by extracting information from electronic medical records. Numerous textbooks and tutorials describe specific algorithms and applications for text processing, yet algorithmic knowledge is only one ingredient of a successful NLP project. Drawing on the available literature, this paper presents a stepwise approach that applies the Systems Development Life Cycle (SDLC) to projects that rely on data extraction through language processing.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

64,215 characters extracted from source content.

Expand or collapse full text

Do It Right! A Methodology for Successful NLP System Development Patterson OV a,∗ , South B a , Workman TL a , DuVall SL a a [Affiliation placeholder], , , , , Abstract Natural language processing (NLP) is a common method for supplying data to clinical research and decision making by extracting information from elec- tronic medical records [1, 2, 3]. Numerous textbooks and tutorials describe specific algorithms and applications for text processing [4, 5, 6, 7, 8, 9, 10, 11], yet algorithmic knowledge is only one ingredient of a successful NLP project. Drawing on the available literature, this paper presents a stepwise approach that applies the Systems Development Life Cycle (SDLC) to projects that rely on data extraction through language processing. Keywords: clinical natural language processing, systems development life cycle, information extraction, large language models, clinical text 1. Introduction NLP encompasses a broad range of algorithms for the computerized pro- cessing of unstructured text. For roughly its first 40 years after emerging as a discipline in the 1950s, NLP was largely a promise of the future: linguists studied language structure, computer scientists developed algorithms, and hardware engineers expanded computational capacity. Many systems were built, but few crossed from academic research into practical use [2, 4, 5, 12]. Since the mid-1980s, the growth of electronic data, greater computing power, and an expanding open-source community have lowered barriers to entry and broadened NLP’s use, including clinical text processing for health outcomes research [13]. ∗ Corresponding author Email address: ovpatterson@gmail.com (Patterson OV) arXiv:2607.05644v1 [cs.CL] 6 Jul 2026 NLP has long since moved beyond computer science to become central to clinical informatics and biomedical research [7]. As interest has grown, experts have produced a wealth of learning materials covering individual tasks, including parsing, part-of-speech tagging, and semantic role labeling, as well as machine learning and system architecture [4, 5, 6, 7, 8, 9, 10, 11]. This abundance of resources and freely available implementations can suggest that building an information extraction system for a given use case requires little specialized effort. The recent proliferation of large language models (LLMs) has renewed this impression. Models capable of answering clinical questions, summarizing notes, and extracting structured data are now accessible to any researcher with an application programming interface (API) key. The apparent ease suggests that reliable extraction is now within routine reach. That picture is an illusion. These models fabricate plausible content not present in the source, omit information that is there, and produce inconsistent results from prompts that say the same thing differently. These models are impressive but do not remove the need for the underlying process. Instead, they make that need less visible. NLP development carries the same project risks as any other software un- dertaking. The clinical literature rarely reports failures, but the Management Information Systems literature documents them extensively [14], and clinical NLP has no particular immunity. Business practitioners address these risks through the Systems Development Life Cycle (SDLC), a formal sequence of steps for developing computerized solutions. Applying the SDLC deliber- ately gives clinical researchers a structured means of reducing the likelihood of failure. Applying clinical NLP also resembles retrospective manual chart abstrac- tion. The primary difference is the use of computerized algorithms in place of human reviewers. The extensive chart-abstraction literature, therefore, offers relevant lessons in project success and failure [15, 16, 17]. The recent arrival of large language models draws this parallel even closer. Where earlier rule- and feature-based systems matched patterns that bore little resemblance to human reading, LLMs interpret narrative text in a way that approximates how a human abstractor reads a chart. Like a human abstractor, they can misread, infer beyond the evidence, or apply an interpretation inconsistently. The similarity is therefore no longer merely conceptual: the same disciplined practices that make manual chart abstraction reliable—such as explicit defi- nitions, reviewer training, adjudication, and measured agreement—apply di- 2 rectly to LLM-based extraction, and the same process discipline is required to keep either from failing. This paper introduces the SDLC steps and offers specific recommenda- tions for carrying them out when building a clinical NLP information extrac- tion system. 2. Systems Development Life Cycle Tutorials, review articles, and textbooks tend to emphasize algorithms while neglecting the fact that application development is a process that must be managed. Publications on failed projects are rare in clinical informat- ics, but the Management Information Systems literature reports many, and mismanagement is the most frequently cited cause of failure [14]. The SDLC has been used since the 1960s, when it was formalized to manage complex software projects for large-scale business systems. Although many variants exist for different project types, all share the same core phases: 1. Planning 2. Analysis 3. Design 4. Implementation 5. Testing 6. Deployment 7. Maintenance The sections below discuss each phase and show how to carry an in- formation extraction project from start to finish. Skipping a required step can jeopardize the entire project or, at minimum, waste time and effort. Our focus is on clinical applications of NLP, and on information extraction specifically. Tasks that involve text generation, such as summarization, intro- duce different design and evaluation requirements and are outside the scope of this methodology. Even a small research project that applies NLP to a single dataset benefits from being managed properly. 3. Planning Planning comes first. The project team must identify the intended pur- pose of the future system, since different goals impose different requirements and constraints, and then assess whether that goal is attainable through a feasibility analysis. 3 3.1. Project Purpose The system’s purpose should be conceptualized during planning. Project sponsors typically begin with a genuine organizational need [18], and the purpose must deliver real value to the organization it serves [19]. Baccarini defines project purpose as “the intended near-term effects on the user of the product as a result of utilizing the project’s outputs,” measured by how well the system meets user needs [20]. Purpose is intertwined with the project’s overall goal and provides the means of achieving it [20]. A project should have a single central purpose so that team effort stays focused [21]. That purpose can be framed as requirements paired with acceptable limitations. Consider three representative scenarios (Table 1). Table 1: Representative project purpose scenarios. PurposePrimary RequirementAcceptable Limitations Novel algorithm orproof-of- concept Original codebase and al- gorithm suitable for pub- lication Slow performance, high error rate, small docu- ment set Applied extrac- tion for clinical research Highly accurate dataset of specific data elements No originality required; code reuse preferred Commercial or enterprise sys- tem Robust, scalable, flexi- ble, with configurable in- put and output Limited scope per de- ployment 1. A novel algorithm or proof-of-concept system. The requirement is an original codebase and algorithm suitable for publication. Depend- ing on the research question, slow performance, a high error rate, or a small document set may be acceptable. Like fundamental research, such systems may have no commercial use case but can advance general understanding of language processing. 2. Applied extraction for a clinical research study. The require- ment is a highly accurate dataset of specific data elements. Here, orig- inality is undesirable and code reuse is preferred, because the system is only a means to obtain the data. Reusing accepted approaches and existing modules yields a custom solution with the least effort, and high accuracy is often achieved by narrowing the clinical subdomain. 4 3. A commercial or enterprise system. These must be robust, scal- able, and flexible enough to process large, diverse document sets with- out system-wide failure and to accept configurable input and output formats. Such qualities are achieved by limiting the system’s scope. In every case, the system should address the specific needs recorded in the research proposal or planning documentation, and success criteria must be defined, including how to measure the extent to which the system meets its goals. 3.2. Project Scope Inadequate requirements and undefined scope are recognized causes of project failure [14, 15, 22]. Identifying target variables and operationalizing their definitions logically and reproducibly is essential to any study. Clinical documents contain vast amounts of information about a patient and their en- vironment, expressed as concepts, values, and relationships. A single concept can be expressed by many different words, phrases, and symbols. The difficulty of a study depends on the complexity of the concept of in- terest. Simple concepts have clear, widely agreed definitions. Patient weight, for example, is a routinely measured vital sign expressed in pounds or kilo- grams. Complex variables either lack a generally accepted definition, such as “culture” [23], or combine multiple simple variables, such as “congestive heart failure” [24], evidence of homelessness [25], or other social determinants of health. Because projects often run for months or years, formal documentation of variable definitions is a required part of planning. A concept sheet records the detailed definition and prevents unplanned drift. Beyond decomposing complex concepts into simple ones, concept sheets specify expected ranges and units for numeric variables and allowable values for categorical variables. Concept sheets are also a key communication device. NLP-supported clinical studies are conducted by multidisciplinary teams combining clinical and technical expertise [1, 2], so the sheets must be shared among all person- nel. Their level of formality can vary with project size, but the definitions must be understood and agreed upon by the whole team and remain read- ily accessible throughout the project. When definitions change, the updates must be distributed immediately. Failures typically stem from problems with people, processes, or product risks [14], and in NLP work people and processes are tightly linked. Kap- pelman et al. list “no change control process” among the top failure risks, an 5 issue that a routinely reviewed concept sheet helps manage, and cite missing risk-analysis documentation as another. Poor estimation of risk is the sin- gle largest source of failure [22], making risk analysis a crucial part of any feasibility assessment. 4. Analysis Once goals are defined, the team must estimate the feasibility of achieving them and, if feasible, define the target corpus. 4.1. Feasibility Analysis Feasibility hinges on two questions, applicable regardless of the extrac- tion approach: (1) Does the information exist in the available clinical text? (2) Can it be extracted with sufficient accuracy? Answering the first question requires manual inspection of the target cor- pus. Retrospective clinical studies rely on data collected for care rather than research, so the variables of interest may or may not be routinely documented. Clinicians in the target subdomain can usually describe how information is recorded, though practices vary across sites and change over time. When the extraction approach uses a large language model, this initial characterization requires particular care. Hallucination—the generation of plausible but un- supported content—means that apparent extraction success is not evidence that the information exists in the text. A lightweight manual chart review of a representative sample can confirm whether target concepts are present. When they are not consistently docu- mented, concept sheets may need to define alternative or surrogate variables. Chest radiograph reports, for instance, rarely state a pneumonia diagnosis outright [26], so identifying pneumonia may instead rely on mentions such as “opacity” and “consolidation”, even though “opacity” is not the diagnosis itself. The second question depends on how well the approach handles ambigu- ity. Two challenges, semantic ambiguity and contextual ambiguity, largely determine whether high-accuracy extraction is achievable by any method, and both are discussed in the subsections below. For LLM-based approaches, published benchmark performance is rarely a reliable guide to accuracy on a specific clinical task. Feasibility must be assessed on representative examples from the target corpus, not inferred from general capability. 6 4.2. Semantic Ambiguity A term is a string in text that represents a relevant clinical entity. “LVEF,” for example, is a term for an echocardiogram measurement. A concept is a standardized representation of that entity, linked to all terms that can express it: “ef,” “LVEF,” “lv ej frac,” and “ejection fraction” are distinct terms for the same concept, left ventricular ejection fraction. Semantic ambiguity is the number of concepts that a given term can map to. Short abbreviations are especially ambiguous: “PE” can mean pulmonary edema, peripheral edema, pulmonary embolism, physical education, or phys- ical examination, among others. Resolving such cases may require extensive word-sense disambiguation. Fully spelled-out terms are far less ambiguous. “Left ventricular ejection fraction,” for example, has only one meaning. The more possible concepts a term maps to, the harder disambiguation becomes and the less feasible high performance is. In extreme cases, even a human reader cannot reliably interpret a term without contextual clues. The prevalence of the competing meanings matters as much as their num- ber. “CCS,” for example, can mean Canadian Cardiovascular Society score or cubic centimeters. Cubic centimeters appears constantly as a unit of mea- sure, whereas the severity score is stated very rarely, so a system seeking the score must sift through millions of mentions to find the fewer than 0.1% that are relevant. 4.3. Contextual Ambiguity Once a term is mapped to a concept, the context in which it appears must be identified. Three contextual axes dominate the clinical literature: assertion (also called negation), temporality, and experiencer. Assertion may be positive, negative, or possible. Temporality may be historical, current, or hypothetical. The experiencer may be the patient or someone else. Other axes, such as severity or anatomic location, may be relevant to a particular study. Some concepts use unambiguous terms but appear in highly variable con- texts. “Fever,” for example, is semantically clear (core body temperature above98.6 ◦ F, setting aside phrases like “yellow fever”) yet can occur in many contexts (see Table 2). Conversely, rare concepts that closely resemble others—such as specific HIV gene mutations—are mentioned only when found present. In the passage “A genotype . . . showed significant HIV drug resistance, including the follow- ing mutations in HIV reverse transcriptase: M41L, L210W, and T215Y,” the 7 Table 2: Contextual variation for the term “fever.” Example mentionAssertion Temporality Experiencer “The patient has a fever.”PositiveCurrentPatient “No fever noted.”NegativeCurrentPatient “Possible low-grade fever.”PossibleCurrentPatient “History of fever.”PositiveHistoricalPatient “Mother had fever during preg- nancy.” PositiveHistoricalOther “Watch for fever if symptoms worsen.” Possible HypotheticalPatient possible mutations are too numerous to enumerate negatively, so a mention is always affirmed, current, and about the patient. Manual chart review is often used for feasibility analysis, confirming both that target information is documented and that it is documented in a machine-accessible way. Agreement among multiple abstractors on the same documents provides an upper bound on achievable NLP performance. Feasibility analysis may show that a fully automated system is imprac- tical, requiring the project definition to change. When target concepts are present but highly ambiguous, automated extraction produces a high error rate. Alternatives include fully manual or NLP-assisted manual review. Fully manual extraction is accurate but slow, suitable only for small datasets. For large datasets, a high-recall NLP system can preselect instances for manual disambiguation. In our work on advanced basal cell carcinoma, a common cancer that rarely reaches advanced stage, fully automatic extraction was insufficiently accurate given the variability and low prevalence of relevant mentions, so we used a simple NLP system to label all mentions and then curated them manually to detect advanced stage. When target concepts are neither well documented nor tractable at scale, structured-data surrogates may replace the NLP solution altogether. Once feasibility is established, analysis turns to document selection. 4.4. Document Selection NLP uses narrative text as its primary source. The electronic medical record (EMR) accumulates documents from every patient encounter, and pa- tients with complex histories may have thousands of progress notes, nursing 8 notes, and lab and pathology reports. Selecting relevant documents reduces both the volume to be processed and the semantic variability of the corpus. Documents must also be selected for system training and testing. Document selection for processing. The most basic approach limits the corpus to documents belonging to the study cohort and generated within the study period. Filtering by document title is also common and works well where document titles are strictly controlled. The Logical Observation Identifiers Names and Codes (LOINC) document ontology was created to standardize this, classifying documents along five axes: subject-matter do- main, author role, clinical setting, type of service, and kind of document. The Observational Medical Outcomes Partnership (OMOP) common data model recommends standard document titles to promote harmonization, and the Health Level 7 (HL7) standard requires LOINC. In practice, however, many EMRs allow custom titles. The Veterans Affairs (VA) VistA system, for ex- ample, permits custom document titles across more than 50,000 clinicians and 130 disconnected installations, so the number of titles keeps growing. Title-based selection is viable where HL7 is followed but unreliable where naming is not enforced. When computational cost is high and relevant mentions are rare, ad- ditional information-retrieval steps may be needed to pre-filter the corpus before processing. This consideration is especially important for LLM-based approaches, where most providers charge per token and costs scale directly with corpus size. Passing large volumes of irrelevant clinical text through an LLM is expensive and unnecessary. A lightweight pre-filtering step, such as keyword matching or document classification, can substantially reduce the number of documents processed and the cost. Document selection for annotation. Careful selection reduces the expense of manual review. Selecting a random sample is simplest but can be suboptimal when target information is infrequent or the corpus is linguis- tically diverse. Stratified random sampling is an alternative when language is expected to vary across identifiable strata, such as clinical subdomain, lo- cality, or time, as linguistic drift introduces new terms. Because language is more consistent within than across strata, sampling within each stratum captures a broader range of examples, which is especially useful when strata differ in size. Sample size. In NLP studies, much as in qualitative research, training-set size is often determined by a maximum-variation approach: positive cases are reviewed and annotated until no new concepts, vocabulary, or lexical variants 9 emerge [27]. The examples in this paper focus on supervised NLP development, in which training data is manually labeled by human reviewers, as opposed to unsupervised approaches that learn patterns statistically. Many real systems combine supervised, unsupervised, and semi-supervised methods. We use su- pervised examples for simplicity. Supervised development typically requires a manually reviewed truth set produced through annotation. 5. Annotation NLP development requires adequate data for training and evaluation, and manual annotation is a common way to produce it. Annotation is a schema-based manual review that identifies spans of text representing tar- get information classes and may also assign attributes or identify relations between classes. It is, in effect, human information extraction, and projects may require annotating anywhere from small to very large document sets. Annotated corpora tell both programmers and machines what to extract or classify. Annotation is time-consuming, expensive, and error-prone, and demands considerable reviewer effort, but human review of some kind is needed to un- derstand variable definitions, refine target classifications, and further define scope. It also reveals the kinds of examples automated extraction will require. These steps are typically iterative: identify examples, refine the guidelines, and reassess reviewer agreement. Such examples are essential for confirming that a system works as expected before it is tested on held-out or previously unseen documents. Annotation can be cognitively demanding. Some tasks require expert judgment while others can be done by non-experts. For annotation to function as a valid reference standard, the data avail- able to annotators must match the data the NLP system will see. The most useful annotation replicates the pipeline exactly: annotators work from the same source documents, in the same form, and record output in the same schema the system will produce. When annotators have access to additional context—such as structured data fields, prior notes, or other documents out- side the system’s scope—the reference standard reflects capabilities the sys- tem does not possess. Measured performance against such a standard will be lower than annotation agreement suggests is achievable, obscuring where the system actually fails and making targeted improvement more difficult. 10 Annotation design should therefore begin with an explicit specification of what the NLP system will and will not have access to. 5.1. Annotation Guidelines Once extraction targets are identified, an annotation guideline is writ- ten with the study investigators to specify the targets, define concepts and attributes, and describe relationships between concepts. Good guidelines provide clear examples of what to annotate (inclusions), how to classify it, and what to leave out (exclusions). For most tasks, individual mentions are the targets the system will iden- tify and extract. Mentions may carry one or more attributes describing contextual features or state. The most common attribute types—negation, experiencer, and temporality—describe context, and the guideline specifies exactly how each is assigned. Tasks may also identify logical relations be- tween mentions, such as a disease and the drugs used to treat it. Guidelines may additionally cover use of the annotation tools. The SDLC integrates nat- urally here, particularly in developing and refining guidelines and running an annotation campaign. 6. Design System design identifies the key elements of the future system along with their inputs, outputs, and processing steps. A complex NLP problem be- comes manageable when broken into its most basic tasks. Just as mat- ter reduces to a few kinds of atoms, NLP problems reduce to two atomic tasks: search and classification. A third task, summarization, is some- times treated as atomic alongside these two. Summarization, however, com- bines extraction with generation—it identifies relevant content and produces new text to represent it. Generation is a fundamentally different kind of task from extraction and is outside the scope of this paper. Inefficiency arises when a designer treats a set of distinct tasks as a single process. Several design approaches are possible. In a well-documented domain, existing resources such as dictionaries and ontologies can supply the logi- cal units. Data-driven approaches instead rely on manual annotations for examples. Ensemble systems apply multiple methods in parallel. An information model represents the concepts, relationships, rules, constraints, and operations that define the data semantics of a domain. A 11 knowledge base encodes that model for a specific implementation. Knowl- edge bases are commonly built either by having experts specify rules or by learning a model from human-annotated text. Five methods for identify- ing and interpreting concepts are in common use: rules, patterns, machine learning, large language models, and hybrid systems that combine these ap- proaches. When a large language model is chosen, the prompt takes on the role of both the concept sheet and the annotation guideline. It must define the target concept, specify what counts as an instance, and describe how to handle ambiguous cases [28, 29]. Retrieval-augmented generation, which grounds LLM output in a curated external knowledge base, is the modern counterpart to the information model [30, 31]. For named-entity recognition and classification tasks, fine-tuned domain-specific encoder models continue to outperform general-purpose LLMs, and the choice of method should be driven by task requirements, not by the novelty of the technology [28, 29]. Design also operates at different levels of analysis. • Mention level. The most common approach: human reviewers clas- sify each mention to a concept, and that training data teaches the system to identify and classify spans of text. A document may contain many mentions, and they need not agree. • Document level. The entire document is classified. A patient may have from a handful to thousands of documents. • Episode level. At this level, mentions across multiple documents are aggregated to capture a clinical event at a specific date or over a defined window, such as an episodic flare in Crohn’s disease or cholecystitis. A patient may have many events or none, and mentions need not agree. Disagreements are reconciled as part of the extraction logic. • Patient level. All mentions and document classifications are com- bined into a single patient-level classification. Extraction results at every level are compared against a human-defined truth set. Once the required tasks and goals are set, the full sequence of steps can be designed. This sequence, called the pipeline, is what each document passes through, with each step adding metadata about document structure or content. A pipeline may include preprocessing to normalize formatting, 12 remove extraneous characters, or combine documents from different special- ties, tables, or EMR locations. The accepted way to store annotations is as metadata: annotations are kept in a separate structure linked to the source document by the start and end indices of each labeled span. This leaves the original document unmod- ified and preserves all data unambiguously. Annotation viewers read both files and combine them visually for human interpretation. LLM-based sys- tems complicate this convention. Unlike rule- or pattern-based systems that identify explicit text spans, an LLM arrives at a conclusion through internal attention mechanisms that are not directly observable. It is therefore not always possible to pinpoint the exact portion of text the model relied on, which makes error analysis more difficult and reduces the interpretability of system output. 7. Implementation Implementation is the focus of most systems-engineering courses, and the first three SDLC phases are largely independent of it. Leidner (2003) summarized the key complexities of building an NLP sys- tem particularly well. Several of his concerns are worth emphasizing (Fig- ure 1). Accuracy. Text processing is inherently difficult, and correct output is never guaranteed. Potential errors must be accounted for whenever NLP output is used. Flexibility. Input and output formats should be configurable rather than hard-coded. Hard-coded values limit utility and cause run-time failures when an expected file is missing. For LLM-based systems, prompts carry the same risk: a prompt embedded in code as a string literal cannot be updated without a code change and is invisible to non-developers. Prompts should be stored as configurable, version-controlled artifacts alongside the system’s other parameters. Scalability. A system should process large volumes within a reasonable time, often through distributed architecture that spreads pipeline instances 1 Source: Leidner JL. Current issues in software engineering for Natural Language Pro- cessing. Proceedings of the HLT-NAACL 2003 Workshop on Software Engineering and Architecture of Language Technology Systems (SEALTS ’03). ACL, Morristown, NJ; 2003. p. 45–50. 13 Figure 1: Concerns of the language engineer (after Leidner, 2003). 1 14 or stages across multiple network nodes. Reuse. Despite many researchers and shared code repositories, reuse of fully configured, comprehensive NLP pipelines is low. Different use cases call for different pipeline designs, so end-to-end systems rarely transfer with- out substantial modification. Only a few individual modules are broadly reused, among them Hugging Face Transformers [32], Stanford’s Stanza [33], and spaCy [34]. Integration remains difficult when developers use different languages, deprecated environments, or divergent paradigms. Leidner lists several barriers to reuse: • Lack of awareness of existing components • Lack of trust in component quality • Mismatch between component properties and project requirements • Licensing, cost, organizational, or technical barriers such as platform incompatibility, dependency conflicts, and poor documentation For LLM-based systems, reusability faces an additional challenge: the pace of new model releases is rapid, and performance varies substantially across models and across versions of the same model. A pipeline validated against one model version may behave differently after an update, and switch- ing to a newer or different model often requires re-evaluation rather than straightforward substitution. This makes LLM components harder to treat as stable, reusable building blocks. Leidner describes this tension through a productivity pyramid that maps implementation approaches along a trade-off between short-term productiv- ity and long-term reusability. At the top of the pyramid are ready-made solutions that reduce development time but offer limited control and cre- ate dependency on external vendors. At the base are custom-built, modular components that require more investment but yield systems that are more controllable, maintainable, and adaptable over time. An increasing number of teams now operate toward the top of the pyramid by building on NLP utilities provided by cloud platforms such as Azure, AWS, Google Cloud, and Databricks. These services offer pre-built functions for common tasks including entity recognition, sentiment analysis, and document classification, reducing development effort at the cost of reduced control over model behav- ior and dependency on vendor roadmaps and pricing. Existing solutions are 15 widely available but rarely do exactly what is needed. Planning, analysis, and validation remain essential regardless of where a team sits on the pyra- mid, and the best outcomes typically come from a flexible framework built from compatible modules rather than a single off-the-shelf system. LLM-based systems introduce a distinct set of implementation decisions. Prompt engineering is the primary tool for steering model behavior without modifying model weights, and few-shot examples embedded in the prompt can substantially improve performance on structured extraction tasks. When prompting alone is insufficient, the model can be fine-tuned on domain- specific examples at a fraction of the cost of full retraining. The choice be- tween local deployment and hosted APIs involves trade-offs in cost, latency, and data governance. Open-source models such as Llama 3 and Mistral can now be run locally using serving frameworks such as Ollama or vLLM, keep- ing clinical text entirely within the organization’s infrastructure and eliminat- ing external data transmission. For organizations preferring hosted services, the same major cloud providers mentioned above also offer LLM-specific ser- vices, including Azure OpenAI, AWS Bedrock, and Google Vertex AI, with Health Insurance Portability and Accountability Act (HIPAA)-eligible con- figurations and signed Business Associate Agreements (BAAs). Consumer- facing APIs, such as the public versions of ChatGPT, Claude, and Gemini, do not offer BAAs and are not appropriate for use with protected health information. Regardless of deployment model, access controls, audit logging, and egress restrictions remain the covered entity’s responsibility. The com- mercial LLM landscape is evolving rapidly, and specific model capabilities, compliance certifications, and vendor offerings described here will continue to change. Teams should verify current status before selecting a deployment approach. 8. Testing Evaluation is integral to the SDLC. Its foundation is comparison of system output against a reference standard, and mishandling that reference standard is a common beginner mistake that invalidates results. Maintaining a strict separation between training and test sets is essen- tial. It is easy to make apparently correct decisions on the test set using information gleaned from it. For example, a developer might treat a term as a good predictor because it works on the test set, or try many parameter values and keep whichever performs best on the test set. As a rule, accuracy 16 on genuinely new data will be much lower than accuracy on a test set the classifier has been tuned to. In a clean experiment, you never run on or even inspect the test set during development. Instead, you reserve a development set for that purpose and run a single final experiment on the untouched test set only after all parameters are fixed. A test set that was never consulted during development is the closest available proxy for real-world performance. Figure 2: Performance factors in system evaluation. Efficacy. The standard measures are precision, recall, and F1. Precision is the fraction of system-identified mentions that are correct—it reflects how often the system is right when it makes an assertion. Recall is the fraction of all true mentions in the text that the system identified—it reflects how completely the system finds what is present. The two measures trade off: a conservative system will favor precision over recall, while an aggressive system will do the reverse. The F1 score is the harmonic mean of precision and recall, combining them into a single value that penalizes large imbalances between the two and facilitates comparison across systems. For LLM-based systems, standard F1 evaluation is complicated by non-determinism and prompt sen- sitivity. The same prompt can produce different outputs across runs, and small changes in prompt wording can shift F1 by a substantial margin with- out any change to the underlying model. Evaluation results are therefore tied to the specific prompt used and should not be generalized to the model as a whole. 17 Efficiency. Efficiency measures computational performance, typically expressed as throughput measured at the level of tokens, sentences, or doc- uments. For traditional NLP pipelines, throughput is predictable and scales with hardware, making benchmarking and optimization straightforward for larger projects. For LLM-based systems, per-document token costs and API latency introduce economic and time constraints that must be factored into deployment planning from the outset, and cost projections should be vali- dated during feasibility analysis rather than discovered at deployment. Effectiveness. The system’s ability to achieve its actual goal. It usually equals efficacy, but for some use cases precision or recall matters more, and extracting some instances with high precision can outweigh extracting all of them. Published work often reports only efficacy, yet for an applied system ef- ficacy may not reflect effectiveness. If the goal is simply to determine whether a patient has a given finding, mention-level counts are irrelevant so long as the finding is identified at least once. Computing effectiveness may require rolling mention-level results up to the document level. A stricter interpre- tation, counting a document as a true positive only when all its mentions are found, would make effectiveness lower than mention-level performance. For LLM-based systems, hallucinated mentions introduce false positives that can inflate apparent recall at the mention level while reducing effectiveness at the document or patient level. Auditing must account for both missed and invented mentions. Efficiency is where the trade-off between traditional and LLM-based ap- proaches is most visible: a well-designed rule-based or encoder-based system can process large clinical corpora at far greater speed and a fraction of the cost of a comparable LLM-based pipeline. Selecting the right tool for the task matters more than the ability to claim cutting-edge technology. When a sim- pler, faster method achieves the required effectiveness even if lower efficacy, it is the better choice. 8.1. Error Analysis A formal error analysis manually examines cases where system output diverged from annotation to find the cause. Errors fall into two kinds: • Systematic errors share a common cause across every instance. A term consistently missed because it is absent from the dictionary, or a sentence structure that the classifier always misreads, are system- atic errors. Because the cause is uniform, the fix is tractable: update 18 the dictionary, revise the rule, or add representative training examples for the pattern. Systematic errors are the most productive target for iteration because a single fix resolves many failures at once. • Random errors, such as novel misspellings, highly ambiguous phrases, or extremely rare contexts, differ from one another and have no sin- gle root cause. They cannot be fixed systematically without adding so many special cases that precision or recall degrades elsewhere. Random errors represent the practical ceiling of the system for a given input do- main. The appropriate response is to characterize them, document the limitation, and determine whether the residual error rate is acceptable for the use case rather than attempting to eliminate them entirely. For LLM-based systems, hallucinations introduce a third category that fits neither type. A hallucinated mention is not caused by a missing rule or an unusual input pattern—it is generated by the model independently of what the source text actually contains. Because hallucinations can be fluent and contextually plausible, they are not always apparent on casual inspection and require deliberate auditing against the source document. Error analysis for LLM-based systems must therefore examine both false negatives (missed mentions) and false positives (invented ones), and should test whether error patterns correlate with prompt wording, input length, or concept type. Er- rors that shift with prompt changes are a signal that the system is sensitive to surface form rather than grounded in the clinical content. 9. Deployment In software development, deployment (or release) is the milestone at which the application is delivered to the customer. For research information extraction, deployment is when the system is run on the target corpus and the data is extracted. Several decisions must be resolved. Data access. Clinical data is governed by privacy and confidentiality regulations, so any person or process that runs the system must be autho- rized to access the data. Clinical EMRs are often physically and logically separate from the research environment. In the VA, for instance, clinicians read and write the live EMR through VistA. To avoid accidental corrup- tion of active records, most retrospective studies draw from a read-only copy maintained in the Corporate Data Warehouse (CDW), which is refreshed 19 on a regular schedule and requires separate authorization through Institu- tional Review Board (IRB) approval and a data use agreement. Access to the CDW does not carry over from access to the live system. This separation of live clinical records from research data is a common governance pattern across health systems and should be confirmed as part of feasibility planning. The research copy may also be transformed relative to the source—mapped to standard terminologies, de-identified, or restructured—and those trans- formations must be understood before building or deploying an extraction system against it. Computing environment. Depending on data volume and process- ing frequency, deployment may require a scalable, distributed environment. A single machine may suffice for a small corpus, but larger volumes ben- efit from processing spread across multiple nodes. With HIPAA-compliant cloud services now available, many organizations deploy large processing jobs to the cloud rather than buying expensive hardware for one-time use. For LLM-based systems handling sensitive clinical text, local deployment on or- ganizational infrastructure is a viable alternative that eliminates protected health information (PHI) transmission to external services entirely. Repeatability. When a system will be reused, a compiled distribution package and clear documentation allow others to deploy and use it without the original developer. Documentation should be written for two distinct audiences with different needs. Developer documentation covers system de- ployment: environment requirements, configuration, execution steps, and known failure modes. Data user documentation covers working with sys- tem output and should be grounded in the concept sheet developed during analysis: each output field should be traceable to a defined concept, with the same definitions, scope boundaries, and inclusion criteria the annotators used. It should also report all performance characteristics—precision, recall, F1, and effectiveness at the relevant level of analysis—so that downstream users understand the expected error rate before drawing conclusions from the data. Data transformations applied during extraction or post-processing should be described explicitly, as should known limitations for specific use cases. Conflating these two audiences in a single document serves neither well. A data analyst consuming extraction results does not need to know how to configure a pipeline, and a developer redeploying the system does not need a clinical interpretation guide. Targeted documentation for each audience reduces misuse of output and lowers the barrier to reuse. Human oversight. LLM-based systems should not be deployed in fully 20 automated mode for clinical extraction without human review. The risk of hallucinated mentions that are fluent and contextually plausible means that automated output cannot be treated as ground truth. A structured review step—sampling output against source documents at regular intervals—is the minimum acceptable safeguard. Systems should also log inputs, outputs, and model version at each run to support error tracing and revalidation when model behavior changes. 10. Maintenance An NLP system should keep performing at or above its deployment level, but several hard-to-predict factors affect performance, chiefly unforeseen changes in input and the fluid nature of language itself. Language is not static. It is a dynamic, largely ergodic process [35]. Sapir introduced the notion of unconscious linguistic drift from his study of dialect evolution [36]. Language changes through geographic separation and adaptation to new circumstances, including new communication tech- nologies [37], and diversification can be morphologic (word forms), syntactic (sentence structure), or semantic (meaning). Using word embeddings and mixed linear regression, Hamilton et al. [38] measured semantic drift in verbs and nouns via cosine similarity across corpora stratified by decade, finding greater change in verbs globally and in nouns locally. They concluded that shifting verb usage reflects traditional linguistic drift, while shifting noun usage reflects cultural change, such as new words for new technologies [39]. The biomedical domain shows the same pattern: new terms such as “Zika” enter the record, and coding systems evolve (International Classification of Diseases, Ninth Revision, Clinical Modification [ICD-9-CM] to ICD-10-CM). Environment and subdomain shape clinical language as well. Harris demon- strated the existence of sublanguages mathematically [40], and Friedman et al. applied his work to the clinical domain [41], showing how semantic word classes and specialized term meanings mark sublanguages that can be differ- entiated by clinical scope [42], specialty [43], and disease [44]. Systems used over time, or across inputs that vary in sublanguage (such as text from different institutions), must be validated. Rule-based systems require their rules to be re-examined and tested regularly. Machine-learning systems must be periodically re-evaluated and likely retrained. Carrell et al. documented the difficulty of adapting a system across health care envi- ronments [45]. In short, linguistic drift, new lexicon (Ebola, Zika), evolving 21 coding systems, changing documentation guidelines, and sublanguage varia- tion across subdomains all mean that rules, patterns, and models must be updated, and systems applied over time require regular revalidation. LLM-based systems are subject to an additional source of drift that has no equivalent in traditional NLP: the underlying model can change without any change to the system code. Hosted model providers update, deprecate, and replace model versions on their own schedules, and a system validated against one version may produce materially different output after an update. This model version drift is harder to detect than linguistic drift because it requires no change in the input data to manifest. Regular revalidation is therefore even more critical for LLM-based systems than for traditional ones, and system logs must record the exact model version used at each run so that performance changes can be traced to their source. 10.1. The SDLC Is a Cycle Although presented sequentially, the SDLC is a cycle (Figure 3). Deploy- ment ends one cycle. Maintenance, which itself requires planning, analysis, design, implementation, and testing, begins the next. Depending on what planning and analysis reveal, a new cycle may call for a major redesign or a minor update. Within each cycle the arrows are bidirectional. If analysis reveals that concepts are too ambiguous or too poorly documented to extract, planning for new concepts may be needed. Design may demand further analysis, and implementation may force design changes. Development also proceeds through smaller iterations. When error anal- ysis finds a systematic error, the team assesses the benefit of fixing it, designs and implements the fix, then re-tests and re-analyzes. Iterations continue un- til no more feasibly fixable systematic errors remain. Overall, building an information extraction system is inherently iterative. 11. Conclusions Regardless of a project’s size, following the steps of the Systems Devel- opment Life Cycle improves the chances of success. This paper has demon- strated how the SDLC applies to developing clinical NLP systems, from planning and feasibility analysis through annotation, design, implementa- tion, testing, deployment, and ongoing maintenance, reframing information 22 Figure 3: The Systems Development Life Cycle as an iterative cycle. extraction not as an algorithmic exercise but as a process to be managed deliberately from start to finish. The rapid expansion of large language model capabilities over the last several years has made this argument both more timely and more urgent. A well-prompted model can return structured output from clinical text within minutes, without annotation, without a reference standard, and without a test set. That speed creates the impression that the process is optional. It is not. LLMs are genuinely capable tools that expand what is possible in clinical NLP, but they introduce their own failure modes that the SDLC is designed to surface and manage: hallucination, prompt sensitivity, non-determinism, model version drift, and data governance constraints. Feasibility analysis determines whether a concept is extractable. Annotation establishes the reference standard that makes evaluation meaningful. Testing exposes relia- bility problems that a single prototype cannot reveal. Deployment planning governs who and what can access clinical data. Maintenance anticipates the drift that makes a validated system degrade over time. Quick results are not the same as accurate results, and a system that has not been analyzed, designed, validated, and governed is not a useable system regardless of how compelling its output appears on first inspection. The SDLC is what converts a promising prototype into a clinical tool that 23 performs reliably, operates within appropriate data governance constraints, and can be trusted by the researchers and clinicians who depend on it. CRediT authorship contribution statement Olga V. Patterson: Conceptualization, Writing – original draft. South B: Writing – review & editing. Workman TL: Writing – review & editing. DuVall SL: Writing – review & editing. Declaration of generative AI and AI-assisted technologies The original intellectual content of this manuscript was written by the authors. Editorial revisions, structural improvements, and formatting were supported by Claude (Anthropic), a large language model assistant. The authors reviewed and take full responsibility for all content. References [1] A. Névéol, P. Zweigenbaum, Clinical Natural Language Processing in 2014: Foundational Methods Supporting Efficient Healthcare., Year- book of medical informatics 10 (1) (2015) 194–8. doi:10.15265/ IY-2015-035. URL http://w.ncbi.nlm.nih.gov/pubmed/26293868http://w. pubmedcentral.nih.gov/articlerender.fcgi?artid=PMC4587052 [2] A. Névéol, P. Zweigenbaum, Clinical Natural Language Processing in 2015: Leveraging the Variety of Texts of Clinical Interest., Yearbook of medical informatics (1) (2016) 234–239. doi:10.15265/IY-2016-049. URL http://w.ncbi.nlm.nih.gov/pubmed/27830256 [3] C. Friedman, S. B. Johnson, Natural Language and Text Processing in Biomedicine, Springer New York, 2006, p. 312–343. doi:10.1007/ 0-387-36278-9_8. URL http://link.springer.com/10.1007/0-387-36278-9_8 [4] K. S. Jones, Natural language processing: a historical review, in: Cur- rent Issues in Computational Linguistics: in Honour of Don Walker, Vol. 7, 1994, p. 3–16. doi:10.1007/978-0-585-35958-8_1. 24 [5] N. Sager, C. Friedman, M. S. Lyman, Medical Language Processing: Computer Management of Narrative Data, New York: Addison-Wesley, Reading, Mass, 1987. URL http://w.worldcat.org/oclc/13947442http://w. aclweb.org/anthology/J/J89/J89-3009.pdfhttp://portal.acm. org/citation.cfm?id=49103.1046397 [6] S. Doan, M. Conway, T. M. Phuong, L. Ohno-Machado, Natural language processing in biomedicine: a unified system architecture overview., Methods in molecular biology (Clifton, N.J.) 1168 (2014) 275–94. arXiv:1401.0569, doi:10.1007/978-1-4939-0847-9_16. URL http://w.ncbi.nlm.nih.gov/pubmed/24870142%5Cnhttp: //arxiv.org/abs/1401.0569http://w.ncbi.nlm.nih.gov/ pubmed/24870142http://arxiv.org/abs/1401.0569http: //dx.doi.org/10.1007/978-1-4939-0847-9_16 [7] P. M. Nadkarni, L. Ohno-Machado, W. W. Chapman, Natural language processing: an introduction, Journal of the Amer- ican Medical Informatics Association 18 (5) (2011) 544–551. doi:10.1136/amiajnl-2011-000464. URL http://jamia.bmj.com/cgi/doi/10.1136/ amiajnl-2011-000464http://jamia.oxfordjournals.org/lookup/ doi/10.1136/amiajnl-2011-000464 [8] R. Grishman, Information extraction: Techniques and challenges, Infor- mation Extraction A Multidisciplinary Approach to an Emerging Infor- mation Technology (1997) 10–27doi:10.1007/3-540-63438-X_2. [9] M. J. Schuemie, J. A. Kors, B. Mons, Word sense disambiguation in the biomedical domain: an overview., Journal of computational biology : a journal of computational molecular cell biology 12 (5) (2005) 554–65. doi:10.1089/cmb.2005.12.554. URL http://w.ncbi.nlm.nih.gov/pubmed/15952878 [10] S. Sarawagi, Information Extraction, Foundations and Trends in Databases 1 (3) (2008) 261–377. doi:10.1561/1900000003. URL http://w.nowpublishers.com/product.aspx?product=DBS& doi=1900000003 25 [11] S. M. Meystre, G. K. Savova, K. C. Kipper-Schuler, J. F. Hurdle, Ex- tracting information from textual documents in the electronic health record: a review of recent research., Yearbook of medical informatics (2008) 128–44. URL http://w.ncbi.nlm.nih.gov/pubmed/18660887 [12] D. Demner-Fushman, W. W. Chapman, C. J. McDonald, What can natural language processing do for clinical decision support?, Journal of biomedical informatics 42 (5) (2009) 760–72. doi:10.1016/j.jbi. 2009.08.007. URL http://w.ncbi.nlm.nih.gov/pubmed/19683066 [13] P. Spyns, Natural language processing in medicine: an overview., Methods of information in medicine 35 (4-5) (1996) 285–301. URL http://w.ncbi.nlm.nih.gov/pubmed/9019092?dopt= Citationhttp://w.ncbi.nlm.nih.gov/pubmed/9019092 [14] L. A. Kappelman, R. McKeeman, L. Zhang, Early Warn- ing Signs of it Project Failure: The Dominant Dozen, In- formation Systems Management 23 (4) (2006) 31–36. doi: 10.1201/1078.10580530/46352.23.4.20060901/95110.4. URL http://web.a.ebscohost.com/ehost/detail/detail?sid= 27084ed3-2b86-4b57-8350-737aef6f1883%40sessionmgr4010&vid= 13&hid=4112&bdata=JnNpdGU9ZWhvc3QtbGl2ZQ%3D%3D#AN=22291664& db=cphhttp://search.ebscohost.com/login.aspx?direct=true& db=cph&AN=22291664&site=ehost [15] M. Vassar, M. Holzmann, The retrospective chart review: important methodological considerations, J Educ Eval Health Prof 10 (2013). doi: 10.3352/jeehp.2013.10.12. [16] R. E. Gearing, I. A. Mian, J. Barber, A. Ickowicz, A methodology for conducting retrospective chart review research in child and adolescent psychiatry., Journal of the Canadian Academy of Child and Adolescent Psychiatry = Journal de l’Académie canadienne de psychiatrie de l’enfant et de l’adolescent 15 (3) (2006) 126–34. URL http://w.ncbi.nlm.nih.gov/pubmed/18392182http: //w.pubmedcentral.nih.gov/articlerender.fcgi?artid= PMC2277255http://w.pubmedcentral.nih.gov/articlerender. fcgi?artid=2277255&tool=pmcentrez&rendertype=abstract 26 [17] L. A. Knake, M. Ahuja, E. L. McDonald, K. K. Ryckman, N. Weathers, T. Burstain, J. M. Dagle, J. C. Murray, P. Nadkarni, Quality of EHR data extractions for studies of preterm birth in a tertiary care center: guidelines for obtaining reliable data., BMC pediatrics 16 (2016) 59. doi:10.1186/s12887-016-0592-z. URL http://w.ncbi.nlm.nih.gov/pubmed/27130217http://w. pubmedcentral.nih.gov/articlerender.fcgi?artid=PMC4851819 [18] W. Yeoh, A. Koronios, Critical success factors for business intelligence systems, in: Journal of computer information systems, Vol. 50, 2010, p. 23–32. doi:abs/10.1080/08874417.2010.11645404. URL https://w.tandfonline.com/doi/abs/10.1080/08874417. 2010.11645404 [19] A. Asosheh, S. Nalchigar, M. Jamporazmey, Information technology project evaluation: An integrated data envelopment analysis and balanced scorecard approach, Expert Systems with Applications 37 (8) (2010) 5931–5938. doi:10.1016/j.eswa.2010.02.012. URL http://linkinghub.elsevier.com/retrieve/pii/ S0957417410000515 [20] D. Baccarini, Logical Framework Method for Defining Project Success, Project Management Journal (1999). URL https://w.pmi.org/learning/library/ logical-framework-method-defining-project-success-5309 [21] K. Davis, Logical framework analysis: a methodology to turn vision into reality, in: AIPM National Conference, 2005. [22] R. R. Nelson, IT Project Management: Infamous Failures, Classic Mistakes, and Best Practices, MIS Quarterly Executive 6 (2) (2007) 67–78. URL http://misqe.org/ojs2/index.php/misqe/article/view/ 128http://w2.comm.virginia.edu/cmit/Research/MISQE6-07. pdf [23] D. K. Iwamoto, W. M. Liu, The impact of racial identity, ethnic iden- tity, asian values and race-related stress on Asian Americans and Asian international college students’ psychological well-being., Journal of coun- seling psychology 57 (1) (2010) 79–91. doi:10.1037/a0017393. 27 URL http://w.ncbi.nlm.nih.gov/pubmed/20396592http://w. pubmedcentral.nih.gov/articlerender.fcgi?artid=PMC2854415 [24] S. M. Meystre, Y. Kim, G. T. Gobbel, M. E. Matheny, A. Redd, B. E. Bray, J. H. Garvin, Congestive heart failure information extraction framework for automated treatment performance measures assessment., Journal of the American Medical Informatics Association : JAMIA 24 (e1) (2017) e40–e46. doi:10.1093/jamia/ocw097. URL https://academic.oup.com/jamia/article-lookup/doi/ 10.1093/jamia/ocw097http://w.ncbi.nlm.nih.gov/pubmed/ 27413122 [25] G. Divita, T. E. Workman, M. E. Carter, A. Redd, M. H. Samore, A. V. Gundlapalli, PlateRunner: A Search Engine to Identify EMR Boilerplates., Studies in health technology and informatics 226 (2016) 33–6. URL http://w.pubmedcentral.nih.gov/articlerender.fcgi? artid=3900197&tool=pmcentrez&rendertype=abstracthttp://w. ncbi.nlm.nih.gov/pubmed/24551356http://w.pubmedcentral. nih.gov/articlerender.fcgi?artid=PMC3900197http://w.ncbi. nlm.nih.gov/pubmed/273504 [26] V. Liu, M. P. Clark, M. Mendoza, R. Saket, M. N. Gardner, B. J. Turk, G. J. Escobar, Automated identification of pneumonia in chest radio- graph reports in critically ill patients., BMC medical informatics and decision making 13 (2013) 90. doi:10.1186/1472-6947-13-90. URL http://w.ncbi.nlm.nih.gov/pubmed/23947340http://w. pubmedcentral.nih.gov/articlerender.fcgi?artid=PMC3765332 [27] M. Topaz, K. Lai, D. Dowding, V. J. Lei, A. Zisberg, K. H. Bowles, L. Zhou, Automated identification of wound information in clinical notes of patients with heart diseases: Developing and validating a nat- ural language processing application., International journal of nursing studies 64 (2016) 25–31. doi:10.1016/j.ijnurstu.2016.09.013. URL http://w.sciencedirect.com/science/article/pii/ S0020748916301602http://linkinghub.elsevier.com/retrieve/ pii/S0020748916301602http://w.ncbi.nlm.nih.gov/pubmed/ 27668855 28 [28] Z. Guo, et al., Improving large language models for clinical named en- tity recognition via prompt engineeringPMC11339492. Venue pending verification. (2024). [29] O. Syrstad, et al., Harnessing large language models for efficient data extraction in systematic reviews: the role of prompt engineer- ingPMC12559671. Venue pending verification. (2024). [30] Authors TBD, Retrieval augmented generation for large language mod- els in healthcare: a systematic reviewPMC12157099. Venue pending ver- ification. (2024). [31] Authors TBD, Retrieval-augmented generation (RAG) in healthcare: a comprehensive review, AI (MDPI) 6 (9) (2025) 226. [32] T. Wolf, L. Debut, V. Sanh, et al., HuggingFace’s Transformers: State- of-the-art natural language processing, in: Proceedings of the 2020 Con- ference on Empirical Methods in Natural Language Processing: System Demonstrations, Association for Computational Linguistics, 2020, p. 38–45. [33] P. Qi, Y. Zhang, Y. Zhang, J. Bolton, C. D. Manning, Stanza : A Python natural language processing toolkit for many human languages, arXiv (2020). arXiv:2003.07082, doi:10.18653/v1/2020.acl-demos.14. [34] M. Honnibal, I. Montani, S. Van Landeghem, A. Boyd, spaCy: Industrial-strength natural language processing in Python, Zenodo, 2020. doi:10.5281/zenodo.1212303. [35] M. J. Jurafsky D., Speech and Language Processing, 2nd (2008). [36] E. Sapir, Language: An introduction to the study of speech, 1921. arXiv:arXiv:1011.1669v3, doi:10.2307/3713880. [37] G. Lupyan, R. Dale, The role of adaptation in understanding linguistic diversity, Language Structure and Environment Social, cultural, and natural factors Edited by Rik De Busser and Randy J. LaPolla (2015) 184doi:10.1075/clscc.6.11lup. URL https://benjamins.com/catalog/clscc.6.11lup 29 [38] W. L. Hamilton, J. Leskovec, D. Jurafsky, Cultural Shift or Linguistic Drift? Comparing Two Computational Measures of Semantic Change., Proceedings of the Conference on Empirical Methods in Natural Lan- guage Processing. Conference on Empirical Methods in Natural Lan- guage Processing 2016 (2016) 2116–2121. URL http://w.ncbi.nlm.nih.gov/pubmed/28580459http://w. pubmedcentral.nih.gov/articlerender.fcgi?artid=PMC5452980 [39] D. Gentner, I. M. France, The Verb Mutability Effect: Stud- ies of the Combinatorial Semantics of Nouns and Verbs, in: Lexical Ambiguity Resolution, Elsevier, 1988, p. 343–382. doi:10.1016/B978-0-08-051013-2.50018-5. URL http://linkinghub.elsevier.com/retrieve/pii/ B9780080510132500185 [40] Z. S. Harris, A Theory of Language and Information: A Mathematical Approach, Clarendon Press, 1991. URL http://w.dmi.columbia.edu/zellig/langinf.html [41] C. Friedman, P. Kra, A. Rzhetsky, Two biomedical sublanguages: a description based on the theories of Zellig Harris., Journal of biomed- ical informatics 35 (4) (2002) 222–35. doi:10.1016/S1532-0464(03) 00012-1. URL http://w.ncbi.nlm.nih.gov/pubmed/12755517 [42] O. Patterson, J. F. Hurdle, Document clustering of clinical narratives: a systematic study of clinical sublanguages., AMIA ... Annual Sympo- sium proceedings / AMIA Symposium. AMIA Symposium 2011 (2011) 1099–1107. URL http://w.pubmedcentral.nih.gov/articlerender.fcgi? artid=3243234&tool=pmcentrez&rendertype=abstracthttp://w. ncbi.nlm.nih.gov/pubmed/22195171http://w.pubmedcentral. nih.gov/articlerender.fcgi?artid=PMC3243234 [43] K. Doing-Harris, O. Patterson, S. Igo, J. Hurdle, Document Sublan- guage Clustering to Detect Medical Specialty in Cross-institutional Clinical Texts., in: Proceedings of the ACM ... International Workshop on Data and Text Mining in Biomedical Informatics . ACM Interna- tional Workshop on Data and Text Mining in Biomedical Informatics, 30 Vol. 2013, ACM Press, New York, New York, USA, 2013, p. 9–12. doi:10.1145/2512089.2512101. URL http://dl.acm.org/citation.cfm?doid=2512089. 2512101https://w.ncbi.nlm.nih.gov/pmc/articles/ PMC4827341/http://w.ncbi.nlm.nih.gov/pubmed/27077137http: //w.pubmedcentral.nih.gov/articlerender.fcgi?artid= PMC4827341 [44] P. Bernhardt, S. M. Humphrey, T. C. Rindflesch, Determining promi- nent subdomains in medicine, AMIA Annu Symp Proc (2005) 46–50. URL http://w.ncbi.nlm.nih.gov/pubmed/16778999?dopt= Citation [45] D. S. Carrell, R. E. Schoen, D. A. Leffler, M. Morris, S. Rose, A. Baer, S. D. Crockett, R. A. Gourevitch, K. M. Dean, A. Mehrotra, Challenges in adapting existing clinical natural language processing systems to multiple, diverse health care settings., Journal of the American Medical Informatics Association : JAMIA 24 (5) (2017) 986–991. doi:10.1093/jamia/ocx039. URL https://academic.oup.com/jamia/article-lookup/ doi/10.1093/jamia/ocx039http://academic. oup.com/jamia/article/24/5/986/3737804/ Challenges-in-adapting-existing-clinical-naturalhttps: //oup.silverchair-cdn.com/oup/backfile/Content_public/ Journal/jamia/24/5/1 31

Instant research discovery