Paper deep dive
FlowExtract: Procedural Knowledge Extraction from Maintenance Flowcharts
Guillermo Gil de Avalle, Laura Maruster, Eric Sloot, Christos Emmanouilidis
Intelligence
Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 97%
Last extracted: 4/10/2026, 3:42:54 AM
Summary
FlowExtract is a hybrid pipeline for extracting directed graphs from ISO 5807-standardized maintenance flowcharts. It combines YOLOv8 for node detection and EasyOCR for text extraction with a novel arrowhead-anchored edge detection method that traces lines backward to source nodes, significantly outperforming vision-language models in connectivity reconstruction.
Entities (5)
Relation Signals (3)
FlowExtract → uses → YOLOv8
confidence 100% · The first stage detects and classifies flowchart elements using YOLOv8
FlowExtract → uses → EasyOCR
confidence 100% · The second stage extracts textual content using EasyOCR
FlowExtract → implements → ISO 5807
confidence 90% · We present FlowExtract, a pipeline for extracting directed graphs from ISO 5807-standardized flowcharts.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Maintenance procedures in manufacturing facilities are often documented as flowcharts in static PDFs or scanned images. They encode procedural knowledge essential for asset lifecycle management, yet inaccessible to modern operator support systems. Vision-language models, the dominant paradigm for image understanding, struggle to reconstruct connection topology from such diagrams. We present FlowExtract, a pipeline for extracting directed graphs from ISO 5807-standardized flowcharts. The system separates element detection from connectivity reconstruction, using YOLOv8 and EasyOCR for standard domain-aligned node detection and text extraction, combined with a novel edge detection method that analyzes arrowhead orientations and traces connecting lines backward to source nodes. Evaluated on industrial troubleshooting guides, FlowExtract achieves very high node detection and substantially outperforms vision-language model baselines on edge extraction, offering organizations a practical path toward queryable procedural knowledge representations. The implementation is available athttps://github.com/guille-gil/FlowExtract.
Tags
Links
- Source: https://arxiv.org/abs/2604.06770v1
- Canonical: https://arxiv.org/abs/2604.06770v1
Trouble viewing inline? Open PDF directly →
Full Text
39,745 characters extracted from source content.
Expand or collapse full text
FlowExtract: Procedural Knowledge Extraction from Maintenance Flowcharts Guillermo Gil de Avalle 1[0009−0004−1538−2256] , Laura Maruster 1[0000−0002−6588−7648] , Eric Sloot 2[0009−0009−2419−6371] , and Christos Emmanouilidis 1[0000−0003−4335−6915] 1 University of Groningen, Nettelbosje 2, Groningen, The Netherlands g.gil.de.avalle,l.maruster,c.emmanouilidis@rug.nl 2 Philips Consumer Lifestyle B.V., Oliemolenstraat 5, Drachten, The Netherlands eric.sloot@philips.com Abstract. Maintenance procedures in manufacturing facilities are often documented as flowcharts in static PDFs or scanned images. They en- code procedural knowledge essential for asset lifecycle management, yet inaccessible to modern operator support systems. Vision-language mod- els, the dominant paradigm for image understanding, struggle to recon- struct connection topology from such diagrams. We present FlowExtract, a pipeline for extracting directed graphs from ISO 5807-standardized flowcharts. The system separates element detection from connectivity re- construction, using YOLOv8 and EasyOCR for standard domain-aligned node detection and text extraction, combined with a novel edge de- tection method that analyzes arrowhead orientations and traces con- necting lines backward to source nodes. Evaluated on industrial trou- bleshooting guides, FlowExtract achieves very high node detection and substantially outperforms vision-language model baselines on edge ex- traction, offering organizations a practical path toward queryable pro- cedural knowledge representations. The implementation is available at https://github.com/guille-gil/FlowExtract. Keywords: Procedural knowledge· Knowledge extraction· Mainte- nance documentation· Flowchart understanding· Computer vision 1 Introduction Flowcharts encode procedural knowledge as step-by-step instructions represent- ing decision logic and action sequences, in a visual format that humans parse effortlessly. We trace arrows between boxes, interpreting shapes as actions or decisions and connections as control flow. This understanding relies on both semantic interpretation of node content and spatial reasoning about how ele- ments relate [1]. Automated systems face a harder task: parsing flowcharts re- quires text recognition, identification of visual elements, and reconstruction of the connections between them [2]. Although standards such as ISO 5807 [3] de- fine unambiguous symbol conventions, procedural documentation often remains arXiv:2604.06770v1 [cs.CV] 8 Apr 2026 2G. Gil de Avalle et al. as scanned images or static PDF files, posing challenges for organizations seeking to extract the knowledge encoded within [4]. This challenge is particularly acute for asset lifecycle management. Mainte- nance procedures guide operators and technicians through equipment diagnos- tics, corrective actions, or preventive routines, encoding operational knowledge developed over decades of experience [5]. However, this knowledge typically re- mains trapped in vast libraries of unorganized documentation that staff must manually navigate [4]. Converting these procedures into queryable knowledge representations would enable intelligent decision support [6], but manual digi- tization is prohibitively expensive at scale [5], creating demand for automated extraction methods that can reliably parse the flowchart structure. Vision-language models (VLMs) have emerged as the dominant paradigm for joint document and image understanding [7], encoding visual features and tex- tual information through transformer architectures. VLMs demonstrate strong visual analysis capabilities, and can even interpret basic diagrams [2]. However, when applied to flowcharts with dense layouts, overlapping arrows, and/or com- plex branching structures, performance suffers substantially. Systematic evalua- tion reveals that while VLMs handle node and text detection reasonably well, they consistently fail to reconstruct the connection topology [2]. Pipeline ap- proaches combining object detection with language models face similar limita- tions, as converting spatial layouts into textual descriptions discards positional information essential for determining connectivity [8]. The conclusion is clear: connectivity extraction remains an important bottleneck in flowchart digitiza- tion, one that semantic understanding alone cannot resolve. Recognizing these limitations, prior work on engineering diagram digitiza- tion suggests an alternative: hybrid architectures that separate element detec- tion from connectivity reconstruction, reserving neural methods for discrete ob- jects, while using classical techniques for line tracing [9,10]. We implement this principle in FlowExtract, combining YOLOv8 [11] for node detection and Easy- OCR [12] for text extraction with a novel edge detection method. Where prior approaches detect line networks first and attach directional information after- ward [10], our method anchors on arrowheads as hooks for connection detection. It analyzes their orientations to determine the direction of the edges, and then traces the connecting lines backward to identify the source nodes. This design prioritizes precision over recall, proposing edges only where directional indicators have been detected rather than inferring connectivity from generic line extraction alone. We evaluate FlowExtract on industrial troubleshooting diagrams from a con- sumer electronics manufacturing facility. These documents present challenges including dense technical terminology, tightly spaced nodes, and overlapping edges. FlowExtract achieves near-perfect node detection and substantially out- performs VLM baselines on edge extraction, with a precision-oriented profile where detected connections are typically correct. This profile fits well within human-in-the-loop digitization workflows [13], where the system provides a reli- able structural skeleton as a directed graph, allowing human validators to extend FlowExtract: Procedural Knowledge Extraction3 the extraction rather than detect and correct errors that could propagate into the final knowledge base [14]. Our contributions are twofold. First, we introduce a novel arrowhead-anchored edge detection method for flowchart digitization that prioritizes precision over recall, producing directed graph representations where detected connections are reliable. Second, we provide empirical evidence that hybrid architectures sep- arating element detection from connectivity reconstruction substantially out- perform vision-language models on procedural flowcharts. Taken together, these contributions offer a starting point for organizations seeking to unlock procedu- ral knowledge from legacy maintenance documentation, enabling its integration into modern decision support systems. 2 Background 2.1 Vision-Language Model-Based Extraction Vision-language models have achieved strong performance on combined document- image interpretation, inferring semantically meaningful descriptions of visual content [7]. However, this semantic focus encounters challenges when connec- tion topology becomes complex. Systematic evaluation on a dataset of 13,858 flowcharts, comprising both scientific diagrams and synthetic examples, reveals a consistent pattern across both closed models (GPT-4V, Claude-3, Gemini-Pro) and open models (LLaVA, Qwen-VL, InternLM, DeepSeek). While VLMs han- dle node and text detection reasonably well, they consistently fail to reconstruct connection topology, with edge-level F1 scores remaining below 0.30 across all tested architectures [2]. These limitations persist amid strong performance on related tasks, also requiring visual reasoning and text recognition. Hybrid approaches, which combine the analytical abilities of VLMs or text- only LLMs with other methods that provide enhanced visual cues, also face similar limitations. A representative example uses the Segment Anything Model for zero-shot segmentation, OCR for text extraction, and GPT-3.5-Turbo for graph reconstruction from element descriptions [8]. This conversion from spatial layout to unordered textual descriptions discards the essential positional infor- mation for determining connectivity. Arrow-guided prompting takes an alterna- tive approach by explicitly encoding arrow directions into VLM prompts [15], demonstrating improvements over unaugmented baselines. However, even with explicit directional cues, VLMs struggle with overlapping or densely packed ar- row patterns. The common limitation across these approaches is the reliance on language models to infer connection topology from representations that encode spatial information implicitly or incompletely. Textual descriptions inherently lose the precise positional relationships of 2D layouts, and proximity becomes ambiguous when elements are described sequentially rather than spatially. One noteworthy exception applies the Relationformer neural architecture to jointly extract symbols and connections from piping and instrumentation di- agrams, outperforming modular baselines by over 25% in edge detection [16]. 4G. Gil de Avalle et al. However, the studied pre-trained on over 170,000 samples and fine-tuned on tens of thousands more, with ablation studies showing catastrophic performance degradation when training data was reduced. Such data requirements are infea- sible in most industrial digitization projects, where labeled diagrams are much more limited. These constraints motivate approaches that leverage the domain knowledge characteristics for methodological design, rather than learning con- nectivity patterns from data alone. 2.2 Spatial Structure Extraction Where language models infer connectivity using their internal understanding and/or fine-tuning, alternative approaches usually exploit the geometric struc- ture of diagrams directly. Diagrams across technical domains share a common vi- sual grammar wherein discrete nodes are joined by connections, generally lines or arrows. This convention appears in flowcharts standardized under ISO 5807 [3], in piping and instrumentation schematics, and in hand-drawn sketches alike. Nodes can be understood as bounded regions with characteristic shapes and fixed aspect ratios. This is where neural network architectures excel, given their capacity for learned pattern matching across symbol detection and text recogni- tion tasks [9]. Connecting lines, by contrast, vary in length and orientation but can be compactly described by geometric parameters, making them amenable to line tracing techniques such as the Hough transform [17] that follow paths through pixel space rather than recognizing learned patterns. Recognizing this asymmetry, recent systems have adopted hybrid architectures, reserving neu- ral networks for discrete elements while relying on classical image processing techniques for line extraction [10]. Within this decomposition, different methods emphasize different elements, though arrowhead detection has emerged as a particularly effective strategy. Ar- rowheads can be treated as nodes themselves, since they are bounded regions with characteristic shapes amenable to pattern recognition. Work on handwrit- ten diagram recognition has demonstrated that extending object detectors with keypoint predictors to localize arrowheads and tails as spatial coordinates can double overall edge prediction accuracy to 78.6% [18]. Even the pipeline ap- proach described in the previous section employed arrows as explicit cues for vision-language models, though without achieving comparable performance [15]. The consistency of these results likely stems from the ubiquity of arrowheads as standardized directional cues, which justifies their use as explicit anchors from which the remainder of each connection can be inferred, thereby improving overall directed graph extraction. 2.3 Human-in-the-loop Extraction Even where decomposition approaches prove more effective, they do not guaran- tee operational reliability. In industrial maintenance contexts, where extracted FlowExtract: Procedural Knowledge Extraction5 Fig. 1. Human-in-the-loop workflow for knowledge extraction from legacy maintenance documentation. procedural knowledge informs decisions with safety consequences on top of ef- fectiveness and efficiency requirements, fully automated extraction alone re- mains insufficient to obtain the data quality that would effectively mitigate such risks [6]. Human-in-the-loop workflows address this gap by positioning domain experts that validate outputs and contribute contextual knowledge, which au- tomated methods cannot reliably capture [5,13], thus maximizing the quality of extraction. Figure 1 illustrates such workflow, in which automated extraction provides a preliminary extraction that experts correct and complete to obtain the final digital queryable knowledge that may be used in operator support systems. However, effective oversight of automated AI systems presents cognitive chal- lenges that constrain how such human-in-the-loop workflows should be designed, even in the conception of the initial automated extraction. Research on automa- tion bias, the tendency to over-rely on automated recommendations, demon- strates that validators often accept incorrect outputs without sufficient verifi- cation [14]. This bias manifests itself asymmetrically. Commission errors, where validators accept incorrect outputs, prove harder to detect than omission er- rors, where validators notice missing information [19]. Training interventions can reduce commission errors but show limited effectiveness against omission errors [14]. Systems producing false positives therefore impose heavier cognitive burdens than systems producing false negatives; the former requires detecting errors embedded within plausible outputs, while the latter requires extending an incomplete but reliable skeleton. We therefore posit that extraction systems designed to fit within human-in-the-loop workflows should adopt conservative ex- traction strategies, prioritizing precision over recall to position validators where human cognitive abilities excel. 6G. Gil de Avalle et al. Table 1. Class Distribution Across Dataset Splits ClassTraining Validation Test Total Arrowhead 3574893 498 Process1772345 245 Document 1191138 168 Decision1181529 162 Terminator 3549 48 Connector1725 24 Total823103 219 1145 3 Methodology 3.1 Dataset The present study involves a collection of troubleshooting maintenance diagrams collected from an operational consumer electronic domestic devices manufactur- ing facility. Using real-world operational data ensures our evaluation reflects the actual challenges of production environments, including authentic visual char- acteristics and domain-specific terminology. The dataset comprises 35 diagrams written in Dutch, consisting of 1145 nodes (flowchart elements, such as process boxes, decision diamonds, and terminators), edges (directional arrows connect- ing nodes), and decision labels (text annotations “ja”/“nee” in Dutch, appearing along edge paths to indicate branching logic). This dataset was manually annotated using Label Studio [20], an open-source data labeling tool, for bounding box detection of all element types and ground truth specifying node connection and edge labels. In total, 1145 node instances were captured across all diagrams, plus an extra 62 “ja”/“nee” annotations. We split the dataset into 25 training images (823 instances, Table 1), 3 validation images (103 instances), and 7 test images (219 instances). The test set is held out during training and used only for final evaluation. Note that although the operational dataset is small, each image provides multiple instances of symbols, offering a somewhat richer basis for training, validation, and testing. 3.2 FlowExtract Pipeline Our pipeline processes troubleshooting diagram images and outputs directed graphs representing procedural flow. Fig. 2 illustrates the processing workflow: (a) the three-stage pipeline overview comprising node detection, text extraction, and edge detection; and (b) the edge detection methodology showing arrowhead orientation analysis and line tracing. Node Detection The first stage detects and classifies flowchart elements using YOLOv8 [11], chosen for its maturity, extensive documentation, and established track record in domain adaptation that facilitates the inclusion of custom sym- bol sets. We define six element classes based on the subset of ISO 5807 [3] FlowExtract: Procedural Knowledge Extraction7 Fig. 2. FlowExtract pipeline overview (a) and edge detection methodology (b), showing arrowhead orientation analysis and line tracing for straight, L-shaped, and multi-branch connections. symbols present in our diagrams, plus arrowheads which enable subsequent edge detection, as shown in Fig. 3: Process (rectangular boxes containing action de- scriptions), Decision (diamond shapes containing yes/no questions), Document (rectangles with wavy bottom edges referencing external procedures), Termina- tor (rounded rectangles marking start/end points), Connector (circular elements with identifiers for cross-page references), and Arrowhead (arrow tips indicating edge direction, essential for edge detection but not included as graph nodes). We initialize YOLOv8s (small variant) with weights pre-trained on the COCO dataset [21], a large-scale object detection benchmark containing 80 object cat- egories across 330K images, providing general-purpose visual features, offering some transferability to our domain. Given the limited size of the training in- stances, we use the small variant to reduce overfit risk, as smaller models with fewer parameters are less prone to memorizing training examples when data is limited. However, our data set exhibits severe class imbalance, with 21 arrow- heads (most common class; 357 samples) for every connector (rarest class, 24 samples), as seen in Table 1. We address this through a Data Augmentation 8G. Gil de Avalle et al. Strategy, combining Mosaic augmentation [22], which combines four training images per sample, thereby increasing our effective dataset size, with Brightness variation (HSV, brightness metric, at±30%) to simulate varying scan conditions. We intentionally avoid other types of augmentation (e.g., geometric, which ro- tates, translates, and flips the image) due to tight bounding box annotations, so as to prevent loss of spatial data characteristics. Text Extraction The second stage extracts textual content using EasyOCR [12], a deep learning-based OCR system that provides multilingual support and han- dles the text characteristics common in technical diagrams (different font sizes, orientations, and image quality of scanned documents). For each detected ele- ment (excluding arrowheads), we crop the corresponding image region and apply OCR to extract the contained text. Additionally, we perform a full-image scan to detect decision labels (“ja”/“nee” in Dutch, corresponding to yes/no) that appear along arrow paths rather than within bounded elements. These labels are stored separately for later assignment to the edges during edge detection. Edge Detection The third stage reconstructs the directed graph structure by determining which elements are connected by arrows. As pointed during the Background section, we derive connections from detected arrowheads and line segments. Detected arrowhead nodes from the first stage are used to derive edges, as shown in Fig. 2(b). For each detected arrowhead, we determine its orientation by identifying which end is closest to a node element: the “pointy” end (closest to a box) indicates the target node, while the “blunt” end indicates the direction toward the source. From the blunt end of each arrowhead, we trace connected line segments back toward potential source nodes by applying the Probabilistic Hough Transform [17] to detect line segments in a preprocessed binary image. Starting from the arrowhead position, we perform a graph-based traversal of detected segments, following connected paths until reaching proximity to a node element. This approach handles straight connections, L-shaped paths, and multi- branch configurations as illustrated in Fig. 2(b). Once the source and target nodes are identified for an arrowhead, we create a directed edge. For decision nodes with multiple outgoing edges, we assign labels (ja/nee) based on the spatial relationship between detected label positions and arrow directions. Specifically, we score each label by its distance to the arrow midpoint and then run the text extractor to identify whether the label corre- sponds to ja or nee. Decision nodes may have multiple outgoing edges sharing a common source segment before branching; our traversal algorithm handles this by exploring all connected paths from each arrowhead independently, allowing reconstruction of complex branching structures. This arrowhead-anchored ap- proach prioritizes precision over recall by proposing edges only where directional indicators have been explicitly detected, rather than inferring connections from line segments that may represent diagram borders, visual artifacts, or other non- connective structures. FlowExtract: Procedural Knowledge Extraction9 Fig. 3. Flowchart symbols taxonomy (based on ISO 5807:1985) and arrowheads. 3.3 Training Configuration We train YOLOv8s for 250 epochs with image size 640×640, batch size 8, and co- sine annealing learning rate scheduling with initial learning rate of 0.01, following YOLOv8 default recommendations [11]. The image size balances detection ac- curacy with computational efficiency given our hardware constraints (Apple M3 chip with 8 CPU cores and 10 GPU cores), completing training in approximately 40 minutes. The pipeline outputs a JSON structure containing nodes (list of elements with unique ID, type, bounding box coordinates, and extracted text) and edges (list of directed connections with source ID, target ID, edge type, and optional label). This representation enables direct ingestion into knowledge constructs: nodes map to ontological concepts (e.g., process steps as procedural actions), edges represent causal or temporal relationships, and the JSON-LD format facilitates RDF conversion for semantic web integration [5]. 3.4 Evaluation We evaluate each pipeline component separately, structuring them as individual detection and classification tasks. For node detection, we measure precision, recall, and F1 for detecting the five node classes (excluding arrowheads), along with classification accuracy for assigning correct types to detected nodes. For edge detection, we measure precision, recall, and F1 for recovering directed edges between nodes, along with accuracy of assigning correct labels (ja/nee/none) to detected edges. OCR accuracy is measured as character-level match rate between extracted and ground truth text. For detection metrics, we match predicted and ground truth bounding boxes using intersection-over-union (IoU), the ratio of overlap area to union area. A detection is considered correct if IoU> 0.5, similar to comparable papers [16]. Higher thresholds would demand stricter localization, while lower values would risk accepting misaligned boxes. To contextualize our results, we compare them against VLM baselines eval- uated on the same set of documents in previous work [23]. Two state-of-the-art 10G. Gil de Avalle et al. Table 2. Detection Performance on Test Set TaskPrecision Recall F1 Accuracy Node Detection98.4% 99.2% 98.8%– Node Classification–97.6% Text Extraction–99.2% Edge Detection85.5% 54.6% 66.7%– Arrowhead Detection–73.1% Edge Label Assignment–73.8% Table 3. Results Comparison with VLM Baselines from [23] MethodNode F1 Edge F1 Qwen2-VL-7B 0.340 0.107 Pixtral-12B0.295 0.015 FlowExtract 0.988 0.667 open models were tested: Pixtral-12B and Qwen2-VL-7B. Models were prompted to extract entities (nodes) and relations (edges) from flowchart images in an instruction-guided setting, as seen in the original work [23]. The prompt template requested structured JSON output containing node types, bounding boxes, text content, and directional connections. No token limit was imposed, and models received the full flowchart image with the extraction task description. Evaluation used the same ground truth annotations as our pipeline, with matching based on textual closeness of extracted results and category selection correctness. 4 Results Table 2 presents the main evaluation results on the held-out test set. Our pro- posed pipeline achieves near-perfect node detection with 98.8% F1, compared to 34.0% for the best VLM baseline (Table 3). The different node classes achieve detection rates above 97%, with the exception of arrowheads, which only achieve 73.1% accuracy. This demonstrates that YOLOv8 with mosaic augmentation ef- fectively handles the severe class imbalance, successfully detecting rare classes (terminator, connector), amid limited training examples. Node type classification accuracy reaches 97.6%, indicating that detection errors are primarily localiza- tion failures rather than misclassification. For edge detection, our approach achieves 66.7% F1 versus 10.7% for the best VLM, representing a six-fold improvement. The results show high asymme- try between precision (85.5%) and recall (54.6%), indicating that when edges are detected, they are usually correct, but many ground truth edges are missed. This asymmetry reflects the arrowhead-anchored design, which proposes edges only where directional indicators have been explicitly detected rather than inferring connectivity from line structures alone. For detected edges, label assignment (ja/nee) achieves 73.8% accuracy. Analysis reveals that edge recall is directly limited by arrowhead detection performance: since edge detection requires de- FlowExtract: Procedural Knowledge Extraction11 tecting the arrowhead that indicates connection direction, missed arrowheads directly translate to missed edges. Individual analysis reveals that missed arrow- heads tend to be either small at standard zoom levels or partially occluded by overlapping line segments. Text extraction achieves 99.2% character-level accu- racy, indicating that OCR is not a significant error source; the high performance reflects the clean, printed nature of the source documents. Fig. 4. Example of extraction results from one of the maintenance diagrams. The orig- inal textual content within the nodes has been computationally redacted (opaque fills) to anonymize proprietary procedural data, while preserving the structural morphology. Figure 4 provides a qualitative visualization of the extraction pipeline applied to a sample maintenance diagram. The visual overlay confirms the node detection stage successfully isolates discrete elements within dense layouts. More critically, the extracted connections (magenta lines) demonstrate the system’s high preci- sion to trace procedural flow, even in long documents. It also seems to handle multi-branching logic, though capturing seems to be limited to a maximum of 2-3 connections into a single node when dealing with highly dense clusters of edges, thus affecting recall. Furthermore, the figure confirms that when connections are established, the spatial heuristics accurately anchor the path to the correct source and target elements without generating hallucinatory cross-links, as seen by the fact that no magenta lines seem to fall out of the connection paths. This confirms that, while arrowhead omissions restrict total recall (such as missing branches within dense graphical clusters), the successfully parsed connections draft a precise flowchart skeleton. 12G. Gil de Avalle et al. 5 Discussion The results provide empirical support for the architectural principle identified in Section 2.2, namely that standardized technical diagrams benefit from separat- ing element detection from connectivity reconstruction. Node detection achieves very high performance at 98.8% F1, demonstrating that element recognition is effectively solved for this diagram class when approached with conventional object detection methods. The challenge lies entirely in reconstructing connec- tivity. This confirms the pattern documented across engineering diagram do- mains, where symbol detection and text recognition have advanced substantially through neural architectures while connection extraction remains the primary bottleneck [9]. Our contribution is therefore not only an ISO 5807-based dia- gram parser, but also targeted evidence that the separation principle transfers successfully to procedural flowcharts, with implications for how extraction sys- tems should be architected when standardized visual conventions are present. The 66.7% edge F1 could reflect either a failure of structural reasoning or a failure of perception. The precision and recall split reveals the answer. Edge detection achieves 85.5% precision but only 54.6% recall, indicating that when arrowheads are detected, the Hough-based line tracing reliably recovers the cor- rect source node. The bottleneck is upstream: arrowheads are missed due to small apparent size or partial occlusion by overlapping line segments. This finding is sharper than the general observation that vision-language models struggle with spatial reasoning. We have isolated the performance ceiling to a specific percep- tual task rather than to the graph reconstruction logic itself. The line tracing component functions correctly; improvements to arrowhead detection translate directly into system-level gains without requiring changes to the connectivity algorithm. The arrowheads-first architecture reflects a deliberate design choice rather than the only viable option. An alternative strategy, exemplified in recent P&ID digitization work [10], detects the line network first and attaches directional in- formation afterward. That approach would likely achieve higher recall, since a missed arrow does not eliminate the underlying line structure from consideration. However, lines-first introduces ambiguity in dense diagram regions where multi- ple lines converge toward shared junction points, requiring additional heuristics to determine which line segments constitute which edges. Arrowheads-first guar- antees that each detected arrowhead corresponds to exactly one directed edge, eliminating edge-count ambiguity at the cost of recall when arrowheads go un- detected. For procedural flowcharts where directionality carries semantic weight and edge identity must be unambiguous, we argue this tradeoff is appropriate. The choice should be revisited for diagram types where the underlying network matters independently of connection direction, which may not present arrow- heads. The precision-oriented performance profile directly implements the conser- vative extraction philosophy established in Section 2.3. High precision ensures that detected edges are trustworthy, allowing human validators to extend the extracted graph rather than correct it. This aligns with the principle of posi- FlowExtract: Procedural Knowledge Extraction13 tioning human and algorithmic components where each exhibits comparative cognitive advantage [6,13]. The system contributes a reliable structural skele- ton; human reviewers contribute completeness and contextual judgment that automated methods cannot reliably provide. Cognitive load during validation is reduced because reviewers are not second-guessing system outputs but rather filling gaps that the system has conservatively declined to fill. For asset lifecycle management applications where propagated errors carry operational and poten- tial safety consequences, this division of labor supports the kind of human-AI collaboration that industrial knowledge management increasingly requires [5,13]. Notwithstanding the encouraging results, we also acknowledge some limita- tions. Firstly, arrowhead detection recall directly caps edge performance and cannot be circumvented through downstream improvements. Alternative archi- tectures such as lines-first detection offer different tradeoffs but not unambigu- ous improvements; the appropriate choice depends on whether precision or recall matters more for the downstream application. A secondary limitation is scope. FlowExtract targets ISO 5807-standardized flowcharts with clean print and line work, and would require adaptation for documents employing non-standard sym- bol conventions (especially if not using arrows as connectors), other standards, or highly degraded documents. Finally, the pipeline processes documents in iso- lation and does not resolve cross-document references encoded through connec- tor symbols. Future work could extend the system to match connector labels across documents. Merging partial graphs would enable extraction of complete procedures stemming from different documents; a common pattern in industrial maintenance where troubleshooting guides often reference sub-procedures stored separately. Assessment and validation with human operators is also among the plans for further work. 6 Conclusion We introduced FlowExtract, a pipeline for extracting directed graphs from stan- dardized maintenance flowcharts. The system separates element detection from connectivity reconstruction, combining standard domain-adapted node detection with a novel arrowhead-anchored edge detection method that traces connect- ing lines backward from detected directional indicators. Evaluated on industrial troubleshooting diagrams, FlowExtract achieves near-perfect node detection and outperforms vision-language model baselines on edge extraction by a factor of six, with a precision-oriented profile where detected connections are typically correct. These results validate the architectural principle that standardized technical diagrams benefit from treating element detection and connectivity reconstruc- tion as distinct tasks, with neural methods suited to bounded symbols and geo- metric methods suited to continuous line structures. The precision-oriented ex- traction profile fits well within human-in-the-loop workflows, where conservative automated extraction reduces the cognitive burden of validation by position- ing human reviewers to extend an incomplete but reliable skeleton rather than 14G. Gil de Avalle et al. detecting commission errors. For organizations seeking to digitize legacy main- tenance documentation, this hybrid pipeline offers a practical starting point for unlocking procedural knowledge and integrating it into asset lifecycle manage- ment systems. Acknowledgments. This research was funded by the European Union’s Horizon Eu- rope research and innovation programme under the AIXPERT project (Grant Agree- ment No. 101214389), which aims to develop an agentic, multi-layered, GenAI-powered framework for creating explainable, accountable, and transparent AI systems. The au- thors wish to extend special thanks to Philips Lifestyle Consumer B.V. for the valuable collaboration and support in this work. Disclosure of Interests. The authors have no competing interests to declare that are relevant to the content of this article. References 1. Chen, B., Xu, Z., Kirmani, S., Ichter, B., Driess, D., Florence, P., Sadigh, D., Guibas, L., Xia, F.: SpatialVLM: Endowing Vision-Language Models with Spatial Reasoning Capabilities. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), p. 14455–14465 (2024) 2. Pan, H., Zhang, Q., Caragea, C., Dragut, E., Latecki, L.J.: FlowLearn: Evaluating Large Vision-Language Models on Flowchart Understanding. In: Proceedings of the 27th European Conference on Artificial Intelligence (ECAI), p. 73–80. IOS Press (2024) 3. ISO: ISO 5807:1985 Information Processing – Documentation Symbols and Con- ventions for Data, Program and System Flowcharts, Program Network Charts and System Resources Charts. International Organization for Standardization (1985) 4. Mentzas, G., Hribernik, K., Stahre, J., Romero, D., Soldatos, J.: Editorial: Human- Centered Artificial Intelligence in Industry 5.0. Front. Artif. Intell. 7, 1429186 (2024) 5. Celino, I., Carriero, V.A., Azzini, A., Baroni, I., Scrocca, M.: Procedural knowledge management in Industry 5.0: Challenges and opportunities for knowledge graphs. Web Semantics: Science, Services and Agents on the World Wide Web 84, 100850 (2025) 6. Emmanouilidis, C., Pistofidis, P., Bertoncelj, L., Katsouros, V., Fournaris, A.P., Koulamas, C., Ruiz-Carcel, C.: Enabling the human in the loop: Linked data and knowledge in industrial cyber-physical systems. Annual Reviews in Control 47, 249– 265 (2019) 7. Huang, Y., Lv, T., Cui, L., Lu, Y., Wei, F.: LayoutLMv3: Pre-training for Document AI with Unified Text and Image Masking. In: Proceedings of the ACM International Conference on Multimedia, p. 4083–4091 (2022) 8. Arbaz, A., et al.: GenFlowchart: Parsing and Understanding Flowchart Using Gen- erative AI. In: Proceedings of the International Conference on Knowledge Science, Engineering and Management (KSEM), p. 95–107 (2024) 9. Jamieson, L., Moreno-García, C.F., Elyan, E.: A review of deep learning methods for digitisation of complex documents and engineering diagrams. Artificial Intelligence Review 57(136) (2024) FlowExtract: Procedural Knowledge Extraction15 10. Park, S., Kim, H., Paik, S., Kim, K.: Deep Learning-Based Method to Recognize Line Objects and Flow Arrows from Image-Format Piping and Instrumentation Diagrams for Digitization. Applied Sciences 11(21), 10054 (2021) 11. Jocher, G., Chaurasia, A., Qiu, J.: Ultralytics YOLOv8. https://github.com/ ultralytics/ultralytics (2023) 12. JaidedAI: EasyOCR: Ready-to-use OCR with 80+ Supported Languages. https: //github.com/JaidedAI/EasyOCR (2020) 13. Emmanouilidis, C., Waschull, S., Bokhorst, J.A.C., Wortmann, J.C.: Human in the AI Loop in Production Environments. In: Dolgui, A., Bernard, A., Lemoine, D., von Cieminski, G., Romero, D. (eds.) APMS 2021. IFIP AICT, vol. 633, p. 331–342. Springer, Cham (2021) 14. Parasuraman, R., Manzey, D.H.: Complacency and bias in human use of automa- tion: An attentional integration. Human Factors 52(3), 381–410 (2010) 15. Hu, X., Lin, Z., Zeng, F., Lee, J., Keutzer, K., Tomizuka, M., Zhan, W.: Enhancing Flowchart Understanding in Vision-Language Models through Arrow-Based Aug- mentation. arXiv preprint arXiv:2501.08829 (2025) 16. Stürmer, J.M., Graumann, M., Koch, T.: From Engineering Diagrams to Graphs: Digitizing P&IDs with Transformers. In: Proceedings of the IEEE 12th International Conference on Data Science and Advanced Analytics (DSAA). IEEE (2025) 17. Duda, R.O., Hart, P.E.: Use of the Hough Transformation to Detect Lines and Curves in Pictures. Communications of the ACM 15(1), 11–15 (1972) 18. Schäfer, B., Keuper, M., Stuckenschmidt, H.: Arrow R-CNN for Handwritten Di- agram Recognition. International Journal on Document Analysis and Recognition 24, 3–17 (2021) 19. Lyell, D., Coiera, E.: Automation bias and verification complexity: A systematic review. J. Am. Med. Inform. Assoc. 24(2), 423–431 (2017) 20. Heartex: Label Studio: Data labeling software. https://github.com/ heartexlabs/label-studio (2023) 21. Lin, T.Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., Zitnick, C.L.: Microsoft COCO: Common Objects in Context. In: European Conference on Computer Vision (ECCV), p. 740–755 (2014) 22. Bochkovskiy, A., Wang, C.Y., Liao, H.Y.M.: YOLOv4: Optimal Speed and Accu- racy of Object Detection. arXiv preprint arXiv:2004.10934 (2020) 23. Gil de Avalle, G., Maruster, L., Emmanouilidis, C.: Procedural Knowledge Extrac- tion from Industrial Troubleshooting Guides Using Vision Language Models. arXiv preprint arXiv:2601.22754 (2026)