Paper deep dive
Workflow Cards: Structured Summaries of Workflow Executions Using Provenance Data
Nicola Giuseppe Marchioro, Gabriele Padovani, Amal Gueroudji, Rafael Ferreira da Silva, Wesley Brewer, Valentine Anantharaj, Sandro Fiore, Renan Souza
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 89%
Last extracted: 8/16/2026, 3:43:53 AM
Summary
The paper introduces Workflow Cards, a structured documentation format that summarizes machine-readable provenance data from workflow executions into human- and LLM-readable summaries. It addresses the gap left by Model and Data Cards, which focus on static artifacts rather than execution details. The authors define a template based on common provenance questions and evaluate its effectiveness through two benchmarks: one assessing informational complementarity against existing cards, and another comparing Workflow Cards to schema-based provenance querying. Results indicate that Workflow Cards provide unique execution-level information and nearly double the answer quality for LLMs compared to schema-based querying.
Entities (10)
Relation Signals (8)
Workflow Cards → isasummaryof → Provenance Data
confidence 95% · Workflow Cards: structured summaries that condense the machine-readable provenance data of a workflow execution
Workflow Cards → improvesanswerqualityover → schema-based querying
confidence 92% · Workflow Cards nearly double answer quality compared with schema-based querying
Workflow Cards → complements → Model Cards
confidence 90% · Workflow Cards complement rather than replace provenance traces... complementing... Model and Data Cards
Workflow Cards → complements → Data Cards
confidence 90% · Workflow Cards complement rather than replace provenance traces... complementing... Model and Data Cards
Workflow Cards → isevaluatedin → Benchmark II
confidence 90% · Benchmark II evaluates their accessibility advantage for LLMs compared to a lightweight schema-based interface
Workflow Cards → isevaluatedin → Benchmark I
confidence 90% · Benchmark I assesses whether Workflow Cards provide execution-level information absent from existing Model and Data Cards.
Workflow Cards → isgeneratedby → Flowcept
confidence 85% · integrated with the provenance systems yProv4ML [34, 37] and Flowcept [49, 47]
Workflow Cards → isgeneratedby →
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Model Cards and Data Cards have demonstrated the value of structured, human-readable documentation for machine learning artifacts, capturing their context, parameters, limitations, and intended use. However, these practices remain focused on static artifacts (the datasets and trained models themselves) while overlooking the workflow executions that produce, transform, and evaluate them. Such executions hold critical details about data preparation, parameter choice, runtime behavior, resource use, and intermediate transformations, precisely where bias, performance variation, and reproducibility gaps tend to originate. To close this gap, we introduce Workflow Cards: structured summaries that condense the machine-readable provenance data of a workflow execution into a form both humans and large language models (LLMs) can read and analyze. This paper has two main parts. First, it defines a Workflow Card template informed by a representative set of provenance questions that surface from the execution-level data missing from Model and Data Cards. Second, it evaluates how effectively LLMs use Workflow Cards to understand workflow executions compared with querying provenance databases through a schema-based interface. Results show that Workflow Cards provide execution-level information absent from existing card types, such as Model Cards and Data Cards, thereby filling an important documentation gap; and that Workflow Cards nearly double answer quality compared with schema-based querying, consistently across LLM-as-a-Judge and human assessments.
Tags
Links
- Source: https://arxiv.org/abs/2608.11022v1
- Canonical: https://arxiv.org/abs/2608.11022v1
Trouble viewing inline? Open PDF directly →
Full Text
47,100 characters extracted from source content.
Expand or collapse full text
Workflow Cards: Structured Summaries of Workflow Executions Using Provenance Data Thanks: This manuscript has been authored in part by UT-Battelle, LLC, under contract DE-AC05-00OR22725 with the US Department of Energy (DOE). The publisher, by accepting the article for publication, acknowledges that the U.S. Government retains a non-exclusive, paid up, irrevocable, world-wide license to publish or reproduce the published form of the manuscript, or allow others to do so, for U.S. Government purposes. The DOE will provide public access to these results in accordance with the DOE Public Access Plan (http://energy.gov/downloads/doe-public-access-plan).+These authors contributed equally to this work. Work done while at ORNL. Nicola Giuseppe Marchioro+1 Gabriele Padovani+1 Amal Gueroudji2 Rafael Ferreira da Silva3 Wesley Brewer3 Valentine Anantharaj3 Sandro Fiore1 Renan Souza3 Affiliation: 1University of Trento, Trento, Italy 2Argonne National Laboratory, Lemont, IL, USA 3Oak Ridge National Laboratory, Oak Ridge, TN, USA Abstract Model Cards and Data Cards have demonstrated the value of structured, human-readable documentation for machine learning artifacts, capturing their context, parameters, limitations, and intended use. However, these practices remain focused on static artifacts (the datasets and trained models themselves) while overlooking the workflow executions that produce, transform, and evaluate them. Such executions hold critical details about data preparation, parameter choice, runtime behavior, resource use, and intermediate transformations, precisely where bias, performance variation, and reproducibility gaps tend to originate. To close this gap, we introduce Workflow Cards: structured summaries that condense the machine-readable provenance data of a workflow execution into a form both humans and large language models (LLMs) can read and analyze. This paper has two main parts. First, it defines a Workflow Card template informed by a representative set of provenance questions that surface from the execution-level data missing from Model and Data Cards. Second, it evaluates how effectively LLMs use Workflow Cards to understand workflow executions compared with querying provenance databases through a schema-based interface. Results show that Workflow Cards provide execution-level information absent from existing card types, such as Model Cards and Data Cards, thereby filling an important documentation gap; and that Workflow Cards nearly double answer quality compared with schema-based querying, consistently across LLM-as-a-Judge and human assessments. Index Terms: Workflows, Provenance, Agentic workflows, Metadata, Cards, Large Language Models I Introduction Modern workflows generate increasingly large amounts of provenance data describing execution steps, input and output artifacts, computational environments and infrastructure, workflow parameters, Key Performance Indicators (KPIs), and interactions between components [44, 45, 18]. This information plays a fundamental role in supporting reproducibility, traceability, auditing, and workflow-level transparency [26, 30], concerns closely aligned with the broader effort to treat computational workflows as FAIR scientific assets [50]. Several provenance-aware systems and workflow systems have been developed to record detailed execution traces and data lineage (i.e., data provenance) in workflow runs [13, 29, 42, 49]. These systems commonly expose the captured provenance through graph-based representations, structured databases, or machine-readable serialization formats such as those defined by the W3C PROV family of specifications [25]. Although such representations provide extensive traceability, extracting meaningful workflow-level information from them typically requires specialized query tooling, familiarity with graph traversal techniques, and a deep understanding of the underlying provenance schema, barriers that substantially limit the practical accessibility of provenance information [38]. A parallel line of research has introduced human-readable artifacts such as Model Cards [27] and Data Cards [40] to document AI components [15, 2, 52, 33]. However, these frameworks focus on static artifacts and offer limited insight into the executions that produce them, even though many critical provenance questions arise at runtime, when users must understand how outputs were derived, what infrastructure was employed, and how parameter choices influenced the result [44, 10, 7, 45]. The captured provenance holds these answers, but reaching them has required those mentioned specialized query tooling. Recent works explore LLM agents that translate natural-language questions into structured provenance queries [48]; this interactive querying improves accessibility, yet it still requires the LLM agents to locate and assemble the relevant records on demand, leaving a gap for a compact, directly readable summary of the same provenance that humans and LLMs can consume. To address this gap, we introduce Workflow Cards, whose primary objective is to lower the barrier to accessing workflow provenance for both human users and LLMs. Each card is a structured documentation artifact that represents workflow execution provenance through a question-oriented template. The template is derived from recurring provenance questions drawn from prior work [45, 46, 48, 43, 39, 4, 9, 3] and from our experiments, which reveal the fields that contribute workflow-level provenance beyond what other card types, such as Model and Data Cards, already capture. This question-oriented approach follows user-centered design, where framing outputs around user questions improves interpretability [24]. A Workflow Card surfaces a workflow execution’s data flow provenance, execution context, infrastructure, workflow activities, and generated artifacts. Because LLMs excel at document-based question answering [6], the same card serves as ready context for answering provenance questions. Workflow Cards complement rather than replace provenance traces and databases: they can be generated automatically from provenance systems yet remain independent of any storage backend, and may be produced via manual curation or metadata extraction pipelines. The contributions of this work are as follows. We (i) introduce Workflow Cards as workflow-oriented documentation artifacts for exposing workflow provenance in a human- and LLM-readable format. We (i) organize common workflow provenance questions from the literature into three categories (input/dataset, workflow, output/model) that guide the Workflow Card design. We (i) implement an open-source provenance-aware Workflow Card generation approach integrated with the provenance systems yProv4ML [34, 37] and Flowcept [49, 47]. We (iv) evaluate Workflow Cards both as complementary documentation artifacts alongside Model and Data Cards and as an LLM-facing provenance representation, showing on a real HPC ML workflow that Workflow Cards nearly double answer quality for LLM-based provenance question answering compared with schema-based querying. I Related Work I-A Documentation Artifacts for AI Systems Structured documentation artifacts significantly improve AI system transparency. Frameworks such as Datasheets for Datasets [15], Model Cards [27], Data Cards [40], and FactSheets [2] establish standard reporting practices for data collection, model behavior, and service pipelines. Recent efforts explore deeper interaction through Interactive Model Cards [12] and qualitative natural-language summaries via Report Cards [51]. These artifacts enjoy broad adoption as lightweight metadata interfaces, evidenced by analyses of thousands of repositories on platforms such as Hugging Face [23]. However, their focus remains primarily on static inputs and outputs, leaving the execution workflows connecting them largely undocumented. I-B Capturing, Querying, and Representing Provenance Provenance systems track data lineage, parameters, and component interactions to ensure reproducibility and traceability [44, 14, 38]. Workflow frameworks like Taverna [29], Pegasus [13], and provenance-aware ML tooling [42, 35] automatically record these traces, often utilizing the W3C PROV specifications [25] for interoperability. Building on these foundations, researchers have formalized provenance querying to extract derivation relationships and outcomes [7, 10], expanding recently into machine learning lifecycles [41, 45] and agentic workflows [46, 16, 17]. Although provenance-aware systems provide extensive traceability capabilities, prior surveys consistently identify accessibility and usability as major limitations of existing provenance representations [38], since provenance is typically exposed through graph structures, database schemas, or low-level serialization formats optimized for storage and machine querying rather than for human interpretation. Consequently, while provenance systems preserve workflow information faithfully, they rarely offer lightweight interfaces for communicating provenance semantics to human users or LLM-based systems. A complementary line of work packages an entire workflow run together with its provenance. The Workflow Run RO-Crate profile [21] standardizes machine-readable, FAIR packaging of workflow executions. Workflow Cards are orthogonal to, not a replacement for, such formats: rather than exhaustively packaging a run, a Workflow Card extracts its provenance into a compact, query-driven summary intended for direct human reading and LLM consumption. This positioning distinguishes Workflow Cards from the broader ecosystem of machine-oriented metadata schemas for ML assets, such as Croissant [1] for datasets, FAIR4ML [8] for models, and PROV-ML [45] for ML provenance, which standardize structured, machine-readable descriptions. Workflow Cards are complementary to these schemas: they sit one layer above by condensing the provenance data that such schemas help represent into a compact, human- and LLM-readable summary of workflow executions, and can coexist with them [19]. I Methodology We propose Workflow Cards to summarize workflow execution provenance data in a lightweight, interpretable format, and evaluate them across two independent benchmarks sharing the same protocol. Benchmark I assesses whether Workflow Cards provide execution-level information absent from existing Model and Data Cards. Benchmark I evaluates their accessibility advantage for LLMs compared to a lightweight schema-based interface to a provenance database. Two LLMs act as judges throughout: a smaller model, nemotron-nano-3 [28] (∼ 30B parameters), and a substantially larger one, gpt-oss-120b [31](∼ 120B parameters). We use an LLM-as-a-Judge approach because it scales evaluation across the full question set and has been shown to correlate strongly with human judgment on answer-rating tasks [22, 53], and we additionally validate it against an independent human rater in Benchmark I. The answering LLMs, however, differ by design. In Benchmark I, a single separate model (gpt-4o [32]) answers every card configuration, so that performance differences reflect the cards rather than the model. In Benchmark I, the two judge models also serve as answering models, spanning model scale to test whether the Workflow Card advantage holds for both a smaller and a larger model. I-A Categorizing Workflow Provenance Questions Workflow Cards organize relevant information around common provenance access patterns rather than exposing exhaustive traces. To identify these patterns, we analyzed prior work [45, 46, 48, 43, 39, 4, 9, 3] and extracted recurring questions, grouping them into three macro-categories: input (dataset), workflow, and output (model) provenance. These categories served as a conceptual guide for the template design. Starting from these categories, we iteratively refined the Workflow Card template and manually designed a new question set of 33 questions spanning workflow (13), model (10), and data (10) aspects, released with our templates and reproduction code [36]. These questions examine which information is common across documentation artifacts and which information is specific to particular artifact types. During the design process, we established three principles for the Workflow Card. First, it should expose workflow-level provenance rather than replicate metadata already documented in Model or Data Cards. For the representative use case of foundation model fine-tuning, with an input Model Card, one or more Data Cards, and an output Model Card, this means information already held by those cards, such as a detail in the pretrained Model Card, need not be repeated in the Workflow Card. Second, it should represent an immutable execution, so each Workflow Card reflects one specific run and does not evolve after generation. Metadata unrelated to the run itself, such as administrative or policy information that may change independently, is therefore excluded. Third, like the model and Data Cards, it should offer a high-level overview with the smallest footprint possible while maintaining strong question-answering performance. I-B Benchmark I: Cross-Artifact Information Complementarity The first benchmark focused on evaluating the amount of provenance information exposed by Workflow Cards and measuring the overlap between Workflow Cards and existing documentation artifacts. While many different types of cards have been proposed in the literature, this study was restricted to machine learning workflows and focused exclusively on Model Cards and Data Cards, with one card per pipeline artifact: a pretrained Model Card, a fine-tuning Data Card, and a fine-tuned Model Card. To evaluate informational overlap across documentation artifacts, we constructed a set of simulated machine learning fine-tuning workflows using publicly available Hugging Face repositories, referenced in Table I. TABLE I: Overview of Benchmarked Use Cases, divided between Input Model Cards, Data Cards, and Output Model Cards Input Model Data Card Output Model meta-llama/ Llama-3.2- 3B-Instruct avaliev/ chat_doctor prithivMLmods/ Llama-Doctor-3.2- 3B-Instruct nvidia/ Nemotron-Cascade- 2-30B-A3B nvidia/ Nemotron-Cascade- 2-RL-data empero-ai/ openNemo-Cascade- 2-30B-A3B ibm-nasa- geospatial/ Prithvi-100M ANI00/ Crop-Health-Monitor ibm-nasa-geospatial/ Prithvi-100M-multi- temporal-crop- classification distilbert/ distilbert-base- uncased cybersectony/ PhishingEmail Detectionv2.0 cybersectony/ phishing-email- detection- distilbert_v2.4.1 meta-llama/ Llama-3.2-3B ai4privacy/ pii-masking-200k chinu-codes/ llama-3.2-3b- pii-redactor-lora For each workflow configuration, we collected the pretrained Model Card, associated Data Cards, and fine-tuned Model Card. Since public Workflow Cards do not yet exist, we simulated executions and generated Workflow Cards with the proposed template. Workflow-level fields were synthetically generated using Claude Sonnet 4.6, constrained by the real Model and Data Cards. Benchmark I is therefore a structural complementarity analysis: it tests whether the template can represent execution-level information absent from Model and Data Cards, not the factual correctness of synthetic fields. Benchmark I complements it with real provenance captured by Flowcept and yProv4ML (Figure 2). We evaluated provenance question answering under three different information settings: (i) Full Knowledge, in which all documentation artifacts were concatenated and provided simultaneously to the LLM. This configuration provides complete workflow-level context and is considered the top-level answer; (i) Single-card, in which each card was evaluated independently by providing only that documentation artifact; (i) Leave-one-out, where all cards except one were provided to the LLM. This configuration evaluates the informational contribution of the omitted artifact. In the context of this benchmark, information is defined in terms of the ability to answer a question. A documentation artifact is considered informative with respect to a question if an LLM can correctly answer that question using only the information contained within the artifact itself. The prompts instructed the LLMs to avoid inferring or hallucinating. The Full Knowledge configuration establishes the upper-bound informational context available for each workflow execution and was additionally double-checked, corrected, and improved with any missing information. This reference therefore represents the best answer obtainable from the full documentation set, curated and completed by the authors, and serves as the comparison target for the other configurations. Generated answers were evaluated by the two LLM judges. Each LLM rated the answer from 0.0 to 1.0 against the reference answer, where 1.0 denotes full consistency with the reference, 0.0 an incorrect or unsupported answer, and intermediate values partial correctness. Each LLM had its temperature set to 0.0 for the queries, and an additional statistical check was repeated five times to ensure that answer variability did not produce outliers. To additionally characterize the information carried by each card, we computed the pairwise cross-entropy across the documentation artifacts. I-C Benchmark I: Workflow Cards vs. Schema-based Provenance Querying Benchmark I evaluates whether Workflow Cards improve accessibility compared to schema-based querying of the underlying provenance data. It uses the Workflow Card template to describe a scientific machine learning pipeline for the DLESyM [11] climate forecasting model. The workflow spans four steps: (i) pre-processing raw data using the REDI [5] tool, (i) fine-tuning DLESyM, (i) forecasting, and (iv) evaluating forecasts via Empirical Orthogonal Function [20]. The workflow was instrumented using both Flowcept and yProv4ML to capture provenance across all steps. The card generator maps captured entities, activities, agents, task timestamps, resource records, and I/O metadata into the template blocks. Fine-grained task records are grouped by workflow stage, start and end times are reduced to activity durations, status values are summarized as counts, resource telemetry is aggregated at run and activity level, and selected inputs and outputs populate the Significant Workflow Artifacts block. From the benchmark question set, a 17-question subset targeting mainly workflow provenance was used, since questions outside that scope would have shifted the comparison away from information accessibility. We compared two representations of this same captured provenance, differing only in how it is presented to the LLM. The first is a lightweight schema-based querying approach [48], where the LLM is only given the schema and structural provenance rather than the raw records. Data, in this case, is stored into JSONL buffers, and retrieved from a MongoDB database. The second is a Workflow Card generated from the same provenance data and supplied directly as context. This comparison contrasts a pre-computed representation with on-demand retrieval: the Workflow Card packages the relevant provenance ahead of time as static context, whereas schema-based querying requires the model to navigate the interface and assemble the necessary provenance per question. Both representations were answered by the two models and scored under the same LLM-as-a-Judge protocol. A human expert evaluator independently assessed the Benchmark I answers as a calibration check for the automated judges. The human ratings are used to measure whether the LLM-as-a-Judge scores follow the same trend as manual assessment, while the LLM judges provide scalable scoring across the benchmark. IV Results The results are organized into three parts. Section IV-A presents the final template. Section IV-B reports Benchmark I, comparing the information carried by Dataset, Model, and Workflow Cards. Section IV-C reports Benchmark I, comparing Workflow Cards with schema-based provenance querying. IV-A Workflow Card Template Fig. 1: Template structure of a Workflow card, illustrating the key fields and layout used to represent workflow information. Available online [36]. The template was produced through an iterative process of drafting, testing on both benchmarks, and analyzing the results. The final Workflow Card template is shown in Figure 1 and is organized as a top-down summary that begins with a high-level view of the run and progressively descends into per-activity detail. It opens with a short Workflow block that names and describes the run in plain language, followed by a Summary block that records the identity and lifecycle of the execution, including a unique execution identifier, the workflow version, start and end timestamps, total duration, overall status, and the entrypoint repository, branch, and commit that pin the exact code used. An Infrastructure block then captures the computational environment, covering the operating system, hardware, runtime, resource manager, and a snapshot of the software dependencies, so that the run can be situated and, where possible, reproduced. The Workflow Overview gathers run-level aggregates such as activity and task counts, governing arguments, aggregate resource usage across CPU, memory, GPU, disk, and network, and a free-text Observations field for anomalies or decisions made mid-run, before the Activities block repeats a compact record for each stage that reports its tasks, timing, status, hosts, and summarized inputs and outputs. A final Significant Workflow Artifacts block lists only the inputs and outputs needed to reproduce or interpret the run, each identified by a short name and a resolvable reference. Every field is marked as Required or Optional, and a tilde denotes a value that was not captured, which keeps the card lightweight while signaling missing information rather than hiding it. Because each card describes a single immutable execution, it deliberately omits metadata that may evolve independently of the run, and it favors aggregate, high-level descriptions over exhaustive traces so that it stays small enough to read directly and to pass to an LLM as context. Fig. 2: Example Workflow card describing an inference step, performing weather forecasting based on an initial atmospheric condition. Available online [36]. IV-B Benchmark I The first benchmark quantifies the drop in performance when only a single card is provided (the Data Card, pretrained or fine-tuned Model Card, or Workflow Card), compared to the full set of all of them concatenated. The least informative card, the one with the largest average drop from the all-cards reference, is the pretrained Model Card. The Workflow Card, by contrast, matches the full set (Figure 3), reaching 0.619 against the all-cards reference of 0.611 while being, on average, 33.2 kB smaller, showing the same answer quality at a fraction of the context size, roughly 8k fewer tokens at about four characters per token. In these figures, the score is calculated as the average of the LLM-as-a-Judge ratings. Variability for these two initial charts is high, as each column encodes a large number of disparate questions, repeated for five different use cases. Fig. 3: Degradation of the mean of LLM as judge score across different single-card contexts, compared to the all-cards reference. Higher is better. Then, we tested the inverse: the drop when all cards but one are provided (Figure 4), e.g., leaving out one of the Model, Data, or Workflow cards compared to the reference set. Although higher scores generally indicate better performance, in this analysis, the focus is on the impact of removing individual artifacts. A larger decrease in score when an artifact is omitted suggests information that is not easily recovered from the remaining artifacts, making it more informative. Fig. 4: Leave-one-out performance analysis. Although higher scores indicate better performance, the key signal is the drop relative to the full setup. Removing the Data Card or pretrained Model Card has minimal impact, and removing the fine-tuned Model Card a moderate one, whereas excluding the Workflow Card causes the largest decrease, indicating it contains the most unique and relevant information for answering the evaluation questions. In this leave-one-out setting, removing the Workflow Card produces the largest drop-off, followed by the Fine-tuned Model Card, which contains information about the workflow’s final output. Quantitatively, the execution-level information unique to the Workflow Card more than doubles accuracy over the remaining cards: removing it drops the mean score from the all-cards reference of 0.6110.611 to 0.2650.265 (a 2.3×2.3× drop), by far the largest effect of any artifact; removing the Data Card or the Pretrained Model Card, by contrast, changes the score only marginally. To explain these differences, we computed the pairwise cross-entropy between cards (Figure 5). It accounts for the small effect of removing the Pretrained Model Card: much of its information is also captured by the Fine-tuned Model Card, but not the reverse, an asymmetry reflected in the cross-entropy between the two cards. Fig. 5: Matrix of mean cross-entropy scores across all use cases evaluated between different pairs of provenance card types. A lower score indicates less ”surprising” information. Finally, we quantified how well the answering models perform when given only one artifact type and asked questions targeting a specific card type (e.g., dataset-related questions given a Data Card). Table I reports this setting; its columns are the all-cards reference (All Cards) and the four single-card conditions: the Data Card, the fine-tuned Model Card (FT MC), the pretrained Model Card (PT MC), and the Workflow Card. Among the Data and Model Cards, the card matching a question’s focus performs best (e.g., the Data Card on data questions). However, the Workflow Card alone matches or exceeds the All-Cards reference on every question type (Data, Model, and Workflow), despite using less context. Two checks guarded the comparison. First, no artifact exceeded 25 kB, keeping card sizes comparable. Second, the models maintained their answer quality on questions a human rater judged difficult. The reported scores are means over the five use cases: single-card results are stable across them (standard deviations near 0.06, with the pretrained Model Card more variable at 0.13), while the leave-one-out conditions vary more (standard deviations up to 0.27), since the effect of removing a card depends on the specific workflow. TABLE I: Performance isolated per question focus, compared against the all-cards reference. Higher is better. Question Focus All Cards Only DataCard Only FT MC Only PT MC Only Workflow Data 0.6612 0.6154 0.4744 0.2736 0.6884 Model 0.6035 0.4190 0.5716 0.3488 0.6688 Workflow 0.3999 0.1172 0.2626 0.1088 0.4005 IV-C Benchmark I We first checked for large self-preference effects in the LLM-as-a-Judge ratings. Each model rated both systems’ responses on a 0.0 to 1.0 scale, and an independent human rating was included as a calibration reference. Figure 6 shows no clear self-preference pattern: nemotron-nano-3 rates its own answers lower than gpt-oss-120b’s, while gpt-oss-120b grades slightly more generously overall. The human ratings follow the same trend as the model judges, supporting the use of the LLM judges as a calibrated automated evaluation. Fig. 6: Comparison of mean ratings between gpt-oss-120b answers and nemotron-nano-3 answers across different judges and human baselines. Fig. 7: Overall mean ratings for gpt-oss-120b and nemotron-nano-3 responses under the two provenance representations (schema-based querying vs. Workflow Card), as evaluated by both models with the LLM-as-a-Judge framework and by human raters. We then compared question answering under schema-based querying and under Workflow Cards. On average, the card setting is noticeably better (Figure 7): across both answering models, moving from schema-based querying to Workflow Cards nearly doubles the mean rating (from ≈ 0.45 to ≈ 0.87), an improvement on which the LLM-as-a-Judge and human evaluations agree. The spread of ratings is similar for both. Agreement among judges is strongest when cards are the input. For the schema-based approach, the LLM judges score considerably harsher than the human rater, likely because raw schema-and-structure responses are harder for an LLM to interpret than a pre-generated card. V Discussion, Future Work, and Artifact Availability Our results show that pre-summarizing provenance into a card substantially eases LLM consumption compared to navigating raw schemas. Because Workflow Cards are themselves generated from the provenance captured by systems such as Flowcept, the two representations are complementary rather than competing: a provenance agent can emit a Workflow Card and use it as context, or consume one to answer queries. Integrating on-demand card generation into interactive provenance agents is a promising direction for future work. Our study has limitations. Benchmark I evaluates the informational structure of Workflow Cards on synthetically generated content rather than factual provenance, while Benchmark I complements it by validating on an instrumented HPC workflow. Benchmark I, in turn, covers a single workflow assessed by one human evaluator, and because the LLM judges also produced the answers they rate, the human ratings (Fig. 6) serve to validate the automated judgments. Quantitatively, the human ratings agree closely with the LLM judges (Pearson r=0.74r=0.74 for the mean of the two judges, and 0.660.66 and 0.770.77 individually; mean absolute difference 0.130.13 on the 0 to 1 scale). Finally, both benchmarks target machine-learning workflows: fine-tuning pipelines and one climate-ML workflow. The template is designed around execution concepts shared by many workflow systems, including run identity, infrastructure, activities, timing, status, parameters, and input/output artifacts. In non-ML domains such as genomics, computational fluid dynamics, or weather simulation, the Significant Workflow Artifacts block would document domain outputs such as FASTA, HDF5, Zarr, or NetCDF files rather than model weights. Validating these adaptations on non-ML scientific workflows remains future work. To support reproducibility, the Workflow Card template, benchmark question sets, LLM prompts, and the code to reproduce the benchmarks are publicly available [36]. Additionally, the updated versions of Flowcept [47] and yProv4ML [37] featuring Workflow Card generation support are open-source. VI Conclusion Understanding how artifacts were created is critical for identifying biases, preventing misuse, and improving overall explainability. We introduced Workflow Cards as interpretable documents of execution processes connecting datasets and models, filling a vital gap in current reporting standards. Our evaluation demonstrates that Workflow Cards capture unique, process-level provenance absent from existing artifacts and, on a real HPC workflow, nearly double LLM answer quality on provenance questions compared with schema-based querying. By making execution provenance directly readable to both people and LLMs, this work takes one more step toward the transparency and reproducibility of scientific workflows. Acknowledgment This work was partially funded under the National Recovery and Resilience Plan (NRRP), Mission 4 Component 2 Investment 1.4 - Call for tender No. 1031 of 17/06/2022 of Italian Ministry for University and Research funded by the European Union – NextGenerationEU (proj. nr. CN_00000013), and the RI-SCALE project (Grant Agreement 101188168). This research used resources of the Oak Ridge Leadership Computing Facility at the Oak Ridge National Laboratory, supported by the Office of Science of the U.S. Department of Energy under Contract No. DE-AC05-00OR22725. This material is based upon work supported by the U.S. Department of Energy (DOE), Office of Science, Office of Advanced Scientific Computing Research, under Contract DE-AC02-06CH11357. The authors used LLMs (Claude Sonnet 4.6 and GPT-5.5) to assist with grammar, sentence structure, and length control across sections. As described in the paper, an LLM was additionally used to generate synthetic metadata for the Benchmark I. These LLMs produced no scientific analyses, interpretations, or conclusions. References [1] M. Akhtar, O. Benjelloun, C. Conforti, L. Foschini, J. Giner-Miguelez, P. Gijsbers, S. Goswami, N. Jain, M. Karamousadakis, M. Kuchnik, S. Krishna, S. Lesage, Q. Lhoest, P. Marcenac, M. Maskey, P. Mattson, L. Oala, H. Oderinwale, P. Ruyssen, T. Santos, R. Shinde, E. Simperl, A. Suresh, G. Thomas, S. Tykhonov, J. Vanschoren, S. Varma, J. van der Velde, S. Vogler, C. Wu, and L. Zhang (2024) Croissant: a metadata format for ml-ready datasets. In Advances in Neural Information Processing Systems, Datasets and Benchmarks Track, Note: Spotlight Cited by: §I-B. [2] M. Arnold, R. K. E. Bellamy, M. Hind, S. Houde, S. Mehta, A. Mojsilović, R. Nair, K. Natesan Ramamurthy, A. Olteanu, D. Piorkowski, D. Reimer, J. Richards, J. Tsay, and K. R. Varshney (2019) FactSheets: increasing trust in AI services through supplier’s declarations of conformity. IBM Journal of Research and Development 63 (4/5), p. 6:1–6:13. External Links: Document Cited by: §I, §I-A. [3] T. Auge, G. Bali, M. Klettke, B. Ludäscher, W. Söldner, S. Weishäupl, and T. Wettig (2023) Provenance for lattice qcd workflows. In Companion Proceedings of the ACM Web Conference 2023, W ’23 Companion, New York, NY, USA, p. 1524–1530. External Links: ISBN 9781450394192, Document Cited by: §I, §I-A. [4] K. Belhajjame, H. Mezrioui, and Y. Zhao (2025) In-memory indexing and querying of provenance in data preparation pipelines. External Links: 2511.03480 Cited by: §I, §I-A. [5] W. Brewer, P. Widener, V. Anantharaj, F. Wang, T. Beck, A. Shankar, and S. Oral (2026) Data readiness pipeline patterns for scientific ai at scale: insights from climate, fusion, life sciences, and materials. AI Magazine 47 (1), p. e70056. External Links: Document Cited by: §I-C. [6] T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. M. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, and D. Amodei (2020) Language models are few-shot learners. In Advances in Neural Information Processing Systems (NeurIPS), Vol. 33, p. 1877–1901. Cited by: §I. [7] P. Buneman, S. Khanna, and W. C. Tan (2001) Why and where: a characterization of data provenance. In Proceedings of the 8th International Conference on Database Theory (ICDT), p. 316–330. External Links: Document Cited by: §I, §I-B. [8] L. J. Castro, D. Garijo, D. Rebholz-Schuhmann, D. Solanki, J. T. Ciuciu-Kiss, D. Katz, L. Eklund, G. Bharathy, and Research Data Alliance FAIR4ML Task Force (2024) FAIR4ML-schema. Note: Research Data Alliance External Links: Document, Link Cited by: §I-B. [9] A. Chebotko, E. De Hoyos, C. Gomez, A. Kashlev, X. Lian, and C. Reilly (2012) UTPB: a benchmark for scientific workflow provenance storage and querying systems. In 2012 IEEE Eighth World Congress on Services, Vol. , p. 17–24. External Links: Document Cited by: §I, §I-A. [10] J. Cheney, L. Chiticariu, and W. C. Tan (2009) Provenance in databases: why, how, and where. Foundations and Trends in Databases 1 (4), p. 379–474. External Links: Document Cited by: §I, §I-B. [11] N. Cresswell-Clay, B. Liu, D. R. Durran, Z. Liu, Z. I. Espinosa, R. A. Moreno, and M. Karlbauer (2025) A deep learning earth system model for efficient simulation of the observed climate. AGU Advances 6 (4), p. e2025AV001706. External Links: Document Cited by: §I-C. [12] A. Crisan, M. Drouhard, J. Vig, and N. Rajani (2022) Interactive model cards: a human-centered approach to model documentation. In Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency, p. 427–439. External Links: Document Cited by: §I-A. [13] E. Deelman, K. Vahi, G. Juve, M. Rynge, S. Callaghan, P. J. Maechling, R. Mayani, W. Chen, R. Ferreira da Silva, M. Livny, and K. Wenger (2015) Pegasus, a workflow management system for science automation. Future Generation Computer Systems 46, p. 17–35. External Links: Document Cited by: §I, §I-B. [14] J. Freire, D. Koop, E. Santos, and C. T. Silva (2008) Provenance for computational tasks: a survey. Computing in Science & Engineering 10 (3), p. 11–21. External Links: Document Cited by: §I-B. [15] T. Gebru, J. Morgenstern, B. Vecchione, J. W. Vaughan, H. Wallach, H. Daumé I, and K. Crawford (2021) Datasheets for datasets. Communications of the ACM 64 (12), p. 86–92. External Links: Document Cited by: §I, §I-A. [16] A. Gueroudji, A. Kamatar, S. Chaouche, R. Ross, K. Chard, and I. Foster (2026) Beyond centralized labs: federating the co-scientist. In Proceedings of the Supercomputing Asia and International Conference on High Performance Computing in Asia Pacific Region Workshops, SCA/HPCAsiaWS ’26, New York, NY, USA, p. 443–449. External Links: ISBN 9798400723285, Document Cited by: §I-B. [17] A. Gueroudji, T. Mallick, R. Souza, R. F. Da Silva, R. Ross, M. Dorier, P. Carns, K. Chard, and I. Foster (2025) ControlA: agentic workflow control mechanisms for reliable science. In 2025 IEEE International Conference on eScience (eScience), Vol. , p. 415–426. External Links: Document Cited by: §I-B. [18] A. Gueroudji, C. Phelps, T. Z. Islam, P. Carns, S. Snyder, M. Dorier, R. B. Ross, and L. C. Pouchard (2024) Performance characterization and provenance of distributed task-based workflows on hpc platforms. In SC24-W: Workshops of the International Conference for High Performance Computing, Networking, Storage and Analysis, Vol. , p. 2032–2039. External Links: Document Cited by: §I. [19] O. J. R. Gustafsson, S. R. Wilkinson, F. Bacall, S. Soiland-Reyes, S. Leo, L. Pireddu, S. Owen, N. Juty, J. M. Fernández, T. Brown, et al. (2025) WorkflowHub: a registry for computational workflows. Scientific Data 12 (1), p. 837. Cited by: §I-B. [20] A. Hannachi, I. T. Jolliffe, and D. B. Stephenson (2007) Empirical orthogonal functions and related techniques in atmospheric science: a review. International Journal of Climatology 27 (9), p. 1119–1152. External Links: Document Cited by: §I-C. [21] S. Leo, M. R. Crusoe, L. Rodríguez-Navas, R. Sirvent, A. Kanitz, P. De Geest, R. Wittner, L. Pireddu, D. Garijo, J. M. Fernández, I. Colonnelli, M. Gallo, T. Ohta, H. Suetake, S. Capella-Gutiérrez, R. de Wit, B. de Paula Kinoshita, and S. Soiland-Reyes (2024) Recording provenance of workflow runs with RO-Crate. PLOS ONE 19 (9), p. e0309210. External Links: Document Cited by: §I-B. [22] H. Li, Q. Dong, J. Chen, H. Su, Y. Zhou, Q. Ai, Z. Ye, and Y. Liu (2024) LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methods. External Links: 2412.05579 Cited by: §I. [23] W. Liang, N. Rajani, X. Yang, E. Ozoani, E. Wu, Y. Chen, D. S. Smith, and J. Zou (2024) What’s documented in AI? systematic analysis of 32K AI model cards. arXiv preprint arXiv:2402.05160. Cited by: §I-A. [24] Q. V. Liao, D. Gruen, and S. Miller (2020) Questioning the AI: informing design practices for explainable AI user experiences. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems, p. 1–15. External Links: Document Cited by: §I. [25] P. Missier, K. Belhajjame, and J. Cheney (2013) The W3C PROV family of specifications for modelling provenance metadata. In Proceedings of the 16th International Conference on Extending Database Technology (EDBT), p. 773–776. External Links: Document Cited by: §I, §I-B. [26] P. Missier, S. Woodman, H. Hiden, and P. Watson (2016) Provenance and data differencing for workflow reproducibility analysis. Concurrency and Computation: Practice and Experience 28 (4), p. 995–1015. External Links: Document Cited by: §I. [27] M. Mitchell, S. Wu, A. Zaldivar, P. Barnes, L. Vasserman, B. Hutchinson, E. Spitzer, I. D. Raji, and T. Gebru (2019) Model cards for model reporting. In Proceedings of the Conference on Fairness, Accountability, and Transparency (FAT*), p. 220–229. External Links: Document Cited by: §I, §I-A. [28] NVIDIA, A. Blakeman, A. Grattafiori, A. Basant, A. Gupta, A. Khattar, A. Renduchintala, A. Vavre, A. Shukla, A. Bercovich, et al. (2025) Nemotron 3 nano: open, efficient mixture-of-experts hybrid mamba-transformer model for agentic reasoning. External Links: 2512.20848 Cited by: §I. [29] T. Oinn, M. Addis, J. Ferris, D. Marvin, M. Senger, M. Greenwood, T. Carver, K. Glover, M. R. Pocock, A. Wipat, and P. Li (2004) Taverna: a tool for the composition and enactment of bioinformatics workflows. Bioinformatics 20 (17), p. 3045–3054. External Links: Document Cited by: §I, §I-B. [30] W. M. d. Oliveira, D. de Oliveira, and V. Braganholo (2018) Provenance analytics for workflow-based computational experiments: a survey. ACM Computing Surveys 51 (3), p. 53:1–53:25. External Links: Document Cited by: §I. [31] OpenAI, S. Agarwal, L. Ahmad, J. Ai, S. Altman, A. Applebaum, E. Arbus, R. K. Arora, Y. Bai, B. Baker, et al. (2025) Gpt-oss-120b & gpt-oss-20b model card. External Links: 2508.10925 Cited by: §I. [32] OpenAI (2024) GPT-4o system card. External Links: 2410.21276 Cited by: §I. [33] E. Ozoani, M. Gerchick, and M. Mitchell (2022) Model card guidebook. Hugging Face. External Links: Link Cited by: §I. [34] G. Padovani, V. Anantharaj, and S. Fiore (2025) yProv4ML: effortless provenance tracking for machine learning systems. SoftwareX 31, p. 102298. External Links: ISSN 2352-7110, Document Cited by: §I. [35] G. Padovani, V. Anantharaj, and S. Fiore (2025) Provenance tracking in large-scale machine learning systems. In Workshop Proceedings of the 54th International Conference on Parallel Processing, p. 167–174. Cited by: §I-B. [36] G. Padovani, N. G. Marchioro, R. Souza, et al. (2026) Workflow cards: template, examples, question sets, LLM prompts, and benchmark code. Note: https://github.com/data-cards/workflow-provenance-cardv0.0.1 Cited by: §I-A, Fig. 1, Fig. 2, §V. [37] G. Padovani et al. (2026) yProv4ML: a unified interface for logging and tracking provenance information in machine learning experiments. Note: https://github.com/ORNL/flowceptv3.0 Cited by: §I, §V. [38] J. F. Pimentel, J. Freire, L. Murta, and V. Braganholo (2019) A survey on collecting, managing, and analyzing provenance from scripts. ACM Computing Surveys 52 (3), p. 47:1–47:38. External Links: Document Cited by: §I, §I-B, §I-B. [39] D. Pina, L. Kunstmann, A. Chapman, D. de Oliveira, and M. Mattoso (2025) DLProv: a suite of provenance services for deep learning workflow analyses. PeerJ Computer Science 11, p. e2985. External Links: Document Cited by: §I, §I-A. [40] M. Pushkarna, A. Zaldivar, and O. Kjartansson (2022) Data cards: purposeful and transparent dataset documentation for responsible AI. In Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency (FAccT), p. 1776–1826. External Links: Document Cited by: §I, §I-A. [41] M. Schlegel and K. Sattler (2022) Management of machine learning lifecycle artifacts: a survey. ACM SIGMOD Record 51 (4), p. 18–35. External Links: Document Cited by: §I-B. [42] M. Schlegel and K. Sattler (2023) MLflow2PROV: extracting provenance from machine learning experiments. In Proceedings of the Seventh Workshop on Data Management for End-to-End Machine Learning (DEEM @ SIGMOD), External Links: Document Cited by: §I, §I-B. [43] M. Schlegel and K. Sattler (2025) Capturing end-to-end provenance for machine learning pipelines. Information Systems 132, p. 102495. External Links: ISSN 0306-4379, Document Cited by: §I, §I-A. [44] Y. L. Simmhan, B. Plale, and D. Gannon (2005) A survey of data provenance in e-science. ACM SIGMOD Record 34 (3), p. 31–36. External Links: Document Cited by: §I, §I, §I-B. [45] R. Souza, L. G. Azevedo, V. Lourenço, E. Soares, R. Thiago, R. Brandão, D. Civitarese, E. V. Brazil, M. Moreno, P. Valduriez, M. Mattoso, R. Cerqueira, and M. A. S. Netto (2022) Workflow provenance in the lifecycle of scientific machine learning. Concurrency and Computation: Practice and Experience 34 (14), p. e6544. External Links: Document Cited by: §I, §I, §I, §I-B, §I-B, §I-A. [46] R. Souza, A. Gueroudji, S. DeWitt, D. Rosendo, T. Ghosal, R. Ross, P. Balaprakash, and R. F. Da Silva (2025) PROV-AGENT: Unified provenance for tracking AI agent interactions in agentic workflows. In 2025 IEEE International Conference on eScience (eScience), p. 467–473. External Links: Document Cited by: §I, §I-B, §I-A. [47] R. Souza et al. (2026) Flowcept: a multiworkflow provenance integration and observability system. GitHub. Note: https://github.com/ORNL/flowceptv0.10.6 Cited by: §I, §V. [48] R. Souza, T. Poteet, B. Etz, D. Rosendo, A. Gueroudji, W. Shin, P. Balaprakash, and R. F. Da Silva (2025) LLM agents for interactive workflow provenance: reference architecture and evaluation methodology. In Workflows in Support of Large-Scale Science (WORKS) co-located with the ACM/IEEE International Conference for High Performance Computing, Networking, Storage, and Analysis (SC), St Louis, MO, USA, p. 2257–2268. External Links: Document Cited by: §I, §I, §I-A, §I-C. [49] R. Souza, T. J. Skluzacek, S. R. Wilkinson, M. Ziatdinov, and R. F. da Silva (2023) Towards lightweight data integration using multi-workflow provenance and data observability. In IEEE International Conference on e-Science, External Links: Document Cited by: §I, §I. [50] S. R. Wilkinson, M. Aloqalaa, K. Belhajjame, M. R. Crusoe, B. de Paula Kinoshita, L. Gadelha, D. Garijo, O. J. R. Gustafsson, N. Juty, S. Kanwal, F. Z. Khan, J. Köster, K. Peters-von Gehlen, L. Pouchard, R. K. Rannow, S. Soiland-Reyes, N. Soranzo, S. Sufi, Z. Sun, B. Vilne, M. A. Wouters, D. Yuen, and C. Goble (2025) Applying the FAIR principles to computational workflows. Scientific Data 12 (1), p. 328. External Links: Document Cited by: §I. [51] B. Yang, F. Cui, K. Paster, J. Ba, P. Vaezipoor, S. Pitis, and M. R. Zhang (2024) Report cards: qualitative evaluation of llms using natural language summaries. In Workshop on Socially Responsible Language Modelling Research (SoLaR), Note: Spotlight Cited by: §I-A. [52] X. Yang, W. Liang, and J. Zou (2024) Navigating dataset documentations in AI: a large-scale analysis of dataset cards on Hugging Face. arXiv preprint arXiv:2401.13822. Cited by: §I. [53] L. Zheng, W. Chiang, Y. Sheng, S. Zhuang, Z. Wu, Y. Zhuang, Z. Lin, Z. Li, D. Li, E. P. Xing, H. Zhang, J. E. Gonzalez, and I. Stoica (2023) Judging LLM-as-a-Judge with MT-Bench and chatbot arena. External Links: 2306.05685 Cited by: §I.