Paper deep dive

Scalable AI-assisted Workflow Management for Detector Design Optimization Using Distributed Computing

Derek Anderson, Amit Bashyal, Markus Diefenthaler, Cristiano Fanelli, Wen Guan, Tanja Horn, Alex Jentsch Meifeng Lin, Tadashi Maeno, Kei Nagai, Hemalata Nayak, Connor Pecar, Karthik Suresh, Fang-Ying Tsai, Anselm Vossen, Tianle Wang, Torre Wenaus

Year: 2026Venue: arXiv preprintArea: cs.DCType: PreprintEmbeddings: 22

Intelligence

Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 96%

Last extracted: 4/1/2026, 1:48:47 AM

Summary

The paper presents an AI-assisted framework for detector design optimization, integrating multi-objective Bayesian optimization with the PanDA and iDDS workflow management systems. By leveraging a 'Function-as-a-Task' paradigm, the framework enables scalable, automated, and distributed execution of computationally intensive simulations across heterogeneous resources, demonstrated through applications like the ePIC and dRICH detectors for the Electron-Ion Collider.

Entities (6)

iDDS · software-system · 100%PanDA · software-system · 100%Electron-Ion Collider · facility · 98%AID(2)E · project · 95%dRICH · detector · 95%ePIC · detector · 95%

Relation Signals (3)

PanDA → integrateswith → iDDS

confidence 100% · Coupled with its intelligent Distributed Dispatch and Scheduling (iDDS) component, PanDA supports AI/ML-driven workflows

AID(2)E → optimizes → ePIC

confidence 95% · We demonstrate the framework using benchmark problems and realistic studies of the ePIC and dRICH detectors

AID(2)E → utilizes → PanDA

confidence 95% · The AID(2)E framework is structured around three key components... PanDA-based runner integrates with the iDDS workflow orchestrator

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:The Production and Distributed Analysis (PanDA) system, originally developed for the ATLAS experiment at the CERN Large Hadron Collider (LHC), has evolved into a robust platform for orchestrating large-scale workflows across distributed computing resources. Coupled with its intelligent Distributed Dispatch and Scheduling (iDDS) component, PanDA supports AI/ML-driven workflows through a scalable and flexible workflow engine. We present an AI-assisted framework for detector design optimization that integrates multi-objective Bayesian optimization with the PanDA--iDDS workflow engine to coordinate iterative simulations across heterogeneous resources. The framework addresses the challenge of exploring high-dimensional parameter spaces inherent in modern detector design. We demonstrate the framework using benchmark problems and realistic studies of the ePIC and dRICH detectors for the Electron-Ion Collider (EIC). Results show improved automation, scalability, and efficiency in multi-objective optimization. This work establishes a flexible and extensible paradigm for AI-driven detector design and other computationally intensive scientific applications.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

21,512 characters extracted from source content.

Expand or collapse full text

Prepared for submission to JINST Scalable AI-assisted Workflow Management for Detector Design Optimization Using Distributed Computing D. Anderson 4 A. Bashyal 1 M. Diefenthaler 4 C. Fanelli 5 W. Guan 1 T. Horn 2 A. Jentsch 1 M. Lin 1 T. Maeno 1 K. Nagai 3 H. Nayak 5 C. Pecar 3 K. Suresh 5 F.Y. Tsai 6 A. Vossen 3,4 T. Wang 1 T. Wenaus 1 (AID(2)E collaboration) 1 Brookhaven National Laboratory, Upton, New York, USA 2 The Catholic University of America, Washington, DC, USA 3 Duke University, Durham, North Carolina, USA 4 Thomas Jefferson National Accelerator Facility, Newport News, Virginia, USA 5 William & Mary, Williamsburg, Virginia, USA 6 Stony Brook University, Stony Brook, New York, USA E-mail: wguan2@bnl.gov Abstract: The Production and Distributed Analysis (PanDA) system, originally developed for the ATLAS experiment at the CERN Large Hadron Collider, has evolved into a robust platform for orchestrating large-scale workflows across distributed computing resources. Coupled with its intelligent Distributed Dispatch and Scheduling (iDDS) component, PanDA now supports AI/ML- driven workflows through a scalable, flexible workflow engine. We present an AI-assisted framework for detector design optimization that integrates multi- objective Bayesian optimization with the PanDA–iDDS workflow engine to coordinate iterative simulations across heterogeneous resources. The framework addresses the challenge of exploring high-dimensional parameter spaces inherent in modern detector design. We demonstrate the framework using benchmark problems and realistic studies of the ePIC / dRICH detector for the Electron–Ion Collider. Results show improved automation, scalability, and efficiency in multi-objective optimization. This work establishes a flexible and extensible paradigm for AI-driven detector design and other computationally intensive scientific applications. Keywords: Artificial Intelligence, Distributed Computing, Detector Design, Electron Ion Collider, Workflow Management arXiv:2603.30014v1 [cs.DC] 31 Mar 2026 Contents 1 Introduction1 2 Distributed workflow orchestration2 3 Experiments4 4 Discussion and Future Work6 1 Introduction The Production and Distributed Analysis (PanDA) system [1, 2] is a mature workload management platform designed to handle large-scale data processing across heterogeneous and geographically distributed computing resources. By abstracting the underlying infrastructure, PanDA provides users with a unified interface for workload submission and management, enabling transparent and efficient utilization of distributed environments without requiring detailed knowledge of resource configurations. The intelligent Distributed Dispatch and Scheduling (iDDS) [3, 4] extends PanDA with ad- vanced workflow orchestration capabilities for complex and dynamic applications. iDDS supports Directed Acyclic Graph (DAG)-based workflows, conditional execution, iterative processing, and polymorphic workloads, allowing flexible expression and automation of sophisticated computa- tional pipelines. Together, PanDA and iDDS form a robust framework for distributed workflow management, widely adopted in production systems for large-scale data processing and analysis, as shown in Fig. 1. Notable applications include the ATLAS [5] experiment at the Large Hadron Collider [6] and the Vera C. Rubin Observatory [7, 8], where they support diverse workloads such as data carousels, hyperparameter optimization, active learning, and DAG-based data processing pipelines [9]. Detector design represents a particularly demanding application domain, characterized by high-dimensional parameter spaces, complex constraints, and competing optimization objectives. Performance metrics such as spatial and momentum resolution, particle identification efficiency, and geometric acceptance must be evaluated through computationally intensive simulations, making ex- haustive parameter exploration infeasible. Recent advances in machine learning, particularly multi- objective Bayesian optimization, offer powerful strategies for exploring these high-dimensional design spaces efficiently. However, integrating ML-driven optimization with distributed comput- ing infrastructures remains a significant challenge, especially when coordinating large numbers of simulation and reconstruction tasks. We present a scalable framework developed within the AI-assisted Detector Design for the Electron–Ion Collider (AID(2)E) project [10, 11]. The AID(2)E framework is structured around three key components: flexible detector configurations, AI-driven optimization, and adaptable – 1 – execution backends. The detector configuration layer is designed to be extensible, supporting a wide range of use cases—from specific systems such as dRICH and BIC to more general detector design problems. On top of this, AID(2)E explores multiple AI-driven optimization strategies, including multi-objective Bayesian optimization (MOBO) and multi-objective genetic optimization (MOGO), to efficiently navigate high-dimensional design spaces and balance competing objectives. To support diverse computational environments, the framework integrates multiple execution backends. A local joblib-based runner enables rapid prototyping on a single machine, while a SLURM-based runner supports scalable execution on clusters and high-performance computing systems. For large-scale, geographically distributed workloads, a PanDA-based runner integrates with the iDDS workflow orchestrator to coordinate execution across heterogeneous resources. This multi-runner design enables seamless portability of workflows across local, HPC, and distributed infrastructures. In this work, we focus on the PanDA-based execution model, which represents the most complex and scalable deployment scenario. We investigate the behavior of the PanDA scheduler and iDDS workflow orchestration under distributed conditions, including task concurrency, scheduling overhead, and system scalability. By coupling AI-driven optimization with distributed workflow management, the framework enables automated, end-to-end optimization across heterogeneous resources, providing a flexible and extensible paradigm for large-scale detector design and other computationally intensive scientific applications. Figure 1: An integrated workflow with PanDA and iDDS, where iDDS automates complex and dynamic workflows, and PanDA schedules workloads to large-scale distributed heterogeneous computing resources. 2 Distributed workflow orchestration In this section, we present an overview of the distributed workflow architecture underlying the AID(2)E framework. We begin by describing the system design that integrates AI-driven opti- mization with the PanDA/iDDS infrastructure. We then outline the role of PanDA in large-scale workload management and the Function-as-a-Task paradigm in iDDS for workflow orchestration. – 2 – Finally, we describe how these components are combined to enable scalable, AI-assisted detector design. Figure 2: AI integration with PanDA/iDDS. iDDS maps AI pipeline functions to remote tasks executed by PanDA across distributed resources and asynchronously aggregates results, enabling a workflow that behaves like local function execution. Architecture As the scope and complexity of scientific workflows continue to grow, managing intricate dependencies and execution logic becomes increasingly challenging. In particular, adapting AI pipelines to offload selected components as remotely executed tasks often requires significant restructuring, making the process cumbersome and error-prone. To address this challenge, iDDS introduces a Python-based abstraction that leverages decora- tors to transform local functions into distributed PanDA workloads. This approach enables users to define workflows and dependencies using familiar programming constructs, while seamlessly executing selected components across distributed resources with minimal modification to the orig- inal AI pipeline. In addition, iDDS asynchronously aggregates the resulting outputs, allowing computationally intensive workloads to be transparently offloaded to remote resources. This design provides a high-level abstraction that encapsulates distributed execution, enabling the workflow to exhibit semantics analogous to local function execution from the user’s perspective. The architecture combines three layers (Fig. 2): (1) an AIoptimization layer using Ax/BoTorch62 for candidate generation and surrogate model updates; (2) iDDS as workflow orchestrator,handling packaging, deployment, and parallel execution; and (3) PanDA as the workload manager across grid, cloud, HPC, and Kubernetes platforms. Distributed workload management PanDA provides a unified workload management framework for large-scale distributed computing. It offers a consistent interface through which users from different institutions can submit and manage jobs via HTTP-based services, supported by common – 3 – authentication mechanisms such as X.509 certificates or OpenID Connect [12]. PanDA integrates heterogeneous computing resources—including grid, cloud, Kubernetes, and high-performance computing systems—while abstracting site-specific differences in software stacks and schedulers such as Slurm, HTCondor, and PBS. This abstraction significantly reduces user complexity, enabling seamless execution across geographically distributed resources. Proven at scale in the ATLAS experiment at the Large Hadron Collider, where it serves thousands of users across more than 170 sites, and in the Rubin Observatory, PanDA has demonstrated robust performance and scalability, and is now being extended to support emerging use cases at the Electron–Ion Collider. Workflow orchestration iDDS employs the Function-as-a-Task paradigm to orchestrate work- flows, comprising three main steps. First, user source code and execution context are packaged and uploaded to a cache service, from which a function wrapper initializes the runtime environment on remote resources; this environment may be container-based, Conda-based [13], or a standard Python installation. Second, decorated functions and their parameters are serialized into PanDA jobs, which are scheduled and executed on distributed resources, where the wrapper reconstructs and runs the original functions. Third, outputs are asynchronously returned to the caller, enabling non-blocking execution and iterative processing. Result retrieval supports both STOMP [14]- and HTTP REST [15]-based communication, with STOMP via ActiveMQ [16] as the default for efficient transfer and REST as a fallback for robustness across environments. This paradigm allows transparent execution of user-defined functions on distributed resources through PanDA. Leveraging existing infrastructure, it requires minimal configuration while achiev- ing high scalability. Asynchronous, publish–subscribe-based result handling further decouples task execution from workflow control, making the approach well suited for large-scale, AI-driven optimization workflows. AID(2)E workflow The AID(2)E framework addresses the limitations of traditional detector design methods, which often rely on subsystem-level tuning or brute-force approaches like grid search. It implements multi-objective optimization to efficiently explore competing goals—such as performance, resolution, and cost—while uncovering complex correlations in high-dimensional parameter spaces. By integrating machine learning with realistic simulation pipelines, AID(2)E enables AI-driven, system-level optimization of the full detector. The workflow leverages PanDA and iDDS to distribute computationally intensive simulation and reconstruction tasks across heterogeneous resources. Results are returned asynchronously to guide iterative, adaptive optimization, allowing scalable and automated exploration of complex detector designs. This combination of AI and distributed computing ensures efficient resource utilization while controlling computational cost. 3 Experiments The AID(2)E project employs a sequence of closure tests to validate both the optimization method- ology and the distributed workflow infrastructure. Closure test 1 verifies the convergence of multi-objective optimization algorithms using benchmark problems with known Pareto fronts, es- – 4 – (a) Baseline AID(2)E workflow(b) AID(2)E with Function-as-a-Task Figure 3: AID(2)E workflow. (a) The AI-driven framework proposes detector design parameters for multiple objectives, which are evaluated through simulation. (b) With the Function-as-a-Task paradigm, local functions are transformed into PanDA jobs and executed on distributed resources, enabling scalable and transparent workflow execution. High-level Workflow Figure 4: High-level workflow of AID(2)E. Closure test 1 validates AI-assisted optimization using benchmark problems with known Pareto fronts. Closure test 2 evaluates distributed execution across heterogeneous resources using PanDA/iDDS. In the full integration stage, benchmark functions are replaced with compute-intensive ePIC simulation and reconstruction tasks. tablishing a baseline for performance. This work focuses on closure test 2, which evaluates the scalability and efficiency of distributed workflow execution. DTLZ2 benchmark The DTLZ2 [11] problem is used in closure test 2 to evaluate the perfor- mance of the PanDA/iDDS system on distributed resources. Benchmarks were conducted across configurations with varying numbers of objectives and design parameters; here, we present a representative case with five objectives and 100 parameters. This benchmark provides a controlled environment for validating distributed workflow orches- – 5 – (a) Hypervolume vs. trials(b) Hypervolume vs. time Figure 5: DTLZ2 benchmark results for 5 objectives and 100 parameters. Distributed execution achieves similar convergence while enabling higher concurrency. Runtime is dominated by opti- mization overhead rather than objective evaluation. tration prior to full detector simulations. As shown in Fig. 5, comparable optimization perfor- mance is observed between local and distributed execution in terms of hypervolume convergence. Distributed execution significantly increases concurrency, while overall runtime is dominated by optimizer overhead associated with trial generation rather than objective evaluation. These results demonstrate that the PanDA/iDDS framework can scale effectively without degrading optimization performance. dRICH detector optimization To demonstrate applicability to realistic detector design, the work- flow is applied to the dual-radiator Ring Imaging Cherenkov (dRICH) detector in the ePIC exper- iment as outlined in [11]. In this setup, multi-objective Bayesian optimization is coupled with PanDA/iDDS to orchestrate detector simulation, objective evaluation, and result aggregation across distributed resources. The optimization targets key physics performance metrics, including pion–kaon and kaon–proton separation and detector acceptance over a broad momentum range. The design space consists of seven parameters, including aerogel geometry, mirror configuration, and photosensor placement, subject to geometric constraints and overlap checks. As shown in Fig. 6, the hypervolume increases monotonically with the number of trials, in- dicating continuous improvement of detector configurations. The distributed workflow enables efficient scaling across resources, improving throughput and automation while maintaining robust- ness through asynchronous execution. These results demonstrate that the AID(2)E framework effectively integrates AI-driven opti- mization with distributed computing infrastructure, enabling scalable and automated detector design for realistic experimental systems. 4 Discussion and Future Work Performance evaluation demonstrates that the PanDA/iDDS framework scales effectively with con- current execution, while also highlighting overheads from trial generation, scheduling latency, and – 6 – Figure 6: dRICH multi-objective optimization. Hypervolume as a function of optimization trials for the distributed workflow, demonstrating progressive improvement of detector configurations. resource contention in distributed environments. Asynchronous execution and parallel evaluation substantially improve efficiency, particularly for computationally intensive simulations, making the framework most advantageous when evaluation costs dominate optimizer overhead—typical of realistic detector-design studies. The integration of AI-driven optimization with distributed computing enables efficient explo- ration of high-dimensional parameter spaces, reduces time-to-solution, and enhances automation, reproducibility, and overall workflow robustness. By coupling multi-objective optimization with scalable workflow orchestration, the framework provides a practical solution for end-to-end opti- mization of complex detector systems. Future work will focus on extending the framework to support larger machine learning mod- els, additional detector components, and leveraging large language models for advanced workflow automation and decision support. Furthermore, ongoing developments aim to generalize the ap- proach for broader applications and to extend to the scheduling to SLURM based runners, enabling AI-assisted optimization across other detectors and experiments at the Electron–Ion Collider. This scalable, distributed, and AI-driven methodology promises to accelerate detector design cycles, uncover optimal trade-offs among competing objectives, and establish a foundation for future large- scale nuclear and particle physics experiments. Acknowledgments C.F., K.S. and H.N. were supported by the Office of Nuclear Physics of the U.S. Department of Energy under Grant Contract No. DE-SC0024625. C.P. and K.N. were supported by the Office of Nuclear Physics of the U.S. Department of Energy under Grant Contract No. DE-SC0024478. – 7 – M.D. was supported by the U.S. Department of Energy Office of Science, Office of Nuclear Physics contract number DE-AC05-06OR23177, under which Jefferson Science Associates, LLC operates Jefferson Lab. T.H. was supported by DOE U.S. Department of Energy Office of Science, Office of Nuclear Physics contract number DE-SC-0024691. This work was supported by the U.S. Department of Energy, Office of Science, under Brookhaven Science Associates (BSA) contract number DE-SC0012704. References [1] Tadashi Maeno, Aleksandr Alekseev, Fernando Harald Barreiro Megino, Kaushik De, Wen Guan, Edward Karavakis, Alexei Klimentov, Tatiana Korchuganova, FaHui Lin, Paul Nilsson, Torre Wenaus, Zhaoyu Yang, and Xin Zhao. Panda: Production and distributed analysis system. Computing and Software for Big Science, 8(1):4, 2024. ISSN 2510-2044. doi: 10.1007/s41781-024-00114-3. URL https://doi.org/10.1007/s41781-024-00114-3. [2] T. Maeno et al. Overview of ATLAS PanDA Workload Management. J.Phys.: Conf. Ser., 331: 072024, 2011. doi: 10.1088/1742-6596/331/7/072024. [3] Wen Guan, Tadashi Maeno, Aleksandr Alekseev, Fernando Harald Barreiro Megino, Kaushik De, Edward Karavakis, Alexei Klimentov, Tatiana Korchuganova, FaHui Lin, Paul Nilsson, Torre Wenaus, Zhaoyu Yang, and Xin Zhao. idds: intelligent distributed dispatch and scheduling for workflow orchestration. The European Physical Journal C, 86(1):66, Jan 2026. ISSN 1434-6052. doi: 10.1140/epjc/s10052-025-15275-7. URL https://doi.org/10.1140/epjc/s10052-025-15275-7. [4] W. Guan et al. An intelligent Data Delivery Service for and beyond the ATLAS experiment. EPJ Web Conf., 251:02007, 2021. doi: 10.1051/epjconf/202125102007. [5] ATLAS Collaboration. The ATLAS Experiment at the CERN Large Hadron Collider. J. Inst., 3: S08003, 2008. doi: 10.1088/1748-0221/3/08/S08003. [6] L. Evans and P. Bryant (editors). LHC Machine. J. Inst., 3:S08001, 2008. doi: 10.1088/1748-0221/3/08/S08001. [7] Z. Ivezic et al. LSST: From Science Drivers To Reference Design And Anticipated Data Products. Astrophys. J.,, 873(2):111, 2019. doi: 10.3847/1538-4357/ab042c. [8] LSST Science Collaboration. Lsst science book, version 2.0. 2009. doi: 10.48550/arXiv.0912.0201. [9] W. Guan et al. Distributed Machine Learning Workflow with PanDA and iDDS in LHC ATLAS. EPJ Web Conf., 295:04019, 2024. doi: 10.1051/epjconf/202429504019. [10] C. Fanelli et al. Ai-assisted optimization of the ecce tracking system at the electron ion collider. Nuclear Instruments and Methods in Physics Research Section A: Accelerators, Spectrometers, Detectors and Associated Equipment, 1047:167748, 2023. ISSN 0168-9002. doi: https://doi.org/10.1016/j.nima.2022.167748. URL https://w.sciencedirect.com/science/article/pii/S0168900222010403. [11] M. Diefenthaler, C. Fanelli, L.O. Gerlach, W. Guan, T. Horn, A. Jentsch, M. Lin, K. Nagai, H. Nayak, C. Pecar, K. Suresh, A. Vossen, T. Wang, T. Wenaus, and on behalf of the AID(2)E collaboration. Ai-assisted detector design for the eic (aid(2)e). Journal of Instrumentation, 19(07):C07001, jul 2024. doi: 10.1088/1748-0221/19/07/C07001. URL https://doi.org/10.1088/1748-0221/19/07/C07001. – 8 – [12] Openid connect (oidc). https://openid.net/connect/. [13] Conda package manager. https://docs.conda.io/en/latest/. [14] Simple Text Oriented Messaging Protocol. https://stomp.github.io/. [15] Representational State Transfer. https://fr.wikipedia.org/wiki/Representational_state_transfer. [16] Apache ActiveMQ: Flexible and powerful open source multi-protocol messaging. https://activemq.apache.org/. – 9 –

Instant research discovery