Paper deep dive

DataJoint 2.0: A Computational Substrate for Agentic Scientific Workflows

Dimitri Yatsenko, Thinh T. Nguyen

Year: 2026Venue: arXiv preprintArea: cs.DBType: PreprintEmbeddings: 49

Intelligence

Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 91%

Last extracted: 7/21/2026, 1:27:11 AM

Summary

The paper introduces DataJoint 2.0, a computational substrate for agentic scientific workflows (SciOps) that extends the relational workflow model. It addresses the fragmentation of provenance and lack of transactional guarantees in current scientific data management by unifying data structure, computational dependencies, and integrity constraints. Key innovations include Object-Augmented Schemas (OAS) for unified transactional control over relational and object storage, semantic matching to prevent erroneous joins, an extensible type system, and distributed job coordination.

Entities (12)

DataJoint 2.0 · software-system · 98%DataJoint Inc. · organization · 95%Dimitri Yatsenko · person · 95%Thinh T. Nguyen · person · 95%Object-Augmented Schema · feature · 94%Semantic Matching · feature · 93%Relational Workflow Model · concept · 92%SciOps · concept · 90%Airflow · software-system · 85%Databricks · software-system · 85%LangChain · software-system · 85%Nextflow · software-system · 85%

Relation Signals (10)

Dimitri Yatsenko → affiliatedwith → DataJoint Inc.

confidence 95% · Dimitri Yatsenko... — DataJoint Inc.

Thinh T. Nguyen → affiliatedwith → DataJoint Inc.

confidence 95% · Thinh T. Nguyen — DataJoint Inc.

DataJoint 2.0 → implements → Relational Workflow Model

confidence 95% · DataJoint 2.0 addresses this gap through the relational workflow model

DataJoint 2.0 → hasfeature → Object-Augmented Schema

confidence 94% · DataJoint 2.0 introduces four innovations... Object-Augmented Schema (OAS)

DataJoint 2.0 → hasfeature → Semantic Matching

confidence 93% · DataJoint 2.0 introduces... Semantic Matching: Lineage-based resolution

DataJoint 2.0 → supports → SciOps

confidence 92% · DataJoint creates a substrate for SciOps where agents can participate in scientific workflows

DataJoint 2.0 → comparesto → LangChain

confidence 85% · AI agent frameworks (LangChain)... provide no data integrity guarantees... DataJoint creates a substrate... without risking data corruption

DataJoint 2.0 → comparesto → Databricks

confidence 85% · DataJoint addresses a different part of the data lifecycle... Data lakehouse systems (Databricks...)

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:Operational rigor determines whether human-agent collaboration succeeds or fails. Scientific data pipelines need the equivalent of DevOps -- SciOps -- yet common approaches fragment provenance across disconnected systems without transactional guarantees. DataJoint 2.0 addresses this gap through the relational workflow model: tables represent workflow steps, rows represent artifacts, foreign keys prescribe execution order. The schema specifies not only what data exists but how it is derived -- a single formal system where data structure, computational dependencies, and integrity constraints are all queryable, enforceable, and machine-readable. Four technical innovations extend this foundation: object-augmented schemas integrating relational metadata with scalable object storage, semantic matching using attribute lineage to prevent erroneous joins, an extensible type system for domain-specific formats, and distributed job coordination designed for composability with external orchestration. By unifying data structure, data, and computational transformations, DataJoint creates a substrate for SciOps where agents can participate in scientific workflows without risking data corruption.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

47,294 characters extracted from source content.

Expand or collapse full text

DataJoint 2.0: A Computational Substrate for Agentic Scientific Workflows Dimitri Yatsenko, Thinh T. Nguyen — DataJoint Inc. February 2026 Abstract Operational rigor determines whether human-agent collaboration succeeds or fails. The most productive AI deployments operate within systems that enable safe iteration: version control, testing, branching, reproducibility. In software engineering, these practices constitute DevOps; scientific data pipelines need an equivalent operational transformation—SciOps 11 . Yet common approaches to scientific data management make poor substrates for agentic workflows: file-based systems offer flexibility but fragment provenance; task-centric orchestrators manage execution but remain agnostic to data structure; data lakehouses optimize analytical queries but treat computation as external. None provides the unified substrate that agentic scientific workflows require. DataJoint 2.0 addresses this gap through the relational workflow model: tables represent workflow steps, rows represent artifacts, foreign keys prescribe execution order. The schema specifies not only what data exists but how it is derived—a single formal system where data structure, computational dependencies, and integrity constraints are all queryable, enforceable, and machine-readable. Four technical innovations extend this foundation: (1) object-augmented schemas integrating relational metadata with scalable object storage, (2) semantic matching using attribute lineage to prevent erroneous joins, (3) an extensible type system for domain-specific formats, and (4) distributed job coordination designed for composability with external orchestration. By unifying data structure, data, and computational transformations in a single queryable framework, DataJoint creates a computational substrate for SciOps where agents can participate in scientific workflows without risking data corruption. Keywords: relational workflow model, agentic workflows, SciOps, scientific data pipelines, AI agents, data integrity, computational reproducibility, provenance tracking 1 Introduction The most successful AI agent deployments in software de- velopment share a common pattern: they operate within infrastructure that enforces operational rigor. Version control captures every change. Test-driven development provides continuous validation. Branching allows experi- mentation without risk to production. CI/CD pipelines automate quality gates. Together, these create a “ratch- eting” mechanism—progress is captured, mistakes are reversible, and humans steer while automation executes. This is DevOps; the key insight is that operational infras- tructure determines whether human-agent collaboration succeeds or fails. Scientific research is undergoing the same agentic trans- formation. Early AI co-scientist platforms 10,17 emphasize literature synthesis and work with experimental data in small samples supplied to them. But the frontier is shift- ing toward full integration with scientific pipelines: AI that analyzes results as they are acquired, steers down- stream computations, and evolves the analysis pipeline in response to emerging hypotheses—allowing scientists to interact with their entire study at a higher level of abstraction. This is SciOps 11 : not AI that reads about science, but AI that participates in the scientific workflow. A common belief holds that AI will help make sense of complex data with less need for structure, as long as sufficient metadata is supplied. But operational rigor becomes more critical for AI-driven workflows, not less. Most scientific teams today rely on file-based storage, workflow orchestrators, and ad-hoc scripts—flexible and accessible, but fragmenting provenance across discon- nected systems without transactional guarantees, safe experimentation on live pipelines, reversal after errors, or smooth evolution as collaborative teams grow. Rela- tional databases offer the rigor that agentic workflows demand: referential integrity, atomic transactions, declar- ative queries, machine-readable schemas. Yet they have found limited adoption in science, perceived as rigid, bottleneck-prone, and burdensome to administer. Con- ventional implementations do little to dispel these con- cerns: they cannot store large arrays efficiently, describe only what data exists rather than how it is derived, and disconnect workflows from the data they produce. DataJoint bridges this gap through the relational workflow model, which extends relational foundations to 1 arXiv:2602.16585v1 [cs.DB] 18 Feb 2026 represent not merely data structure but the complete scientific workflow (Section 2). The schema captures workflow specification. Referential integrity prevents corruption. Semantic matching catches erroneous oper- ations. Per-table job coordination enables distributed computation with full provenance. This paper presents DataJoint 2.0, which further extends these capabilities to meet the demands of agentic computation at scale. 1.1 Evolution of DataJoint. DataJoint was first re- leased as an open-source MATLAB toolbox in 2011, with the Python implementation following in 2014. The initial publication 22 introduced practical implementations for managing scientific data in collaborative settings. Sub- sequent work formalized the theoretical foundations: en- tity normalization and a five-operator query algebra 23 . DataJoint Elements 24 demonstrated modular workflow designs for neurophysiology—reusable schema modules (e.g., for electrophysiology, calcium imaging, pose estima- tion, behavioral sequencing) that laboratories customize and compose into complete pipelines, enabling consistent data architecture across independent research groups. 1.2 Contributions. This paper makes one conceptual and four technical contributions. Conceptual contribution. We articulate the re- lational workflow model as a distinct paradigm for un- derstanding relational databases—one that adds an op- erational dimension to classical data modeling. Tables represent workflow steps, rows represent artifacts, and foreign keys prescribe execution order. The schema spec- ifies not only what data exists but how it is derived (Section 2). Technical contributions. Building on this foun- dation, DataJoint 2.0 introduces four innovations for agentic computation: 1. Object-Augmented Schema (OAS): Unified transactional control over relational tuples and ob- ject storage (Section 4) 2.Semantic Matching: Lineage-based resolution of matching attributes in binary query operators, pre- venting erroneous joins on homonymous attributes (Section 5) 3. Extensible Type System: Pluggable codecs for domain-specific formats (Section 6) 4. Automated Job Management: Deterministic per-table distributed computation with provenance tracking (Section 7) 2 The Relational Workflow Model The relational data model 4 remains the most rigorous foundation for data management. Its mathematical grounding in predicate calculus and set theory provides precise semantics for data representation and manipula- tion. Referential integrity, normal forms, and declarative queries enable systems that are consistent, maintainable, and scalable. The model has historically been interpreted through two conceptual frameworks. Codd’s mathematical foun- dation views tables as logical predicates and rows as true propositions—mathematically rigorous but requir- ing abstract reasoning disconnected from domain think- ing. Chen’s Entity-Relationship Model 3 shifted focus to concrete domain modeling with entities, attributes, and relationships—more intuitive, but creating a gap between conceptual diagrams and SQL implementation, and cru- cially lacking any workflow or computational dimension. The relational workflow model 23 introduces a third paradigm: tables represent workflow steps, rows represent workflow artifacts, and foreign key dependencies prescribe execution order. This adds an operational dimension absent from both predecessors. 2.1 Core Principles. •Table tiers: Tables are classified by their data en- try mode. Manual tables receive direct user entry. Lookup tables hold reference data. Imported and Computed tables define computations viamake() methods. The distinction: Imported tables reach out to data sources outside the DataJoint system (instru- ments, electronic lab notebooks, external databases), while Computed tables derive their contents entirely from upstream DataJoint tables. • Declarative computation: Themake()method specifies how each entity is derived. This compu- tation logic is declared within the table definition, making it part of the schema itself rather than an external workflow specification. •Dependencies as foreign keys: Foreign keys de- fine computational dependencies, not only referen- tial integrity. The dependency graph is explicit, queryable, and enforced by the database. •Master-part relationships: A workflow step of- ten produces multiple related items that must be created or deleted atomically—for example, detected peaks within a spectrum, or trial events within a session. Master-part relationships declare this trans- actional grouping directly in the schema: the master table represents the workflow step, while part tables hold the individual items. Insertions and deletions cascade as a unit, enforcing transactional semantics without application code. • Entity normalization: All data is represented as well-formed entity sets with primary keys identifying each entity uniquely. This eliminates redundancy and ensures consistent updates. 2 AspectMathematical (Codd) ER (Chen) Relational Workflow Core QuestionFunctional dependencies Entity types When/how created Workflow Dimension Not addressedNot central Fundamental Implementation Gap HighHighNone Workflow SupportNoneNoneNative Table 1: Three paradigms for interpreting the relational model. 2.2 Active vs. Passive Schemas. The key distinction from classical models: traditional schemas are passive— containers for data produced by external processes. In the relational workflow model, the schema is active— Computed tables declare how their contents are de- rived, making the schema itself the workflow specifi- cation. Schemas are defined as Python classes, and entire pipelines are organized as self-contained code repositories—version-controlled, testable, and deployable using standard software engineering practices. A useful analogy: electronic spreadsheets unified data and computation in a way that now seems obvious. A spreadsheet naturally contains both cells with values and cells with formulas—it would be unnatural to separate them into distinct systems. Yet this same integration never penetrated relational databases in their 50+ years of history. Classical schemas describe data structure; computation lives elsewhere. The relational workflow model brings to databases what spreadsheets brought to tabular calculation: the recognition that data and the computations that produce it belong together. The analogy has limits: spreadsheets’ coupling of data and formulas is also the source of their well-known fragility— hidden dependencies, circular references, “spreadsheet hell.” DataJoint addresses this through formal schema constraints and explicit dependency declaration rather than ad-hoc cell references; the coupling is governed by relational integrity, not implicit in cell addresses. This unification has profound implications. Data man- agement (storage, integrity, queries) and workflow speci- fication (dependency resolution, provenance) become a single formal system. The schema captures the entire scientific process: from raw acquisition through derived analyses. Crucially, this enables a clean separation of con- cerns: scientists define what computations derive from what data (the workflow specification), while DevOps teams handle how computations execute (orchestration)— and can do so uniformly across pipelines without under- standing each pipeline’s particulars. The open-source framework expresses computational dependencies; orches- tration remains a separate operational concern. 2.3 The Workflow Normalization Principle. Database normalization decomposes data into tables to eliminate re- dundancy and ensure that updates propagate consistently. Classical normalization theory 12 achieves this through normal forms based on functional dependencies. Entity normalization 3 asks whether each attribute describes the entity identified by the primary key. Workflow nor- malization extends these principles with an operational dimension: Workflow Normalization Principle: Every table represents an entity type created at a spe- cific workflow step, and all attributes describe that entity as it exists at that step. A Session table contains attributes known when the session is entered (date, experimenter, subject). Analysis parameters determined later belong in Computed tables that depend on Session. This workflow discipline prevents the “kitchen sink” tables that plague scientific databases— tables that accumulate attributes from different workflow stages, obscuring provenance and complicating updates. 2.4 Query Algebra. DataJoint provides a five- operator algebra embedded in Python: RestrictFilter entities by attribute values or member- ship in other relations ProjectSelect and rename attributes, compute derived values Join Combine related entities across relations AggregateGroup entities and compute summary statis- tics Union Combine entity sets with compatible structure The algebra achieves algebraic closure: every operator produces a valid entity set with a well-defined primary key, enabling unlimited composition. For most operators, the result’s primary key equals the left operand’s primary key; Join is the exception, where the result’s primary key depends on functional dependencies between operands. This preservation of entity integrity—every query result is itself a proper entity set with clear identity—distinguishes DataJoint’s algebra from SQL, where query results lack both a well-defined primary key and a clear entity type: they are simply a “bag of rows,” not a proper entity set. Queries compose naturally as expressions. The host language’s abstraction mechanisms—functions, classes, modules—organize query logic. Scientists work with fa- miliar data structures; the framework handles translation to SQL. 3 2.5 Diagram Notation. DataJoint provides auto- generated diagrams that visualize data workflows—the directed flow of information through a pipeline (Figure 1). The notation encodes workflow structure at a glance: Table tierShape and color distinguish manual tables (green rectangles), lookup tables (gray), imported tables (blue ellipses), and computed tables (red el- lipses). Part tables appear as plain text attached to their master. Dependency typeSolid lines indicate identity inheri- tance: the child’s primary key contains the parent’s primary key. Dashed lines indicate references with- out identity inheritance. Directed layoutSource tables appear at the origin; derived results flow toward computed tables. The layout reveals the complete dependency graph and workflow progression. Dependencies between tables form a directed acyclic graph (DAG); aggregated dependencies between schemas likewise form a DAG. Unlike task DAGs in workflow managers (Nextflow, CWL, Airflow), these are relational schema DAGs—they define data structure and relation- ships, not just execution steps. This structure enables automated reasoning about execution order and parallel scheduling. The diagrams also highlight schema dimen- sions—entities that inject new primary key attributes (e.g.,Subject,Session,Trial), rendered with under- lined names in the diagram. These primary key sources cascade through dependent tables via identity inheritance. Dimensions define the “shape” of data in a pipeline: a ta- ble inheriting from bothSubjectandSessioncontains one row per subject-session combination. This dimension- ality is explicit in the schema and visible in the diagram, enabling agents to reason about data granularity without inspecting table contents. This notation contrasts with traditional ER diagrams: it shows workflow progression (not just static structure), encodes identity inheritance explicitly (not just cardinal- ity), and is auto-generated from code (always reflecting the actual schema). 3 Background and Related Work 3.1 Related Systems. SQL remains the universal inter- face to relational databases, but its limitations are well- known: verbose syntax, poor composability, error-prone semantics (NULL handling, implicit coercion, name-based matching), and results that are “bags” rather than proper relations. DataJoint retains SQL databases as the storage and transaction engine but provides a different interface: the five-operator algebra described in Section 2. Against this backdrop, several categories of systems ad- dress parts of the scientific data management problem. File-based workflow systems (Nextflow 8 , Snake- make 16 , Arvados 6 , Cromwell) focus on computational orchestration—scheduling containerized tasks that trans- form input files into output files. While these systems excel at managing how computations run, they treat data as opaque file collections without structural constraints. Metadata is limited to key-value tags; there are no for- eign keys, no referential integrity, and no way to query into files. Provenance tracks which files produced which files, but not attribute-level lineage. The Common Work- flow Language (CWL) provides portability across these systems but inherits their file-centric limitations. DataJoint offers concrete advantages over file-based workflows: queryable intermediate results (SQL access to any pipeline stage, not just final outputs), entity-level provenance (track which subject or session was affected, not just which workflow ran), incremental recomputation (re-run only affected entities when upstream data changes, not entire workflows), and AI agent integration (agents can introspect schema structure and query data directly, rather than parsing opaque files and logs). Decoupled architectures offer genuine advantages that explain their dominance in bioinformatics: independent evolution of data and computation layers (update anal- ysis code without touching the data model), toolchain flexibility (swap compute engines freely—Spark, Dask, GPU clusters—because any tool that reads files works), and natural alignment with organizational boundaries (data engineers, scientists, and DevOps evolve indepen- dently). CWL optimized for portability as its top priority and achieved it. DataJoint accepts tighter coupling as a deliberate trade-off: framework commitment in exchange for a single formal system where data structure, com- putational dependencies, and integrity constraints are all queryable, enforceable, and machine-readable. For agentic workflows where agents must reason about data structure, introspect dependencies, and operate safely, the decoupled approach cannot provide what the rela- tional workflow model offers. Importantly, choosing DataJoint does not preclude CWL interoperability. Schema-addressed storage orga- nizes objects in paths that mirror primary key structure (e.g.,subject/session/recording.nwb), matching the directory conventions CWL workflows expect. Migration from CWL to DataJoint can be largely AI-assisted; export from DataJoint back to CWL is straightfor- ward since the schema already encodes explicit structure. Hybrid operation—wheremake()methods invoke CWL tools or CWL workflows populate DataJoint tables— enables incremental adoption. There is no platform lock- in. General-purpose workflow orchestrators (Airflow, Prefect, Flyte, Dagster) provide task-centric execution management: scheduling DAGs of Python functions, man- aging retries and resources, and monitoring execution 4 subject session scan SubjectSample Instrument Session Acquisition Acquisition.Scan MassAnalysis AcquisitionMethod MassAnalysis.Spectrum PeakDetection PeakDetectionParams PeakDetection.Peak Figure 1: DataJoint diagram of a liquid chromatography–mass spectrometry (LC-MS) data processing pipeline. Green rectangles are manual tables, gray boxes are lookup tables, blue/red ellipses are imported/computed tables. Solid lines indicate identity inheritance; dashed lines indicate references. Part tables (e.g.,Acquisition.Scan) appear as plain text. The workflow flows left-to-right: biological samples→ instrument sessions→ scan acquisition→ spectral analysis→ peak detection. See github.com/datajoint/lcms-demo. state. These systems excel at the operational when and where of computation but are agnostic to data structure— tasks pass opaque artifacts, and the orchestrator tracks task-level provenance (which DAG ran when) rather than data-level provenance (which inputs produced which out- puts). The key distinction: orchestrators schedule tasks; DataJoint identifies which data entities need compu- tation based on schema state. DataJoint’s job tables, make()methods, and foreign key dependency graph map naturally onto orchestration primitives—queues, tasks, and DAGs—making integration with existing orchestra- tion systems straightforward. The open-source library intentionally leaves orchestration (scheduling, resource al- location, retry policies) to external systems; the managed platform integrates these capabilities directly. Data lakehouse systems (Databricks, Snowflake, Delta Lake, Iceberg) unify structured analytics with scal- able object storage using columnar formats optimized for distributed query processing. These systems excel at analytical throughput across massive datasets but treat computation as external to the data model—lineage is captured as metadata rather than enforced by schema constraints, and provenance tracks table-level dependen- cies rather than attribute-level derivation. DataJoint addresses a different part of the data lifecycle: govern- ing how data is derived rather than optimizing how it is queried at scale. In practice, DataJoint pipelines produce curated datasets in lakehouse-compatible for- mats (e.g., Delta tables via the extensible type system and schema-addressed storage on shared object storage), enabling downstream analytical consumption with full upstream provenance. Array databases (SciDB 1 , TileDB 18 ) optimize for array operations but have weak relational semantics and limited support for heterogeneous scientific metadata. Data versioning systems (DVC, lakeFS) provide ver- sion control for data but lack integrated query languages and computational coordination. AI agent frameworks (LangChain) enable tool use but provide no data integrity guarantees or transactional safety for database operations. 4 Object-Augmented Schema Model 4.1 The Scientific Data Challenge. Scientific datasets combine structured metadata (subjects, sessions, param- eters) with large arrays (images, time series, recordings). Relational databases excel at the former: enforcing rela- tionships, maintaining integrity, enabling precise queries. Object stores excel at the latter: efficient storage and access patterns for large binary data. The common workaround—storing file paths in database columns— fragments integrity across two systems: files can be or- phaned when records are deleted, references can point to missing objects, and no transactional guarantees span both systems. This is a fundamental, long-standing prob- lem in scientific data management. The object-augmented schema solves this by extending referential integrity to encompass both relational tuples and stored objects under unified transactional control. 4.2 Proper Representation. DataJoint 2.0 assigns data to the appropriate storage component based on its nature and access patterns: Relational Database Structural information, meta- data, and small values belong in the relational database, which enforces data integrity through ref- erential constraints and accelerates queries through indexing. Object StoreLarge scientific objects—arrays, images, recordings—reside in the object store, which affords efficient access patterns: memory mapping, chunking, lazy loading, and parallel access. Across both storage components, ACID transactional integrity is maintained: the relational database holds lightweight references that ensure atomicity and consis- tency of all operations. 5 Within the object store, two addressing schemes serve different needs: Hash-addressed storage names objects by content hash, enabling automatic deduplication across the entire database. This scheme is appropriate when the same bi- nary object may appear in multiple records—for example, shared configuration files, reference images, or replicated experimental stimuli. Schema-addressed storage organizes objects in paths that incorporate schema names, table names, and primary key values—enabling direct filesystem naviga- tion without database queries. This naming convention is compatible with file-based workflow systems (CWL, Nextflow, Snakemake), facilitating hybrid operation and migration. The scheme accommodates complex scientific objects potentially composed of multiple parts—chunked arrays (Zarr), hierarchical containers (HDF5), or multi- file datasets. 4.3 Unified Referential Integrity. All stored objects participate in referential integrity. Deleting a record cascades to dependent tables and removes associated objects. Garbage collection performs an exhaustive check of all references before deleting any object—ensuring that shared hash-addressed objects are removed only when truly unreferenced. Agents can delete data safely; integrity constraints prevent orphans. 4.4 Transactional Guarantees. Object operations are transactional: inserts write objects before committing tuples; deletes remove tuples before garbage collecting objects; failures trigger automatic rollback of partial operations. 5 Semantic Matching As schemas grow, attributes with identical names may have unrelated meanings. Name-based matching— used by all traditional databases—cannot detect when session_idin one table derives from a different source thansession_idin another, producing spurious joins that silently corrupt results. AI agents face acute risk: they cannot intuit semantic relationships from names alone. DataJoint 2.0 tracks attribute provenance through foreign key chains. In binary operators (join, restriction, aggregation), same-named attributes match only if they share a common ancestor in the lineage graph; otherwise the query raises an error requiring explicit resolution. Agents receive immediate feedback on query validity—no silent data corruption from semantic mismatches. 6 Extensible Type System 6.1 Three-Layer Architecture. Layer 1: Native Types (Discouraged) Raw SQL types (TEXT,TIME). Backend-specific, not portable. Allowed for legacy compatibility. Layer 2: Core Types (Recommended)Portable types:int64,float64,varchar(n),datetime, uuid,json. Consistent semantics across MySQL and PostgreSQL. Layer 3: Codec Types (Extensible) Programmable serialization with lazy loading and streaming support. Entry-point based plugin discovery enables domain-specific formats. 6.2 Lazy References. Large arrays return lazy refer- ences instead of loading data. Metadata (shape, dtype) is accessible without I/O; explicit materialization loads from storage. This enables agents to inspect data prop- erties before committing to expensive operations. 7 Automated Job Management 7.1 Deterministic Computation. Derived data requires computation. DataJoint’s job management system is fully deterministic and automated: given a schema with defined dependencies, the system identifies pending work, coordinates distributed workers, handles failures, and tracks provenance—all without requiring intelligent co- ordination. This determinism is precisely what makes the system reliable for agentic workflows: agents can trigger computations or introduce new data and pipeline branches, relying on the job system to execute derived computations correctly. 7.2 Job Coordination. Each computed table has an associated job table tracking reservation status, worker identity, and errors—isolating jobs per computation type with no contention between unrelated pipelines. Workers reserve jobs via optimisticINSERT(conflicts indicate an- other worker claimed the job); thepopulate()method releases connections during long computations, reacquir- ing only to insert results. 7.3 Architectural Boundary. This job management sys- tem provides workflow coordination—identifying pending work, reserving jobs, tracking status—but intentionally excludes workflow orchestration: scheduling policies, re- source allocation, retry strategies, alerting, and moni- toring dashboards. This separation is deliberate. Data- Joint specifies what needs to be computed and why (be- cause upstream data exists); orchestration systems handle when and where computations run. The job tables and populate()mechanism are designed to be composable with external orchestration—Airflow, Prefect, or custom schedulers can read job tables to discover pending work, invokepopulate()calls as tasks, and monitor status through DataJoint’s observable state. The managed platform (Section 9) integrates orchestration directly, re- moving the need for external orchestration infrastructure for teams that prefer a unified solution. 6 8 Suitability for Agentic Workflows Unlike conventional databases, which describe data struc- ture but not computation, and unlike workflow man- agers, which orchestrate computation but remain agnos- tic to data structure, DataJoint brings together data structure, data itself, and computational transformations into a single queryable framework. This unification is what makes the system a natural substrate for AI agents. Schemas are self-describing: agents can introspect table structure, dependencies, and data state programmatically. Integrity constraints make operations safe by default— invalid joins, type mismatches, and referential violations fail cleanly rather than corrupting data silently. The dependency graph is explicit, enabling agents to reason about execution order without implicit knowledge. Core operations are idempotent, so agents can retry on failure without side effects. And all state—job status, compu- tation progress, errors—is queryable, giving agents the observability they need to monitor and react. Together, these properties mean that agents can participate in sci- entific workflows with the same transactional guarantees that protect human-initiated operations. 9 Platform Architecture A complete DataJoint deployment integrates three core components around the open-source library (Figure 2): 9.1 Open-Source Core. The DataJoint Python li- brary 25 implements the relational workflow model: declarative table definitions as Python classes with make()methods specifying derivation logic, the five- operator query algebra for composable data access, job coordination viapopulate()with automatic dependency resolution, and storage abstraction via fsspec supporting local filesystem, S3, GCS, Azure Blob, and custom back- ends. A pipeline project is a standard Python package—a code repository containing schema definitions, computa- tion logic, and configuration—managed with the same version control, testing, and deployment practices as any software project. SQL backends include MySQL and PostgreSQL. Complete documentation is available on- line 7 . 9.2 Managed Platform. Production deployments at scale require infrastructure beyond the open-source li- brary: managed databases and object storage, compute orchestration across distributed workers, observability for pipeline health, integration with laboratory informa- tion management systems and electronic lab notebooks, and API endpoints enabling AI agents to operate on live pipelines. A managed platform, initially developed with funding from the National Institute of Neurologi- cal Disorders and Stroke (NINDS) and now backed by venture capital, addresses these operational requirements for teams across academia, biotech, pharma, and other industries. A growing catalog of existing pipelines pro- vides a fast path to deployment for common experimental modalities. The open-source library and the managed platform share the same pipeline code, allowing teams to choose the deployment model that fits their needs. 10 Evaluation DataJoint has supported neuroscience research contin- uously since 2011. Over 70 peer-reviewed publications describe data pipelines built with the framework, appear- ing in journals including Nature, Cell, Science, Nature Neuroscience, and Nature Communications. 1 The following projects in life science and precision medicine research illustrate the range of applications: MICrONS 15 (multimodal neurophysiology and connec- tomics of mouse visual cortex), Aeon 2 (continuous long- term behavioral observation), Spyglass 13 (reproducible hippocampal neuroscience), SCENE 19 (ecological neuro- science across species), Hussain Shuler Lab 20 (multimodal electrophysiology and behavioral tracking), ORION 14 (brain organoid characterization), PosePipe 5,9 (clinical markerless motion capture), UCSF Cadwell Lab (multi- modal cell-type connectivity), and Harvard Mouse Be- havior Core 21 (behavioral syllable extraction via MoSeq). This adoption across independent groups—from millisec- ond electrophysiology to months-long ethological studies to in vitro systems biology—validates the relational work- flow model as a practical foundation for scientific data management. 11 Discussion 11.1 Trade-offs. Additional machinery (lineage track- ing, per-table job tables) adds overhead but prevents error classes that are catastrophic at scale. Schema con- straints limit ad-hoc operations but ensure data quality over project lifetimes. Production deployments surface practical costs: worker lifecycle management for long- running computations (e.g., GPU model fitting exceeding one hour requires careful timeout configuration), object store tuning for large intermediate artifacts, and con- nection pool management during multi-hour distributed processing. These operational concerns are inherent to any system that enforces transactional guarantees across distributed computation, but they require expertise be- yond the schema definition itself. The relational workflow model requires learning a uni- form framework for data modeling and computation spec- ification. However, teams need to model data and estab- lish computational conventions regardless—DataJoint simply provides a consistent way to do so. The pay- off is immediate comprehension: researchers who know DataJoint can switch between projects and immediately 1 A curated list is maintained athttps://docs.datajoint.com/ about/publications/. 7 Figure 2: DataJoint platform architecture. The open-source Python library provides the relational workflow model—schema definition, query algebra, and distributed computation. This core integrates with a relational database (system of record), object storage (scalable data), and code repositories (version-controlled pipeline definitions). The managed platform adds infrastructure, observability, and orchestration for production deployments. understand how data is structured and computations or- ganized, without deciphering project-specific conventions. The division of labor becomes cleaner: researchers define the pipeline (both data structure and computational de- pendencies), while DevOps teams handle orchestration in a domain-agnostic manner using standard tools. Concerns about framework commitment are often over- stated. Data lives in standard SQL tables and object stores accessible by any tool. Well-structured pipelines keep analysis logic in independent libraries thatmake() methods invoke—these remain portable. What is Data- Joint-specific is the workflow specification: schema def- initions declaring computational dependencies. This parallels any workflow system (Nextflow workflows are Nextflow-specific), but DataJoint’s specification lives alongside the data it governs, enabling integrity guaran- tees and machine-readable semantics for agentic work- flows. An important architectural trade-off: the MySQL/- PostgreSQL backends are row-oriented, optimized for transactional workloads—inserting rows, reading rows, enforcing referential integrity. This is precisely what DataJoint’s workflow model requires. However, row- oriented storage is fundamentally not optimized for ana- lytical workloads: scanning millions of rows across a few columns, large-scale aggregations, or the columnar access patterns that make Spark and lakehouse systems fast. DataJoint’s query algebra (Section 2) is designed for workflow composition (“give me all sessions for subject X with these parameters”), not warehouse-scale analytics (“average firing rate across 50 million neurons grouped by brain region”). DataJoint is a pipeline governance and provenance system, not a big-data analytics engine. For large-scale analytics, derived datasets can be exported to lakehouse-compatible formats via the extensible type system and schema-addressed storage, enabling analyti- cal tools to consume governed data with full upstream provenance. 11.2 Future Directions: Agentic Data Science. The relational workflow model positions DataJoint as a natural substrate for agentic data science. The schema serves as a machine-readable specification that agents can introspect, reason about, and safely modify. Referential integrity and semantic matching prevent data corruption, while job tables and progress tracking provide observable state for monitoring and failure recovery. The managed platform is actively developing agen- tic capabilities: natural language interfaces for pipeline exploration, agent-assisted schema development, and au- tonomous pipeline monitoring. These operational fea- tures build on the conceptual foundations described in this paper. Looking forward, we see the opportunity for collabora- tive human-agent workflows: interfaces where scientists specify intent at a high level and agents handle imple- mentation details, with the schema serving as the shared contract between human understanding and machine ex- ecution. The goal is not to replace scientific judgment but to amplify it—letting researchers focus on experi- 8 mental design and interpretation while agents handle the engineering of data workflows. 12 Conclusion The relational workflow model offers a new way to under- stand relational databases—not merely as storage systems but as computational substrates. By interpreting tables as workflow steps and foreign keys as execution depen- dencies, the schema becomes a complete specification of how data is derived, not just what data exists. This con- ceptual shift, implicit in DataJoint since its inception, is here articulated as a distinct paradigm alongside Codd’s mathematical foundation and Chen’s entity-relationship framework. DataJoint 2.0 extends this foundation for agentic computation: integrating object storage with transac- tional guarantees, preventing semantic errors through lineage tracking, and enabling distributed computation with full provenance. By unifying data structure, data, and computational transformations in a single queryable framework, DataJoint offers a template for building systems that humans and machines can trust. A Key Terminology Relational Workflow ModelTables represent work- flow steps, rows represent artifacts, foreign keys pre- scribe execution order. Adds an operational dimen- sion to Codd’s mathematical foundation and Chen’s ER model. Workflow NormalizationEvery table represents an entity type created at a specific workflow step; all attributes describe that entity as it exists at that step. Semantic JoinMatches attributes by shared lineage through foreign key chains, not merely shared names. Prevents invalid joins on homonymous but unrelated attributes. Object-Augmented Schema (OAS)Integrates rela- tional tables with object storage under unified trans- actional control and referential integrity. Master-Part Relationship A workflow step that pro- duces multiple related items as a compositional unit. Insertions and deletions cascade atomically. Acknowledgments This work was supported by NIH grant U24 NS116470 (DataJoint Pipelines for Neurophysiology), NIH SBIR Direct-to-Phase I grant R44 NS129492 (DataJoint SciOps), and NIH grant R01 NS123849 (ORION). Data- Joint’s continued development is supported by seed fund- ing from Nina Capital, Inoca Capital Partners, and Cap- ital Factory. We thank DataJoint Inc. engineers and scientists Milagros Marín and Kushal Bakshi for their contributions, as well as open-source consultant Davis Bennett. We also thank the DataJoint open-source com- munity and the many neuroscience laboratories whose adoption and feedback have shaped the system’s evolu- tion. References [1] P. G. Brown. SciDB: A database management sys- tem for applications with complex analytics. In Computing in Science & Engineering, volume 12, pages 34–43. IEEE, 2010. [2]D. Campagner, J. Bhagat, G. Lopes, L. Calcaterra, A. G. Pouget, A. Almeida, T. T. Nguyen, C. H. Lo, T. Ryan, B. Cruz, F. J. Carvalho, Z. Li, A. Erskine, J. Rapela, O. Folsz, M. Marín, J. Ahn, S. Nier- wetberg, S. C. Lenzi, J. D. S. Reggiani, and SWC GCNU Experimental Neuroethology Group. Aeon: an open-source platform to study the neural basis of ethological behaviours over naturalistic timescales. bioRxiv, 2025. doi: 10.1101/2025.07.31.664513. Con- tinuous long-term behavioral monitoring platform using DataJoint. [3] P. P.-S. Chen. The entity-relationship model— toward a unified view of data. ACM Transactions on Database Systems, 1(1):9–36, 1976. [4]E. F. Codd. A relational model of data for large shared data banks. Communications of the ACM, 13(6):377–387, 1970. [5] R. J. Cotton, A. DeLillo, A. Cimorelli, K. Shah, J. Peiffer, S. Anarwala, K. Ab- dou, and T. Karakostas. Markerless motion capture and biomechanical analysis pipeline. arXiv preprint arXiv:2303.10654, 2023. doi: 10.48550/arXiv.2303.10654. PosePipe: markerless motion capture for clinical gait analysis. [6]Curii Corporation. Arvados: An open source plat- form for managing and analyzing biomedical big data.https://arvados.org, 2014. Open-source platform combining Keep (content-addressed stor- age) with Crunch (CWL workflow execution). [7]DataJoint. DataJoint documentation.https:// docs.datajoint.com, 2026. Official documentation for DataJoint 2.0. [8]P. Di Tommaso, M. Chatzou, E. W. Floden, P. P. Barja, E. Palumbo, and C. Notredame. Nextflow en- ables reproducible computational workflows. Nature biotechnology, 35(4):316–319, 2017. [9]S. Donahue, I. Djuraskovic, K. Shah, F. Sinz, R. Chafetz, and R. J. Cotton. Calibrated un- certainty for trustworthy clinical gait analysis us- 9 ing probabilistic multiview markerless motion cap- ture. arXiv preprint arXiv:2601.22412, 2026. doi: 10.48550/arXiv.2601.22412. Probabilistic extension with calibrated uncertainty quantification. [10]K. Huang, S. Zhang, H. Wang, Y. Qu, Y. Lu, Y. Roohani, R. Li, L. Qiu, J. Zhang, Y. Di, et al. Biomni: A general-purpose biomedical AI agent. bioRxiv, 2025. doi: 10.1101/2025.05.30.656746. General-purpose biomedical AI agent from Stanford. [11]E. C. Johnson, T. T. Nguyen, B. K. Dichter, F. Zap- pulla, M. Kosma, K. Gunalan, Y. O. Halchenko, S. Q. Neufeld, K. Ratan, N. J. Edwards, S. Ressl, S. R. Heilbronner, M. Schirner, P. Ritter, B. Wester, S. Ghosh, M. E. Martone, F. Pestilli, and D. Yat- senko. SciOps: Achieving productivity and reli- ability in data-intensive research. arXiv preprint arXiv:2401.00077, 2024. doi: 10.48550/arXiv.2401. 00077. SciOps framework: DevOps principles ap- plied to scientific research. [12]W. Kent. A simple guide to five normal forms in relational database theory. Communications of the ACM, 26(2):120–125, 1983. doi: 10.1145/358024. 358054. [13] K. H. Lee, E. L. Denovellis, R. Ly, J. Magland, J. Soules, A. E. Comrie, D. P. Gramling, J. A. Guidera, R. Nevers, P. Adenekan, et al. Spy- glass: a framework for reproducible and share- able neuroscience research. bioRxiv, 2024. doi: 10.1101/2024.01.25.577295. DataJoint-based frame- work for hippocampal neuroscience at UCSF. [14] M. Marín, J. Werth, H. Ullah, M. Cuevas, M. de Godoy, T. Nguyen, J. Kubanek, D. Yatsenko, and A. Shcheglovitov. ORION: An open-source pipeline for reproducible brain organoid generation and multimodal electrophysiological characteriza- tion. Manuscript in preparation, 2026. [15] MICrONS Consortium. Functional connectomics spanning multiple areas of mouse visual cortex. Na- ture, 2025. doi: 10.1038/s41586-025-08790-w. Func- tional neurophysiology component managed with DataJoint. [16]F. Mölder, K. P. Jablonski, B. Letcher, M. B. Hall, C. H. Tomkins-Tinch, V. Sochat, J. Forster, S. Lee, S. O. Twarber, A. Kanitz, et al. Sustainable data analysis with Snakemake. F1000Research, 10, 2021. [17]NovelSeek Team, B. Zhang, S. Feng, X. Yan, J. Yuan, Z. Yu, X. He, et al. NovelSeek: When agent be- comes the scientist—building closed-loop system from hypothesis to verification. arXiv preprint arXiv:2505.16938, 2025. doi: 10.48550/arXiv.2505. 16938. Unified closed-loop multi-agent framework for autonomous scientific research. [18]S. Papadopoulos, K. Datta, S. Madden, and T. Matt- son. TileDB: Array data management for massive genomics. In Proceedings of the VLDB Endowment, volume 10, pages 349–360, 2016. [19] Simons Foundation. Simons collaboration on eco- logical neuroscience (SCENE), 2025. URLhttps: //w.simonsfoundation.org/neuroscience/ simons-collaboration-on-ecological-neuroscience/ . Multi-institutional research initiative. [20]E. Sutlief, S. Zhang, K. Foresberg, R. Kaneko, and M. G. Hussain Shuler. Reward-reset interval timing drives patch foraging decisions through neural state transitions in dorsomedial striatum. bioRxiv, 2025. doi: 10.1101/2025.09.29.679309. Multimodal inte- gration of electrophysiology, fiber photometry, and behavior using DataJoint. [21]C. Weinreb, M. A. M. Osman, L. Kumar, and S. R. Datta. Keypoint-MoSeq: parsing behavior by linking point tracking to pose dynamics. Nature Methods, 21: 1329–1339, 2024. doi: 10.1038/s41592-024-02318-2. Unsupervised behavioral syllable extraction from pose tracking. [22]D. Yatsenko, J. Reimer, A. S. Ecker, E. Y. Walker, F. Sinz, P. Berens, A. Hoenselaar, R. J. Cotton, A. S. Siapas, and A. S. Tolias. DataJoint: managing big scientific data using MATLAB or Python. bioRxiv, page 031658, 2015. doi: 10.1101/031658. Original DataJoint toolbox introduction. [23]D. Yatsenko, E. Y. Walker, and A. S. Tolias. Data- Joint: A simpler relational data model. arXiv preprint arXiv:1807.11104, 2018. Theoretical foun- dations: entity normalization, five-operator algebra. [24]D. Yatsenko, T. Nguyen, S. Shen, K. Gunalan, C. A. Turner, R. Guzman, M. Sasaki, D. Sitonic, J. Reimer, E. Y. Walker, and A. S. Tolias. Data- Joint Elements: Data workflows for neurophysiol- ogy. bioRxiv, page 2021.03.30.437358, 2021. doi: 10.1101/2021.03.30.437358. Modular workflow com- ponents for neurophysiology. [25] D. Yatsenko et al. DataJoint for Python.https:// github.com/datajoint/datajoint-python , 2011– 2026. Open-source implementation. Apache 2.0 li- cense. 10

Instant research discovery