Paper deep dive

Toward Controlling Biology with Language:Offline Learning of Prompt-Conditioned Interventions for Cells, Organoids, and Biobots

Nam H. Le, Douglas Blackiston, Michael Levin, Josh Bongard

Year: 2026Venue: arXiv preprintArea: q-bio.QMType: PreprintEmbeddings: 48

Intelligence

Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 94%

Last extracted: 10/5/2026, 5:13:12 AM

Summary

This paper presents an offline learning framework to map natural language instructions to biological interventions for xenobots (synthetic multicellular organisms). By treating existing experimental archives as fixed datasets, the authors use a Vision-Language Model (VLM) as a zero-shot reward judge to train a Prompt-to-Intervention (P2I) network without new wet-lab experiments. The model achieves 80.0% held-out accuracy in predicting interventions for new instructions, demonstrating that VLM-derived rewards can effectively substitute for ground-truth labels in controlling living systems.

Entities (6)

Xenobot · biological-system · 98%Prompt-to-Intervention (P2I) Network · machine-learning-model · 97%Vision-Language Model (VLM) · ai-model · 96%Offline Learning · methodology · 95%SentenceBERT · algorithm · 92%Segment Anything Model (SAM) · algorithm · 90%

Relation Signals (5)

Prompt-to-Intervention (P2I) Network → controls → Xenobot

confidence 96% · Toward Controlling Biology with Language... mapping a natural-language description of desired behavior to the physical intervention that elicits it

Prompt-to-Intervention (P2I) Network → uses → SentenceBERT

confidence 95% · We embed prompts with SentenceBERT... The P2I policy network... takes this embedding as input

Vision-Language Model (VLM) → providesrewardfor → Prompt-to-Intervention (P2I) Network

confidence 94% · using a vision-language model’s own judgment as the sole training reward

Segment Anything Model (SAM) → usedfortracking → Xenobot

confidence 93% · We track position per-frame using the Segment Anything Model (SAM)... to reject false positives.

Offline Learning → enables → Prompt-to-Intervention (P2I) Network

confidence 92% · Here we show that a natural-language interface... can be learned entirely offline this way

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:Artificial intelligence increasingly serves as a natural-language interface to complex technical systems, letting people accomplish sophisticated tasks by describing what they want rather than specifying how to do it. Extending this interface to living systems is harder: unlike code or images, a biological intervention has no closed-form linguistic meaning, and the paired language-intervention-outcome data needed to learn such a mapping is expensive to collect, since each example requires its own wet-lab experiment. One way around this is to treat an existing archive of interventions and their already-observed outcomes as a fixed, offline dataset, and use a vision-language model to judge, without any new experiments, whether an archived outcome matches a natural-language description. But whether that judgment is reliable enough to train a language-to-intervention mapping on -- without new experiments and without human validation -- has remained unclear. Here we show that a natural-language interface for a living organism -- a xenobot, a synthetic multicellular construct with no nervous system -- can be learned entirely offline this way, using a vision-language model's own judgment as the sole training reward: an instruction is mapped to the intervention already on record as producing the described behavior. This mapping generalizes to entirely new instructions, evaluated against archive data withheld from training (80.0% held-out accuracy vs a $66.7% chance baseline, matching a network trained directly on ground-truth labels).

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

45,042 characters extracted from source content.

Expand or collapse full text

Toward Controlling Biology with Language: Offline Learning of Prompt-Conditioned Interventions for Cells, Organoids, and Biobots Nam H. Le Affiliation: University of Vermont Email: namlehai90@gmail.com Douglas Blackiston Michael Levin Affiliation: Tufts University, Allen Discovery Center Josh Bongard Affiliation: University of Vermont Abstract Artificial intelligence increasingly serves as a natural-language interface to complex technical systems, letting people accomplish sophisticated tasks by describing what they want rather than specifying how to do it. Extending this interface to living systems is harder: unlike code or images, a biological intervention has no closed-form linguistic meaning, and the paired language-intervention-outcome data needed to learn such a mapping is expensive to collect, since each example requires its own wet-lab experiment. One way around this is to treat an existing archive of interventions and their already-observed outcomes as a fixed, offline dataset, and use a vision-language model to judge, without any new experiments, whether an archived outcome matches a natural-language description. But whether that judgment is reliable enough to train a language-to-intervention mapping on – without new experiments and without human validation – has remained unclear. Here we show that a natural-language interface for a living organism – a xenobot, a synthetic multicellular construct with no nervous system – can be learned entirely offline this way, using a vision-language model’s own judgment as the sole training reward: an instruction is mapped to the intervention already on record as producing the described behavior. This mapping generalizes to entirely new instructions, evaluated against archive data withheld from training (80.0%80.0\% held-out accuracy vs. a 66.7%66.7\% chance baseline, matching a network trained directly on ground-truth labels). These results indicate that offline archives, combined with automated judgment, can substitute for costly new experiments when learning to ground natural language in a living system’s own recorded behavior – an intermediate step toward directing living matter with the same ease as code, though whether these predicted interventions actually reproduce their intended behavior on live, newly-stimulated cells remains to be tested. 1 Introduction Imagine a clinician or bioengineer simply describing what they want – calm this inflamed tissue, direct these cells to close a wound, steer this synthetic biobot toward a target – and having cells, organoids, or synthetic biobots (Kriegman et al., 2021; Blackiston et al., 2021b; Levin, 2021) carry out that intention directly, the way a modern software system already responds to a natural-language request. Controlling living systems this way is a long-standing goal across biology, medicine, and synthetic biology, yet the interface remains fundamentally low-level: practitioners specify interventions parameter by parameter, through trial and error informed by domain expertise. In other technical domains, natural language has emerged as a high-level interface to complex systems: users specify a goal – generate this code (Fried et al., 2023), design this molecule (Bengio et al., 2021), plan this trajectory (Huang et al., 2022) – and a learned model translates intent into low-level action. This raises a question central to applied machine learning (ML): can a similar interface be learned for biological systems, mapping a natural-language description of desired behavior to the physical intervention that elicits it? Two properties make this hard. First, collecting paired (prompt, intervention, outcome) data online is prohibitively expensive: each datum is a wet-lab experiment, ruling out approaches that need a fresh experiment per prompt or per objective. Second, the objective space is inherently multi-task and contains opposing goals – e.g. “stop” versus “accelerate” – and a single policy trained across such goals is known to suffer objective interference, where the gradient for one goal actively degrades another (Yu et al., 2020; Chen et al., 2018), so naively pooling all objectives into one unconstrained shared model is not a safe default either. Together, these rule out the two most direct approaches: brute-forcing the mapping with enough paired data, and training one policy across all objectives without accounting for how they interact. We address both properties by treating language-to-intervention as an offline learning problem. Archives of intervention-outcome pairs already exist, collected for purposes other than language-conditioned learning; we treat such an archive as a fixed dataset over which a policy is trained without any new experiments. To supply a reward signal in the absence of labeled prompt-outcome alignment, we use a pretrained VLM as a zero-shot scorer (Rocamonde et al., 2023), judging how well an archived outcome matches a natural-language description, and check that judgment against independent, non-VLM ground truth rather than trust it by assumption. The policy uses one output head per behavioral category over a shared backbone. Figure 1: Overview of the pipeline. (a) Tracked trajectory of a xenobot before stimulation (PRE), with a zoomed view of the construct. (b) The same trajectory, marking the moment the electrical stimulus is applied. (c) The trajectory continuing after stimulation (POST), showing the resulting change in behavior. (d) An example natural-language prompt. (e) The Prompt-to-Intervention (P2I) network: a frozen SBERT sentence embedding is passed through a small MLP that outputs a single predicted duration. (f) The predicted duration is matched to its nearest neighbor on the archive’s duration axis; the matched batch’s own intervention and trajectory, as in (b)–(c), are what a vision-language model scores against the prompt, and that score is the reward used to train P2I. Contributions. This work makes three key contributions, each tested directly in Section 4 rather than only asserted: (1) we demonstrate an archive-based learning paradigm for language-to-intervention translation, using a vision-language model’s own judgment as the reward in place of new experimental feedback; (2) we validate this approach end-to-end, generalizing to entirely new instructions on archive data withheld from training at 80.0%80.0\% ground-truth accuracy, against a 66.7%66.7\% chance baseline; (3) we show this VLM-derived reward is not redundant even with only two behaviors currently distinguished: replacing it with a single fixed duration chosen directly from real velocity data trains to a comparable loss but collapses held-out generalization to 59.1%59.1\%, below chance, while the VLM-fit reward reaches 80.0%80.0\%. 2 Related Work Large language models serve as general-purpose interfaces to complex domains, generating functional code (Chen et al., 2021), aiding molecular design (Jumper et al., 2021), and assisting scientific discovery (Zheng et al., 2025), typically letting users specify goals in language rather than low-level parameters. Biological control is harder: language has no closed-form semantics for what an electrical stimulus should mean, and paired (prompt, intervention, outcome) data is expensive to collect. Synthetic biology has produced xenobots, living Xenopus laevis cell assemblies exhibiting coordinated motion (Kriegman et al., 2021; Blackiston et al., 2021b), where targeted electrical or optogenetic interventions elicit reproducible behavioral change (Zajdel et al., 2020; Levin, 2021; Emiliani et al., 2022), but control remains manual, tuned parameter-by-parameter without a language interface. Vision-language models can serve as zero-shot reward models (Rocamonde et al., 2023; Radford et al., 2021; Liang et al., 2023; Wang et al., 2024), though rarely validated against ground truth independent of the VLM itself – a gap we address directly. Learning across opposing behavioral goals is a multi-task problem, for which task-specific heads over a shared backbone are a standard remedy (Yu et al., 2020; Chen et al., 2018; Ruder, 2017; Caruana, 1997). Training from a fixed archive with only terminal rewards places this work in the offline, contextual-bandit regime (Levine et al., 2020; May et al., 2012), for which derivative-free search such as CMA-ES (Hansen and Ostermeier, 2001) is standard; we show the reward is in fact differentiable, so both backpropagation and gradient-free search apply, and use their agreement to validate the optimum found. Two prior studies explore the underlying idea of prompt-to-intervention learning as a proof-of-concept in simulated, artificial-life cells; we position this work explicitly against the specific gaps each left open. Le et al. (2025) introduced a P2I network mapping a prompt embedding to a spatial vector field steering simulated 2D cell collectives, scored by a VLM against a fixed one-word vocabulary (e.g. “clustering” vs. “scattering”); the reward was treated as non-differentiable by design, optimized only with (1+1)(1+1)-ES and a genetic algorithm, and the paper flags gradient interference between opposing objectives as an open problem at multi-prompt scale. Le et al. (2026) extends this, replacing the fixed vocabulary with a continuous, free-form VLM score in [0,1][0,1] – closer in spirit to the reward used here – but trained on a single prompt (“form a cluster”), again assumed non-differentiability, and lists gradient-trainable optimization and real-cell validation as explicit future work. This paper answers both directly rather than by assumption: the VLM-derived reward is fit as a smooth, differentiable function and trained by direct backpropagation, cross-validated against four independent optimizers; we train and evaluate on 33 prompts rather than one; and every experiment here is on a real, living, non-neural organism rather than a physics simulation. 3 Methods 3.1 Problem setup We are given a fixed archive =(di,xi)i=1NA=\(d_i,x_i)\_i=1^N of N previously-recorded interventions on a xenobot: di∈ℝ>0d_i _>0 is the real electrical-stimulation duration applied in trial i, and xix_i is that trial’s real, tracked PRE/POST trajectory; A is fixed throughout training and evaluation, with no new intervention ever applied to collect it. A user supplies a natural-language prompt p∈p∈P describing a desired change in behavior (e.g. “get the bot to slow down”). We seek a policy fθ:→ℝ>0f_θ:P _>0 that maps a prompt to a predicted duration d^=fθ(p) d=f_θ(p); at evaluation time, d d is rounded to its nearest real, archived duration (Section 3.4), and judged against that intervention’s actual outcome. Because no ground-truth (p,d)(p,d) pairs exist – no human ever paired a prompt with a duration – we cannot supervise fθf_θ directly. Instead a vision-language model acts as a judge J⁡(p,x)∈[0,1]J(p,x)∈[0,1], scoring how well a real trajectory x matches what p describes; J never sees d, only trajectory xix_i and prompt text, so its judgment can depend on duration only through what the trajectory itself reveals. Composing J with the archive gives a per-prompt reward over real, already-observed outcomes, ri(p)=J⁡(p,xi)r_i(p)=J(p,x_i) for each archive point i, computed once, offline, and never queried live during training. The remainder of this section gives, in order: the real data A is built from and the train/test archive split (Section 3.2), the architecture of fθf_θ (Section 3.3), how ri(p)\r_i(p)\ becomes a training signal (Section 3.4), and the held-out evaluation protocol (Section 3.5). 3.2 Dataset and trajectory extraction The archive consists of videos of individually-tracked xenobots undergoing a three-phase protocol: a PRE baseline period, a DURING period (an electrical stimulus of controlled duration d, the sole intervention parameter here), and a POST period. We refer to each such recorded trial as a batch b throughout (a full tracked trial, not an ML minibatch). From raw video to trajectory. Extracting the trajectory used throughout this paper (Figure 2) requires suppressing visual noise specific to this imaging setup – stimulus-induced air bubbles, specular reflection, and debris – that a naive brightness- or contour-based tracker confuses with the xenobot itself. We track position per-frame using the Segment Anything Model (SAM) (Kirillov et al., 2023) across three complementary detection strategies – bounding-box re-detection, optical-flow-guided prompting, and grid-search prompting – each cross-checked against dish geometry and the tracked xenobot’s own appearance to reject false positives. This yields, for each batch, a PRE-phase and a POST-phase position trace, each a 2D trajectory in real pixel coordinates. This – not a precomputed velocity – is what the VLM judge J reads (Section 3.4): if J were instead shown a ready-made velocity number, judging a motion prompt could reduce to comparing two numbers, and a plain numerical rule could do that job just as well, defeating the point of testing whether a VLM reading raw trajectories is useful at all. Section 4.3 makes this comparison directly, training against a purely numerical, velocity-only target instead of J’s judgment. Frames where all three detection tiers fail are excluded as missing data rather than treated as stillness. Figure 2: From raw video to trajectory, real batch. (a) A raw frame, cropped, with a stimulus-induced bubble chain near the dish edge. (b) Optical flow magnitude, one of three detection tiers used despite this noise – motion concentrates at the electrodes, with the true signal (bright point near center) far weaker. (c, d) Per-frame x/y position across the full PRE/DURING/POST protocol: a noisy PRE baseline settles into a distinct POST position – the behavior change the VLM judge assesses from the corresponding 2D trajectory. Assessment window. Classifying a batch requires a PRE baseline window and a POST assessment window, each a fixed number of frames – the last few PRE frames and the first few POST frames – which aggregate (mean or median) to a pre-window velocity vbprev^pre_b and a post-window (assessment) velocity vbassessv^assess_b per batch. A window whose length instead scaled with d would confound duration with the amount of time available to observe behavior, since xenobots naturally decelerate after any stimulus, so we fix both window lengths to a constant, independent of d. The two window lengths and the aggregation statistic are themselves chosen by a grid search (7 pre-window sizes × 8 post-window sizes × 2 aggregation statistics, 112 combinations), maximizing how well vbassess−vbprev^assess_b-v^pre_b separates real ‘faster’- from ‘slower’-labeled batches by duration (Mann–Whitney test). We validate the selected configuration with a permutation test: shuffling the (batch, duration) pairing and rerunning the full 112-combination search 200 times against the real result. The full grid, the permutation protocol, and both results are in Section 4.2 and the Supplementary Material. After quality filtering, 101 batches remain classified (26 ‘faster’, 75 ‘slower’) out of 149 tracked; 48 of 149 are excluded as unreliable rather than force-classified. Each of the 149 tracked batches corresponds to a distinct, individually stimulated xenobot construct – no construct is reused across batches, so batches constitute genuine biological replicates rather than repeated trials on the same organism. We use a 60/40 train/test split (61/40: 16/45 faster/slower train, 10/30 faster/slower test), stratified at the sub-class level so the rarest sub-class is represented in both splits. 3.3 Architecture We embed prompts with SentenceBERT (Reimers and Gurevych, 2019), a frozen 384-dimensional sentence encoder. The P2I policy network (Figure 1e) takes this embedding as input and outputs a predicted intervention duration, through a fully-connected backbone: 384→64384→ 64 (ReLU, dropout 0.2) →32→ 32 (ReLU, dropout 0.2), with one sigmoid output head per behavior category (26,78626,786 parameters total for 2 heads) – one head per behavioral direction, so that opposing objectives (e.g. “stop” vs. “accelerate”) never share an output. A single shared head is a real alternative, not ruled out by assumption: we test it directly, with a distributional (μ,σ)(μ,σ) output and regularization tuned specifically for it, and it remains both worse and less stable on unseen language than 2 heads (Supplementary Material) – so we use 2 heads throughout. 3.4 Reward, baselines, and optimizer comparison Given a prompt p, fθf_θ outputs a predicted duration d^=fθ(p) d=f_θ(p). Training requires an error signal computed from real data, without running any new experiment. For each of the |train|=33|P_train|=33 training prompts, every archive point is scored by the VLM judge: ri(p)=J⁡(p,xi)r_i(p)=J(p,x_i) for i=1,…,Ni=1,…,N (N=101N=101), giving the full reward table ri(p)i,p\r_i(p)\_i,p, ||×|train|=101×33=3,333|A|×|P_train|=101× 33=3,333 real (archive point, prompt) pairs, computed once and reused for every subsequent optimization step. As an ablation, J is also queried from a precomputed velocity-vs-time plot instead of raw positions, to test whether withholding that signal and forcing J to infer pace from position and time alone produces a more reliable judge. This table admits two different training signals for θ (Figure 3). (a) Discrete route. Following prior work in this line of research (Le et al., 2025; Le et al., 2026), the predicted duration is snapped to its nearest archive point, i∗(d^)=argmini∈1,…,N⁡|di−d^|,i^*( d)= _i∈\1,…,N\ |d_i- d |, (1) and its score ri∗(d^)(p)r_i^*( d)(p) is used directly as the reward, so θ is optimized to maximize p∼train[ri∗(fθ(p))(p)]E_p P_train\! [r_i^*(f_θ(p))(p) ]. Because the argmin in Equation 1 is piecewise-constant, this reward is non-differentiable with respect to d d almost everywhere; we reproduce this route as an ablation, optimizing it with two methods that do not require a gradient through it – REINFORCE (policy gradient) and CMA-ES (Hansen and Ostermeier, 2001) (Ros and Hansen, 2008, diagonal-covariance variant;) (full hyperparameters in the Supplementary Material). (b) Differentiable route. For each prompt p, we fit a logistic curve to its (di,ri(p))i=1N\(d_i,r_i(p))\_i=1^N pairs, r^p(d)=σ⁡(wpd~+bp),d~=d−dmindmax−dmin, r_p(d)=σ\! (w_p\, d+b_p ), d= d-d_ d_ -d_ , (2) with (wp,bp)(w_p,b_p) fit per prompt by logistic regression and dmin,dmaxd_ ,d_ the archive’s own minimum and maximum duration. We reduce r^p(⋅) r_p(·) to a single fixed regression target d†(p)d (p) – the duration at the midpoint of r^p r_p’s own achieved range, clipped into the nearest real, archive-verified interval where every point is correctly labeled for p’s intended behavior if it would otherwise fall outside one (construction detail in the Supplementary Material) – and train minθ⁡p∼train[(fθ(p)−d†(p))2] _θ\;E_p P_train\! [ (f_θ(p)-d (p) )^2 ] (3) by ordinary backpropagation (Adam). No VLM, archive lookup, or simulation is queried during a training step under this route – d†(p)d (p) is computed once, offline, directly from the already-computed reward table. The same table independently validates both routes’ outcomes against ground truth (Section 4.2). Figure 3: The two ways to turn the precomputed VLM-score table into a training signal, for the same real prompt. (a) Discrete nearest-neighbor reward: a predicted duration snaps to its nearest real archive batch, and that batch’s raw score is read off directly as the reward – the argmin is discrete, so this route needs REINFORCE or CMA-ES. (b) Differentiable curve-fit target: the same real points, fit into one smooth curve and reduced to a single fixed target, so the error is an ordinary mean-squared error and the whole thing trains by backpropagation. Agreement between all three resulting optimizers – one gradient-based (Equation 3), one policy-gradient, one derivative-free, all consuming the identical reward table – validates the optimum each finds rather than assuming one is necessary. 3.5 Evaluation protocol We verify no held-out prompt duplicates a training prompt and bound cross-split similarity (full list in the Supplementary Material). A held-out prompt’s output head is inferred from its embedding via nearest-centroid similarity to training-prompt centroids, never looked up from a known category. Let =train⊔testP=P_train P_test and =train⊔testA=A_train A_test be the disjoint prompt and archive splits (Section 3.2). Four evaluation cells follow from the product of prompt side × archive side: Train=train×train,GT3=train×test,GT1=test×train,GT2=test×test.Train=P_train\!×\!A_train, 3=P_train\!×\!A_test, 1=P_test\!×\!A_train, 2=P_test\!×\!A_test. (4) For a cell ′×′P ×A , a prediction fθ(p)f_θ(p), p∈′p∈P , is scored by k-nearest-neighbor vote (k=3k=3) among di:i∈′\d_i:i∈A \ against each neighbor’s true, independently-tracked ‘behavior_group’ label, and is correct if the majority vote matches p’s intended direction. GT2 is the only cell where both p∉trainp∉P_train and every i∈′i∈A satisfies i∉traini∉A_train – generalization along both axes at once. Chance on this task is 66.7%66.7\%, not 50%50\%: prompts are drawn evenly from three buckets (motion_increase, motion_reduction, stop), but ‘stop’ and ‘motion_reduction’ are both scored against the same ‘slower’ archive direction, so two of every three prompts need ‘slower’ – a baseline that always answers ‘slower’ is right two-thirds of the time by construction, not by tracking any real signal. Two further checks reuse this setup without retraining: an 8×8×-paraphrase stress test (Section 4.1) and a post-hoc sub-label split of ‘slower’ into genuine stops vs. mild slowdowns (Section 4.4). 4 Results Each subsection below tests one of the three contributions claimed in the introduction directly, rather than asserting it: Section 4.1 tests contribution (2), the headline generalization number, and is itself the direct evidence for contribution (1) – that training works at all without any new experiment; Section 4.2 checks the precondition contribution (1) depends on, that J’s judgment and the archive it draws from are themselves trustworthy; Section 4.3 tests contribution (3), that this reward is not redundant with a simpler numerical alternative; Section 4.4 is further robustness analysis not tied to a specific claimed contribution; Section 4.5 shows the same reference model end to end as a live interface. 4.1 Main result: generalization to held-out prompts and archives One of the contributions we state in the introduction is that this offline paradigm generalizes to entirely new instructions at well above chance; this section runs that experiment directly. Table 1 shows Backprop outperforming the two cross-validating optimizers, CMA-ES and REINFORCE, on every evaluation cell, each averaged over independent seeds (full analysis in the Supplementary Material). We use Backprop as the reference model for the rest of this paper. Table 1: Ground-truth accuracy, all four evaluation cells. 30 seeds for all three optimizers. Train GT3 GT1 GT2 Backprop (reference model) 89.1±1.0%89.1 1.0\% 88.9±0.0%88.9 0.0\% 81.1±5.6%81.1 5.6\% 80.0±0.0%80.0 0.0\% CMA-ES 66.7±0.0%66.7±0.0\% 66.5±1.0%66.5±1.0\% 62.4±7.1%62.4±7.1\% 62.9±6.7%62.9±6.7\% REINFORCE 69.4±10.2%69.4±10.2\% 69.4±10.2%69.4±10.2\% 66.9±9.0%66.9±9.0\% 66.9±8.7%66.9±8.7\% These four cells test generalization along two independent axes: unseen prompts and unseen archive data. Accuracy degrades gracefully across all three held-out cells relative to Train, and GT2 – stressing both axes at once – stays well above the 66.7%66.7\% chance baseline. This is neither a leakage nor a small-sample artifact: no held-out prompt duplicates a training prompt (max 0.600.60 Jaccard overlap), and stress-testing the single saved, already-trained model at 8×8× scale (120 new paraphrases, no retraining) holds accuracy at 85.0%85.0\%, close to the original 80.0%80.0\%, with ‘stop’ the most reliable bucket (full detail in the Supplementary Material). Figure 4 visualizes this same result alongside additional baselines (random, constant, untrained network, nearest-neighbor-trained; full description in the Supplementary Material) on a different, oracle-probability metric – included for a broader sense of where chance-level and naively-trained baselines fall, not for a direct numeric comparison to Table 1. Figure 5 makes the headline number concrete: for three real, held-out prompts, the untrained network’s prediction lands on one archive trajectory and the trained network’s prediction lands on another – visibly different real trajectories, not just different numbers. Figure 4: Ground-truth accuracy across the four evaluation cells for Backprop and REINFORCE, against chance and additional baselines (random, constant, untrained network, nearest-neighbor-trained; described in full in the Supplementary Material). Backprop retains the largest margin over chance at every cell, including GT2, the fully held-out one. Figure 5: Untrained vs. trained, same held-out prompt, real archive trajectories. Top: motion_increase (“speed up”); middle: motion_reduction (“slower”); bottom: stop. Gold = PRE, colored = POST; triangle = start, square = end. Top and bottom rows show a clean wrong-vs-right contrast; the middle row’s untrained and trained predictions both land on the correct direction for this particular prompt (detail in the Supplementary Material), included for honesty rather than omitted for effect. 4.2 Validating the VLM judge and the archive Figure 6: Net VLM score s⁡(b)s(b) per batch (jittered for visibility) against true, independently-tracked behavior label. Open circles: the 3 sign-disagreement batches. We first ask whether J’s judgment tracks a batch’s real, independently-tracked behavior at all. For batch b, define its net score s⁡(b)s(b) as the mean of rb(p)r_b(p) across motion_increase prompts minus the mean across motion_reduction/stop prompts, so s⁡(b)>0s(b)>0 means J judged b as more consistent with speeding up than slowing down. Figure 6 plots s⁡(b)s(b) for every batch against that batch’s true, independently-tracked behavior label. The two agree at r=0.843r=0.843 with 97%97\% sign agreement; every disagreement sits within ±0.02± 0.02px/s of a true zero velocity change – genuinely ambiguous cases, not J misreading a clear signal. Correlating s⁡(b)s(b) against the raw velocity delta itself, instead of the binary label, gives a markedly weaker r=0.49r=0.49: J reliably separates the two behavior categories but does not linearly track fine-grained speed within one, consistent with how the reward is used throughout this paper (a per-prompt target duration, not a speed regression). Querying J from a precomputed velocity-vs-time plot instead of the raw trajectory reaches only r=0.49r=0.49–0.660.66 per category – withholding the precomputed answer and forcing genuine inference produces a more reliable J, not merely a harder-to-game one. Figure 7 shows the reward table’s structure directly: all 33 fitted curves r^p(d) r_p(d) (Equation 2), one independently-fit curve per prompt. Curves within a category are visually near-indistinguishable – every prompt’s own independently-fit curve peaks at the same archive-supported duration – a category-level signal consistent with J scoring near-synonymous prompts near-identically against the same real batch, rather than a within-category wording sensitivity (detail in the Supplementary Material). Figure 7: All 33 fitted VLM-score-vs-duration curves r^p(d) r_p(d) (sigmoid regression, one curve per prompt, 90 points each), grouped by behavior category. The archive itself is equally load-bearing. The selected window configuration (30 pre-frames, 50 post-frames, median aggregation) separates db\d_b\ by behavior at Mann–Whitney p=1.57×10−4p=1.57× 10^-4, against a best-achievable p=1.6×10−5p=1.6× 10^-5 elsewhere in the same 112-point grid – trading a little separability for more readable per-batch plots for J. But this configuration was chosen by searching all 112 combinations rather than fixed in advance, so the achieved p-value alone is not trustworthy: searching many configurations and reporting whichever looks best is a form of p-hacking, where a small p-value can reflect the size of the search rather than a real effect. We check for exactly this with a permutation test: shuffle the (b,db)(b,d_b) pairing, rerun the full 112-way search on the shuffled data, and keep only its best-achievable p-value; repeating this 200 times gives a null distribution of what the search alone could produce by chance, with no real effect present. The real configuration’s p=1.57×10−4p=1.57× 10^-4 is more extreme than all 200 shuffled best-case results (Figure 8), so the separation reflects a real effect and not an artifact of having searched over many configurations. Figure 8: Null distribution of the best p-value achievable by the full 112-way grid search across 200 random duration-label shuffles, against the real result. 4.3 VLM reward vs. a numerical target The purpose of this section is to check whether the VLM-derived target d†(p)d (p) (Equation 3) is doing real work, or whether a much simpler numerical target – with no VLM involved at all – would train fθf_θ just as well. With only two archive behaviors currently distinguished, this is a real possibility worth ruling out directly rather than assuming. We replace d†(p)d (p) with a single fixed duration per behavior direction, chosen purely from real velocity data: the training-archive batch whose measured velocity change (vbassess−vbprev^assess_b-v^pre_b) is strongest in that direction, no VLM or per-prompt differentiation involved. Every ‘faster’ prompt is given the same target duration, and every ‘slower’ prompt the same (different) target duration. fθf_θ is trained against these fixed targets with the identical objective as Equation 3 (mean squared error, Adam, 30 seeds), so training loss and held-out accuracy are directly comparable to the VLM-fit target. Figure 9: Training loss (seed 42, log scale) for the VLM-fit target d†(p)d (p) vs. the purely numerical, velocity-only target – same architecture, optimizer, and epochs. Both converge to a comparable near-zero loss; the numerical target is not harder to fit. Figure 9 shows both targets converge to a comparable near-zero training loss, so the numerical target is not simply easier or harder to fit than d†(p)d (p), and optimization difficulty cannot explain a generalization gap between them if one exists. Table 2 shows one does. Table 2: Same architecture and optimizer (Backprop), 30 seeds; only the training target differs. Train GT3 GT1 GT2 VLM reward (Table 1) 89.1±1.0%89.1 1.0\% 88.9±0.0%88.9 0.0\% 81.1±5.6%81.1 5.6\% 80.0±0.0%80.0 0.0\% Numerical, velocity-only 88.9±0.0%88.9±0.0\% 66.7±0.0%66.7±0.0\% 79.6±1.7%79.6±1.7\% 59.1±2.3%59.1±2.3\% Train accuracy is nearly identical (88.9%88.9\% vs. 89.1%89.1\%), but GT2 falls to 59.1±2.3%59.1± 2.3\% – below the 66.7%66.7\% chance baseline. Per-prompt inspection shows why: the single chosen ‘faster’ duration happens to land in a region of the held-out archive’s duration axis dominated by ‘slower’-labeled real batches, so every held-out ‘faster’ prompt is misclassified as ‘slower’ – not a training failure but a placement failure, invisible from training accuracy alone. A curve fit to the full VLM-scored distribution avoids this; a single extremal real data point does not. Even with only two behaviors currently defined, the VLM reward is not redundant. 4.4 Revisiting ‘stop’ vs. ‘slower’ This section is further robustness analysis, not a test of a claimed contribution. ‘Stop’ and ‘slower’ prompts are merged throughout this paper into one ‘motion_reduction’ class because the data cannot support a clean 3-way split: a direct stop/mild-slower/faster classifier reaches only 0.4770.477 balanced accuracy (barely above chance), against 0.7130.713 once merged. Is that gap because stop and mild slowdown are genuinely indistinguishable, or simply data-starved? Retrospectively splitting ‘slower’ by a sub-label rule (velocity within 10%10\% of baseline at any point) finds a real, significant separation between genuine stops (n=48n=48) and mild slowdowns (n=14n=14; p=7.5×10−6p=7.5× 10^-6, non-overlapping interquartile ranges) – the distinction exists in the data. But three separate attempts to learn it from language all collapsed to the same majority answer regardless of prompt, consistent with data scarcity (only 14 mild-slower examples) rather than a model limitation (detail in the Supplementary Material). This does not affect the paper’s main result, which only relies on the coarser merge, and is left for future work. A downstream consequence. Checking where each held-out ‘stop’/‘motion_reduction’ prediction actually lands (90 held-out prompts; Table 3): ‘stop’ is highly reliable (97.8%97.8\% land on a true stop); ‘motion_reduction’ over-shoots to full-stop behavior in 33.3%33.3\% of cases, traced to the same class imbalance within held-out ‘slower’ data (1919 true-stop vs. 66 true-mild-slower) that motivated the merge in the first place. Table 3: Where each held-out prompt’s predicted duration lands, against the archive’s true sub-label (n=45n=45 each; full per-source breakdown in the Supplementary Material). prompt intent lands on true STOP lands on true MILD-SLOWER ‘stop’ prompt 97.8%97.8\% (44/45) 2.2%2.2\% (1/45) ‘motion_reduction’ prompt 33.3%33.3\% (15/45) 66.7%66.7\% (30/45) 4.5 The system as a live interface The results above are numbers; Figure 10 shows what the trained reference model actually returns for three real prompts, end to end: a prompt is typed in, fθf_θ predicts a duration, and the matched archive batch’s real PRE/POST trajectories and measured velocities are shown back to the user (video demo: zapgpt_sequence.mp4, Supplementary Material). Figure 10: Real PRE/POST trajectories and measured velocities returned for three prompts, same reference model as Table 1. Top to bottom: “speed up” (d^=74.0 d=74.0s, PRE 1.171.17px/s → POST 1.811.81px/s); “slow down” (d^=131.0 d=131.0s, PRE 1.091.09px/s → POST 0.300.30px/s); “stop” (d^=194.0 d=194.0s, PRE 2.842.84px/s → POST 0.150.15px/s, POST trajectory collapses to a single point). 5 Discussion and Future Work In this paper, we have demonstrated that a natural-language interface for a living organism can be learned entirely offline, from an existing archive and a vision-language model’s own judgment, without any new experiments. The resulting P2I mapping generalizes to entirely new instructions at 80.0%80.0\% ground-truth accuracy against a 66.7%66.7\% chance baseline, holding across three independent optimizers sharing the same VLM reward; this reward is also not redundant with a simpler numerical alternative, which collapses held-out generalization to 59.1%59.1\%, below chance (full analysis in the Supplementary Material). This paper’s scope has real limits: one intervention parameter, a small archive (101 of 149 tracked batches), predefined behavior categories, and no main-text result from a new physical experiment. The system retrieves rather than generates – every prediction is the archive’s own closest already-observed match, with compositional recombination beyond it left untested (Supplementary Material). The most direct next step is prospective: does a predicted intervention, applied to a newly-stimulated xenobot never seen in training, actually reproduce its intended behavior? Beyond xenobots, the same offline, archive-plus-VLM-judgment paradigm applies wherever an intervention-outcome archive already exists – other synthetic living machines, cell and tissue cultures under electrical or optogenetic control, and, further out, clinical bioelectric therapy (Ebrahimkhani and Levin, 2021; Zajdel et al., 2020; Blackiston et al., 2021a; Levin, 2021) – with this offline half a tractable place to start. AI use statement A vision-language model (Claude) is a core methodological component of this work: it supplies the training reward for P2I by judging real archive trajectories against natural-language prompts (Section 3.4), and its reliability as a judge is itself directly evaluated against independent, non-VLM ground truth (Section 4.2) rather than assumed. Separately, in preparing this manuscript, a generative AI tool (Claude) was used to assist with paraphrasing. All AI-assisted text was reviewed by the authors. We take responsibility for the final content of this work, including text, claims, and artifacts produced with the aid of generative AI. References Bengio et al. (2021) E. Bengio, M. Jain, M. Korablyov, D. Precup, and Y. Bengio Flow network based generative models for non-iterative diverse candidate generation. In Advances in Neural Information Processing Systems, Vol. 34. Cited by: §1. Blackiston et al. (2021a) D. J. Blackiston, A. K. Trinh, B. Tran, and M. Levin Stimulating neuroplasticity to restore vision. Eneuro 8 (6). Cited by: §5. Blackiston et al. (2021b) D. Blackiston, E. Lederer, S. Kriegman, S. Garnier, J. Bongard, and M. Levin A cellular platform for the development of synthetic living machines. Science Robotics 6 (52), p. eabf1571. Cited by: §1, §2. Caruana (1997) R. Caruana Multitask learning. Machine learning 28 (1), p. 41–75. Cited by: §2. Chen et al. (2021) M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. D. O. Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374. Cited by: §2. Chen et al. (2018) Z. Chen, V. Badrinarayanan, C. Lee, and A. Rabinovich Gradnorm: gradient normalization for adaptive loss balancing in deep multitask networks. In International conference on machine learning, p. 794–803. Cited by: §1, §2. Ebrahimkhani and Levin (2021) M. R. Ebrahimkhani and M. Levin Synthetic living machines: a new window on life. Iscience 24 (5). Cited by: §5. Emiliani et al. (2022) V. Emiliani, E. Entcheva, R. Hedrich, P. Hegemann, K. R. Konrad, C. Lüscher, M. Mahn, Z. Pan, R. R. Sims, J. Vierock, et al. Optogenetics for light control of biological systems. Nature Reviews Methods Primers 2 (1), p. 55. Cited by: §2. Fried et al. (2023) D. Fried, A. Aghajanyan, J. Lin, S. Wang, E. Wallace, F. Shi, R. Zhong, S. Yih, L. Zettlemoyer, and M. Lewis InCoder: a generative model for code infilling and synthesis. In International Conference on Learning Representations, Cited by: §1. Hansen and Ostermeier (2001) N. Hansen and A. Ostermeier Completely derandomized self-adaptation in evolution strategies. Evolutionary computation 9 (2), p. 159–195. Cited by: §2, §3.4. Huang et al. (2022) W. Huang, P. Abbeel, D. Pathak, and I. Mordatch Language models as zero-shot planners: extracting actionable knowledge for embodied agents. In International Conference on Machine Learning, p. 9118–9147. Cited by: §1. Jumper et al. (2021) J. Jumper, R. Evans, A. Pritzel, T. Green, M. Figurnov, O. Ronneberger, K. Tunyasuvunakool, R. Bates, A. Žídek, A. Potapenko, et al. Highly accurate protein structure prediction with alphafold. nature 596 (7873), p. 583–589. Cited by: §2. Kirillov et al. (2023) A. Kirillov, E. Mintun, N. Ravi, H. Mao, C. Rolland, L. Gustafson, T. Xiao, S. Whitehead, A. C. Berg, W. Lo, P. Dollár, and R. Girshick Segment anything. In International Conference on Computer Vision, p. 4015–4026. Cited by: §3.2. Kriegman et al. (2021) S. Kriegman, D. Blackiston, M. Levin, and J. Bongard Kinematic self-replication requires transient breaking of symmetry and multibody coordination. Proceedings of the National Academy of Sciences 118 (9), p. e2011466118. Cited by: §1, §2. Le et al. (2025) N. Le, P. Erickson, Y. Zhang, M. Levin, and J. Bongard Giving simulated cells a voice: evolving prompt-to-intervention models for cellular control. In Proceedings of the Genetic and Evolutionary Computation Conference Companion, p. 2327–2335. Cited by: §2, §3.4. Le et al. (2026) N. H. Le, P. Erickson, Y. Zhang, M. Levin, and J. Bongard ZapGPT: free-form language prompting for simulated cellular control.. In Proceedings of the 2025 8th International Conference on Computational Intelligence and Intelligent Systems, CIIS ’25, New York, NY, USA, p. 119–126. External Links: ISBN 9798400719332, Link, Document Cited by: §2, §3.4. Levin (2021) M. Levin Bioelectric signaling: reprogrammable circuits underlying embryogenesis, regeneration, and cancer. Cell 184 (8), p. 1971–1989. Cited by: §1, §2, §5. Levine et al. (2020) S. Levine, A. Kumar, G. Tucker, and J. Fu Offline reinforcement learning: tutorial, review, and perspectives on open problems. arXiv preprint arXiv:2005.01643. Cited by: §2. Liang et al. (2023) J. Liang, W. Huang, F. Xia, P. Xu, K. Hausman, B. Ichter, P. Florence, and A. Zeng Code as policies: language model programs for embodied control. In 2023 IEEE International conference on robotics and automation (ICRA), p. 9493–9500. Cited by: §2. May et al. (2012) B. C. May, N. Korda, A. Lee, D. S. Leslie, and N. Cesa-Bianchi Optimistic bayesian sampling in contextual-bandit problems.. Journal of Machine Learning Research 13 (6). Cited by: §2. Radford et al. (2021) A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, p. 8748–8763. Cited by: §2. Reimers and Gurevych (2019) N. Reimers and I. Gurevych Sentence-bert: sentence embeddings using siamese bert-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing, Cited by: §3.3. Rocamonde et al. (2023) J. Rocamonde, V. Montesinos, E. Nava, E. Perez, and D. Lindner Vision-language models are zero-shot reward models for reinforcement learning. arXiv preprint arXiv:2310.12921. Cited by: §1, §2. Ros and Hansen (2008) R. Ros and N. Hansen A simple modification in cma-es achieving linear time and space complexity. In International Conference on Parallel Problem Solving from Nature, p. 296–305. Cited by: §3.4. Ruder (2017) S. Ruder An overview of multi-task learning in deep neural networks. arXiv preprint arXiv:1706.05098. Cited by: §2. Wang et al. (2024) Y. Wang, Z. Sun, J. Zhang, Z. Xian, E. Biyik, D. Held, and Z. Erickson Rl-vlm-f: reinforcement learning from vision language foundation model feedback. arXiv preprint arXiv:2402.03681. Cited by: §2. Yu et al. (2020) T. Yu, S. Kumar, A. Gupta, S. Levine, C. Finn, and K. Hausman Gradient surgery for multi-task learning. Advances in Neural Information Processing Systems. Cited by: §1, §2. Zajdel et al. (2020) T. J. Zajdel, G. Shim, L. Wang, A. Rossello-Martinez, and D. J. Cohen SCHEEPDOG: programming electric cues to dynamically herd large-scale cell migration. Cell systems 10 (6), p. 506–514. Cited by: §2, §5. Zheng et al. (2025) Y. Zheng, H. Y. Koh, J. Ju, A. T. Nguyen, L. T. May, G. I. Webb, and S. Pan Large language models for scientific discovery in molecular property prediction. Nature Machine Intelligence 7 (3), p. 437–447. Cited by: §2. Appendix A Appendix Full methodology, prompt lists, VLM query template, held-frame investigation, generalization matrices, and additional ablations are provided in the Supplementary Material accompanying this submission.

Instant research discovery