Paper deep dive

SpikeWorld: Fast-State Adaptation for Frozen Spiking World Models

Ziqiao Yu

Year: 2026Venue: arXiv preprintArea: cs.CVType: PreprintEmbeddings: 49

Intelligence

Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 91%

Last extracted: 8/11/2026, 3:54:04 AM

Summary

The paper introduces SpikeWorld, a 1.45M-parameter sparse spiking world model that jointly trains for heterogeneous sensory prediction, semantics, image-text binding, and action-conditioned dynamics. The core innovation is a deployment-time adaptation mechanism where all trained model parameters remain frozen. Instead, delayed next-state residuals update two external paths: a cumulative fixed-bank loss for bounded action correction and route-specific residual matrices for refining next-state prediction. This approach avoids the need for optimizer state or weight updates, preserving the learned representation while adapting to changed dynamics (e.g., shear, attenuation) in Meta-World environments.

Entities (8)

SpikeWorld · model · 98%Meta-World · environment · 95%Spike Transformer · architecture · 92%Attenuation Dynamics · dynamics-type · 90%Shear Dynamics · dynamics-type · 90%RLS Estimator · algorithm · 88%Fixed-Bank Losses · mechanism · 85%Residual Matrices · mechanism · 85%

Relation Signals (7)

SpikeWorld → evaluatedon → Meta-World

confidence 95% · In a six-arm study comprising 450 new Meta-World trajectories... SpikeWorld raises frozen-policy reward

SpikeWorld → uses → Spike Transformer

confidence 95% · audio, rendered text, images, video events, and state–action sequences pass through modality adapters into one two-layer Spike Transformer.

SpikeWorld → handles → Attenuation Dynamics

confidence 90% · On held-out shear and attenuation streams, the combined external state improves aggregate prediction

SpikeWorld → handles → Shear Dynamics

confidence 90% · On held-out shear and attenuation streams, the combined external state improves aggregate prediction

SpikeWorld → employsmechanism → Fixed-Bank Losses

confidence 88% · cumulative fixed-bank losses select the bounded action correction

SpikeWorld → employsmechanism → Residual Matrices

confidence 88% · route-specific residual matrices refine next-state prediction

RLS Estimator → comparedwith → SpikeWorld

confidence 85% · A 16-byte RLS estimator obtains the highest non-oracle reward on linear attenuation, showing that the contribution is not superior linear identification, but its integration with a frozen multimodal spiking checkpoint.

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:A predictive model receives a self-supervised signal whenever the consequence of an action is observed. Using that signal after deployment is difficult when dynamics and semantics share parameters: freezing prevents adaptation, whereas weight updates require optimizer state and may alter the learned representation. Here we introduce SpikeWorld, a 1.45M-parameter sparse spiking model jointly trained for heterogeneous sensory prediction, semantics, image-text binding and action-conditioned dynamics. At deployment, all trained parameters are frozen. Delayed next-state residuals update two external paths: cumulative fixed-bank losses select the bounded action correction, while route-specific residual matrices refine next-state prediction. Neither path uses labels, teacher outputs, rewards, success signals or the true shift value. Joint optimization improves action next-state MSE by 17.10\% while also improving multimodal prediction, semantic accuracy and image-text retrieval. On held-out shear and attenuation streams, the combined external state improves aggregate prediction by 5.48\% and 30.01\%; its fixed-bank action path improves tracking by 24.20\% and 3.94\%, respectively. In a six-arm study comprising 450 new Meta-World trajectories (75 per arm), SpikeWorld raises frozen-policy reward by 7.90 (95\% CI [2.48, 14.06]); the 13.33-point success difference is descriptive (CI [0, 40]). For identical sensory inputs, model parameters and inherited semantic outputs remain bitwise unchanged. A 16-byte RLS estimator obtains the highest non-oracle reward on linear attenuation, showing that the contribution is not superior linear identification, but its integration with a frozen multimodal spiking checkpoint. Reference code is publicly available at this https URL.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

47,872 characters extracted from source content.

Expand or collapse full text

SpikeWorld: Fast-State Adaptation for Frozen Spiking World Models Ziqiao Yu DiDi International Business Group yuziqiao@didiglobal.com Abstract A predictive model receives a self-supervised signal whenever the consequence of an action is observed. Using that signal after deployment is difficult when dynamics and semantics share parameters: freezing prevents adaptation, whereas weight updates require optimizer state and may alter the learned representation. Here we introduce SpikeWorld, a 1.45M-parameter sparse spiking model jointly trained for heterogeneous sensory prediction, semantics, image–text binding and action-conditioned dynamics. At deployment, all trained parameters are frozen. Delayed next-state residuals update two external paths: cumulative fixed-bank losses select the bounded action correction, while route-specific residual matrices refine next-state prediction. Neither path uses labels, teacher outputs, rewards, success signals or the true shift value. Joint optimization improves action next-state MSE by 17.10% while also improving multimodal prediction, semantic accuracy and image–text retrieval. On held-out shear and attenuation streams, the combined external state improves aggregate prediction by 5.48% and 30.01%; its fixed-bank action path improves tracking by 24.20% and 3.94%, respectively. In a six-arm study comprising 450 new Meta-World trajectories (75 per arm), SpikeWorld raises frozen-policy reward by 7.90 (95% CI [2.48, 14.06]); the 13.33-point success difference is descriptive (CI [0, 40]). For identical sensory inputs, model parameters and inherited semantic outputs remain bitwise unchanged. A 16-byte RLS estimator obtains the highest non-oracle reward on linear attenuation, showing that the contribution is not superior linear identification, but its integration with a frozen multimodal spiking checkpoint. Reference code is publicly available at https://github.com/Oooorca/SpikeWorld. 1 Introduction A world model makes a testable claim each time it predicts the consequence of an action. Once the consequence is observed, the prediction residual is available without a label, demonstration or reward. This makes prediction an appealing source of deployment-time supervision. The same signal is also dangerous: if predictive dynamics and semantic readouts share a representation, updating the model to fit a changed environment can alter the representation on which perception and binding depend. Freezing the model preserves semantics but leaves the changed dynamics uncorrected. Updating all predictive weights has the opposite profile and additionally requires persistent optimizer state. This tension is particularly relevant for spiking models. Membrane state gives them an explicit causal representation, while sparse spike communication can reduce attention interactions. These properties do not, by themselves, answer two questions. A collection of modality-specific checkpoints is not one multimodal model, and a smaller prediction error does not imply a useful or safe action correction. Both properties must therefore be tested directly. Here we introduce SpikeWorld, a sparse spiking world model designed around a separation between state formation and state adaptation. Offline, audio, rendered text, images, video events, and state–action sequences pass through modality adapters into one two-layer Spike Transformer. Next-slice prediction, semantic classification, image–text binding and action-conditioned next-state prediction jointly update the same temporal core. At deployment, the resulting 1.45M-parameter checkpoint is immutable. Only an external fast-state module changes, using the discrepancy between a predicted endpoint and the endpoint observed eight environment steps later. A frozen router selects among a fixed source, shear, attenuation and noise families, and a bounded inverse map converts the estimated dynamics into an action correction. Figure 1: SpikeWorld forms one predictive–semantic state offline and adapts only external state at deployment. A, causal adapters map audio, rendered text, images, video and state–action prefixes to a common sequence interface. Next-slice prediction, semantic classification, image–text binding and action-conditioned next-state prediction jointly update the same sparse spiking core. B, the deployed policy, world model, semantic readouts and fixed-bank router are frozen. After the endpoint at t+8t+8 is observed, its discrepancy from the previously issued prediction is routed to a 24,384-byte external module. Cumulative candidate scores determine the bounded correction for the next action, whereas route-specific residual matrices correct subsequent state predictions. The first four actions are unchanged; observation four produces the first write, so decision five is the earliest corrected action. The bottom cards summarize joint training, registered-shift adaptation and closed-loop evaluation; they report within-method effects rather than a ranking over all baselines. Model-parameter and same-input semantic-logit hashes remain unchanged. The stated 24,384-byte budget excludes the episodic routing buffer, which is reported separately in Section˜A.3. The experiments examine three linked properties. First, we verify joint formation rather than checkpoint packaging: both multimodal and action losses have nonzero gradients to the same core, and the joint objective improves all four aggregate metrics across five action-model initializations. This establishes the integration context; it does not by itself show that multimodal training is necessary for the later adaptation gain. Second, on 80 held-out Meta-World episodes, the frozen model plus fast state improves aggregate prediction and action tracking under shear and attenuation; correct event–action pairing is stronger than a shuffled control. Third, the six-arm closed-loop study contains 450 new trajectories (75 per arm). The first corrected action occurs only after four observations, after which mean reward rises by 7.90; the observed success difference is 13.33 percentage points (CI [0, 40]). The online interface is audited to exclude labels, teachers, rewards, success signals and true shift values. Our contributions are threefold: 1. We demonstrate one jointly optimized spiking checkpoint whose shared core supports heterogeneous sensory prediction, semantics, cross-modal binding and action-conditioned dynamics. 2. We develop a prediction-driven deployment state that adapts the evaluated actuator changes while all trained weights and inherited semantic outputs remain bitwise unchanged for identical sensory inputs. 3. We show that this correction transfers from offline dynamics metrics to reward, with a descriptive success-rate change for a frozen policy, and audit its information access, mutable storage, latency and sparse attention execution. The claim is intentionally narrower than general continual learning. A 16-byte RLS estimator achieves higher mean reward on constant linear attenuation, and our router does not discover arbitrary new mechanisms. SpikeWorld instead shows that prediction-driven adaptation can be attached to a shared predictive–semantic spiking state without rewriting that state at deployment. 2 Related Work Predictive state and world models. Predictive-state representations define state by the distribution of future observations (Littman et al., 2002); learned world models use related objectives for latent simulation and control (Ha and Schmidhuber, 2018). Feature-predictive video models further show that useful future structure need not be expressed as pixel reconstruction (Bardes et al., 2024; Assran and others, 2025). SpikeWorld adopts this predictive view but studies a different transition: after offline representation formation, can an observed endpoint residual update a compact deployment state without modifying the learned model? The control result concerns correction of a frozen policy under changed actuator dynamics, not policy learning. Event-driven spiking models. Event sensors encode changes rather than complete repeated frames (Lichtsteiner et al., 2008; Gallego et al., 2022). Surrogate gradients make temporal spike patterns trainable (Neftci et al., 2019; Zenke and Ganguli, 2018) and can recover information that is unavailable to rate codes (Yu et al., 2026). Spiking Transformers extend this computation to attention architectures (Zhou et al., 2023b; Zhu et al., 2023), while EventCLIP, EventBind and masked event modeling connect event streams to semantic or self-supervised objectives (Wu et al., 2023; Zhou et al., 2023a; Klenk et al., 2022). Dual-memory spiking architectures separate a compact contextual state from fast event-driven processing (Sun et al., 2026). Our separation acts across the deployment boundary: a shared spiking predictor is formed jointly offline, then kept immutable while a low-capacity residual state remains plastic. Test-time adaptation and system identification. Test-time adaptation typically changes selected model parameters using an unlabeled surrogate objective. TENT minimizes entropy (Wang et al., 2021); CoTTA and EATA add restoration or sample selection to limit drift (Wang et al., 2022; Niu et al., 2022). SpikeWorld instead freezes every trained parameter and uses the supervised-in-time prediction residual to update external state. This setting also overlaps classical system identification, which estimates dynamics from observed inputs and outputs. We therefore include recursive least squares rather than treating neural adaptation as the default baseline. Its strong performance on linear attenuation identifies where a specialized estimator is preferable. The multimodal checkpoint is the integration context for our adaptation study, not an experimentally isolated cause of the control gain. Context adaptation and fast state for control. Fast weights provide variables that change between neural activity and learned parameters in time scale (Ba et al., 2016). In control, latent-context methods infer task or environment variables from recent experience (Rakelly et al., 2019), model-based meta-learning adapts dynamics online (Nagabandi et al., 2019), and Rapid Motor Adaptation predicts extrinsic context for a fixed base policy (Kumar et al., 2021). SpikeWorld is narrower than these general adaptation frameworks: its router selects from a fixed actuator bank. Its technical distinction is an auditable causal update driven by delayed endpoint prediction error, attached to an otherwise frozen predictive–semantic spiking checkpoint. The closed-loop study therefore asks whether this integration is operational, not whether it supersedes matched system identification or learned adaptive policies. 3 One Predictive–Semantic Spiking State 3.1 Heterogeneous event streams, shared temporal computation The sensory frontends map their inputs to length-16 causal sequences. Audio is encoded as binned cochlear events. Handwritten digits and rendered digit words become signed retinal events. Video is represented by pooled causal feature changes, whereas a continuous state–action pair is represented by a two-token prefix through the same width-96 core. A learned modality token identifies the source. Thus, “multimodal” refers to shared temporal computation and parameters; it does not imply identical sensors, eventizers or input dimensions. The shared core contains two Transformer layers of width 96, four attention heads and a feed-forward width of 192. Leaky integrate-and-fire units evolve as vtℓ v_t =βut−1ℓ+Itℓ, =β u_t-1 +I_t , (1) stℓ s_t =[vtℓ>Vth], =1[v_t >V_ th], (2) utℓ u_t =vtℓ−Vthstℓ, =v_t -V_ ths_t , (3) where surrogate derivatives are used only during offline training. Each query attends to the four most recent admissible keys. Continuous membrane features carry the predictive state; thresholded spikes gate communication between layers. 3.2 Joint formation objective Let x1:Ts∈ℝT×Dsx^s_1:T ^T× D_s denote a sensory stream. Define its normalized next-slice error as ℓs=1(T−1)DsLpersists‖x^2:Ts−x2:Ts‖22. _s= 1(T-1)D_sL^s_ persist\| x^s_2:T-x^s_2:T\|_2^2. (4) The implemented objective averages four modality groups, ℒnext=14[12(ℓSHD+ℓSSC)+ℓtext+ℓimage+ℓvideo]L_ next= 14[ 12( _ SHD+ _ SSC)+ _ text+ _ image+ _ video]; the persistence normalizer places different input widths on a comparable scale. A coupled semantic head reads the shared hidden sequence and averages five streams, ℒsem=15∑s∈SHD,SSC,text,image,videoCE⁡(hs(Zs),ys).L_ sem= 15 _s∈\SHD,SSC,text,image,video\CE(h_s(Z^s),y^s). (5) For paired retinal images and rendered words, a multi-positive contrastive loss aligns samples that share a digit identity. With I and T denoting image and text anchors, respectively, ℒbind=12(ℒI→T+ℒT→I),ℒI→T=−1N∑ilog⁡∑j:yj=yiesim⁡(ziI,zjT)/τ∑jesim⁡(ziI,zjT)/τ,L_ bind= 12(L_I→ T+L_T→ I), _I→ T=- 1N _i _j:y_j=y_ie^sim(z_i^I,z_j^T)/τ _je^sim(z_i^I,z_j^T)/τ, (6) with ℒT→IL_T→ I defined symmetrically. For control state qtq_t, task index c, action ata_t, and the state observed eight environment steps later, the same core predicts a standardized change, ℒact=139‖gθ(qt,at,c)−qt+8−qt−μΔσΔ‖22.L_ act= 139 \|g_θ(q_t,a_t,c)- q_t+8-q_t- _ _ \|_2^2. (7) One balanced optimizer step minimizes ℒjoint=ℒnext+ℒsem+0.25ℒbind+4ℒact.L_ joint=L_ next+L_ sem+0.25L_ bind+4L_ act. (8) The raw semantic bank inherited from multimodal pretraining is fixed, whereas the predictive core, coupled semantic readouts, binding path and control adapter are optimized together. Sharing an architecture is weaker than sharing a model. We therefore test whether both ℒactL_ act and the multimodal terms in Equation˜8 deliver finite, nonzero gradients to the same core tensors, and whether those tensors change. We additionally compare joint optimization with action-only updates to the shared core. These controls distinguish genuine joint state formation from placing independent modules in one checkpoint. 4 Prediction-Driven Fast-State Adaptation 4.1 A frozen deployment graph After joint training, every model parameter, semantic readout and router parameter is frozen. At macro-step t, the world model receives state qtq_t, commanded action ata_t and task index c, then predicts the endpoint after eight environment steps. Once that endpoint is observed, the standardized residual is rt=gθ0(qt,at,c)−qt+8−qt−μΔσΔ.r_t=g_ _0(q_t,a_t,c)- q_t+8-q_t- _ _ . (9) Unlike reward or task success, rtr_t follows directly from a prediction that was issued before the transition. The accounted fast-state module contains 6,096 FP32 values (24,384 bytes) and retains no optimizer moments or replay buffer. Its principal components are two route-specific matrices, Cshear,Catten∈ℝ96×30C_ shear,C_ atten ^96× 30, together with 32 cumulative candidate scores, a count, and padding used to match the comparison budget. The reference runner also stores per-transition router signatures and candidate losses in an episodic Python buffer. This buffer is reported separately from the 24,384-byte module budget and grows with episode length. Given frozen hidden state ht∈ℝ96h_t ^96, mean h¯∈ℝ96 h ^96 and fixed basis B∈ℝ30×96B ^30× 96, route ρt _t, and affine output head (Wout,bout)(W_ out,b_ out), the selected candidate first transforms the action before the core produces hth_t. For ρt∈shear,attenuation _t∈\shear,attenuation\, the corresponding matrix then corrects the hidden state as h~t=ht+CρtB(ht−h¯),d^t=Wouth~t+bout. h_t=h_t+C_ _tB(h_t- h), d_t=W_ out h_t+b_ out. (10) For source and noise, Cρt=0C_ _t=0. Let dt=(qt+8−qt−μΔ)/σΔd_t=(q_t+8-q_t- _ )/ _ and ℓt=139‖d^t−dt‖22 _t= 139\| d_t-d_t\|_2^2. For an active shift route, the selected matrix takes one normalized, projected residual step after the endpoint becomes available, Cρt←Π‖C‖F≤0.20(Cρt−0.02∇Cρtℓtmax⁡(‖∇Cρtℓt‖F,10−12)).C_ _t← _\|C\|_F≤ 0.20 (C_ _t-0.02 _C_ _t _t (\| _C_ _t _t\|_F,10^-12) ). (11) No persistent optimizer state is needed. The first four actions are unchanged; observation four triggers routing and the first matrix write, so decision five is the earliest action that can change. 4.2 A fixed bank of dynamics families The mechanism bank represents source dynamics and two actuator changes, Tsource(a) T_ source(a) =a, =a, (12) Tshear(a;k)y T_ shear(a;k)_y =ay+kax, =a_y+ka_x, (13) Tatten(a;g)x,y T_ atten(a;g)_x,y =gax,y. =ga_x,y. (14) Each candidate prediction is a function only of the pre-transition state, action and task, although the reference evaluator computes the batch after loading the stored endpoint. The endpoint enters only when the predictions are scored, and the resulting losses are combined with the source residual, state change, absolute-value and action–change cross statistics. A router trained offline maps the four-transition average to source, shear, attenuation or noise. The deployment code never receives the mechanism label or its true value. Cumulative candidate loss then selects k k or g g within the routed family. More explicitly, for candidate j, d^t(j)=gθ0(qt,clip⁡(Tj(at),−1,1),c),Sj←Sj+139‖d^t(j)−dt‖22, d_t^(j)=g_ _0(q_t,clip(T_j(a_t),-1,1),c), S_j← S_j+ 139\| d_t^(j)-d_t\|_2^2, (15) At observation four, a shear or attenuation route retroactively adds all four pending loss vectors to S; later observations add one vector at a time. Source and noise routes do not update S or the residual matrices. Within an active family, the selected value is j∗=arg⁡minj:family(j)=ρt⁡Sjj^*= _j:\,family(j)= _tS_j. It affects both prediction, through the transformed action supplied to the frozen core, and control, through the bounded inverse. The residual matrix supplies an additional prediction-only correction. The registered-shift adaptation study uses a prequential offline protocol. It loads a transition produced by a stored behavior action, predicts from its pre-transition fields, scores after the endpoint is available, and records a counterfactual bounded command. The tracking metric is computed only after the online pass, using mechanism annotations unavailable to the update. The closed-loop control study evaluates the same update rule in a live loop: the resulting bounded command is executed from decision five onward. In neither protocol can an endpoint alter the earlier action that generated it. This mechanism bank is deliberately finite. It tests whether causal residuals can select and adapt a fixed dynamics family, not whether the model can discover an arbitrary physical law. Moreover, earlier source-return streams did not select source reliably; we therefore make no claim of automatic write suppression after unrestricted repeated shifts. 4.3 From a dynamics estimate to a safe action For desired policy action at∗a_t^*, the selected family proposes a~y a_y =ay∗−k^ax∗ =a_y^*- ka_x^* (shear), (shear), (16) a~x,y a_x,y =ax,y∗/max⁡(g^,0.2) =a_x,y^*/ ( g,0.2) (attenuation). (attenuation). (17) The proposed action is bounded coordinate-wise, at=at∗+clip⁡(a~t−at∗,−0.05,0.05).a_t=a_t^*+clip( a_t-a_t^*,-0.05,0.05). (18) If the candidate leaves [−1,1]4[-1,1]^4, the method proposes at∗a_t^* instead. In the closed-loop study the proposal is executed by the live environment; in the registered-shift study it is evaluated counterfactually. This guard is part of the method rather than a cosmetic constraint: an earlier uncapped inverse reduced pooled prediction error but failed the prespecified shear action gate. Finally, the external state is absent from every inherited semantic readout. Together with frozen model parameters, this makes current-step semantic logits identical for identical sensory inputs. Adapted actions can alter later observations, so this structural statement does not imply trajectory-level semantic invariance. The method protects the learned readout rather than adding new concepts during deployment. 5 Experimental Protocol The experiments address the three linked questions summarized along the bottom of Figure˜1: whether the checkpoint is jointly formed, whether its prediction residual can adapt changed dynamics without weight updates, and whether the resulting correction changes closed-loop behavior. Joint multimodal–action training. The multimodal cache contains balanced SHD and SSC spike sequences (Cramer et al., 2022), paired retinal events from handwritten digits and rendered digit words, and a content-disjoint Something-Something-V2 causal-event subset. Every sequence has 16 slices. Action-conditioned data are Meta-World state, action and delayed-endpoint streams from reach, push, pick-place and door-open (Yu et al., 2019). Five action-model initializations inherit the same multimodal foundation checkpoint and train independent control adapters. We compare the sequential frozen-core checkpoint, 200 action-only shared-core updates and 200 joint updates of Equation˜8. This study therefore tests robustness to action-path initialization, not five independent multimodal pretraining runs. Registered-shift adaptation. Before data collection we freeze the jointly connected model, mechanism grid, router, update rule and safety bound. We then collect 80 held-out, irreversible episodes from five environment replicas, four tasks and counterbalanced shear and attenuation orders. Each of five model seeds evaluates every stream. The online files expose current state, behavior action, observed endpoint, episode seed, step and stream index. Mechanism annotations reside in a separately hashed audit manifest that is opened only after all online arms finish. Primary metrics are relative reductions in next-state and action-tracking MSE. An action-shuffled arm tests whether improvement depends on the correct temporal pairing rather than marginal event statistics. Closed-loop control. The closed-loop study uses reach, push and pick-place, for which the frozen policy retains measurable headroom under attenuation (g=0.4g=0.4). Five model seeds, three tasks, five new environment seeds and six paired arms produce 450 live trajectories. The arms are frozen behavior, SpikeWorld fast state, full prediction-path tuning, recursive least squares (RLS), bounded compressed replay and an exact-inverse oracle. Full tuning changes 434,412 predictive-path parameters and retains Adam state. RLS is a deliberately strong baseline for a constant linear fault. All non-oracle corrections use the same 0.05 bound. Reward and binary success are evaluation-only and never enter an update. Deployment audit. All intervals use 20,000 bootstrap draws at the prespecified unit of analysis; raw environment steps are not treated as independent samples. We report per-task and per-seed effects, mutable and optimizer-state bytes, measured software update latency, parameter hashes, semantic-output hashes, environment instantiation counts, dependency hashes and every field available to the online process. A separate length-16 proxy records executed attention-score pairs and software forward latency. It is not a chip-energy measurement. 6 Results 6.1 One checkpoint is jointly predictive, semantic and action-conditioned Table˜1 asks first whether action dynamics enter the shared multimodal state. Relative to the sequential frozen-core checkpoint, joint optimization reduces action next-state MSE by 17.10% (95% CI [14.04, 20.04]) and normalized multimodal next-slice MSE by 0.75% [0.68, 0.83]. Semantic accuracy increases by 1.80 percentage points [1.22, 2.44], and image–text class R@1 by 1.81 points [0.19, 3.44]. For every action-model seed, both losses deliver finite, nonzero gradients to the same core tensors and the resulting core hash changes. Table 1: Joint multimodal–action formation. Means over five action-model initializations. Lower is better for MSE; higher is better for accuracy and retrieval. Training protocol Action MSE Norm. next MSE Sem. acc. (%) Bind R@1 (%) Sequential, frozen core 0.1463 0.6776 19.15 10.00 Action-only core update 0.1174 0.6786 19.27 8.50 Joint objective 0.1212 0.6725 20.94 11.81 The action-only update achieves 3.30% lower action MSE than the joint objective, but image–text retrieval falls from 10.00% to 8.50%. Joint optimization instead reaches 11.81%, a paired advantage of 3.31 points [0.94, 5.38] over action-only updating. Thus, the shared objective trades a small amount of action specialization for a measurably broader multimodal state. It is not simply the best action model with unrelated sensory heads attached. 6.2 Causal residuals adapt two actuator shifts from a fixed bank We next freeze the full checkpoint and expose it to held-out shear and attenuation streams. Both prediction and action-tracking gains have strictly positive aggregate intervals (Table˜2). Action improvement is positive for every one of the five model seeds and for all four tasks under both shifts. The correctly paired update also exceeds the action-shuffled control: the paired action-MSE advantage is 0.004126 [0.002958, 0.005314] for shear and 0.001651 [0.000256, 0.003184] for attenuation. Table 2: Held-out fixed-bank shift adaptation. Relative reductions in prediction and action-tracking MSE with 95% bootstrap intervals. Shift Prediction improvement Action improvement Router acc. Shear 5.48% [3.78, 7.09] 24.20% [13.52, 33.05] 76.5% Attenuation 30.01% [27.55, 32.29] 3.94% [3.52, 4.37] 94.0% The two metrics expose different constraints. Attenuation is identified accurately from prediction, but the 0.05 cap permits only a modest action change. Shear is routed less accurately, yet the selected correction produces a larger tracking gain. Shear prediction is negative on the push task, so the evidence does not support a per-task universal prediction claim. The consistent action result is the more relevant outcome of the bounded inverse. 6.3 Fixed-bank correction improves closed-loop behavior In a six-arm study comprising 450 new trajectories (75 per arm), SpikeWorld raises mean cumulative reward from 414.68 to 422.58 and success from 53.33% to 66.67% (Tables˜3 and 2). The paired reward increase is 7.90, 95% CI [2.48, 14.06]. The 13.33-point success difference is descriptive because its interval is [0, 40]. Reward gains are positive for every task—+2.31 on pick-place, +11.61 on push and +9.79 on reach—and for all model seeds, ranging from +7.19 to +8.66. The first four macro-transitions change no action; separation begins at decision five after the required observations have occurred. Table 3: Closed-loop control on new attenuation trajectories. Reward and success are evaluation-only. Times are measured on the recorded CPU run. Method Reward Success Update Accounted state Frozen 414.68 53.33% – 0 SpikeWorld 422.58 66.67% 1.48 ms 24 KB† Full prediction tuning 422.19 66.67% 3.57 ms 1.74 MB + 3.48 MB opt. RLS 434.41 65.33% 1.79 ms 16 B Replay 428.59 60.00% 1.79 ms ≤ 24 KB Oracle inverse 485.50 80.00% – privileged †Fast-state module only; the reference runner’s episodic routing buffer is excluded. Figure 2: The prediction-driven correction transfers to task behavior. A, cumulative reward is identical to frozen behavior during the four observation-only decisions and separates only after the first corrected action. B, SpikeWorld’s reward gain is positive on every evaluated task. C, observed success is 20 points higher on push and reach and remains unchanged on the already-strong pick-place task. Curves in A are means over five model seeds, three tasks and five new environment seeds; the oracle is a privileged diagnostic. Fast-state adaptation and full prediction tuning reach the same aggregate success. Their paired reward difference is +0.40 [-0.30, 1.63]. Full tuning, however, changes the coupled semantic-output hash in every model–task run, whereas SpikeWorld leaves all parameter and direct semantic-output hashes bitwise unchanged for identical inputs. Aggregate semantic accuracy does not fall during this short full-tuning stream; the supported observation is representation drift, not semantic collapse. RLS attains the highest non-oracle mean reward on constant linear attenuation, with a gain of 19.73 [1.24, 54.19] over frozen behavior. The result is expected for a 16-byte estimator matched to a linear fault. SpikeWorld therefore does not dominate classical identification. Its distinct contribution is that one frozen predictive–semantic model handles both attenuation and shear, preserves its checkpoint, and yields a useful closed-loop correction. The present data do not isolate multimodal pretraining or spiking computation as the cause of this correction; they establish that the correction can coexist with the jointly formed checkpoint. 6.4 Deployment audit: state size, sparsity and information access The complete checkpoint contains 1,451,388 parameters (5.81 MB in FP32). The accounted 24,384-byte fast-state module is 0.420% of that storage and has no optimizer state. The reference runner additionally retains an episodic Python routing buffer; 24,384 bytes is therefore a module-parameter budget, not total runtime memory. Its measured update takes 1.476 ms, compared with 3.568 ms for full prediction tuning, a 2.42-fold difference in this implementation. In a matched length-16 attention proxy, 58 admissible QK pairs replace 256 dense pairs, a 77.34% reduction. Median full-forward time changes from 24.12 to 21.10 ms (1.14×). These are software measurements and do not imply a measured neuromorphic energy advantage. All 20 registered-shift online shards contain only observations, actions, endpoints, seeds, steps and stream indices. In the closed-loop study, the controller update receives exactly state, command, task and endpoint; a pre-environment lock precedes all 450 environment instances, and dependency hashes remain unchanged. Together with the unchanged parameter and direct semantic-output hashes, these audits establish the claimed label-, teacher-, reward-, success- and mechanism-value-free deployment boundary. 7 Discussion and Conclusion Predictive models expose an unusual deployment signal: their error becomes known after time advances. The central result of this study is that this signal can improve action without being allowed to rewrite the model that produced it. SpikeWorld first forms one shared spiking state from sensory prediction, semantics, cross-modal binding and action-conditioned dynamics, then freezes that state. At deployment, accumulated fixed-bank losses select a bounded action inverse, while route-specific residual matrices refine prediction. The former improves reward; the combined state also reduces next-state error. The success-rate change is descriptive under the reported interval. The experiments suggest a simple design principle: when a world model also carries semantics, deployment-time plasticity should be placed in a state with a narrow read–write interface rather than distributed across the trained representation. For identical inputs, that separation gives exact direct-readout non-interference and removes persistent optimizer memory. It also makes causal auditing possible: the first correction follows four observed transitions, and the complete set of online fields can be enumerated. The principle is functional rather than biological; the present eventizers and router are engineered components. The RLS comparison is equally informative. On constant linear attenuation, a 16-byte estimator obtains a larger mean reward gain than SpikeWorld. A compact neural state is therefore not justified by linear identification alone. Its potential value lies in attaching adaptation to a model that also carries heterogeneous predictive and semantic information and distinguishes more than one fixed dynamics family. Nonlinear and compound shifts are required to test whether that broader state yields an advantage beyond this initial case. Several limitations define the scope of the result. The router selects among families learned offline and does not discover an arbitrary new mechanism. Earlier source-return studies did not reliably suppress subsequent writes, so the experiments do not establish unrestricted continual learning. The five The joint-training runs share one multimodal foundation checkpoint, whose semantic accuracy is modest. The visual and text eventizers are hand-designed, the video frontend is not fully spiking, and no physical asynchronous sensor or neuromorphic chip is evaluated. Finally, the policy is fixed: SpikeWorld adapts action execution, not the policy objective. Within these boundaries, the experiments establish a specific loop. A shared multimodal spiking predictor can issue a causal forecast, compare it with the observed transition, update fixed-bank scores and prediction residual matrices, and improve a frozen policy through a bounded inverse while every trained parameter and identical-input semantic output remains unchanged. This provides a concrete basis for studying richer deployment-time adaptation without conflating environmental calibration with representation rewriting. References M. Assran et al. (2025) V-JEPA 2: self-supervised video models enable understanding, prediction and planning. arXiv preprint arXiv:2506.09985. Cited by: §2. J. Ba, G. E. Hinton, V. Mnih, J. Z. Leibo, and C. Ionescu (2016) Using fast weights to attend to the recent past. In Advances in Neural Information Processing Systems, Vol. 29. External Links: Link Cited by: §2. A. Bardes, Q. Garrido, J. Ponce, X. Chen, M. Rabbat, Y. LeCun, M. Assran, and N. Ballas (2024) Revisiting feature prediction for learning visual representations from video. arXiv preprint arXiv:2404.08471. Cited by: §2. B. Cramer, Y. Stradmann, J. Schemmel, and F. Zenke (2022) The heidelberg spiking data sets for the systematic evaluation of spiking neural networks. IEEE Transactions on Neural Networks and Learning Systems 33 (7), p. 2744–2757. External Links: Document Cited by: §5. G. Gallego, T. Delbruck, G. Orchard, C. Bartolozzi, B. Taba, A. Censi, S. Leutenegger, A. J. Davison, J. Conradt, K. Daniilidis, and D. Scaramuzza (2022) Event-based vision: a survey. IEEE Transactions on Pattern Analysis and Machine Intelligence 44 (1), p. 154–180. External Links: Document, Link Cited by: §2. D. Ha and J. Schmidhuber (2018) World models. arXiv preprint arXiv:1803.10122. External Links: Link Cited by: §2. S. Klenk, D. Bonello, L. Koestler, N. Araslanov, and D. Cremers (2022) Masked event modeling: self-supervised pretraining for event cameras. arXiv preprint arXiv:2212.10368. External Links: Link Cited by: §2. A. Kumar, Z. Fu, D. Pathak, and J. Malik (2021) RMA: rapid motor adaptation for legged robots. In Robotics: Science and Systems XVII, External Links: Document, Link Cited by: §2. P. Lichtsteiner, C. Posch, and T. Delbruck (2008) A 128×128 120 db 15 μ latency asynchronous temporal contrast vision sensor. IEEE Journal of Solid-State Circuits 43 (2), p. 566–576. External Links: Document, Link Cited by: §2. M. L. Littman, R. S. Sutton, and S. Singh (2002) Predictive representations of state. In Advances in Neural Information Processing Systems, Vol. 14, p. 1555–1561. Cited by: §2. A. Nagabandi, I. Clavera, S. Liu, R. S. Fearing, P. Abbeel, S. Levine, and C. Finn (2019) Learning to adapt in dynamic, real-world environments through meta-reinforcement learning. In International Conference on Learning Representations, External Links: Link Cited by: §2. E. O. Neftci, H. Mostafa, and F. Zenke (2019) Surrogate gradient learning in spiking neural networks: bringing the power of gradient-based optimization to spiking neural networks. IEEE Signal Processing Magazine 36 (6), p. 51–63. External Links: Document, Link Cited by: §2. S. Niu, J. Wu, Y. Zhang, Y. Chen, S. Zheng, P. Zhao, and M. Tan (2022) Efficient test-time model adaptation without forgetting. In Proceedings of the 39th International Conference on Machine Learning, Proceedings of Machine Learning Research, Vol. 162, p. 16888–16905. External Links: Link Cited by: §2. K. Rakelly, A. Zhou, C. Finn, S. Levine, and D. Quillen (2019) Efficient off-policy meta-reinforcement learning via probabilistic context variables. In Proceedings of the 36th International Conference on Machine Learning, Proceedings of Machine Learning Research, Vol. 97, p. 5331–5340. External Links: Link Cited by: §2. P. Sun, Z. Su, J. Achterberg, G. Indiveri, D. F. M. Goodman, and D. Akarca (2026) Algorithm–hardware co-design of neuromorphic networks with dual memory pathways. Nature Machine Intelligence 8, p. 901–912. External Links: Document, Link Cited by: §2. D. Wang, E. Shelhamer, S. Liu, B. Olshausen, and T. Darrell (2021) Tent: fully test-time adaptation by entropy minimization. In International Conference on Learning Representations, External Links: Link Cited by: §2. Q. Wang, O. Fink, L. V. Gool, and D. Dai (2022) Continual test-time domain adaptation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 7201–7211. Cited by: §2. Z. Wu, X. Liu, and I. Gilitschenski (2023) EventCLIP: adapting CLIP for event-based object recognition. arXiv preprint arXiv:2306.06354. External Links: Link Cited by: §2. T. Yu, D. Quillen, Z. He, R. Julian, A. Narayan, H. Shively, A. Bellathur, K. Hausman, C. Finn, and S. Levine (2019) Meta-World: a benchmark and evaluation for multi-task and meta reinforcement learning. arXiv preprint arXiv:1910.10897. External Links: Link Cited by: §5. Z. Yu, P. Sun, and D. F. M. Goodman (2026) Beyond rate coding: surrogate gradients enable spike timing learning in spiking neural networks. Neuromorphic Computing and Engineering 6, p. 014016. External Links: Document, Link Cited by: §2. F. Zenke and S. Ganguli (2018) SuperSpike: supervised learning in multilayer spiking neural networks. Neural Computation 30 (6). External Links: Document, Link Cited by: §2. J. Zhou, X. Zheng, Y. Lyu, and L. Wang (2023a) EventBind: learning a unified representation to bind them all for event-based open-world understanding. arXiv preprint arXiv:2308.03135. External Links: Link Cited by: §2. Z. Zhou, Y. Zhu, C. He, Y. Wang, S. Yan, Y. Tian, and L. Yuan (2023b) Spikformer: when spiking neural network meets transformer. In International Conference on Learning Representations, External Links: Link Cited by: §2. R. Zhu, Q. Zhao, G. Li, and J. K. Eshraghian (2023) SpikeGPT: generative pre-trained language model with spiking neural networks. arXiv preprint arXiv:2302.13939. External Links: Link Cited by: §2. Appendix A Implementation and Extended Evidence A.1 Realized architecture and offline data The complete checkpoint has 1,451,388 parameters: 1,442,527 inherited multimodal parameters and 8,861 action-path parameters. The sparse core uses two layers, width 96, four heads, FFN width 192, membrane decay 0.85, sensory sequence length 16 and top-k 4 causal attention. The control path uses two tokens. The representative joint-training checkpoint is stored in FP32 and has SHA-256 6513bc37d9cd3a5a91dedb478ccd54ce87850e6cd06cb1f3fb9437767dece1f. The multimodal cache uses balanced SHD and SSC examples, paired retinal digit–word examples and video-event examples. The retinal source is sklearn.datasets.load_digits; text is rendered as spatial word events with a modality-independent scanpath. Video events are pooled causal feature changes from a content-disjoint Something-Something-V2 cache. The joint-training study reuses this foundation checkpoint across action seeds, a limitation stated in the main text. A.2 Joint optimization details The action adapter is first trained for 800 updates. The joint-training study then performs 200 shared-core updates with batch size 32 and learning rate 10−410^-4. The action, next-slice, semantic and binding weights are 4, 1, 1 and 0.25. Joint optimization exposes 605,079 trainable parameters and an estimated 4.84 MB of Adam moment state. The stable raw semantic bank remains frozen. For the five action-model seeds, joint action-MSE improvements are 15.68, 11.73, 22.63, 16.13 and 19.30%. Next-slice improvement is positive for all five seeds. Binding change is positive for four seeds and −0.63-0.63 p for one; the paired mean and bootstrap interval remain positive. Relative to action-only shared updating, joint action MSE is 1.08–5.87% higher, within the registered 10% non-inferiority margin. A.3 Fast-state accounting and update The accounted fast-state module contains 6,096 FP32 values: two 96×3096× 30 route-specific residual matrices (5,760 values), 32 cumulative candidate scores, one transition count, and 303 inert entries that preserve the 24,384-byte comparison budget. Of these, 5,793 values (23,172 bytes) are active; the remainder is padding. The module retains no optimizer moments or replay. The reference runner additionally appends a 247-value router signature and 32 candidate losses per transition to Python lists. These episodic buffers are not included in the 24,384-byte module accounting and grow with episode length; at the four-observation routing point they contain 1,116 FP32 values (4,464 bytes), excluding Python-object overhead. The candidate bank contains source identity, 15 shear values from 0.05 to 0.75, and 16 attenuation values from 0.20 to 0.95. For each causal transition the evaluator first issues a prediction, then receives the endpoint, updates cumulative candidate losses and, if routed to a registered shift, takes one normalized-gradient residual-matrix step with learning rate 0.02 and radius 0.20. The first four actions are unchanged; observation four completes the router signature and triggers the first write. The safe inverse caps every coordinate correction at 0.05 and abstains if the proposed command exceeds the action range. A.4 Registered-shift per-task and per-seed checks Shear action-tracking improvement by task is 15.46% (door-open), 24.79% (pick-place), 9.78% (push) and 46.75% (reach). Its five model-seed means are 32.45, 21.16, 29.13, 17.71 and 20.53%. Attenuation task means are 2.18, 1.03, 0.88 and 11.67%, and model-seed means range from 3.58 to 4.13%. The action advantage over action-shuffled controls is 0.004126, CI [0.002958, 0.005314] for shear and 0.001651, CI [0.000256, 0.003184] for attenuation. A.5 Closed-loop task-level details The paired safe-fast reward gains are +2.31 on pick-place, +11.61 on push and +9.79 on reach. Frozen/safe-fast success rates are respectively 80/80%, 60/80% and 20/40%. The five model-seed reward gains are 7.55, 8.52, 7.61, 7.19 and 8.66. SpikeWorld and full tuning have identical aggregate success; their paired reward difference is +0.40, CI [-0.30, 1.63]. The adaptation advantage is zero through decision four; the first changed-action effect appears at decision five after four observations. Mean cumulative reward advantages at decisions 5, 8, 11 and 13 are 0.06, 2.41, 5.54 and 7.90. The observed success difference appears at decision 12 and reaches 13.33 p at decision 13. A.6 Deployment and reproducibility audit The registered-shift study collects 80 fresh episodes and stores twenty online shards with fields current_observations, behavior_actions, observed_endpoint_observations, episode_seeds, steps and stream_indices. Mechanism values and stage labels are confined to a separately hashed audit manifest. The closed-loop study creates exactly 450 environment instances after writing a pre-environment lock. Its controller update function receives four positional objects: state, commanded action, task and endpoint. Parameter and dependency hashes are checked again after the run. The complete machine-readable evidence is: • sensory/manifest.json; • results/joint.json; • results/adaptation.json; • results/control.json; • results/deployment_audit.json. Code availability. The reference implementation, final experiment configurations, deployment audit, tests and locked paper-result summary are publicly available at https://github.com/Oooorca/SpikeWorld. Large checkpoints and raw trajectories are described by the repository artifact manifest and are not stored in Git. A.7 Component ablations and diagnostic controls Table˜4 collects the component evidence used in the paper. The three blocks answer different questions and are not intended as one cross-protocol ranking: joint training isolates offline shared-core optimization, closed-loop control compares online alternatives under constant attenuation, and the registered-shift study diagnoses routing and action safety. Table 4: Component ablations and controls. The reported values are means for their stated protocol. “Unchanged” refers to frozen parameter and identical-input semantic-output hashes. Protocol Arm Observed consequence Joint training Sequential frozen core Action MSE 0.1463; binding 10.00%. Action-only update Action MSE 0.1174; binding falls to 8.50%. Joint objective Action/next MSE 0.1212/0.6725; semantic/binding 20.94/11.81%. Closed-loop Frozen Reward 414.68; success 53.33%. Full tuning Reward 422.19; success 66.67%; semantic hashes change. RLS Reward 434.41; success 65.33%; strongest linear-fault reward. Replay Reward 428.59; success 60.00%; reward CI versus frozen crosses zero. Fast-state module Reward 422.58; success 66.67%; frozen hashes unchanged. Uncapped predecessor Uncapped inverse Fresh shear action gate fails. Shift adaptation Bounded inverse Both shifts have positive action means for every model seed and task. Action-shuffled Correct pairing advantages: 0.004126 shear; 0.001651 attenuation. A.8 Negative results and claim boundary Uncapped inversion failed the registered shear action metric in a held-out predecessor study even though pooled absolute tracking improved. Earlier continual streams failed to suppress writes on return to the source environment, with suppression between 9% and 51%. Finally, closed-loop RLS has a larger mean reward gain than SpikeWorld on constant attenuation. These observations exclude claims of automatic prediction-to-action transfer, generic source recognition, open-world mechanism discovery and dominance over classical linear identification. “Unified” denotes joint optimization through one shared core; it does not denote identical sensor physics. “Sparse” denotes executed attention pairs; it does not imply measured chip energy. “Online adaptation” denotes external state estimation under registered shifts; it does not denote online policy learning or biologically realistic continual perception.

Instant research discovery