Paper deep dive
NeuronSpark: A Spiking Neural Network Language Model with Selective State Space Dynamics
Zhengzheng Tang
Intelligence
Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 93%
Last extracted: 3/22/2026, 5:37:47 AM
Summary
NeuronSpark is a 0.9B-parameter Spiking Neural Network (SNN) language model trained from random initialization. It utilizes selective state-space spiking dynamics, leakage-current inter-layer communication, and PonderNet adaptive timesteps to achieve language modeling capabilities without Transformer distillation. The architecture incorporates stabilization techniques like residual centering and lateral-inhibition normalization, implemented with custom Triton-fused kernels for efficiency.
Entities (5)
Relation Signals (3)
NeuronSpark → implementstechnique → PonderNet
confidence 95% · We design PonderNet adaptive timesteps at each sublayer
NeuronSpark → usesarchitecture → PLIF
confidence 95% · All neurons in NeuronSpark follow the Parametric Leaky Integrate-and-Fire (PLIF) model.
NeuronSpark → developedby → Zhengzheng Tang
confidence 90% · NeuronSpark: A Spiking Neural Network Language Model with Selective State Space Dynamics Zhengzheng Tang
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:We ask whether a pure spiking backbone can learn large-scale language modeling from random initialization, without Transformer distillation. We introduce NeuronSpark, a 0.9B-parameter SNN language model trained with next-token prediction and surrogate gradients. The model combines selective state-space spiking dynamics, leakage-current inter-layer communication, PonderNet adaptive timesteps, fused Triton PLIF kernels, and stabilization techniques (residual centering, lateral-inhibition normalization, and natural-gradient compensation). Under a constrained budget (about 1.4B pretraining tokens and 6.5K SFT steps), NeuronSpark-0.9B reaches 3.6 pretraining loss and shows early multi-turn dialogue behavior after SFT. These results support the feasibility of end-to-end language modeling with a pure SNN architecture at this scale.
Tags
Links
- Source: https://arxiv.org/abs/2603.16148v1
- Canonical: https://arxiv.org/abs/2603.16148v1
Trouble viewing inline? Open PDF directly →
Full Text
45,938 characters extracted from source content.
Expand or collapse full text
NeuronSpark: A Spiking Neural Network Language Model with Selective State Space Dynamics Zhengzheng Tang111Code: https://github.com/Brain2nd/NeuronSpark-V1.NeuronSpark-0.9B (HuggingFace): https://huggingface.co/Brain2nd/NeuronSpark-0.9B.NeuronSpark-0.9B (ModelScope): https://w.modelscope.ai/models/Brain2nd/NeuronSpark-0.9B.NeuronSpark-0.9B-Chat (HuggingFace): https://huggingface.co/Brain2nd/NeuronSpark-0.9B-Chat.NeuronSpark-0.9B-Chat (ModelScope): https://w.modelscope.ai/models/Brain2nd/NeuronSpark-0.9B-Chat. Boston University zztangbu@bu.edu Abstract We ask whether a pure spiking backbone can learn large-scale language modeling from random initialization, without Transformer distillation. We introduce NeuronSpark, a 0.9B-parameter SNN language model trained with next-token prediction and surrogate gradients. The model combines selective state-space spiking dynamics, leakage-current inter-layer communication, PonderNet adaptive timesteps, fused Triton PLIF kernels, and stabilization techniques (residual centering, lateral-inhibition normalization, and natural-gradient compensation). Under a constrained budget (about 1.4B pretraining tokens and 6.5K SFT steps), NeuronSpark-0.9B reaches 3.6 pretraining loss and shows early multi-turn dialogue behavior after SFT. These results support the feasibility of end-to-end language modeling with a pure SNN architecture at this scale. NeuronSpark: A Spiking Neural Network Language Model with Selective State Space Dynamics Zhengzheng Tang111Code: https://github.com/Brain2nd/NeuronSpark-V1.NeuronSpark-0.9B (HuggingFace): https://huggingface.co/Brain2nd/NeuronSpark-0.9B.NeuronSpark-0.9B (ModelScope): https://w.modelscope.ai/models/Brain2nd/NeuronSpark-0.9B.NeuronSpark-0.9B-Chat (HuggingFace): https://huggingface.co/Brain2nd/NeuronSpark-0.9B-Chat.NeuronSpark-0.9B-Chat (ModelScope): https://w.modelscope.ai/models/Brain2nd/NeuronSpark-0.9B-Chat. Boston University zztangbu@bu.edu 1 Introduction Large language models (LLMs) based on Transformers (Vaswani et al., 2017) have achieved remarkable success across natural language processing tasks. However, their quadratic attention mechanism and dense floating-point computation raise fundamental questions about computational efficiency and biological plausibility. Meanwhile, spiking neural networks (SNNs) (Maass, 1997) — the “third generation” of neural networks — process information through discrete spikes and temporal dynamics, offering potential advantages in energy efficiency and neuromorphic hardware deployment. Despite significant progress in SNN-based vision models, SNN language modeling remains underdeveloped. This gap is important because language is a central benchmark for general sequence modeling; without evidence at language-model scale, claims about SNNs as a practical alternative to dense Transformer computation remain limited. Existing approaches such as SpkGPT (Zhu et al., 2024), SpkBERT (Bal and Sengupta, 2024), and SpkBERT-110M (Lv et al., 2023) either rely on distillation from pretrained Transformers, retain non-spiking components in critical stages, or remain at relatively small model scale. Consequently, the field still lacks a clear answer to the following question: Can a pure SNN architecture learn language from random initialization at meaningful scale under standard next-token training? In this work, we address this gap by introducing NeuronSpark, a 0.9B-parameter SNN language model trained from random initialization. Given the available compute budget (8× RTX 4090 GPUs), we train on approximately ∼ 1.4B tokens from a 10B-token corpus; despite this constraint, the model exhibits non-trivial language generation and dialogue behavior. Our key technical insight is that the membrane potential dynamics of Leaky Integrate-and-Fire (LIF) neurons can be formulated as a selective state space model (Gu and Dao, 2024), where the decay rate β, input gain α, and firing threshold VthV_th serve as input-dependent gating mechanisms analogous to Mamba’s selection mechanism. This perspective enables us to design an end-to-end spiking language architecture that is both trainable at scale and interpretable through the SSM lens. A key modeling choice is to treat layer-to-layer signals as floating-point leakage-current signals, while retaining 0/1 spikes as the internal neuronal event process; this distinction avoids the expressivity bottleneck of purely binary inter-layer communication. Contributions. 1. We propose the Selective State Space SNN Block with 7 parallel projection paths, computing dynamic β(t),α(t),Vth(t)β(t),α(t),V_th(t) from input signals through learned modulation networks with structured initialization, establishing a formal SNN–SSM duality (Section 3.4). 2. We introduce leakage-current activation (1−β)⋅Vpost(1-β)· V_post as the default inter-layer signal for PLIFNode boundaries, which naturally emphasizes fast-responding neurons and provides implicit temporal-scale weighting (Section 3.3). 3. We design PonderNet adaptive timesteps at each sublayer, enabling per-token dynamic SNN computation depth with geometric-distribution weighting and ponder cost regularization (Section 3.6). 4. We develop Triton-fused PLIF kernels with per-element and row-parameter variants, performing the entire PLIF forward/backward (including surrogate gradient) in a single kernel launch (Section 4.3). 5. We introduce residual centering and lateral inhibition normalization as SNN-native stabilization techniques, along with a two-phase natural gradient compensation for modulation parameters (Sections 4.1–4.4). 6. We train and release NeuronSpark-0.9B under a constrained data budget, and provide evidence that a pure SNN can acquire non-trivial language modeling ability from random initialization (Section 5). Collectively, these contributions target a single bottleneck in prior work: the lack of a scalable, end-to-end spiking recipe for language modeling that is both theoretically grounded and practically trainable. Figure 1: NeuronSpark architecture overview. The residual stream carries continuous values h; PLIFLeak denotes PLIF neurons with leakage activation (1−β)⋅Vpost(1-β)· V_post. PonderNet aggregation (applied per sublayer) collapses K frames per token with learned geometric-distribution weights. Inter-layer communication uses floating-point leakage-current signals; binary spikes are internal firing events rather than the default layer-to-layer representation. The decode stage uses uniform K-frame mean. Residual centering (subtract per-token mean) is applied before each residual addition. 2 Related Work Spiking Neural Networks for Language. Prior work can be grouped by the specific gap it leaves unaddressed. Distillation dependence: SpkBERT (Bal and Sengupta, 2024) and SpkBERT-110M (Lv et al., 2023) transfer representations from pretrained ANN/Transformer models, which reduces evidence that language competence can emerge from fully spiking training dynamics. Partial spiking pipelines: SpkGPT (Zhu et al., 2024) demonstrates generative behavior with spike-based hidden computation, but still retains non-spiking components (e.g., embedding/output stages), leaving end-to-end spiking feasibility unresolved. Scale limitations: existing studies are typically limited to ≤ 216M parameters, well below contemporary language-model regimes. Our work targets these three gaps jointly by training a 0.9B model from random initialization with standard next-token prediction and spiking dynamics throughout the core sequence-processing stack. State Space Models. Structured State Spaces (S4) (Gu et al., 2022) introduced efficient linear recurrence for sequence modeling. Mamba (Gu and Dao, 2024) added input-dependent selection, achieving Transformer-competitive performance. Mamba-2 (Dao and Gu, 2024) established a formal duality between SSMs and attention. We observe that SNN membrane dynamics V[t]=β(t)⋅V[t−1]+α(t)⋅I[t]V[t]=β(t)· V[t-1]+α(t)· I[t] are structurally identical to the selective SSM recurrence, with β as the decay coefficient and α as the input gate. The spike-and-reset mechanism adds a discrete nonlinearity absent in continuous SSMs. Adaptive Computation. Adaptive Computation Time (ACT) (Graves, 2016) allows networks to vary computation per input. PonderNet (Banino et al., 2021) improved upon ACT with a geometric distribution prior. We apply PonderNet at the SNN timestep level within each sublayer: the K frames per token are aggregated with learned halt probabilities, enabling different tokens to use 1 to KmaxK_ effective SNN steps. Surrogate Gradient Training. The non-differentiability of spike generation (Θ(V−Vth) (V-V_th)) is addressed by surrogate gradient methods (Neftci et al., 2019; Zenke and Vogels, 2021), which replace the Heaviside derivative with smooth approximations. NeuronSpark uses Sigmoid surrogate gradients (α=4.0α=4.0) throughout, implemented in the SpikingJelly framework (Fang et al., 2023), with custom Triton kernels that fuse the surrogate computation into the sequential scan. 3 Architecture This section is organized around one central method question: how to make a pure SNN language model simultaneously expressive, trainable, and scalable. Our design follows a four-step logic: (1) define a stable neuron-level state update, (2) choose an inter-layer signal that avoids binary-communication bottlenecks, (3) build a selective sequence block on top of that signal, and (4) add system-level training stabilizers so optimization remains tractable at 0.9B scale. 3.1 Overview NeuronSpark follows a three-stage pipeline (Figure 1): (1) Encode: Token IDs → embedding (D-dim) → repeat K times, producing a (T⋅K,B,D)(T·K,B,D) tensor. Gradients flow directly through embedding; the output head reuses the embedding matrix (weight tying (Press and Wolf, 2017)). (2) SNN Forward: L=20L=20 decoder layers with gradient checkpointing, each containing an SNNBlock (attention analogue) and an SNNFFN (MLP analogue), with PonderNet adaptive K-frame aggregation. All neuron states reset per sequence. (3) Decode: RMSNorm (Zhang and Sennrich, 2019) → output PLIFNode (leakage) → K-frame uniform mean → projection → lateral inhibition (Carandini and Heeger, 2012) → tied head → logits. Each decoder layer follows a Pre-LN residual pattern matching Qwen3/LLaMA (Touvron et al., 2023; Yang et al., 2025): ←+center(OutProj(PonderAgg(SNNBlock(⋯)))) +center (OutProj(PonderAgg(SNNBlock(·s))) ) (1) ←+center(OutProj(PonderAgg(SNNFFN(⋯)))) +center (OutProj(PonderAgg(SNNFFN(·s))) ) (2) where center()=−mean()center(x)=x-mean(x) is residual centering (Section 4.1). The residual stream ∈ℝTK×B×Dh ^TK× B× D carries continuous values throughout; only the SNN sublayers operate on spike/membrane dynamics. The remainder of this section instantiates this logic in order: PLIF dynamics define the base state transition, leakage-current activation defines the default inter-layer representation, SNNBlock/SNNFFN define sequence computation, and the final subsections describe optimization-oriented stabilizers. 3.2 PLIF Neuron Dynamics All neurons in NeuronSpark follow the Parametric Leaky Integrate-and-Fire (PLIF) model (Fang et al., 2021). This subsection provides the dynamical foundation on which all later architectural choices are built. We distinguish two variants: PLIFNode (fixed parameters). Used at layer boundaries (input neurons, gate/up neurons, output neuron). Each has D-dimensional (or DffD_f-dimensional) learnable parameters: Vpre[t] V_pre[t] =β⋅Vpost[t−1]+(1−β)⋅x[t] =β· V_post[t-1]+(1-β)· x[t] (3) s[t] s[t] =Θ(Vpre[t]−Vth) = (V_pre[t]-V_th) (4) Vpost[t] V_post[t] =Vpre[t]−Vth⋅s[t] =V_pre[t]-V_th· s[t] (5) where β=σ(w)∈(0,1)β=σ(w)∈(0,1) with w∼(logit(1−1/τ0), 0.5)w (logit(1-1/ _0),\;0.5) and Vth∼(0.5v0, 1.5v0)V_th (0.5v_0,\;1.5v_0). The random initialization creates diversity across dimensions: different neurons have different time constants and firing sensitivities. Equation (5) implements soft reset: the membrane potential is reduced by VthV_th upon firing, preserving residual charge. SelectivePLIFNode (dynamic parameters). Used inside SNNBlock for D⋅ND· N hidden neurons. Parameters β(t),α(t),Vth(t)β(t),α(t),V_th(t) are computed per-step from the input (Section 3.4): V[t] V[t] =β(t)⋅V[t−1]+α(t)⋅I[t] =β(t)· V[t-1]+α(t)· I[t] (6) s[t] s[t] =Θ(V[t]−Vth(t)) = (V[t]-V_th(t)) (7) V[t] V[t] -=Vth(t)⋅s[t] -=V_th(t)· s[t] (8) This is structurally identical to Mamba’s selective SSM recurrence h[t]=A¯(t)⋅h[t−1]+B¯(t)⋅x[t]h[t]= A(t)· h[t-1]+ B(t)· x[t], with the addition of the spike-and-reset nonlinearity. 3.3 Membrane Potential Leakage Activation A critical design choice is the signal transmitted between components. This is the key bridge from neuron dynamics to network-level information flow. Standard SNN practice uses binary spikes s[t]∈0,1s[t]∈\0,1\, but this severely limits gradient flow through the surrogate function’s narrow support. An alternative is the raw membrane potential VpostV_post, but this treats all neurons equally regardless of their temporal dynamics. We use leakage-current activation as the default inter-layer signal. In other words, unless explicitly stated otherwise, downstream layers consume floating-point leakage-current signals (bioelectric-state proxies) rather than binary spikes: leak[t]=(1−β)⋅Vpost[t]leak[t]=(1-β)· V_post[t] (9) This quantity is the amount of membrane potential that will dissipate due to exponential decay before the next input arrives. Biologically, it corresponds to the leak current through the membrane conductance (Hodgkin and Huxley, 1952; Abbott, 1999). This leakage-current activation provides natural temporal-scale weighting: neurons with large (1−β)(1-β) (fast dynamics, short memory) produce proportionally larger signals, while neurons with small (1−β)(1-β) (slow dynamics, long memory) are implicitly attenuated. This reweighting is applied at all PLIFNode outputs: input neurons (2 per layer), gate/up neurons in SNNFFN (2 per layer), and the output neuron. The SelectivePLIFNode hidden neurons inside SNNBlock output raw VpostV_post rather than leakage, because β(t)β(t) is dynamic (varies per step) and cannot be absorbed into a static downstream weight matrix. This is a deliberate design choice: leakage scaling applies only at fixed-β boundaries. 3.4 Selective State Space SNN Block With neuron dynamics and inter-layer signaling fixed, we next define the core sequence module. The SNNBlock is the attention analogue, processing input through D⋅ND· N hidden spiking neurons with input-dependent parameters. It computes seven parallel projections from the input leakage signal — six input projections and one output projection: Input projections (D→D⋅ND→ D· N or D→D→ D): [t] [t] =Win⋅leak[t] =W_in·leak[t] (10) β(t) β(t) =σ(Wβ⋅leak[t]+β) =σ(W_β·leak[t]+b_β) (11) α(t) α(t) =softplus(Wα⋅leak[t]+α) =softplus(W_α·leak[t]+b_α) (12) Vth(t) V_th(t) =Vmin+|Wth⋅leak[t]+th| =V_min+|W_th·leak[t]+b_th| (13) [t] [t] =σ(Wgate⋅leak[t]) =σ(W_gate·leak[t]) (14) skip[t] _skip[t] =Wskip⋅leak[t] =W_skip·leak[t] (15) The modulation projections Wβ,Wα,WthW_β,W_α,W_th are initialized at 0.1×0.1× the scale of WinW_in, ensuring that β(t),α(t),Vth(t)β(t),α(t),V_th(t) are dominated by their respective biases at the start of training, providing a stable initialization. Hidden neuron dynamics. The D⋅ND· N hidden neurons follow SelectivePLIF (Eqs. 6–8), computed via fused Triton PLIF kernels (Section 4.3). Output projection (D⋅N→D· N→ D): out[t]=Wout⋅Vpost[t]⊙[t]+skip[t]out[t]=W_out· V_post[t] [t]+I_skip[t] (16) Note: the output uses VpostV_post (not leakage) from the hidden neurons, because β(t)β(t) is dynamic. The gate g provides multiplicative control over which dimensions pass through. The skip connection skipI_skip ensures gradient flow even when all hidden neurons are silent. Structured initialization. The modulation biases β,α,thb_β,b_α,b_th receive carefully designed initialization (details in Appendix A): βb_β is logit-spaced across N groups targeting β∈[0.80,0.99]β∈[0.80,0.99] (multi-timescale); αb_α is initialized near softplus−1(1.0)softplus^-1(1.0) so initial α≈1α≈ 1; thb_th is calibrated from stationary variance σV=p/3⋅1−β2K _V= p/3· 1-β^2K with target firing rates 25%–8% across N groups; WinW_in rows are scaled by 1−β2 1-β^2 per group; WoutW_out columns are scaled by 1/pfire1/ p_fire per group. 3.5 SNN Feed-Forward Network The SNNFFN mirrors the SwiGLU MLP (Touvron et al., 2023) with spiking neurons replacing the activation function: gate_leak =(1−βg)⋅Vpost(PLIFgate(Wgate⋅leak)) =(1- _g)· V_post(PLIF_gate(W_gate·leak)) (17) up_leak =(1−βu)⋅Vpost(PLIFup(Wup⋅leak)) =(1- _u)· V_post(PLIF_up(W_up·leak)) (18) out =Wdown⋅(gate_leak⊙up_leak) =W_down·(gate\_leak \_leak) (19) +Wskip⋅leak +W_skip·leak (20) The element-wise product of two leakage signals replaces SiLU(x)⊙xSiLU(x) x gating in SwiGLU (Shazeer, 2020). Both PLIF neurons provide implicit nonlinearity through the integrate-fire-reset cycle; their leakage outputs carry temporal dynamics that pure activation functions cannot express. WdownW_down is initialized with 1/L1/ L scaling to prevent gradient explosion through deep residual chains. 3.6 PonderNet Adaptive Timesteps Each token is represented as K SNN frames. Rather than uniformly averaging all K frames, we learn per-frame halt probabilities following PonderNet (Banino et al., 2021): pk p_k =σ(Whalt⋅framek+bhalt)∈(0,1) =σ(W_halt·frame_k+b_halt)∈(0,1) (21) Sk S_k =∏j=1k−1(1−pj)(survival probability) = _j=1^k-1(1-p_j) (survival probability) (22) λk _k =pk⋅Sk,λ^k=λk/∑k′λk′ =p_k· S_k, λ_k= _k/ _k _k (23) output =∑kλ^k⋅framek,[K]=∑k⋅λ^k = _k λ_k·frame_k, [K]= _kk· λ_k (24) [K]E[K] serves as a ponder cost regularizer (λponder=0.01 _ponder=0.01). PonderNet is applied independently at each sublayer (2L=402L=40 aggregation points). WhaltW_halt is initialized with Xavier uniform ×0.01× 0.01 and bhalt=−3.5b_halt=-3.5 (σ(−3.5)≈0.03σ(-3.5)≈ 0.03), so PonderNet starts near-uniform and gradually specializes. After aggregation, the result is projected through OutProj (D→D→ D, no bias), then broadcast back to K frames for residual addition. 4 Stabilization and Efficient Implementation 4.1 Residual Centering Each sublayer’s output projection is mean-subtracted before residual addition: center()=−1D∑d=1Dxdcenter(x)=x- 1D _d=1^Dx_d. This eliminates DC drift that would otherwise accumulate across 20 residual layers. 4.2 Lateral Inhibition Normalization The output layer uses lateral inhibition (divisive normalization): LateralInhib()=γ⋅/1D∑dhd2+ϵLateralInhib(h)=γ·h/ 1D _dh_d^2+ε, where γ∈ℝDγ ^D is a learnable gain. This is mathematically equivalent to RMSNorm (Zhang and Sennrich, 2019) but corresponds to divisive normalization (Carandini and Heeger, 2012). We implement it as a fused Triton kernel. 4.3 Triton Fused PLIF Kernels The PLIF recurrence involves a sequential scan that cannot be trivially parallelized due to spike-and-reset. We implement two variants of fused Triton (Tillet et al., 2019) kernels: Per-element kernel (SelectivePLIFNode, dynamic β[k],Vth[k]β[k],V_th[k]): single-pass sequential scan with inline charge–fire–reset and Sigmoid surrogate gradient in the backward pass. All arithmetic in fp32 with bf16 storage. Row-parameter kernel (PLIFNode, fixed β,Vthβ,V_th): parameters loaded once into registers, reducing global memory reads from 3 per step to 1, yielding ∼ 40% speedup. Backward kernel accumulates ∇β,∇Vth _β, _V_th in registers. CPU fallback: 3-phase approach via Hillis-Steele parallel prefix scan (Blelloch, 1990; Martin and Cundy, 2018), spike fixed-point iteration, and surrogate gradient re-computation. 4.4 Natural Gradient Compensation The modulation biases β,α,thb_β,b_α,b_th suffer from two gradient pathologies. We apply compensation after gradient unscaling and before gradient clipping: Phase 1: Activation saturation. ∇bβ←∇bβ/max(β(1−β), 1/Cmax) _b_β← _b_β/ (β(1-β),\;1/C_ ), effectively performing gradient descent in β-space. Similarly for α: ∇bα←∇bα/max(σ(bα),0.1) _b_α← _b_α/ (σ(b_α),0.1). Phase 2: Cross-layer equalization. For each modulation parameter type, normalize per-layer gradient norms to the geometric mean: ∇layeri←∇layeri⋅GeoMean(‖∇1‖,…,‖∇L‖)/‖∇i‖ _layer_i← _layer_i·GeoMean(\| _1\|,…,\| _L\|)/\| _i\|. 5 Experiments 5.1 Setup Model configuration. NeuronSpark-0.9B: D=896D=896, N=8N=8, K=16K=16, L=20L=20, Dff=2688D_f=2688, 6144-token BPE (Sennrich et al., 2016) vocabulary, 874M parameters. Datasets. Pretraining: Seq-Monkey (Mobvoi, 2023) (∼ 29M samples, ∼ 10B tokens). SFT: BelleGroup train_3.5M_CN (BelleGroup, 2023) (∼ 3.5M conversations). Compute constraints. All training was conducted on 8× NVIDIA RTX 4090 GPUs. Due to limited compute, we train on small subsets: Table 1: Dataset utilization. Stage Full Dataset Actual Used Fraction Pretrain ∼ 10B tokens 85K steps (∼ 1.4B tokens) ∼ 14% SFT ∼ 3.5M conversations 6.5K steps (∼ 42K samples) ∼ 1.2% Training details. Pretraining: Adam (Loshchilov and Hutter, 2019), peak lr=2×10−4=2×10^-4, 1000-step warmup, cosine decay, gradient accumulation 8, effective batch 64, bfloat16 (Micikevicius et al., 2018), gradient checkpointing (Chen et al., 2016). Neuron parameters receive 10×10× base lr. SFT: AdamW (lr=5×10−5=5×10^-5, weight decay 0.01), training only on assistant response tokens. 5.2 Results Table 2: Training results for NeuronSpark-0.9B. Metric Pretrain SFT Training loss 3.6 2.1 Parameters 874M Training steps 85,000 6,500 Tokens seen ∼ 1.4B ∼ 0.4B Hardware 8× NVIDIA RTX 4090 Figure 2: Pretraining loss curve over 85K steps (∼ 1.4B tokens). Loss decreases from 9.0 to ∼ 3.5. Training throughput: ∼ 960 tokens/sec on 8× RTX 4090 GPUs. Qualitative evaluation. After SFT, the model demonstrates basic Chinese dialogue (translated; model outputs in Chinese): Q: What is the capital of China? A: The capital of China is Beijing. Q: Hello! A: How can I help you? These observations suggest that a pure SNN architecture can support coherent language generation from random initialization, even under limited-data training. 5.3 Architecture Ablation via Training Stability During development, we explored multiple architectural variants (each trained 1K–12K steps). Table 3 summarizes 7 variants; Figure 3 shows loss curves. Table 3: Ablation variants. All stagnated above loss 7.0; only the final architecture reached 3.5. Variant Steps Loss What Changed Final V1 85K 3.5 Full architecture MPD-AGL + no Phase 2 4.8K 7.21 Adaptive surrogate gradient, removed cross-layer equalization E[K] floor 1.2K 7.47 Added minimum E[K] floor Bounded α 5.1K 7.47 Bounded gain multiplier HC α (decoupled) 3.7K 7.44 Separate α parameter Sinkhorn health 2.1K 7.62† Sinkhorn-projected health score Cortical lateral 4.1K 7.66† Cross-token causal spike propagation No gradient sync 0.6K NaN Missing gradient synchronization † (loss >>7.5). Figure 3: Training loss: final architecture (blue) vs. 9 ablation variants. None of the ablation variants achieves a loss below 7.0. 5.4 Comparison with Existing SNN Language Models Table 4: Comparison with existing SNN language models. Model Par. From Core Gen. Dia. SpkBERT-110M 110M ✗ ✓ ✗ ✗ SpkBERT 110M ✗ ✓ ✗ ✗ SpkGPT 216M ✓ ✓ ✓ ✗ NeuronSpark-0.9B 874M ✓ ✓ ✓ ✓ To complement aggregate training metrics, we next analyze how computation is allocated internally and what linguistic structure the model has learned. 5.5 Biological Interpretability Analysis We analyze the trained NeuronSpark-0.9B-Chat model to examine whether its learned SNN dynamics exhibit linguistically and biologically meaningful patterns. All analyses are conducted on 40 Chinese sentences spanning science, daily life, education, economics, and complex multi-clause constructions. Figure 4 presents the four main findings. Figure 4: Biological interpretability of NeuronSpark-0.9B-Chat. (a) Per-token E[K]: punctuation receives fewer steps than content words. (b) POS-level E[K]: function words/punctuation are lower by about ∼ 0.7. (c) Per-layer E[K]: SNNBlock increases with depth, SNNFFN stays near 7–8. (d) Learned β distribution: 67.3% fast (<0.9<0.9), 32.7% slow (≥0.9≥ 0.9). Computation allocation is structural, not predictive. A natural hypothesis is that PonderNet allocates more SNN steps to tokens that are harder to predict (high surprisal =−logP(next token)=- P(next token)). Figure 5 tests this directly on 541 tokens (40 sentences). Naïvely, surprisal and E[K] appear negatively correlated (r=−0.50r=-0.50); however, this is entirely driven by the BOS (beginning-of-sequence) sentinel token, which has extremely low E[K] (3.2) and high surprisal (8.9) by construction. Excluding BOS tokens, the correlation drops to r=−0.12r=-0.12 (near zero), and binned analysis confirms that mean E[K] is essentially flat (∼ 7.4–7.9) across all surprisal ranges. This reveals that PonderNet’s computation budget is governed by structural/syntactic role rather than predictive difficulty: punctuation and function words receive fewer steps not because they are easy to predict, but because they play a structurally simpler role in the sequence. This is consistent with biological findings that neural processing effort correlates more with syntactic complexity than with statistical predictability (Neftci et al., 2019). Figure 5: Surprisal vs. E[K] (40 Chinese sentences, 541 tokens). (a) The apparent correlation is dominated by BOS tokens: r=−0.50r=-0.50 overall, r=−0.12r=-0.12 without BOS. (b) Binned E[K] is nearly flat across surprisal, indicating allocation is largely independent of predictive difficulty. Reasoning capability assessment. To further characterize what the model has and has not learned, we test on 28 questions across four categories: arithmetic (8), logical reasoning (6), commonsense (8), and dialogue coherence (6). Figure 6 summarizes the results. The model achieves 0% on arithmetic (unable to perform any calculation), 25% on commonsense (mostly coincidental keyword matches), and 83% on logic (though inspection reveals many “correct” answers arise from the expected keyword appearing in repetitive output rather than genuine inference). By contrast, all 6 coherence tests produce fluent, grammatical Chinese responses, confirming that the model has acquired surface-level language generation ability. Critically, panel (b) shows that E[K] is flat (∼ 7.6) across all categories, regardless of task difficulty. The model does not allocate additional SNN computation for harder reasoning tasks, further confirming that PonderNet’s adaptive computation is driven by structural token properties (Section 3.3) rather than semantic reasoning demands. These results are consistent with the limited training budget: the model has learned structural language patterns (fluency, POS-dependent computation) but has not yet acquired the factual knowledge or compositional reasoning that would require substantially more training data. Figure 6: Reasoning capability assessment (28 questions). (a) Accuracy by category: arithmetic 0%, logic 83% (superficial), commonsense 25%, coherence 6/6. (b) E[K] and surprisal are flat across categories, indicating no adaptive computation increase for harder tasks. The model has learned structural language patterns but not reasoning. Adaptive computation aligns with linguistic complexity. Panel (a) shows that PonderNet assigns systematically fewer SNN timesteps to punctuation (E[K]≈ 5.7) and function words (E[K]≈ 7.4) than to content words (nouns 8.0, verbs 8.0, adjectives 8.2). This pattern — emerging without any explicit linguistic supervision — mirrors the intuition that structurally predictable tokens require less neural computation. The BOS token receives the fewest steps (E[K]=3.2), consistent with it being a fixed sentinel requiring no contextual processing. Depth-dependent computation budget. Panel (c) reveals a striking asymmetry: SNNBlock E[K] increases monotonically with layer depth (from ∼ 4 at layer 2 to ∼ 12.7 at layer 19), while SNNFFN E[K] remains relatively flat (∼ 7–8). This suggests that deeper layers require more SNN timesteps for the attention-analogue computation (SNNBlock) but not for the feed-forward transformation (SNNFFN). A possible interpretation is that deeper layers perform more complex contextual integration, requiring longer membrane-potential evolution, while the point-wise nonlinear transformation in SNNFFN saturates at a fixed computation depth. Multi-timescale neuron specialization. Panel (d) shows that the 143,360 hidden neurons self-organize into fast-responding (β<0.9β<0.9, 67.3%) and slow-memory (β≥0.9β≥ 0.9, 32.7%) populations. This is reminiscent of biological cortical circuits where fast-spiking interneurons coexist with regular-spiking pyramidal cells operating at different timescales. The distribution is unimodal with a long right tail, indicating that the model learns a continuum of timescales rather than a sharp dichotomy, with a preference for faster dynamics. 6 Discussion Our central claim is that large-scale language modeling in a pure SNN regime is not only conceptually plausible but empirically attainable when the architectural design directly addresses the optimization and expressivity gaps left by prior SNN language studies. Beyond feasibility, our interpretability analyses (Section 5.5) reveal that the trained model develops computational strategies with striking parallels to biological neural processing. SNN–SSM duality. The selective PLIF dynamics establish a direct correspondence with Mamba’s selective SSM: β(t)↔A¯(t)β(t) A(t), α(t)↔B¯(t)α(t) B(t), V[t]↔h[t]V[t] h[t]. The spike-and-reset mechanism introduces a hard, input-dependent nonlinearity absent in continuous SSMs. Biological interpretability: structure before semantics. Three complementary experiments (Section 5.5) paint a coherent picture of what the model learns and how it allocates neural resources: (1) Resource allocation mirrors syntactic role, not predictive difficulty. PonderNet assigns systematically fewer SNN steps to punctuation and function words than to content words (nouns, verbs, adjectives), mirroring how biological cortical circuits allocate differential processing effort based on stimulus structural complexity rather than statistical surprise (Carandini and Heeger, 2012). Critically, E[K] is uncorrelated with token surprisal (r=−0.12r=-0.12 after excluding the BOS sentinel), confirming that adaptive computation is governed by syntactic role rather than prediction error — a pattern consistent with neurolinguistic findings that neural processing load correlates more with syntactic complexity than with information-theoretic surprisal. (2) Hierarchical computation depth resembles cortical processing. Deeper layers allocate progressively more SNN timesteps (SNNBlock E[K] increases from ∼ 4 at layer 2 to ∼ 12.7 at layer 19), while SNNFFN E[K] remains stable (∼ 7–8). This asymmetry parallels the cortical hierarchy where higher-order areas exhibit longer temporal integration windows, and point-wise transformations (analogous to SNNFFN) saturate at a fixed processing depth. (3) Multi-timescale neuron specialization. The 143,360 hidden neurons self-organize into fast-responding (β<0.9β<0.9, 67.3%) and slow-memory (β≥0.9β≥ 0.9, 32.7%) populations, reminiscent of the coexistence of fast-spiking interneurons and regular-spiking pyramidal cells in biological cortex. (4) Structural competence without reasoning. The model achieves fluent Chinese generation (6/6 coherence) but fails at arithmetic (0/8), with E[K] flat across all task categories (∼ 7.6). This dissociation between structural fluency and reasoning ability, combined with the structure-driven (not difficulty-driven) computation allocation, suggests that the model has acquired a “structural backbone” of language — analogous to early stages of biological language acquisition where grammatical patterns precede semantic understanding. Continued training on more data would be needed to progress from structural pattern learning to genuine semantic reasoning. Data efficiency. NeuronSpark acquires basic language capabilities with ∼ 14% of pretraining data and ∼ 1.2% of SFT data. The biological interpretability findings above suggest this efficiency may arise from the SNN architecture’s inductive bias toward structural pattern extraction, though controlled Transformer baselines are needed to confirm this hypothesis. Limitations (1) 0.9B parameters, 512-token context. (2) No quantitative benchmarks (C-Eval, CMMLU) or Transformer baselines. (3) Chinese only. (4) Repetition artifacts and no reasoning capability. (5) Interpretability analyses are correlational, not causal. Energy efficiency. The spike-based hidden computation may be amenable to deployment on neuromorphic platforms (e.g., Intel Loihi (Davies et al., 2018)), which could yield substantial energy savings. A rigorous quantitative evaluation remains future work. 7 Conclusion We presented NeuronSpark, a 0.9B-parameter spiking language model that jointly addresses three persistent gaps in prior work: distillation dependence, partially non-spiking pipelines, and limited model scale. By connecting SNN membrane dynamics to selective state space models and introducing leakage-current inter-layer signaling, PonderNet adaptive timesteps, fused Triton PLIF kernels, residual centering, lateral inhibition normalization, and natural-gradient compensation, we show that pure SNN architectures can learn non-trivial language behavior from random initialization under limited training data. Beyond architectural feasibility, our interpretability analyses reveal that the trained model develops biologically plausible computational strategies: structure-driven (not difficulty-driven) resource allocation, hierarchical depth-dependent processing, multi-timescale neuron specialization, and a “structure before semantics” learning progression that parallels biological language acquisition. These findings suggest that SNN architectures may offer not only energy-efficiency potential but also a path toward more interpretable language models grounded in neuroscience principles. Code, weights, and training infrastructure are publicly available at the links in the first-page footnote. References L. F. Abbott (1999) Lapicque’s introduction of the integrate-and-fire model neuron (1907). Brain Research Bulletin 50 (5-6), p. 303–304. Cited by: §3.3. M. Bal and A. Sengupta (2024) SpikingBERT: distilling bert to train spiking language models using implicit differentiation. Proceedings of the AAAI Conference on Artificial Intelligence 38 (10), p. 10998–11006. Cited by: §1, §2. A. Banino, J. Balaguer, and C. Blundell (2021) PonderNet: learning to ponder. International Conference on Machine Learning Workshop on Theoretic Foundation, Criticism, and Application Trend of Explainable AI. Cited by: §2, §3.6. BelleGroup (2023) BelleGroup train_3.5m_cn: chinese instruction-following dataset. External Links: Link Cited by: §5.1. G. E. Blelloch (1990) Prefix sums and their applications. Technical Report CMU-CS-90-190, School of Computer Science, Carnegie Mellon University. Cited by: §4.3. M. Carandini and D. J. Heeger (2012) Normalization as a canonical neural computation. Nature Reviews Neuroscience 13 (1), p. 51–62. Cited by: §3.1, §4.2, §6. T. Chen, B. Xu, C. Zhang, and C. Guestrin (2016) Training deep nets with sublinear memory cost. arXiv preprint arXiv:1604.06174. Cited by: §5.1. T. Dao and A. Gu (2024) Transformers are SSMs: generalized models and efficient algorithms through structured state space duality. International Conference on Machine Learning. Cited by: §2. M. Davies, N. Srinivasa, T. Lin, G. Chinya, Y. Cao, S. H. Choday, G. Dimou, P. Joshi, N. Imam, S. Jain, et al. (2018) Loihi: a neuromorphic manycore processor with on-chip learning. IEEE Micro 38 (1), p. 82–99. Cited by: Energy efficiency.. W. Fang, Y. Chen, J. Ding, Z. Yu, T. Masquelier, D. Chen, L. Huang, H. Zhou, G. Li, and Y. Tian (2023) SpikingJelly: an open-source machine learning infrastructure platform for spike-based intelligence. Science Advances 9 (40), p. eadi1480. Cited by: §2. W. Fang, Z. Yu, Y. Chen, T. Masquelier, T. Huang, and Y. Tian (2021) Incorporating learnable membrane time constant to enhance learning of spiking neural networks. In Proceedings of the IEEE/CVF International Conference on Computer Vision, p. 2661–2671. Cited by: §3.2. A. Graves (2016) Adaptive computation time for recurrent neural networks. arXiv preprint arXiv:1603.08983. Cited by: §2. A. Gu and T. Dao (2024) Mamba: linear-time sequence modeling with selective state spaces. International Conference on Machine Learning. Cited by: §1, §2. A. Gu, K. Goel, and C. Ré (2022) Efficiently modeling long sequences with structured state spaces. International Conference on Learning Representations. Cited by: §2. A. L. Hodgkin and A. F. Huxley (1952) A quantitative description of membrane current and its application to conduction and excitation in nerve. The Journal of Physiology 117 (4), p. 500–544. Cited by: §3.3. I. Loshchilov and F. Hutter (2019) Decoupled weight decay regularization. International Conference on Learning Representations. Cited by: §5.1. C. Lv, T. Xu, J. Li, C. Wang, and J. Liu (2023) SpikeBERT: a language spikformer trained with two-stage knowledge distillation from bert. arXiv preprint arXiv:2308.15122. Cited by: §1, §2. W. Maass (1997) Networks of spiking neurons: the third generation of neural network models. Neural Networks 10 (9), p. 1659–1671. Cited by: §1. E. Martin and C. Cundy (2018) Parallelizing linear recurrent neural nets over sequence length. International Conference on Learning Representations. Cited by: §4.3. P. Micikevicius, S. Narang, J. Alben, G. Diamos, E. Elsen, D. Garcia, B. Ginsburg, M. Houston, O. Kuchaiev, G. Venkatesh, and H. Wu (2018) Mixed precision training. International Conference on Learning Representations. Cited by: §5.1. Mobvoi (2023) Seq-monkey general open corpus. External Links: Link Cited by: §5.1. E. O. Neftci, H. Mostafa, and F. Zenke (2019) Surrogate gradient learning in spiking neural networks: bringing the power of gradient-based optimization to spiking neural networks. IEEE Signal Processing Magazine 36 (6), p. 51–63. Cited by: §2, §5.5. O. Press and L. Wolf (2017) Using the output embedding to improve language models. Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics, p. 157–163. Cited by: §3.1. R. Sennrich, B. Haddow, and A. Birch (2016) Neural machine translation of rare words with subword units. Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, p. 1715–1725. Cited by: §5.1. N. Shazeer (2020) GLU variants improve transformer. arXiv preprint arXiv:2002.05202. Cited by: §3.5. P. Tillet, H. T. Kung, and D. Cox (2019) Triton: an intermediate language and compiler for tiled neural network computations. Proceedings of the 3rd MLSys Conference. Cited by: §4.3. H. Touvron, T. Lavril, G. Izacard, X. Martinet, M. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar, et al. (2023) LLaMA: open and efficient foundation language models. arXiv preprint arXiv:2302.13971. Cited by: §3.1, §3.5. A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin (2017) Attention is all you need. Advances in Neural Information Processing Systems 30. Cited by: §1. A. Yang, B. Yang, B. Zhang, et al. (2025) Qwen3 technical report. arXiv preprint arXiv:2505.09388. Cited by: §3.1. F. Zenke and T. P. Vogels (2021) The remarkable robustness of surrogate gradient learning for instilling complex function in spiking neural networks. Neural Computation 33 (4), p. 899–925. Cited by: §2. B. Zhang and R. Sennrich (2019) Root mean square layer normalization. Advances in Neural Information Processing Systems 32. Cited by: §3.1, §4.2. R. Zhu, Q. Zhao, G. Li, and J. K. Eshraghian (2024) SpikeGPT: generative pre-trained language model with spiking neural networks. IEEE Transactions on Neural Networks and Learning Systems. Cited by: §1, §2. Appendix A Structured Initialization Details The SNNBlock modulation parameters require careful initialization. We target Kref=16K_ref=16 steps and assumed input firing rate p=0.15p=0.15. Multi-timescale β: βn=linspace(0.80,0.99,N) _n=linspace(0.80,0.99,N); bias bβ,n=log(βn/(1−βn))b_β,\,n= ( _n/(1- _n)), repeated across D channels with (0,0.1)N(0,0.1) perturbation. α near unity: bα∼(0.5413,0.1)b_α (0.5413,0.1), giving α≈1.0α≈ 1.0. Threshold calibration: σV(β)=p/3⋅1−β2Kref _V(β)= p/3· 1-β^2K_ref; target firing rates pfire=linspace(0.25,0.08,N)p_fire=linspace(0.25,0.08,N); Vth,n=σV(βn)⋅Φ−1(1−pfire,n)V_th,n= _V( _n)· ^-1(1-p_fire,n). WinW_in scaling: rows scaled by 1−βn2 1- _n^2 per group. WoutW_out balancing: columns scaled by 1/pfire,n1/ p_fire,n (normalized to mean 1). Appendix B Model Configuration Table 5: Detailed model configuration. Hidden dim (D) 896 State expansion (N) 8 Max SNN steps (K) 16 Layers (L) 20 FFN dim (DffD_f) 2688 Vocab 6144 Context 512 Total params 874M Surrogate Sigmoid(α=4.0) VminV_min 0.1 Neuron LR mult 10× Ponder weight 0.01 Output proj init 0.02/2L0.02/ 2L (GPT-2) Appendix C Parameter Breakdown Table 6: Parameter breakdown. SNNBlock dominates (77.2%) due to 7 projections in D×ND×N space. Component Params % Embedding (tied) 5.5M 0.6 SNNBlock ×20 674.8M 77.2 SNNFFN ×20 160.8M 18.4 Residual proj ×40 32.1M 3.7 Other 1.0M 0.1 Total 874.1M 100 Appendix D Engineering Optimizations • Fused modulation: σ,softplus,|⋅|,×σ,softplus,|·|,× fused via torch.compile into single kernel. • Fused halt weights: PonderNet σ→log(1−p)→cumsum→exp→normalizeσ→ (1-p) → fused. • Merged SNNFFN matmul: Wgate,WupW_gate,W_up concatenated into single (2Dff,D)(2D_f,D) matmul. • Merged PLIF scan: Gate/up neurons merged into single 2Dff2D_f-dim scan. • Gradient checkpointing: Each of L=20L=20 layers checkpointed (∼ 60% memory reduction). Appendix E Training Hyperparameters Table 7: Training hyperparameters. Pretrain SFT Optimizer Adam AdamW Peak LR 2×10−42×10^-4 5×10−55×10^-5 Neuron LR 2×10−32×10^-3 5×10−45×10^-4 Schedule Warmup+Cosine Warmup+Cosine Warmup 1000 100 Weight decay 0 0.01 Eff. batch 64 64 Grad clip 1.0 1.0 Precision bf16 bf16