Paper deep dive
Rethinking the Tradeoff Between Temporal Encoding and Nonlinear Computation in Spiking Language Models
Hanfei Liu, Shuchang Feng, Yanxia Chen, Changzeng Fu, Shiqi Zhao
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 94%
Last extracted: 10/9/2026, 8:03:11 AM
Summary
The paper introduces Spora, a spiking language model that addresses the tradeoff between temporal encoding precision and computational cost in attention mechanisms. Spora utilizes Unipolar Binary Spiking (UBS) and Bipolar Binary Spiking (BBS) encodings to convert continuous activations into integer codes, enabling accumulation-and-shift dot products and integer-exponent mappings for attention. This approach replaces costly nonlinear operations like Softmax and multiplication with efficient shift-based approximations, achieving a theoretical 98.53% reduction in core attention energy. Spora outperforms SpikeLM on GLUE benchmarks, achieving a 76.6 average score with four time steps.
Entities (8)
Relation Signals (7)
Spora → achievesscoreon → GLUE
confidence 96% · Spora achieves 76.6 average GLUE score
Spora → uses → Bipolar Binary Spiking
confidence 95% · Spora combines Unipolar Binary Spiking (UBS) and Bipolar Binary Spiking (BBS) encodings.
Spora → uses → Unipolar Binary Spiking
confidence 95% · Spora combines Unipolar Binary Spiking (UBS) and Bipolar Binary Spiking (BBS) encodings.
Spora → achievesscoreon → COLA
confidence 94% · 44.1 CoLA MCC
Spora → outperforms → SpikeLM
confidence 92% · improving over SpikeLM by 1.2 and 6.2 points, respectively.
Unipolar Binary Spiking → enables → integer-exponent mappings
confidence 90% · These representations support accumulation-and-shift dot products and integer-exponent mappings in attention.
Bipolar Binary Spiking → enables → signed activations
confidence 90% · Bipolar Binary Spiking (BBS) separates sign and magnitude and learns a scale for signed activations.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Spiking language models face a tradeoff between representing continuous semantic features over short temporal windows and retaining costly nonlinear attention operations. We introduce Spora, which jointly designs spike encodings and attention operators. Binary temporal weights let $T$ spikes represent compositional values with up to $T$ bits of capacity, compared with $O(\log_2 T)$ bits for spike-count readout. Unipolar Binary Spiking (UBS) uses thresholds and spike-triggered residual decay to produce non-negative integer codes; Bipolar Binary Spiking (BBS) separates sign and magnitude and learns a scale for signed activations. These representations support accumulation-and-shift dot products and integer-exponent mappings in attention. With four time steps, Spora achieves 76.6 average GLUE score and 44.1 CoLA MCC, improving over SpikeLM by 1.2 and 6.2 points, respectively. Extending BBS to six steps raises these scores to 78.2 and 47.4. Conditional-decay analysis, matched-budget activation-quantization comparisons, event-workload statistics, and fixed-point evaluation further characterize the connection between encoding fidelity and computational cost.
Tags
Links
- Source: https://arxiv.org/abs/2610.10933v1
- Canonical: https://arxiv.org/abs/2610.10933v1
Trouble viewing inline? Open PDF directly →
Full Text
53,971 characters extracted from source content.
Expand or collapse full text
Rethinking the Tradeoff Between Temporal Encoding and Nonlinear Computation in Spiking Language Models Hanfei Liu Affiliation: Northeastern University, 110819, Shenyang, China Affiliation: Hebei Key Laboratory of Marine Perception Networkand Data Processing, 066004, Qinhuangdao, China Email: liuhf@mails.neu.edu.cn Shuchang Feng Affiliation: Northeastern University, 110819, Shenyang, China Affiliation: Hebei Key Laboratory of Marine Perception Networkand Data Processing, 066004, Qinhuangdao, China Email: fengshuchang@mails.neu.edu.cn Yanxia Chen Affiliation: Northeastern University, 110819, Shenyang, China Affiliation: Hebei Key Laboratory of Marine Perception Networkand Data Processing, 066004, Qinhuangdao, China Email: chenyanxia@mails.neu.edu.cn Changzeng Fu Affiliation: Northeastern University, 110819, Shenyang, China Affiliation: Hebei Key Laboratory of Marine Perception Networkand Data Processing, 066004, Qinhuangdao, China Email: fuchangzeng@qhd.neu.edu.cn Shiqi Zhao †thanks: Corresponding author. Affiliation: Northeastern University, 110819, Shenyang, China Affiliation: Hebei Key Laboratory of Marine Perception Networkand Data Processing, 066004, Qinhuangdao, China Email: zhaoshiqi@qhd.neu.edu.cn Abstract Spiking language models face a tradeoff between representing continuous semantic features over short temporal windows and retaining costly nonlinear attention operations. We introduce Spora, which jointly designs spike encodings and attention operators. Binary temporal weights let T spikes represent compositional values with up to T bits of capacity, compared with O(log2T)O( _2T) bits for spike-count readout. Unipolar Binary Spiking (UBS) uses thresholds and spike-triggered residual decay to produce non-negative integer codes; Bipolar Binary Spiking (BBS) separates sign and magnitude and learns a scale for signed activations. These representations support accumulation-and-shift dot products and integer-exponent mappings in attention. With four time steps, Spora achieves 76.6 average GLUE score and 44.1 CoLA MCC, improving over SpikeLM by 1.2 and 6.2 points, respectively. Extending BBS to six steps raises these scores to 78.2 and 47.4. Conditional-decay analysis, matched-budget activation-quantization comparisons, event-workload statistics, and fixed-point evaluation further characterize the connection between encoding fidelity and computational cost. Keywords: Spiking neural networks ; Language model ; Temporal encoding ; Approximate attention 1 Introduction Figure 1: Spike encodings: (a) rate, (b) TTFS, (c) rank-order, (d) Few-Spike, (e) UBS, and (f) BBS. Event labels denote readout weights; BBS traces show separate positive and negative inputs. Spiking Neural Networks (SNNs) process information through discrete events and membrane-potential dynamics inspired by biological neurons (Mehonic and Kenyon, 2022; Zhang et al., 2020b; Maass, 1997). Their sparse communication and event-driven computation offer a route to energy-efficient neural processing (Li et al., 2021). Recent advances in spiking vision (Wu et al., 2022; Hu et al., 2023), particularly Transformer-based architectures (Vaswani et al., 2017; Zhou et al., 2023a; Zhou et al., 2023b; Li et al., 2024; Yao et al., 2023), have achieved competitive performance with four or fewer time steps. Motivated by the generalization capabilities of large-scale language models (Du et al., 2022), SpikeLM (Xing et al., 2024b) and SpikeGPT (Zhu et al., 2023) demonstrate the feasibility of spike-driven language processing. Language representations, however, are dense, signed, and context-dependent. Transformer predictions depend on both the precision of these features and their interactions through attention. Mapping continuous activations into short spike sequences can therefore affect not only individual feature values but also the attention distributions that combine information across tokens. Existing spiking language models approach this problem from two directions. Models that introduce spikes into selected linear modules retain floating-point Softmax to preserve attention fidelity (Xing et al., 2024b). Hardware-oriented alternatives replace nonlinear operations with approximations (Tang et al., 2025; Zhou et al., 2026), while temporal budgets remain important for representation precision. Increasing the number of steps provides more encoding opportunities but also increases temporal processing. Although these two categories differ in implementation, they face the same fundamental challenge: preserving fine-grained semantic information within a short temporal window while avoiding costly nonlinear attention computation. Figure 1 illustrates the relevant encoding choices. Rate coding counts events, TTFS uses first-spike latency (Yu et al., 2023), and rank-order coding uses relative firing order across neurons (Rufin Van Rullen, 2001). Weighted-spike and Few-Spike methods assign numerical readout weights to temporal events (Kim et al., 2018; Stöckl and Maass, 2021). Our approach uses binary place values: an event’s time index identifies its numerical contribution. Unlike a spike-count estimate, this readout forms an integer operand that can enter subsequent arithmetic directly. Following this principle, Spora combines Unipolar Binary Spiking (UBS) and Bipolar Binary Spiking (BBS) encodings. UBS configures input partitions through thresholds and spike-triggered residual decay while retaining fixed binary readout weights. BBS separates sign from magnitude and uses a learnable scale to represent signed activations. These complementary codes support accumulation-and-shift dot products and integer-exponent attention mappings. The design follows two requirements: encoding should adapt to activation distributions, and the resulting values should preserve the arithmetic structure needed by the next operator. Conditional decay and learned scales serve the first requirement; binary readout serves the second. This connects the temporal representation to the attention computation that consumes it. Our contributions are: (1) Unipolar encoding. UBS distinguishes the numerical values specified by binary readout from the input intervals determined by thresholds and conditional decay. This provides short-window integer representations for shift-based exponential approximation. (2) Bipolar encoding. BBS applies a shared magnitude process to both signs, with a learnable scale that adapts its finite signed codebook to the feature range. (3) Spiking attention and evaluation. The combined design replaces major attention multiplications and exponential evaluations with accumulation and shifts, yielding a theoretical 98.53% reduction in core attention energy. Four-step Spora reaches 76.6 average GLUE score and 44.1 CoLA MCC, exceeding SpikeLM by 1.2 and 6.2 points. Conditional-decay analysis, matched-budget Static-QAT comparisons, event workloads, and fixed-point experiments connect these task results to encoding fidelity and computational cost. 2 Related Work Spiking neural encoding. Rate coding uses equally weighted events to approximate continuous values. Spike Firing Approximation (Yao et al., 2025) and memory-efficient training methods (Xiao et al., 2022; Meng et al., 2023) reduce training costs, but spike-count readout still limits the number of representable levels. Temporal alternatives include TTFS (Yu et al., 2023; Zhao et al., 2025) and rank-order coding (Rufin Van Rullen, 2001), with tradeoffs in training stability and decoding dynamics (Wei et al., 2023). Weighted-spike and Few-Spike methods introduce temporal numerical weights (Kim et al., 2018; Stöckl and Maass, 2021). Spora uses fixed binary weights and conditional residual decay to obtain integer codes suited to signed features and attention operators. Transformer-based SNNs. Spikformer (Zhou et al., 2023b) replaces Softmax with spiking self-attention and reaches 74.81% ImageNet-1k accuracy (Deng et al., 2009) in four steps. Spike-driven Transformer (Yao et al., 2023) uses additions and masking, while QKformer (Zhou et al., 2024) introduces linear-complexity Q-K attention. MaxFormer (Fang et al., 2026) and SpikePool (Lee et al., 2025) use max-pooling in patch embedding and attention, respectively, to improve signal transmission. Language models such as BERT (Devlin et al., 2019) additionally require precise interactions between context-dependent token features, motivating joint treatment of representation and attention approximation. Hardware-adapted language SNNs. SpikeBERT (Lv et al., 2026) uses Not-XOR spiking attention, and WE-SpikingFormer (Zhou et al., 2026) uses winner-take-all attention. Sorbet (Tang et al., 2025) combines shift-based PTsoftmax and PowerNorm to avoid multiplication, division, and square roots, using sixteen time steps. SpikeLM (Xing et al., 2024b) improves information flow through ternary spikes and adaptive thresholds, but retains exponential attention operations, as does SpikeLLM (Xing et al., 2025). Spora instead links binary temporal readout to integer-exponent attention, addressing short-window precision and nonlinear computation within one encoding design. 3 Method Spora combines BBS-encoded signed features with UBS-based nonlinear mappings (Figure 2). Binary temporal weights give both representations explicit numerical values for accumulation-and-shift computation. Figure 2: Spora architecture. (A) UBS temporal weights; (B) sign-separated BBS; (C) approximate attention, with ShiftApprox denoting shift-based Softmax approximation; (D) feed-forward block, where UBS implements the GELU approximation during inference. 3.1 Representational Capacity and Numerical Roles We measure the size of a decoded alphabet by C=log2||C= _2|C|, where C is the set of distinct readout values. For a binary sequence of length U, count-based decoding retains only K=∑t=1US(t)K= _t=1^US(t), whereas binary-weight decoding retains its place values: count _count =0,…,U, =\0,…,U\, (1) Ccount C_count =log2(U+1), = _2(U+1), binary _binary =0,…,2U−1, =\0,…,2^U-1\, Cbinary C_binary =U. =U. For example, at U=4U=4, count readout maps the sixteen possible spike patterns to five values, while binary readout assigns each pattern a distinct integer. This is an alphabet-capacity comparison: the thresholds and residual dynamics determine which patterns an input distribution actually occupies. UBS uses the larger alphabet for nonlinear mappings whose outputs must have explicit integer meaning. BBS extends this construction to signed features. With M magnitude decisions, the signed alphabet contains at most 2(2M−1)+1=2M+1−12(2^M-1)+1=2^M+1-1 states. Throughout the experiments, T=2MT=2M denotes the BBS configuration with M positive and M negative bit planes. Thus, T=4T=4 and T=6T=6 correspond to seven and fifteen signed states. UBS uses U=4U=4 stages in both configurations. This separates the BBS temporal budget from the UBS exponent-code budget. 3.2 Unipolar Binary Spiking Encoding UBS processes larger-weight decisions before lower-weight refinements. We use U for its number of stages, reserving T for the BBS configuration reported in the experimental tables. For non-negative input x, UBS initializes V(0)=xV(0)=x and uses U temporal weights w(t)=2U−tw(t)=2^U-t. At each step, a threshold comparison generates a binary spike: S(t) S(t) =Θ[V(t−1)−Vth(t)], = \! [V(t-1)-V_th(t) ], (2) S(t) S(t) ∈0,1, ∈\0,1\, where Θ is the Heaviside function. When S(t)=1S(t)=1, the threshold is subtracted and the remaining potential is multiplied by α. When S(t)=0S(t)=0, the potential is passed unchanged to the next comparison. This spike-triggered update is V(t)=α[V(t−1)−Vth(t)],S(t)=1,V(t−1),S(t)=0.V(t)= casesα [V(t-1)-V_th(t) ],&S(t)=1,\\ V(t-1),&S(t)=0. cases (3) The decoded value is UBS(x)=∑t=1Uw(t)S(t)UBS(x)= _t=1^Uw(t)S(t). How conditional decay shapes the code. Binary readout specifies output values, while the residual dynamics determine their input regions. Consider the first two decisions, writing θt=Vth(t) _t=V_th(t). The second spike satisfies S(2)=1⟺x≥θ2,S(1)=0,x≥θ1+θ2/α,S(1)=1. gatheredS(2)=1 \\ casesx≥ _2,&S(1)=0,\\ x≥ _1+ _2/α,&S(1)=1. cases gathered (4) Thus, the effective threshold for a later event depends on whether an earlier event fired. Changing α moves this boundary without changing either event’s readout weight. Over multiple steps, this mechanism shapes input partitions through the preceding spike sequence. The no-decay comparison in Section 4.2 tests the effect of this partitioning on attention fidelity while retaining binary readout. Thresholds and decay are fitted to the target mapping while keeping the readout weights fixed; Section 3.5 gives the search procedure and training gradients. 3.3 Bipolar Binary Spiking Encoding BBS separates sign and magnitude to encode signed activations, such as layer-normalized features, building on bidirectional spike representations in SpikeLM (Xing et al., 2024b). For x∈ℝx , it initializes V(0)=|x|/sV(0)=|x|/s with learnable scale s>0s>0 and applies Equations 2–3 to the magnitude. With M=T/2M=T/2 magnitude steps and binary weights wM(t)=2M−tw_M(t)=2^M-t, the output is Q(|x|) Q(|x|) =∑t=1MwM(t)S(t), = _t=1^Mw_M(t)S(t), (5) BBS(x) (x) =ssign(x)Q(|x|). =s\,sign(x)\,Q(|x|). The T binary spike tensors comprise M positive and M negative pathways; negative-path events contribute with a minus sign. The default T=4T=4 configuration uses magnitude thresholds and weights [2,1][2,1], while T=6T=6 uses [4,2,1][4,2,1]. The scale adapts the fixed-threshold range to activation distributions and can be fused into the corresponding linear weights for accumulation-based inference. Separating polarity from magnitude gives positive and negative inputs the same magnitude-encoding rule. At T=4T=4, the decoded magnitudes range from 0 to 3, giving seven signed states including zero; the T=6T=6 configuration extends this to magnitudes from 0 to 7 and fifteen signed states. The scale controls their spacing in the original activation domain. This provides the representation budgets used in the workload and Static-QAT experiments. BBS therefore combines uniformly spaced reconstruction values with input regions determined by the spike recurrence. A static uniform quantizer can use the same signed values while assigning inputs through uniformly spaced thresholds. Matching the number of states compares equal-sized output alphabets; the downstream experiment examines how these different assignments perform within the complete model. 3.4 Spiking Attention Mechanism Spiking QKV representation. BBS encodes the outputs of the Query, Key, and Value projections. For T=4T=4, define the signed integer tensor Q~ Q by BBS(Q)=sqQ~,Q~=QP1+2QP2−QN1−2QN2,BBS(Q)=s_q Q, Q=Q_P1+2Q_P2-Q_N1-2Q_N2, (6) where QPjQ_Pj and QNjQ_Nj are binary positive and negative tensors with magnitude weight j. Defining K~ K and V~ V analogously gives BBS(Q)BBS(K)T=sqskQ~K~T.BBS(Q)\,BBS(K)^T=s_qs_k\, Q K^T. (7) Expanding the binary planes expresses the product through binary interactions, signed accumulations, and power-of-two shifts, followed by the scale factor. In particular, multiplying two binary-weight contributions produces another power-of-two coefficient. Their magnitudes can therefore be handled by shifts, while the positive and negative pathways determine the accumulation sign. The same signed representation is retained for the Value projection. Spiking approximation of Softmax. The identity ex/d=2x/(dln2)e^x/ d=2^x/( d 2) converts exponential evaluation into an integer-exponent mapping. Using the threshold search above, UBS approximates the exponent by c(x) c(x) =UBS(x)=∑t=1U2U−tS(t)≈⌊xdln2⌋, =UBS(x)= _t=1^U2^U-tS(t)≈ x d 2 , (8) ex/d e^x/ d ≈2c(x)=∏t=1U2 2U−tS(t). ≈ 2^c(x)= _t=1^U2^\,2^U-tS(t). Each active bit contributes a power-of-two shift, and the integer exponent determines the combined shift amount. In this construction, UBS approximates the exponent code; the subsequent power-of-two mapping supplies the nonlinearity used for attention scores. Let C=UBS(sqskQ~K~U)C=UBS(s_qs_k Q K^U) and E=2CE=2^C, with UBS and exponentiation applied elementwise. The normalized attention output is Softmax(QKUd)V≈diag(E)−1EsvV~,Softmax\! ( QK^U d )V (E1)^-1E\,s_v V, (9) where 1 sums over keys. Integer codes support shift-based exponentiation and accumulation with BBS-encoded values; row normalization and scale application remain floating-point operations. At inference, sqsks_qs_k can be absorbed into the UBS thresholds, leaving svs_v on the output. Why evaluate attention-output fidelity? For a reference attention row pj∝2ujp_j 2^u_j, exponent-code errors δj _j yield p^j=pj 2δj∑kpk 2δk, p_j= p_j\,2 _j _kp_k\,2 _k, (10) where the sum is over valid keys. A common exponent offset cancels during normalization, whereas differences between code errors redistribute attention across tokens. The effect on the output also depends on the associated Value vectors. This motivates measuring P^V PV against reference attention outputs in addition to fitting the exponent codes: the former evaluates the quantity passed to the next layer. For training, we use a straight-through estimator (STE) (Bengio et al., 2013) for spike firing. UBS and BBS return decoded integer representations without explicitly unrolling spike propagation through subsequent layers; the trained parameters are used for the corresponding temporal binary encoding at inference. 3.5 Threshold Search and Differentiable Training To approximate a target function f(x)f(x), we optimize four thresholds θ=[θ1,…,θ4]θ=[ _1,…, _4] and α for the decoded output P(x,θ,α)P(x;θ,α): minθ, 0<α≤1ℒ(θ,α)=1N∑i=1N[P(xi,θ,α)−f(xi)]2+λ. _θ,\,0<α≤ 1\;L(θ,α)= 1N _i=1^N[P(x_i;θ,α)-f(x_i)]^2+ . (11) Inputs xix_i are uniformly sampled from the target domain; I indicates violations of threshold ordering and range constraints, penalized by λ. Differential evolution (Storn and Price, 1997) provides a global approximate solution, followed by local grid refinement. The resulting parameters are used for the selected activation and exponential mappings. The global stage searches across threshold configurations, while local refinement adjusts the best candidate’s partitions. Readout weights remain powers of two throughout the search, so fitting the target mapping does not change the arithmetic structure of the resulting code. Offline search. For a chosen target f and domain [a,b][a,b], offline search uses 1,201 uniformly spaced inputs, a population of 125 candidates, and 120 generations. Differential mutation samples F∼(0.5,1)F (0.5,1) and binomial crossover uses probability 0.7, with seed 42. Threshold bounds are [1,60][1,60] and decay bounds are [0.01,1][0.01,1]. After differential evolution, thresholds are rounded to integers and locally refined. The target domain and threshold–weight association are specified for each mapping. This offline function fit supplies encoder parameters; the data-based recalibration in Section 4.2 is a separate mechanism experiment. Algorithm 1 Differential evolution and local refinement for UBS 1: Target f, sampled inputs xk\x_k\, feasible set ℬB, population size NPN_P, generations G 2: Thresholds and decay z⋆=(θ⋆,α⋆)z =(θ ,α ) 3: Initialize z1,…,zNP∈ℬz_1,…,z_N_P ; evaluate the loss in Equation 11 4: for g=1,…,Gg=1,…,G do 5: Let zbestz_best be the lowest-loss candidate 6: for each candidate ziz_i do 7: Select distinct r1,r2≠ir_1,r_2≠ i and sample F∼(0.5,1)F (0.5,1) 8: Form vi=zbest+F(zr1−zr2)v_i=z_best+F(z_r_1-z_r_2) 9: Cross viv_i with ziz_i at rate 0.70.7, retaining at least one coordinate from viv_i 10: Repair the trial to satisfy ℬB and evaluate its loss 11: Retain the trial if its loss is no larger than that of ziz_i 12: end for 13: end for 14: Round the best thresholds to integers and refine locally under the same constraints 15: return the best feasible z⋆z Surrogate gradients. Network training uses straight-through or smooth surrogate derivatives for discrete firing (Bengio et al., 2013). For the attention UBS code, the clipped surrogate is Qatt(x)=clip(x/(dln2),0,n)Q_att(x)=clip(x/( d 2),0,n); for post-activation FFN UBS, it is Qffn(x)=clip(x,0,n)Q_ffn(x)=clip(x,0,n), where n=2U−1n=2^U-1 is the maximum unsigned code. These surrogates propagate gradients through the corresponding code range while the forward computation retains discrete values. For BBS, write u=|x|/su=|x|/s and y=ssign(x)q(u)y=s\,sign(x)q(u), where q is the decoded magnitude from the complete residual recurrence. If qsur′(u)q _sur(u) denotes its surrogate derivative, the scale gradient follows the chain rule: ∂ℒ∂s≈∂ℒ∂ysign(x)[q(u)−uqsur′(u)]. ∂ s≈ ∂ y\,sign(x) [q(u)-u\,q _sur(u) ]. (12) The first term accounts for rescaling the decoded magnitude, and the second accounts for moving the normalized input relative to the thresholds. Smooth spike derivatives provide gradients near decision boundaries and are propagated through the residual updates. A smaller scale increases the normalized magnitude and threshold crossings; a larger scale widens the represented activation range. Learning s therefore adapts the fixed magnitude codebook to each encoder’s feature distribution. 4 Results Benchmarks and metrics. We evaluate eight GLUE datasets spanning grammatical acceptability (CoLA), sentiment classification (SST-2), paraphrase identification (QQP and MRPC), semantic similarity (STS-B), and inference (MNLI, QNLI, and RTE). MNLI provides matched and mismatched development splits, giving nine reported task columns. Classification tasks use accuracy, CoLA uses Matthews correlation, and STS-B uses Spearman correlation, all on a 0–100 scale. The all-task average is the arithmetic mean of these nine columns. The reduced average excludes CoLA and follows the MNLI-split convention marked in Table 1. Training setup. We evaluate Spora on GLUE (Wang et al., 2019) after pre-training on English Wikipedia (Devlin et al., 2019) and BookCorpus (Zhu et al., 2015) at maximum sequence length 128. Training uses AdamW (Loshchilov and Hutter, 2019), a learning rate of 5×10−55× 10^-5, 5,000 warmup steps, and mixed precision (Micikevicius et al., 2018). Multi-level BERT MLM distillation (Sun et al., 2019) transfers the BERT-base teacher’s output distribution, hidden representations, and attention matrices, followed by GLUE fine-tuning. Training uses four 24GB RTX4090 GPUs; layer-local mechanism evaluation uses one RTX4090 D. 4.1 Performance on GLUE Table 1 compares Spora with full-precision BERT (Devlin et al., 2019); quantized models Q2BERT (Shen et al., 2020), BinaryBERT (Bai et al., 2021), BiBERT (Qin et al., 2022), TernaryBERT (Zhang et al., 2020a), BiT (Liu et al., 2022), and BiPET (Xing et al., 2024a); and spiking language models. Among the quantized and spiking baselines, all except Q2BERT, WE-SpikingFormer, and SpikingFormer employ knowledge distillation. Table 1: GLUE development results (0–100). QQP/MRPC: accuracy; STS-B: Spearman’s correlation; CoLA: MCC. Avg. excludes CoLA; Avg.all includes it. Asterisks indicate averages using only one MNLI split. Time denotes time steps. SporaT=4’s QQP/MRPC F1 scores are 85.9/86.6; the reduced average for SporaT=6 uses MNLI-m. Model Time QQP MNLIm/m SST-2 QNLI RTE MRPC STS-B Avg. CoLA Avg.all ANN BERT – 91.3 83.8/83.4 92.3 90.7 69.3 88.2 89.4 86.1 60.0 83.2 Q2BERT – 67.0 47.2/47.3 80.6 61.3 52.7 68.4 4.4 53.6 0.0 47.7 BinaryBERT – 79.9 62.7/63.9 82.5 52.6 52.7 68.3 6.5 58.6 14.6 53.7 BiBERT – 83.7 66.1/67.5 88.7 72.6 57.4 72.5 33.6 67.8 25.4 63.0 TernaryBERT – 63.1 40.3/40.0 80.7 50.0 54.5 68.3 12.4 51.2 0.0 45.5 BiT – 82.9 77.1/77.5 87.7 85.7 58.8 79.7 71.1 77.6 25.1 71.7 BiPET – 83.7 69.5/70.6 86.2 81.7 66.1 76.2 80.2 76.8 22.9 70.8 SNN SpikingFormer 4 83.8 67.8/68.6 82.7 74.6 58.8 74.0 72.3 73.4 16.7 66.6 SpikeLM 4 87.9 76.0/76.9 86.5 84.9 65.3 78.7 84.3 80.1 37.9 75.4 Sorbet 16 86.5 77.3/– 90.4 86.1 60.3 79.9 78.1 79.8∗ – – SpikingBERT 125 86.8 78.1/– 88.2 85.2 66.1 79.2 82.2 80.8∗ – – WE-SpikingFormer 4 85.1 –/70.1 89.0 77.5 55.9 81.6 42.8 71.7 27.9 66.2 SporaT=4 4 89.2 79.2/79.4 89.6 85.4 60.0 80.2 82.5 80.9∗ 44.1 76.6 SporaT=6 6 89.4 80.1/80.6 89.9 86.9 63.2 82.2 84.3 82.3∗ 47.4 78.2 With four time steps, Spora reaches 76.6 average score across all reported GLUE columns and 80.9 under the table’s reduced averaging convention. Relative to SpikeLM (Xing et al., 2024b), it improves QQP by 1.3 points and MNLI-m/m by 3.2/2.5 points. The largest gain is on CoLA: 44.1 MCC versus 37.9 for SpikeLM, 27.9 for WE-SpikingFormer (Zhou et al., 2026), and 16.7 for SpikingFormer (Zhou et al., 2023a). On SST-2, Spora reaches 89.6%, approaching Sorbet’s (Tang et al., 2025) 90.4% with four rather than sixteen steps. It also exceeds SpikingBERT (Bal and Sengupta, 2024) on QQP, MNLI-m, and SST-2 with four rather than 125 steps. Spora also reaches 85.4% on QNLI and 80.2% on MRPC, compared with 84.9% and 78.7% for SpikeLM. On QQP, its 89.2% accuracy is within 2.1 points of full-precision BERT. These improvements span sentence-pair classification, sentiment analysis, and grammatical acceptability, showing that the short-window design benefits several language-understanding tasks rather than a single benchmark. 4.2 Encoding and Attention Analysis Conditional-decay protocol. We evaluate SporaT=6 on CoLA at maximum length 128, comparing original four-step UBS, no-decay UBS (α=1α=1) with recalibrated thresholds, jointly calibrated UBS, and static uniform integer encoding. Calibration minimizes exponent-code MSE on 256 training examples under the same maximum search budget; evaluation uses all 1,043 development examples in FP32, excluding padding. For attention scores Z, let r=round(Z)r=round(Z). The encoder input is x=clip(r,0,60)x=clip(r,0,60), and the calibration target is x/(8ln2)x/(8 2). We compare P^V PV with two reference outputs: PrawV=Softmax(Z/8)V,PsharedV=Softmax(x/8)V. \ aligned P_rawV&=Softmax(Z/8)V,\\ P_sharedV&=Softmax(x/8)V. aligned . (13) The raw-score reference measures total error, while the shared-preprocessing reference measures coding error after the same preprocessing. For either reference P, output NMSE is NMSE=∑‖(P^−P)V‖F2∑‖PV‖F2,NMSE= Σ\|( P-P)V\|_F^2Σ\|PV\|_F^2, (14) where both sums are aggregated over valid queries. Attention-output fidelity. Original UBS lowers error relative to recalibrated no-decay UBS in all 12 layers (Figure 3). Aggregate NMSE decreases from 0.02100 to 0.01834 under the raw-score reference and from 0.01277 to 0.01066 under the shared-preprocessing reference, corresponding to reductions of 12.69% and 16.58%, respectively. Static uniform encoding achieves raw-score NMSE of 0.01805 and shared-preprocessing NMSE of 0.01063. Joint UBS recalibration improves the original by less than 0.2% relative. Conditional decay thus refines input partitions while retaining binary readout, bringing temporal encoding close to direct integer encoding in attention fidelity. The raw and shared references distinguish errors introduced before encoding from those of the integer mapping itself. Improvement under both references connects the residual update to the attention output, while the small recalibration gain indicates that the checkpoint’s UBS configuration is already close to the jointly calibrated setting. Figure 3: Attention-output NMSE for four-step integer encoders: (a) raw-score and (b) shared-preprocessing references. Annotations report aggregate reductions relative to recalibrated no-decay UBS. Comparison with static activation quantization. We next compare the complete BBS/UBS design against Static-QAT on CoLA. Table 2: CoLA comparison under matched representation budgets: 15 signed states and four-bit UBS. Method Acc. (%) MCC ×100× 100 Static-QAT 76.61 40.97 SporaT=6 79.00 47.40 SporaT=6 uses three magnitude steps [4,2,1][4,2,1] with 15 signed states; Static-QAT uses symmetric 4-bit quantization with the same 15 states. Both have a four-bit UBS representation budget. Static-QAT replaces the BBS and UBS encoders with uniform quantizers with learnable scales, retaining the architecture and base-2 attention normalization. It follows the four-epoch MLM distillation schedule with the same BERT-base teacher, temperature 2.0, and MLM/hidden/attention loss weights of 1.0/0.5/5.0, followed by CoLA fine-tuning and development-MCC checkpoint selection. Its best checkpoint is evaluated alongside the existing fine-tuned SporaT=6 checkpoint. Table 2 reports 47.40 MCC and 79.00% accuracy for SporaT=6, exceeding Static-QAT by 6.43 MCC points and 2.39 accuracy points. The two comparisons address successive parts of the design. The layer-local study measures how UBS partitions affect a specific attention operator and shows fidelity close to static integer encoding. The downstream comparison includes signed BBS features, UBS mappings, and their interaction with training. Its 6.43-point MCC gain establishes a task-level advantage for the complete design under the matched representation budget. 4.3 Computational Cost and Numerical Precision Arithmetic energy accounting. We estimate energy as operation counts multiplied by technology-level unit costs (Horowitz, 2014). For FP32 arithmetic, the reference costs are 4.6 pJ per MAC, 3.7 pJ per multiplication, and 0.9 pJ per addition. In the FP32 linear-layer accounting, we use the addition cost as a proxy for a shift. The low-bit attention model uses 0.03 pJ for an 8-bit integer addition and 0.1 pJ for a 32-bit integer addition; exponential evaluation uses the 1.7 pJ estimate adopted in Sorbet (Tang et al., 2025). These are arithmetic energy estimates for the specified operation models. For a linear map with dense MAC count NdenseN_dense, let rtr_t be the active fraction in signed bit plane t and wtw_t its numerical weight. Event-wise accounting gives NAC N_AC =Ndense∑trt, =N_dense _tr_t, (15) Nshift N_shift =Ndense∑t:|wt|>1rt, =N_dense _t:\,|w_t|>1r_t, Eevent E_event =NACEAC+NshiftEshift, =N_ACE_AC+N_shiftE_shift, Edense E_dense =NdenseEMAC. =N_denseE_MAC. The shifted planes are identified by weight magnitude rather than their storage order. For weights [2,1][2,1], only magnitude-two events contribute to the shift term. Sign selects addition or subtraction. This notation connects the energy estimate to the measured event counts in Figure 4. Attention is accounted for component by component: binary-plane products and shifts for QKTQK^T, UBS code generation and shifts for exponentiation, row normalization, accumulation with V, and output scaling. The energy of a component is ∑oNoEo _oN_oE_o over its arithmetic operations. Normalization and value scaling remain explicit in Table 3; the total reduction therefore refers to the complete core-attention accounting shown there. Theoretical attention energy. For BERT-base at sequence length 128, with 12 heads of dimension 64, estimated core attention energy decreases from 116.272 to 1.7083 μ per layer, a 98.53% reduction (Table 3, left). The savings arise from accumulation-and-shift computation for QKTQK^T and AVAV and integer-exponent mappings for Softmax. Query/key scales are absorbed into UBS thresholds; value scaling is applied to the output. Table 3: Left: theoretical attention energy in μ (Conv.: conventional; Approx.: proposed). Right: fixed-point emulation on CoLA. State and Prob. denote BBS membrane-state and attention-probability formats; Sat. is the percentage of clipped state values. Module Conv. Approx. QKTQK^T 57.881 0.303 ex/2UBSe^x/2^UBS 0.334 0.028 Denominator 0.176 0.005 AVAV 57.881 1.008 svs_v scaling – 0.364 Total 116.272 1.7083 Format State Prob. Acc. (%) MCC ×100× 100 Sat. (%) FP32 Float Float 77.95 44.39 – 16-bit Q6.10 U16 77.56 43.44 0.0194 12-bit Q6.6 U12 76.51 40.64 0.0193 8-bit Q6.2 U8 77.28 42.90 0.0184 Workload and precision protocol. We use the 1,043-example CoLA development set at maximum length 128 and the fine-tuned four-step model: two BBS magnitude steps [2,1][2,1], seven signed states −3,−2,−1,0,1,2,3\-3,-2,-1,0,1,2,3\, and four UBS bit-planes. Event workload. Figure 4 aggregates active events and temporal slots across layers. BBS densities range from 32.5% to 44.3%, with balanced positive/negative events (49.5%/50.5%); FFN and attention UBS densities are 1.8% and 1.7%. For spike-to-linear paths, active-event counts multiplied by output fanout estimate event-driven accumulations. Their aggregate ratio to dense MACs is 50.1%, falling to 7.2% for FFN UBS, which offers the sparsest accumulation path. The contrast with FFN BBS’s 88.6% ratio shows where event sparsity contributes most. Binary weights determine how an event is processed, while activation-dependent firing determines how many such operations are needed; these are complementary sources of computational savings. Figure 4: Activation-derived workload on CoLA at padded length 128. Left: event density by component. Right: estimated event-driven accumulations relative to dense MACs in spike-to-linear paths. Fixed-point emulation. We quantize BBS membrane potentials to Q6.(b−6)Q6.(b-6), round the decay coefficient, and uniformly quantize attention probabilities to b∈8,12,16b∈\8,12,16\ bits. Integer thresholds remain exact; learned scales and other operators remain floating point, and automatic mixed precision is disabled. Table 3 (right) reports 43.44/42.90 MCC at 16/8 bits, decreases of 0.95/1.48 points from FP32’s 44.39. The 12-bit configuration reaches 40.64 MCC; saturation remains below 0.02% in all cases. The 16- and 8-bit formats therefore retain most downstream accuracy for low-precision states and probabilities. Whole-model comparison. The FP32 MAC/AC accounting summarized in Table 4 estimates 11.17 mJ for four-step Spora, compared with 51.41 mJ for BERT-base, a ratio of approximately 4.60×4.60×. This whole-model estimate includes computation outside the core attention block and therefore has a different scope from the 98.53% core-attention reduction. Table 4: Whole-model arithmetic energy estimates at the reported configurations. Starred values are inferred from energy ratios in the corresponding papers. The accounting uses FP32 MAC/AC reference costs. Model BERT-base LIF-BERTT=4 SpikeLMT=4 SpikingBERT SpikeBERTT=4 SorbetT=16 SporaT=4 Energy (mJ) 51.41 7.98 13.74 12.85∗ 14.74 24.14∗ 11.17 Table 5: Ablation study on GLUE. Spora1 removes BBS encoding from the Query-layer input. SporaSoftmax restores standard scaled dot-product attention while retaining BBS-encoded Q/K/V. Spora w/o KD removes knowledge distillation. Model QQP MNLIm/m SST-2 QNLI RTE MRPC STS-B CoLA Avg. Spora 89.2 79.2/79.4 89.6 85.4 60.0 80.2 82.5 44.1 76.6 Spora1 88.9 79.3/79.4 89.4 85.2 58.6 80.9 83.2 46.8 76.8 SporaSoftmax 88.8 78.5/79.3 89.7 85.0 62.1 81.2 80.7 45.2 76.7 Spora w/o KD 87.1 76.4/76.7 87.2 83.2 57.1 79.4 79.7 38.5 73.9 Figure 5: Attention score visualization of BERT and Spora with time steps 4 and 6 on the CoLA sample “Janet broke the cup.” across 12 layers. Q denotes the query token and K denotes the key token. 4.4 Ablation Study All variants in Table 5 are pre-trained on the same corpus and fine-tuned on GLUE. Replacing approximate attention with standard Softmax changes the average score from 76.6 to 76.7; removing Query-input BBS changes it to 76.8. The complete model thus retains comparable average accuracy with both encoding and approximate attention enabled. These ablations answer complementary questions: Query-input BBS tests the effect of encoding features before the projection, whereas the Softmax variant tests the nonlinear attention approximation with BBS-encoded Q/K/V retained. The small average differences show that both choices preserve task performance within this architecture. Under the same time-step and training budgets, knowledge distillation increases the average from 73.9 to 76.6 and CoLA MCC from 38.5 to 44.1. Increasing the temporal budget. As shown in Table 1, SporaT=6 improves the all-task average from 76.6 to 78.2, a gain of 1.6 points, and improves every reported task column. CoLA rises by 3.3 points, RTE by 3.2, MRPC by 2.0, and STS-B by 1.8. The additional magnitude bit expands the signed codebook from seven to fifteen states while keeping four-stage UBS attention. These results expose a controllable accuracy–representation tradeoff: four-step Spora provides the shorter configuration, while six-step Spora uses finer signed representations to improve task performance. Attention distribution. Figure 5 compares BERT and Spora across 12 layers. Spora exhibits more dispersed attention than BERT on this example, with further dispersion at T=6T=6. Extending magnitude weights from [1,2][1,2] to [1,2,4][1,2,4] increases representational resolution; the associated task improvements are quantified in Table 1. 5 Discussion and Conclusion Spora gives temporal spike sequences a numerical structure that can be used directly by attention operators. Binary weights preserve place values; conditional decay shapes input partitions; sign-separated BBS and learned scales accommodate the feature distributions of language models. The resulting design links representation to accumulation, shifts, and integer-exponent mappings rather than treating the spike encoder as an isolated discretization step. The experiments connect these mechanisms at several levels. Conditional-decay analysis measures local attention fidelity, the matched-budget Static-QAT comparison evaluates the complete encoding design after training, and GLUE ablations characterize the accuracy cost of the selected projection and attention approximations. Extending BBS from four to six steps raises the all-task average from 76.6 to 78.2 and CoLA MCC from 44.1 to 47.4. Event workloads, arithmetic energy estimates, and fixed-point emulation further describe the computational properties of the representation. An immediate extension is to combine activation encoding with weight quantization so that parameter storage and event arithmetic can be optimized together. Implementations that exploit signed bit planes and event sparsity would also allow the operation-level savings to be studied alongside memory traffic and execution overhead. The present results support a joint design principle: short-window codes should preserve useful semantic distinctions and expose numerical structure that makes the next operation simpler. References Bai et al. (2021) H. Bai, W. Zhang, L. Hou, L. Shang, J. Jin, X. Jiang, Q. Liu, M. Lyu, and I. King BinaryBERT: pushing the limit of BERT quantization. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), C. Zong, F. Xia, W. Li, and R. Navigli (Eds.), Online, p. 4334–4348. External Links: Link, Document Cited by: §4.1. Bal and Sengupta (2024) M. Bal and A. Sengupta SpikingBERT: distilling bert to train spiking language models using implicit differentiation. Proceedings of the AAAI Conference on Artificial Intelligence 38 (10), p. 10998–11006. External Links: Link, Document Cited by: §4.1. Bengio et al. (2013) Y. Bengio, N. Léonard, and A. Courville Estimating or propagating gradients through stochastic neurons for conditional computation. External Links: 1308.3432, Link Cited by: §3.4, §3.5. Deng et al. (2009) J. Deng, W. Dong, R. Socher, L. Li, K. Li, and L. Fei-Fei ImageNet: a large-scale hierarchical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, Vol. , p. 248–255. External Links: Document Cited by: §2. Devlin et al. (2019) J. Devlin, M. Chang, K. Lee, and K. Toutanova BERT: pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), J. Burstein, C. Doran, and T. Solorio (Eds.), Minneapolis, Minnesota, p. 4171–4186. External Links: Link, Document Cited by: §2, §4.1, §4. Du et al. (2022) N. Du, Y. Huang, A. M. Dai, S. Tong, D. Lepikhin, Y. Xu, M. Krikun, Y. Zhou, A. W. Yu, O. Firat, B. Zoph, L. Fedus, M. P. Bosma, Z. Zhou, T. Wang, E. Wang, K. Webster, M. Pellat, K. Robinson, K. Meier-Hellstern, T. Duke, L. Dixon, K. Zhang, Q. Le, Y. Wu, Z. Chen, and C. Cui GLaM: efficient scaling of language models with mixture-of-experts. In Proceedings of the 39th International Conference on Machine Learning, K. Chaudhuri, S. Jegelka, L. Song, C. Szepesvari, G. Niu, and S. Sabato (Eds.), Proceedings of Machine Learning Research, Vol. 162, p. 5547–5569. External Links: Link Cited by: §1. Fang et al. (2026) Y. Fang, D. Zhou, Z. Wang, H. Ren, Z. Zeng, L. Li, shibo zhou, and R. Xu Spiking neural networks need high-frequency information. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, External Links: Link Cited by: §2. Horowitz (2014) M. Horowitz 1.1 computing’s energy problem (and what we can do about it). In 2014 IEEE International Solid-State Circuits Conference Digest of Technical Papers (ISSCC), Vol. , p. 10–14. External Links: Document Cited by: §4.3. Hu et al. (2023) Y. Hu, H. Tang, and G. Pan Spiking deep residual networks. IEEE Transactions on Neural Networks and Learning Systems 34 (8), p. 5200–5205. External Links: Document Cited by: §1. Kim et al. (2018) J. Kim, H. Kim, S. Huh, J. Lee, and K. Choi Deep neural networks with weighted spikes. Neurocomputing 311, p. 373–386. External Links: ISSN 0925-2312, Document, Link Cited by: §1, §2. Lee et al. (2025) D. Lee, A. Sima, Y. Li, P. Stinis, and P. Panda SpikePool: event-driven spiking transformer with pooling attention. External Links: 2510.12102, Link Cited by: §2. Li et al. (2024) Y. Li, Y. Lei, and X. Yang Spikeformer: training high-performance spiking neural network with transformer. Neurocomputing 574, p. 127279. External Links: ISSN 0925-2312, Document, Link Cited by: §1. Li et al. (2021) Y. Li, Y. Guo, S. Zhang, S. Deng, Y. Hai, and S. Gu Differentiable spike: rethinking gradient-descent for training spiking neural networks. In Advances in Neural Information Processing Systems, M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. W. Vaughan (Eds.), Vol. 34, p. 23426–23439. External Links: Link Cited by: §1. Liu et al. (2022) Z. Liu, B. Oguz, A. Pappu, L. Xiao, S. Yih, M. Li, R. Krishnamoorthi, and Y. Mehdad BiT: robustly binarized multi-distilled transformer. In Advances in Neural Information Processing Systems, S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (Eds.), Vol. 35, p. 14303–14316. External Links: Link Cited by: §4.1. Loshchilov and Hutter (2019) I. Loshchilov and F. Hutter Decoupled weight decay regularization. In International Conference on Learning Representations, External Links: Link Cited by: §4. Lv et al. (2026) C. Lv, T. Li, W. Qiao, X. Wang, M. Wu, W. Liu, S. Dou, X. Zheng, and X. Huang SpikeBERT: a language spikformer learned from bert with knowledge distillation. Neural Networks 197, p. 108482. External Links: ISSN 0893-6080, Document, Link Cited by: §2. Maass (1997) W. Maass Networks of spiking neurons: the third generation of neural network models. Neural Networks 10 (9), p. 1659–1671. External Links: ISSN 0893-6080, Document, Link Cited by: §1. Mehonic and Kenyon (2022) A. Mehonic and A. J. Kenyon Brain-inspired computing needs a master plan. Nature 604 (7905), p. 255–260. External Links: Document, Link Cited by: §1. Meng et al. (2023) Q. Meng, M. Xiao, S. Yan, Y. Wang, Z. Lin, and Z. Luo Towards Memory- and Time-Efficient Backpropagation for Training Spiking Neural Networks . In 2023 IEEE/CVF International Conference on Computer Vision (ICCV), Vol. , Los Alamitos, CA, USA, p. 6143–6153. External Links: ISSN , Document, Link Cited by: §2. Micikevicius et al. (2018) P. Micikevicius, S. Narang, J. Alben, G. Diamos, E. Elsen, D. Garcia, B. Ginsburg, M. Houston, O. Kuchaiev, G. Venkatesh, and H. Wu Mixed precision training. In International Conference on Learning Representations, External Links: Link Cited by: §4. Qin et al. (2022) H. Qin, Y. Ding, M. Zhang, Q. Yan, A. Liu, Q. Dang, Z. Liu, and X. Liu BiBERT: accurate fully binarized bert. External Links: 2203.06390, Link Cited by: §4.1. Rufin Van Rullen (2001) S. J. T. Rufin Van Rullen Rate coding versus temporal order coding: what the retinal ganglion cells tell the visual cortex. Neural Computation 13 (6), p. 1255–1283. External Links: Link Cited by: §1, §2. Shen et al. (2020) S. Shen, Z. Dong, J. Ye, L. Ma, Z. Yao, A. Gholami, M. W. Mahoney, and K. Keutzer Q-bert: hessian based ultra low precision quantization of bert. Proceedings of the AAAI Conference on Artificial Intelligence 34 (05), p. 8815–8821. External Links: ISSN 2159-5399, Link, Document Cited by: §4.1. Stöckl and Maass (2021) C. Stöckl and W. Maass Optimized spiking neurons can classify images with high accuracy through temporal coding with two spikes. Nature Machine Intelligence 3 (3), p. 230–238. Cited by: §1, §2. Storn and Price (1997) R. Storn and K. Price Differential evolution–a simple and efficient heuristic for global optimization over continuous spaces. Journal of Global Optimization 11 (4), p. 341–359. External Links: Document Cited by: §3.5. Sun et al. (2019) S. Sun, Y. Cheng, Z. Gan, and J. Liu Patient knowledge distillation for BERT model compression. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), K. Inui, J. Jiang, V. Ng, and X. Wan (Eds.), Hong Kong, China, p. 4323–4332. External Links: Link, Document Cited by: §4. Tang et al. (2025) K. Tang, Z. Yan, and W. Wong Sorbet: a neuromorphic hardware-compatible transformer-based spiking language model. In Forty-second International Conference on Machine Learning, External Links: Link Cited by: §1, §2, §4.1, §4.3. Vaswani et al. (2017) A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin Attention is all you need. In Advances in Neural Information Processing Systems, I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (Eds.), Vol. 30, p. . External Links: Link Cited by: §1. Wang et al. (2019) A. Wang, A. Singh, J. Michael, F. Hill, O. Levy, and S. R. Bowman GLUE: a multi-task benchmark and analysis platform for natural language understanding. In International Conference on Learning Representations, p. 353–355. External Links: Link, Document Cited by: §4. Wei et al. (2023) W. Wei, M. Zhang, H. Qu, A. Belatreche, J. Zhang, and H. Chen Temporal-Coded Spiking Neural Networks with Dynamic Firing Threshold: Learning with Event-Driven Backpropagation . In 2023 IEEE/CVF International Conference on Computer Vision (ICCV), Vol. , Los Alamitos, CA, USA, p. 10518–10528. External Links: ISSN Cited by: §2. Wu et al. (2022) J. Wu, C. Xu, X. Han, D. Zhou, M. Zhang, H. Li, and K. C. Tan Progressive tandem learning for pattern recognition with deep spiking neural networks. IEEE Transactions on Pattern Analysis and Machine Intelligence 44 (11), p. 7824–7840. External Links: Document Cited by: §1. Xiao et al. (2022) M. Xiao, Q. Meng, Z. Zhang, D. He, and Z. Lin Online training through time for spiking neural networks. In Advances in Neural Information Processing Systems, S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (Eds.), Vol. 35, p. 20717–20730. External Links: Link Cited by: §2. Xing et al. (2024a) X. Xing, L. Du, X. Wang, X. Zeng, Y. Wang, Z. Zhang, and J. Zhang BiPFT: binary pre-trained foundation transformer with low-rank estimation of binarization residual polynomials. Proceedings of the AAAI Conference on Artificial Intelligence 38 (14), p. 16094–16102. External Links: Link, Document Cited by: §4.1. Xing et al. (2025) X. Xing, B. Gao, Z. Zhang, D. A. Clifton, S. Xiao, L. Du, G. Li, and J. Zhang SpikeLLM: scaling up spiking neural network to large language models via saliency-based spiking. External Links: 2407.04752, Link Cited by: §2. Xing et al. (2024b) X. Xing, Z. Zhang, Z. Ni, S. Xiao, Y. Ju, S. Fan, Y. Wang, J. Zhang, and G. Li SpikeLM: towards general spike-driven language modeling via elastic bi-spiking mechanisms. In Proceedings of the 41st International Conference on Machine Learning, Proceedings of Machine Learning Research. External Links: Link Cited by: §1, §1, §2, §3.3, §4.1. Yao et al. (2023) M. Yao, J. Hu, Z. Zhou, L. Yuan, Y. Tian, B. XU, and G. Li Spike-driven transformer. In Thirty-seventh Conference on Neural Information Processing Systems, External Links: Link Cited by: §1, §2. Yao et al. (2025) M. Yao, X. Qiu, T. Hu, J. Hu, Y. Chou, K. Tian, J. Liao, L. Leng, B. Xu, and G. Li Scaling spike-driven transformer with efficient spike firing approximation training. IEEE Transactions on Pattern Analysis and Machine Intelligence 47 (4), p. 2973–2990. External Links: ISSN 1939-3539, Link, Document Cited by: §2. Yu et al. (2023) M. Yu, T. Xiang, S. P., K. T. N. Chu, B. Amornpaisannon, Y. Tavva, V. P. K. Miriyala, and T. E. Carlson A ttfs-based energy and utilization efficient neuromorphic cnn accelerator. Frontiers in Neuroscience Volume 17 - 2023. External Links: Link, Document, ISSN 1662-453X Cited by: §1, §2. Zhang et al. (2020a) W. Zhang, L. Hou, Y. Yin, L. Shang, X. Chen, X. Jiang, and Q. Liu TernaryBERT: distillation-aware ultra-low bit BERT. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), B. Webber, T. Cohn, Y. He, and Y. Liu (Eds.), Online, p. 509–521. External Links: Link, Document Cited by: §4.1. Zhang et al. (2020b) Y. Zhang, P. Qu, Y. Ji, W. Zhang, G. Gao, G. Wang, S. Song, G. Li, W. Chen, W. Zheng, F. Chen, J. Pei, R. Zhao, M. Zhao, and L. Shi A system hierarchy for brain-inspired computing. Nature 586 (7829), p. 378–384. External Links: Document Cited by: §1. Zhao et al. (2025) L. Zhao, Z. Huang, J. Ding, and Z. Yu TTFSFormer: a TTFS-based lossless conversion of spiking transformer. In Forty-second International Conference on Machine Learning, External Links: Link Cited by: §2. Zhou et al. (2026) C. Zhou, S. Guo, J. Wang, D. Ma, K. Che, B. Chen, Q. Meng, Z. Ma, and Y. Tian Winner-take-all spiking transformer for language modeling. External Links: 2604.11321, Link Cited by: §1, §2, §4.1. Zhou et al. (2023a) C. Zhou, L. Yu, Z. Zhou, H. Zhang, Z. Ma, H. Zhou, and Y. Tian Spikingformer: spike-driven residual learning for transformer-based spiking neural network. CoRR abs/2304.11954. External Links: Link Cited by: §1, §4.1. Zhou et al. (2024) C. Zhou, H. Zhang, Z. Zhou, L. Yu, L. Huang, X. Fan, L. Yuan, Z. Ma, H. Zhou, and Y. Tian QKFormer: hierarchical spiking transformer using q-k attention. In Advances in Neural Information Processing Systems, A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang (Eds.), Vol. 37, p. 13074–13098. External Links: Document, Link Cited by: §2. Zhou et al. (2023b) Z. Zhou, Y. Zhu, C. He, Y. Wang, S. YAN, Y. Tian, and L. Yuan Spikformer: when spiking neural network meets transformer. In The Eleventh International Conference on Learning Representations, External Links: Link Cited by: §1, §2. Zhu et al. (2023) R. Zhu, Q. Zhao, and J. K. Eshraghian SpikeGPT: generative pre-trained language model with spiking neural networks. CoRR abs/2302.13939. External Links: Link Cited by: §1. Zhu et al. (2015) Y. Zhu, R. Kiros, R. Zemel, R. Salakhutdinov, R. Urtasun, A. Torralba, and S. Fidler Aligning books and movies: towards story-like visual explanations by watching movies and reading books. In Proceedings of the IEEE International Conference on Computer Vision, p. 19–27. External Links: Link Cited by: §4.