Paper deep dive
Recursive transformers for semiconductor thermo-mechanical reliability
Kart-leong Lim
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 92%
Last extracted: 8/1/2026, 1:16:08 AM
Summary
This paper evaluates three recursive transformer architectures—Tiny Recursive Model (TRM), Depth Recursive (DEPTH), and Simple Recursive (SIMPLE)—as efficient surrogate models for semiconductor thermo-mechanical reliability analysis. The authors demonstrate that recursive weight-sharing transformers offer a superior trade-off between prediction accuracy, parameter efficiency, and computational cost (FLOPs) compared to conventional over-parameterized transformers, particularly for low-dimensional engineering datasets like stress/warpage prediction and Laplace PDE solving.
Entities (10)
Relation Signals (6)
Depth Recursive transformer → appliedto → Stress10k
confidence 95% · We validate this principle on two low-dimensional engineering prediction tasks... Stress10k... DEPTH... is the primary factor behind the favorable accuracy-per-parameter... trade-offs achieved for package-level Stress10k
Depth Recursive transformer → appliedto → Warpage10k
confidence 95% · We validate this principle on two low-dimensional engineering prediction tasks... Warpage10k... DEPTH... is the primary factor behind the favorable accuracy-per-parameter... trade-offs achieved for package-level... Warpage10k
Recursive weight-sharing transformers → reduces → Parameter count
confidence 95% · recursive weight-sharing transformers provide an effective and generalizable trade-off between prediction accuracy, parameter efficiency, and computational cost
Recursive weight-sharing transformers → reduces → Computational Complexity
confidence 95% · recursive weight-sharing transformers provide an effective and generalizable trade-off between prediction accuracy, parameter efficiency, and computational cost
Depth Recursive transformer → appliedto → PINN
confidence 90% · 2) Laplace PDE iterative numerical solver for capacitance field... PINN... DEPTH... is the primary factor behind the favorable accuracy-per-parameter... trade-offs achieved for... PINN prediction.
Depth Recursive transformer → outperforms → Tiny Recursive Model
confidence 90% · The proposed DEPTH (M6, M7) deliver the strongest or near-strongest retrieval accuracy... while the dual-latent-state recursion of TRM (M4,M5) is both parameter- and FLOP-expensive without a commensurate accuracy return.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Transformer-based surrogate models are increasingly used to replace expensive first-principles simulation in engineering design. But conventional transformer architectures are often over parameterized for the small, low-dimensional datasets typical of engineering design spaces, where large simulation data is expensive to generate. Under these conditions, excess parameter capacity leads to overfitting rather than improved accuracy, while also incurring unnecessary memory and compute overhead. This motivates a shift towards architectures that focus on additional compute rather than additional learnable parameters. This paper presents a hardware-aware evaluation of three recursive transformer paradigms for surrogate thermo-mechanical analysis of advanced packages: a)Tiny Recursive Model, b) our proposed Depth Recursive transformer, c) and a simple recursive transformer. We systematically compare their predictive performance (Recall, Mean Reciprocal Rank), parameter count, computational complexity (FLOPs), providing practical design guidelines for selecting recursive transformer architectures under resource-constrained scenarios. We validate this principle on two low-dimensional engineering prediction tasks: 1) thermo-mechanical reliability analysis of advanced semiconductor packages, where stress and warpage from thermal cycling must be evaluated repeatedly across a design-of-experiments sweep under costly finite element analysis (FEA). 2) Laplace PDE iterative numerical solver for capacitance field. Overall, recursive weight-sharing transformers provide an effective and generalizable trade-off between prediction accuracy, parameter efficiency, and computational cost for small data engineering surrogate modeling.
Tags
Links
- Source: https://arxiv.org/abs/2607.27251v1
- Canonical: https://arxiv.org/abs/2607.27251v1
Trouble viewing inline? Open PDF directly →
Full Text
21,145 characters extracted from source content.
Expand or collapse full text
Recursive weight-sharing transformers for semiconductor thermo-mechanical reliability Abstract Transformer-based surrogate models are increasingly used to replace expensive first-principles simulation in engineering design. But conventional transformer architectures are often over parameterized for the small, low-dimensional datasets typical of engineering design spaces, where large simulation data is expensive to generate. Under these conditions, excess parameter capacity leads to overfitting rather than improved accuracy, while also incurring unnecessary memory and compute overhead — this motivates a shift towards architectures that focus on additional compute rather than additional learnable parameters. This paper presents a hardware-aware evaluation of three recursive transformer paradigms for surrogate thermo-mechanical analysis of advanced packages: a)Tiny Recursive Model, b) our proposed Depth Recursive transformer, c) and a simple recursive transformer. We systematically compare their predictive performance (Recall@K, Mean Reciprocal Rank), parameter count, computational complexity (FLOPs), providing practical design guidelines for selecting recursive transformer architectures under resource-constrained scenarios. We validate this principle on two low-dimensional engineering prediction tasks: 1) thermo-mechanical reliability analysis of advanced semiconductor packages, where stress and warpage from thermal cycling must be evaluated repeatedly across a design-of-experiments sweep under costly finite element analysis (FEA). 2) Laplace PDE iterative numerical solver for capacitance field. Overall, recursive weight-sharing transformers provide an effective and generalizable trade-off between prediction accuracy, parameter efficiency, and computational cost for small-data engineering surrogate modeling — a regime for which conventional large-parameter transformers are poorly suited — as demonstrated on advanced package reliability prediction and capactior electrostatic field modeling. I Introduction Transformer models have demonstrated strong predictive performance across a wide range of engineering applications, including electronic design automation (EDA) [7], scientific machine learning [16], and digital twins [25]. However, their increasing computational and memory requirements present significant challenges for deployment in hardware-constrained environments, such as on-chip inference, edge accelerators, and real-time design optimization [22, 2]. Consequently, substantial research has focused on reducing transformer computation through improvements at the algorithmic, architectural, and system levels [3, 8, 5, 11]. Thermo-mechanical simulation remains computationally expensive, motivating the use of transformer-based surrogate models for rapid reliability prediction. However, deploying such models within practical EDA workflows requires architectures with low computational complexity, compact memory footprints, and high inference throughput. These requirements have motivated a broad range of transformer compute reduction techniques and can be broadly categorized into seven complementary paradigms according to the aspect of computation they optimize. These include Efficient Attention [14, 3], Token Reduction [21, 1], Conditional Computation [8, 6], Architectural Optimization [5, 15, 24, 23], Model Compression [11, 12, 13], Hardware Optimization [4, 20], and State Space Models [10, 9]. Although these approaches pursue the common objective of improving computational efficiency, they address different computational bottlenecks, ranging from reducing attention complexity and sequence length to redesigning network architectures or replacing attention-based models altogether. Among these paradigms, this work focuses on Architectural Optimization, where computational efficiency is achieved by redesigning the transformer itself. In particular, recursive weight sharing enables deeper computation without a proportional increase in parameters [5, 24, 23], making it particularly attractive for resource-constrained EDA applications. Although numerous approaches have been proposed to reduce transformer computation, their suitability depends on the target application. Efficient attention mechanisms primarily benefit long-sequence models where self-attention dominates the computational cost [14, 3], while token reduction techniques rely on large token sets to achieve meaningful computational savings [21, 1]. Conditional computation and Mixture-of-Experts architectures improve scalability for large foundation models but introduce additional routing complexity and memory overhead [8, 6, 19]. Model compression and hardware-specific optimizations are generally applied after model design or target specific execution platforms [11, 13, 4]. In contrast, our application involves compact transformer surrogates operating on short input sequences under strict memory and computational constraints. Consequently, Architectural Optimization provides the most suitable design strategy, enabling computation to be reduced intrinsically through recursive weight sharing while preserving a compact, hardware-friendly architecture [5, 24, 23]. Figure 1: Tiny Recursive model Figure 2: Depth Recursive model Figure 3: Simple Recursive model I Recursive weight sharing transformers The main concept of recursive transformer is to treat transformer as a reusable template (e.g. RECURRECUR) throughout the model. Multiple replicas of RECUR are used in a model, but they all share the single set of weights, which is updated once during each training iteration. RECURRECUR should be independent of models we choose to deploy in. The simplest configuration of RECURRECUR is a one block transformer in Eqn.1 which consists of Self-Attention ATTN(⋅)ATTN(·), Normalization layer NORM(⋅)NORM(·), Feed-Forward Network (FFN). Conversely, if we want a deeper transformer, we can define RECURRECUR using e.g. two transformer blocks. RECUR= = ATTN(⋅∣WQ,WK,WV,WO) (· W_Q,W_K,W_V,W_O) (1) →ADD1+NORM1(⋅∣γ1,β1) _1+NORM_1(· _1, _1) →FFN(⋅∣W1,W2) (· W_1,W_2) →ADD2+NORM2(⋅∣γ2,β2). _2+NORM_2(· _2, _2). I-A Tiny Recursive Model The Tiny Recursive Model (TRM) uses weight sharing RECUR to cap the parameters while perform AI reasoning on Sudoku and puzzle tasks shown in Fig. 1. Within each iteration, RECURZRECUR_Z updates the latent feature z by feeding the input embeddings ∈ℝL×dtoken=[1,…,L]E ^L× d_token=[e_1,…,e_L]–where L is number of tokens and dtokend_token is the embedding dimension–along with the previous latent feature (n−1)z^(n-1), and the previous output feature (n−1)y^(n-1). The updated latent feature (n)z^(n) is then passed to RECURYRECUR_Y, which updates the output representation (n)y^(n) while masking the input embeddings with zeros. Repeating this coupled update for N times defines the LATENTLATENT operation in Eqn.2. This progressively refine both (n)z^(n) and (n)y^(n). LATENT≜(RECURY∘RECURZ)N (RECUR_Y _Z )^N (2) (n)=RECURZ([1,…,L,(n−1),(n−1)]), ^(n)=RECUR_Z\! ([e_1,…,e_L,z^(n-1),y^(n-1)] ), (n)=RECURY([,…,,(n),(n−1)]). ^(n)=RECUR_Y\! ([0,…,0,z^(n),y^(n-1)] ). Finally to compute TRM (Eqn.3), LATENTLATENT is executed for T times and fed into another RECURRECUR. Increasing either the inner N or the outer T loops increases the computation cost (FLOPs) without increasing the parameters through weight sharing. TRM(,ini,ini)≜RECUR∘(LATENT)T=RECUR(,[(RECURY(,RECURZ(,,),))N]T). split&TRM\! (E,z^ini,y^ini ) (LATENT )^T\\ &=RECUR\! (E, [ (RECUR_Y\! (0,RECUR_Z\! (E,z,y ),y ) )^N ]^T ). split (3) I-B Depth Recursive Model From our domain expertise in advanced packaging, we observe that the dataset exhibits a gradually changing depth-like structure for each fixed design case. Motivated by this observation, we repurpose RECUR by treating sequential depth as a state input while fully retaining the weight sharing objective in Fig.2 We call this approach as the Depth Recursive Model (DEPTH). Inspired by Recurrent Neural Networks (RNNs), DEPTH adapts the notion of hidden states to weight-dependent sequential depth states within RECUR. Using the terminology of RNN for sequential modeling, in DEPTH (Eqn.4) we define sequential state as S(t)S^(t), initial hidden state as embedding input =(0)E=H^(0), and sequential (or recursive) output H(t+1)H^(t+1) for depth t∈1,…,Tt∈\1,…,T\. DEPTH(St⊈E,(1),…,(T))≜(RECUR∘(⋅+(t)s))T=RECUR(+(1)s)→(1)→RECUR((1)+(2)s)→(2)⋯→RECUR((T−1)+(T)s)→(T). array[]lDEPTH\! (E_S^t E,S^(1),…,S^(T) ) (RECUR (·+S^(t)W_s ) )^T\\[5.69054pt] =RECUR\! (E+S^(1)W_s ) ^(1)\\ \! (H^(1)+S^(2)W_s ) ^(2)\\ ·s \! (H^(T-1)+S^(T)W_s ) ^(T). array (4) I-C Simple Recursive model The simple recursive model (SIMPLE) in Fig.3 is seen to repeatedly apply an identical recursive transformation. Using (Eqn. 5), SIMPLE applies RECUR for T recursive iterations to generate the output H(T)H^(T). SIMPLE() (E) ≜(RECUR)T (RECUR )^T (5) =RECUR()→(1) =RECUR(E) ^(1) →RECUR((1))→(2) (H^(1) ) ^(2) …→RECUR((T−1))→(T) … (H^(T-1) ) ^(T) I Experimental Results I-A Baselines and proposed models To thoroughly evaluate the tracking and retrieval performance, we compare our approach against seven baseline configurations categorized across four distinct architectural paradigms: • VANILLA: Consists of M1, which serves as a baseline 1-Block Layer Normalization (LN) Transformer architecture. • SIMPLE: Comprises M2 and M3 for Simple Recursive Model, representing a 2-Block Root Mean Square (RMS) recursive structure configured with recursive steps of T=1T=1 and T=3T=3, respectively. • TRM: Represents the Tiny Recursive Model, where M4 denotes a 1-Block-RMS variant incorporating a latent variable (Z), and M5 denotes a 1-Block-LN variant operating over joint latent configurations (Z,YZ,Y). T=5T=5 and N=3N=3 were used. • DEPTH: Comprises Depth Recursive Model, where M6 specifies a 1-Block-RMS design and M7 represents a 2-Block-RMS architecture. The depth is adapted to T=16T=16 for both the Stress10K and Warpage10K dataset. I-B Datasets description We evaluate recursive models across three datasets with partial DOE factorial cases, N=factorslevelsN=factors^levels—the Stress10k and Warpage10k datasets (N=54×16=10,000N=5^4× 16=10,000 pairs) generated by FEA simulation. Synthetic capacitor electrostatic field dataset generated by physics-informed neural network (PINN) (N=222×15=7,265N=22^2× 15=7,265 pairs). Stress10k, Warpage10k [18]: Advanced packaging design requires understanding how material properties and geometric configurations influence thermo-mechanical behavior across spatial and depth-wise dimensions. Design variables –epoxy molding compound (EMC) coefficient of thermal expansion (CTE) and elastic modulus (E), die sizes (SizediesSize_dies), inter-die spacing (GapdiesGap_dies), and position within the package depth (S(t)S^(t)) –determine the resulting stress and warpage responses obtained from finite element analysis (FEA). =[EMCCTE,EMCE,Sizedies,Gapdies,S(t)]∈ℝ5×1. _stress= bmatrixEMC_CTE,\;EMC_E,\;Size_dies,\;Gap_dies,\;S^(t) bmatrix ^5× 1. (6) =(1),(2),…,(16)∈ℝ13×13×16 _stress= \I^(1),I^(2),…,I^(16) \ ^13× 13× 16 PINN [17]: The electrostatic field within an air-filled capacitor can be modeled by establishing collocation and boundary points across a two-dimensional grid. While the Laplace equation governs the internal collocation points, the system’s boundary conditions define the perimeter. In this study, we implement five distinct boundary conditions (C1 through C5) in [17] governed by boundary parameters (a,b,da,b,d). Rather than treating all boundary metrics as static values, we allow the parameters to vary dynamically. Modifying these specific parameters significantly alters the resulting electrostatic field distribution, as demonstrated in Figure 1. PINN=[a,b,d(t)]∈ℝ3×1. _PINN= bmatrixa,b,d^(t) bmatrix ^3× 1. (7) PINN=(1),(2),…,(15)∈ℝ13×13×15 _PINN= \I^(1),I^(2),…,I^(15) \ ^13× 13× 15 Figure 4: Higher FLOPs means a GPU takes longer to run on a single request. Figure 5: Unlike FLOPs, we cannot ”slowly” load a model into VRAM while running it; the entire model must fit into VRAM before inference begins. I-C Pareto Analysis Figure 4 compares MRR against FLOPs across the three datasets. The Pareto frontiers differ across datasets: Stress10K progresses from M1→ 2→ 3→ 7, indicating that M3 remains a competitive intermediate operating point; PINN follows M1→ 2→ 6→ 7, where depth conditioning first becomes Pareto-optimal. Warpage10K transitions M2→ 7, suggesting that the intermediate recursive variants are dominated. The TRM variants (M4–M5) do not lie on the Pareto frontier for any dataset, indicating a less favorable accuracy-to-computation trade-off. Figure 5 compares MRR against Parameters. Unlike the FLOP analysis, the Pareto frontiers consistently favor DEPTH. For PINN, Stress10K and Warpage10K, the Pareto frontier consists exclusively of the DEPTH models (M6–M7), where additional parameters yield further accuracy improvements while remaining Pareto-optimal. The TRM variants (M4–M5) remain well inside the Pareto region despite their larger parameter count, indicating that increasing parameter alone does not translate into better MRR. IV Conclusion The proposed DEPTH (M6, M7) deliver the strongest or near-strongest retrieval accuracy while using among the fewest parameters and lowest FLOPs of all seven models evaluated. The naive weight-sharing recursion of SIMPLE (M2,M3) is FLOP-efficient but accuracy-limited relative to M7 at comparable compute, while the dual-latent-state recursion of TRM (M4,M5) is both parameter- and FLOP-expensive without a commensurate accuracy return. These results support the central claim that explicitly uses depth as a recursively injected input in which each recursive output contributes an individual loss term during BPTT, is the primary factor behind the favorable accuracy-per-parameter and accuracy-per-FLOP trade-offs achieved for package-level Stress10k, Warpage10k, and PINN prediction. References [1] D. Bolya, C. Fu, X. Dai, P. Zhang, and J. Hoffman (2023) Token merging: your vit but faster. In International Conference on Learning Representations (ICLR), Cited by: §I. [2] C. Cioflan et al. (2024) Distributed inference with minimal off-chip traffic for transformers on low-power mcus. arXiv preprint arXiv:2412.04372. Cited by: §I. [3] T. Dao, D. Y. Fu, S. Ermon, A. Rudra, and C. Ré (2022) FlashAttention: fast and memory-efficient exact attention with io-awareness. In Advances in Neural Information Processing Systems (NeurIPS), Cited by: §I. [4] T. Dao (2024) FlashAttention-2: faster attention with better parallelism. In International Conference on Learning Representations (ICLR), Cited by: §I. [5] M. Dehghani et al. (2019) Universal transformers. In International Conference on Learning Representations (ICLR), Cited by: §I. [6] N. Du et al. (2022) GLaM: efficient scaling of language models with mixture-of-experts. In International Conference on Machine Learning (ICML), Cited by: §I. [7] S. Fan, H. Lu, S. Zhang, N. Cao, X. Zhang, and J. Li (2024) Graph-transformer-based surrogate model for accelerated converter circuit topology design. In Proceedings of the 61st ACM/IEEE Design Automation Conference (DAC), Cited by: §I. [8] W. Fedus, B. Zoph, and N. Shazeer (2022) Switch transformers: scaling to trillion parameter models with simple and efficient sparsity. Journal of Machine Learning Research. Cited by: §I. [9] A. Gu and T. Dao (2024) Mamba: linear-time sequence modeling with selective state spaces. In International Conference on Learning Representations (ICLR), Cited by: §I. [10] A. Gu, K. Goel, and C. Ré (2022) Efficiently modeling long sequences with structured state spaces. In International Conference on Learning Representations (ICLR), Cited by: §I. [11] S. Han, J. Pool, J. Tran, and W. J. Dally (2015) Learning both weights and connections for efficient neural networks. In Advances in Neural Information Processing Systems (NeurIPS), Cited by: §I. [12] G. Hinton, O. Vinyals, and J. Dean (2015) Distilling the knowledge in a neural network. In NIPS Deep Learning Workshop, Cited by: §I. [13] E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen (2022) LoRA: low-rank adaptation of large language models. In International Conference on Learning Representations (ICLR), Cited by: §I. [14] A. Katharopoulos, A. Vyas, N. Pappas, and F. Fleuret (2020) Transformers are rnns: fast autoregressive transformers with linear attention. In International Conference on Machine Learning (ICML), Cited by: §I. [15] Z. Lan, M. Chen, S. Goodman, K. Gimpel, P. Sharma, and R. Soricut (2020) ALBERT: a lite bert for self-supervised learning of language representations. In International Conference on Learning Representations (ICLR), Cited by: §I. [16] Z. Li, K. Meidani, and A. B. Farimani (2023) Transformer for partial differential equations’ operator learning. Transactions on Machine Learning Research. Cited by: §I. [17] K. Lim, R. Dutta, and M. Rotaru (2023) Inverse prediction of capacitor multiphysics dynamic parameters using deep generative model. In 2023 IEEE 73rd Electronic Components and Technology Conference (ECTC), p. 1825–1831. Cited by: §I-B. [18] K. Lim and J. Lin (2025) Deep clustering based boundary-decoder net for inter and intra layer stress prediction of heterogeneous integrated ic chip. In 2025 IEEE 75th Electronic Components and Technology Conference (ECTC), p. 239–245. Cited by: §I-B. [19] Mistral AI (2024) Mixtral of experts. arXiv preprint. Cited by: §I. [20] NVIDIA (2023) TensorRT-llm. Note: GitHub Repository Cited by: §I. [21] Y. Rao et al. (2021) DynamicViT: efficient vision transformers with dynamic token sparsification. In Advances in Neural Information Processing Systems (NeurIPS), Cited by: §I. [22] S. Saha et al. (2025) Vision transformers on the edge: a comprehensive survey of model compression and acceleration strategies. arXiv preprint arXiv:2503.02891. Cited by: §I. [23] Samsung SAIL Research (2025) Tiny recursive models. arXiv preprint. Cited by: §I. [24] L. Yang et al. (2024) Looped transformers are better at learning learning algorithms. In International Conference on Learning Representations (ICLR), Cited by: §I. [25] F. Zhang et al. (2026) Transformer self-attention encoder–decoder with multimodal deep learning for response time series forecasting and digital twin support in wind structural health monitoring. Preprint. Cited by: §I.