Paper deep dive
AEGIS: Scaling Long-Sequence Homomorphic Encrypted Transformer Inference via Hybrid Parallelism on Multi-GPU Systems
Zhaoting Gong, Ran Ran, Fan Yao, Wujie Wen
Intelligence
Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 95%
Last extracted: 4/10/2026, 2:09:39 AM
Summary
AEGIS is a compiler-runtime framework designed to scale long-sequence homomorphic encrypted Transformer inference on multi-GPU systems. It addresses the memory and communication bottlenecks of existing approaches by introducing a dependency-aware workload partitioning strategy that co-locates modulus-coherent and token-coherent data, and a polynomial-operator reordering mechanism to overlap collective communication with computation.
Entities (5)
Relation Signals (3)
AEGIS → optimizes → Transformer
confidence 95% · AEGIS, an Application-Encryption Guided Inference System for scalable long-sequence encrypted Transformer inference
AEGIS → utilizes → CKKS
confidence 95% · AEGIS derives device placement from ciphertext dependencies jointly induced by Transformer dataflow and CKKS polynomial coupling
Transformer → runson → Multi-GPU
confidence 90% · scalable long-sequence encrypted Transformer inference on multi-GPU platforms
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Fully Homomorphic Encryption (FHE) enables privacy-preserving Transformer inference, but long-sequence encrypted Transformers quickly exceed single-GPU memory capacity because encoded weights are already large and encrypted activations grow rapidly with sequence length. Multi-GPU execution therefore becomes unavoidable, yet scaling remains challenging because communication is jointly induced by application-level aggregation and encryption-level RNS coupling. Existing approaches either synchronize between devices frequently or replicate encrypted tensors across devices, leading to excessive communication and latency. We present AEGIS, an Application-Encryption Guided Inference System for scalable long-sequence encrypted Transformer inference on multi-GPU platforms. AEGIS derives device placement from ciphertext dependencies jointly induced by Transformer dataflow and CKKS polynomial coupling, co-locating modulus-coherent and token-coherent data so that communication is introduced only when application dependencies require it, while reordering polynomial operators to overlap the remaining collectives with computation. On 2048-token inputs, AEGIS reduces inter-GPU communication by up to 57.9% in feed-forward networks and 81.3% in self-attention versus prior state-of-the-art designs. On four GPUs, it achieves up to 96.62% scaling efficiency, 3.86x end-to-end speedup, and 69.1% per-device memory reduction. These results establish coordinated application-encryption parallelism as a practical foundation for scalable homomorphic Transformer inference.
Tags
Links
- Source: https://arxiv.org/abs/2604.03425v1
- Canonical: https://arxiv.org/abs/2604.03425v1
Trouble viewing inline? Open PDF directly →
Full Text
90,837 characters extracted from source content.
Expand or collapse full text
Scaling Long-Sequence Homomorphic Encrypted Transformer Inference via Hybrid Parallelism on Multi-GPU Systems Zhaoting Gong zgong6@ncsu.edu North Carolina State UniversityRaleigh, NCUSA , Ran Ran rran@ncsu.edu North Carolina State UniversityRaleigh, NCUSA , Fan Yao fan.yao@ucf.edu University of Central FloridaOrlando, FLUSA and Wujie Wen wwen2@ncsu.edu North Carolina State UniversityRaleigh, NCUSA Abstract. Fully Homomorphic Encryption (FHE) enables privacy-preserving Transformer inference, but long-sequence encrypted Transformers quickly exceed single-GPU memory capacity because encoded weights are already large and encrypted activations grow rapidly with sequence length. Multi-GPU execution therefore becomes unavoidable, yet scaling remains challenging because communication is jointly induced by application-level aggregation and encryption-level RNS coupling. Existing approaches either synchronize between devices frequently or replicate encrypted tensors across devices, leading to excessive communication and latency. We present AEGIS , an Application-Encryption Guided Inference System for scalable long-sequence encrypted Transformer inference on multi-GPU platforms. AEGIS derives device placement from ciphertext dependencies jointly induced by Transformer dataflow and CKKS polynomial coupling, co-locating modulus- and token-coherent data so that communication is introduced only when application dependencies require it, while reordering polynomial operators to overlap the remaining collectives with computation. On 2048 tokens input, AEGIS reduces inter-GPU communication by up to 57.9% in FFNs and 81.3% in self-attention versus prior state-of-the-art designs. On four GPUs, it achieves up to 96.62% scaling efficiency, 3.86× end-to-end speedup, and 69.1% per-device memory reduction. These results establish coordinated application-encryption parallelism as a practical foundation for scalable homomorphic Transformer inference. Homomorphic Encryption, Transformer, Multi-GPU, Communication-Aware Scheduling †ccs: Security and privacy Cryptography†ccs: Computer systems organization Distributed architectures†ccs: Computing methodologies Parallel algorithms 1. Introduction Machine Learning as a Service (MLaaS) has become a widely adopted paradigm that enables users to outsource model inference to cloud platforms equipped with large-scale GPU resources. While this paradigm offers convenient and scalable computation, it inherently exposes user inputs to the service provider during inference, raising significant privacy concerns due to potential data leakage. Homomorphic Encryption (HE) (Gentry, 2009) addresses this issue by allowing arbitrary computations to be performed directly on encrypted data without decryption, thereby ensuring data confidentiality throughout the entire inference process. Several HE schemes based on the hardness of the Ring Learning With Errors (RLWE) problem, including BFV (Fan and Vercauteren, 2012), BGV (Brakerski et al., 2014), and CKKS (Cheon et al., 2016), have been proposed to support encrypted arithmetic with different trade-offs between precision, noise growth, and efficiency. Despite their strong security guarantees, HE-based inference introduces substantial computational and memory overheads. Existing work on HE-based private inference has largely focused on algorithmic optimizations to reduce single-device computation overhead, including ciphertext packing (Kim et al., 2022a; Ran et al., 2022; Zhang et al., 2025; Lee et al., 2024a), polynomial approximations for nonlinearities (Park et al., 2024; Lu et al., 2023), and compiler-level optimizations (Dathathri et al., 2019; Reagen et al., 2021; Dathathri et al., 2020; Lee et al., 2022, 2024b; Cowan et al., 2021; Cheon et al., 2024). While these techniques achieve notable speedups, scaling HE inference to Transformer-based LLMs remains an open challenge due to the memory explosion inherent to encrypted computation. For encrypted Transformer inference in particular, existing state-of-the-art systems that primarily target latency (Zhang et al., 2025; Pang et al., 2023; Moon et al., 2024) often omit detailed analysis of memory bottlenecks. As sequence length increases, both encoded model weights and ciphertext activations rapidly exceed the capacity of a single GPU, as illustrated in Figure 4, making multi-GPU execution unavoidable. Even for moderate workloads, encrypted BERT inference with 128 tokens already requires 4×4× A100 GPUs (Zhang et al., 2025). This challenge is further amplified for long-context models. Under CKKS, weights and activations expand by orders of magnitude due to high-precision modulus chains (e.g., a 16-bit weight expands by approximately 110×110× under a 1763-bit modulus (Zhang et al., 2025)) and packing-induced redundancy. Consequently, even BERT-Base with a 2048-token context requires over 700 GB of encoded weights and 110 GB of ciphertext activations, exceeding the capacity of a single compute node and necessitating efficient multi-device inference. To handle large requests and reduce Time to First Token (TTFT) during prefill, plaintext LLM serving relies on multi-node parallel execution with a rich set of parallelization paradigms, most notably data parallelism (DP) (Dean et al., 2012) and tensor parallelism (TP) (Narayanan et al., 2021), and their hybrid variants (Qin et al., 2025), built atop efficient collective communication (Gropp et al., 1994). However, new challenges arise in encrypted inference. First, plaintext tensor parallelism is ineffective under encryption. These schemes fundamentally assume that (1) tensors can be arbitrarily partitioned and (2) collective communication efficiently synchronizes partial results across devices. Encrypted inference violates both assumptions: ciphertext packing encodes tokens into fixed slot layouts, preventing arbitrary tensor partitioning. As a result, directly adopting plaintext techniques either duplicates large encrypted activations that cause prohibitive memory overhead or yields redundant computation and communication due to slot-format constraints (Section 3.2). Second, HE exposes a native parallelization mechanism based on the Residue Number System (RNS) (Gentry et al., 2012). Commonly referred to as limb parallelism (see Figure 1), which is widely adopted in state-of-the-art HE accelerators (Jayashankar et al., 2025a; Al Badawi et al., 2021b; Wang et al., 2023). However, our profiling shows that this parallelism does not translate to scalable end-to-end encrypted Transformer inference. Despite evenly partitioned memory, performance is dominated by excessive inter-device communication during key switching (Kim et al., 2023), driven by frequent rotations in large and complex encrypted matrix multiplications (see Section 3.2.1). On an RTX A6000 system with NVLink, scaling to 2 GPUs even results in a 2.25× slowdown, demonstrating that it is fundamentally mismatched to Transformer workloads. Third, managing collective communication itself is significantly more challenging than in plaintext. In encrypted inference, even a single token transmission requires sending multiple full ciphertexts, introducing extreme redundancy. For example, transmitting 1024 tokens requires only ∼ 2 MB in plaintext but expands to approximately 1.3 GB under encryption (see Table 2). This communication amplification dramatically increases latency and bandwidth demand, rendering conventional collective strategies ineffective. Together, these challenges demonstrate that no existing solutions offer scalable encrypted Transformer inference, motivating a fundamentally different parallel execution model. Existing systems fail to scale long-sequence encrypted Transformers because application-driven approaches are not encryption-aware, while encryption-driven approaches ignore application-level structure. To bridge this gap, we propose AEGIS , a framework for scalable encrypted Transformer inference on multi-GPU platforms. Our main contributions are as follows: • We present AEGIS , a compiler-runtime co-designed framework for scalable long-sequence encrypted Transformer inference on multi-GPU platforms. It jointly orchestrates application-level data flow and low-level primitive polynomial execution to enable efficient multi-GPU parallelism. • We propose a dependency-aware workload partitioning strategy that bridges application semantics and encryption constraints. AEGIS incorporates both encryption-aware modulus-coherent placement and application-aware token-coherent placement to co-locate slices of the same modulus chain or token, minimizing communication induced by application-level reductions. • We develop a polynomial-operator reordering mechanism that restructures low-level instruction schedules to overlap collective communication with computation, thereby hiding inter-device communication latency. This design enables deterministic communication hiding without relying on runtime heuristics or dynamic synchronization. • Experiments demonstrate that AEGIS consistently outperforms state-of-the-art designs in both latency and memory efficiency, achieving up to 92.98% scaling efficiency and 1.86× end-to-end speed-up on two GPUs, while also reducing per-device memory consumption by up to 50.86% compared to baselines that replicate intermediate ciphertexts. 2. Preliminaries Table 1. Notations and symbols used in this paper Notation Description N Polynomial degree (number of coefficients) S Number of available slots in an encoded message QLQ_L Ciphertext modulus chain q0,q1,…,qL\q_0,q_1,…,q_L\ P Special modulus chain qL+1,qL+2,…\q_L+1,q_L+2,…\ L Length of the modulus chain (maximum multiplicative depth) l∈[0,L)l∈[0,L) Current remaining multiplicative level G Number of GPU devices dmodeld_model Embedding dimension of Transformer layers dheadd_head Dimensionality of each attention head T Input sequence length (number of tokens) stoks_tok Number of elements of each token encoded into one ciphertext ctokc_tok Number of ciphertexts used to represent a single token 2.1. Homomorphic Encryption and Operations RLWE-based HE schemes perform computation over the polynomial residue ring ℤq[x]/(xN+1)Z_q[x]/(x^N+1), where N (a power of two) is the ring dimension and q the ciphertext modulus. In CKKS, a modulus chain Q=∏i=0LqiQ= _i=0^Lq_i is used to support multiple levels of computation. A plaintext vector of length S=N/2S=N/2 is encoded as a complex polynomial and encrypted under a secret key, where S denotes the slot number, which means the number of complex values that can be packed into one ciphertext. The packed values are processed simultaneously, enabling Single Instruction Multiple Data (SIMD)-like computation across slots in subsequent homomorphic operations. CKKS supports linear operations PAdd, CAdd, PMult, and CMult, each multiplication increasing both scale and noise and thereby consuming one level of the modulus chain. The Rescale operation divides by the last modulus qLq_L to reduce scale and remove one prime, while Relinearization (Relin) restores ciphertext size and suppresses noise growth through key switching. Additional primitives include rotations Rotρ, which cyclically shift plaintext slots by offset ρ, and bootstrapping (Bossuat et al., 2021; Chen et al., 2018; Jung et al., 2021) (Boot), which replenishes the modulus chain to enable deeper computation. Scalar operations (P/CAdd, P/CMult) act only on the main primes (q0,…,qL)(q_0,…,q_L), whereas key-switching operations (Relin, Rot, Boot) temporarily extend the modulus to ℤQ⋅PZ_Q· P, where P is the product of special primes used for digit decomposition. After switching, ciphertexts are reduced back to ℤQZ_Q, and the special primes remain active only during key-switching. 2.2. Data Structure in Homomorphic Encryption Figure 1. Illustration of data layout and parallelization models in CKKS computation. (a) Packing that maps plaintext tensors into ciphertext slots. (b) Coefficient-parallel, distributing work across polynomial coefficients. (c) Limb-parallel, distributing work across RNS primes and exposing thread parallel on coefficient. Three-part illustration of CKKS data layout and parallelization. It shows plaintext values packed into ciphertext slots, coefficient-parallel execution across polynomial coefficients, and limb-parallel execution across RNS primes. SOTA CKKS frameworks (37; SEAL (2023); A. A. Badawi, A. Alexandru, J. Bates, F. Bergamaschi, D. B. Cousins, S. Erabelli, N. Genise, S. Halevi, H. Hunt, A. Kim, Y. Lee, Z. Liu, D. Micciancio, C. Pascoe, Y. Polyakov, I. Quah, S. R.V., K. Rohloff, J. Saylor, D. Suponitsky, M. Triplett, V. Vaikuntanathan, and V. Zucca (2022); DESILO (2023)) employ the Chinese Remainder Theorem (CRT) to represent large polynomial coefficients in Residue Number System (RNS) form, enabling efficient arithmetic under large moduli. Each ciphertext component is decomposed over primes (q0,q1,…,qL)(q_0,q_1,…,q_L), where every coefficient c∈ℤQc _Q is stored as residues cmodqi\\,c q_i\,\. The remaining number of primes determines the ciphertext’s available multiplicative depth: fewer primes yield smaller ciphertexts and reduced capacity. As illustrated in Figure 1, all data objects in CKKS including plaintexts, ciphertexts, and keys, share a common algebraic basis ℛQR_Q and can be viewed as structured tensors. The coefficient and modulus dimensions correspond to the polynomial degree N and the modulus chain Q. A ciphertext ct=(c0,c1)∈ℛQ2ct=(c_0,c_1) _Q^2 contains two polynomial components with N coefficients replicated across all primes in Q. Conceptually, it forms a 3D tensor with RNS, coefficient, and polynomial dimensions. GPU implementations typically adopt an RNS-major layout, storing residues under the same modulus contiguously across coefficients, improving coalesced global-memory access, enabling per-prime kernel launches, and aligning with RNS-partitioned execution. 2.3. Parallel Residual Number System Since every encrypted tensor in CKKS is structured within the RNS, multi-GPU execution can be viewed as parallelizing the RNS tensor across devices. Two RNS-native strategies are used: coefficient partitioning (Al Badawi et al., 2021b; Wang et al., 2023), which distributes polynomial coefficients among GPUs (Figure 1(b)), and limb partitioning, which assigns subsets of primes in the modulus chain to different devices (Fan et al., 2023; Jayashankar et al., 2025a; Kim et al., 2022b; Wang et al., 2023; Deng et al., 2024). With typical CKKS parameters containing multiple primes q0,…,qL−1q_0,…,q_L-1, limb partitioning achieves scalable parallelism by distributing prime-indexed tensors across devices (Figure 1(c)), while each device exposes coefficient-level parallelism through its streaming multiprocessors (SMs). Formally, device GPUdGPU_d processes coefficient tensors qi∣imodD=d\\,q_i i D=d\,\, where D is the number of devices. For example, in Figure 1(c), a ciphertext with Q=q0,q1,…,q5Q=\q_0,q_1,…,q_5\ is mapped onto three GPUs: GPU0→q0,q3GPU_0\!→\!\q_0,q_3\, GPU1→q1,q4GPU_1\!→\!\q_1,q_4\, and GPU2→q2,q5GPU_2\!→\!\q_2,q_5\. RNS parallelization is inherently application-independent: it operates directly on the ciphertext’s modular structure and partitions work according to the modulus chain, irrespective of how the plaintext is encoded. Figure 2. Encrypted Transformer inference pipeline with bootstrapping placement (NEXUS (Zhang et al., 2025)). Pipeline diagram of encrypted Transformer inference, showing linear layers, attention, nonlinear layers, and where bootstrapping is inserted to restore multiplicative depth. 2.4. Encrypted Transformer Inference A Transformer block consists of self-attention and feed-forward layers. Given an input tensor X, linear projections produce Q, K, and V, followed by attention computation A=softmax(QK⊤/d)A= softmax(QK / d), aggregation AVAV, and a feed-forward network with activation, normalization, and residual connections. During the prefill phase, inference processes sequences of up to thousands of tokens, and runtime memory is dominated by model parameters and activations. In encrypted inference, the client encrypts X into ciphertexts and sends them to the server for evaluation. As shown in Figure 2, Transformer operations are mapped to homomorphic primitives: linear layers use PMult, attention and activations use CMult, and nonlinear functions are approximated by high-degree polynomials combining both. Since each multiplication consumes one modulus level, bootstrapping (Boot) is periodically inserted to restore noise budget and enable deep Transformer execution. Encrypted inference incurs substantial memory overhead from three sources: ciphertext activations, key-switching keys (for rotations, CMult, and bootstrapping), and pre-encoded plaintext weights. Pre-encoding weights is essential, as it trades storage for reduced computation. For example, reducing a BERT linear projection over 128 tokens from 354.31s354.31\,s to 270.22s270.22\,s. Figure 3. Fundamental parallelisms in plaintext inference. (a) Data parallelism replicates model weights across devices. (b,c) Tensor parallelism partitions weights and aggregates partial results via collectives. Three-panel overview of plaintext inference parallelism. It contrasts data parallelism, which replicates weights, with tensor parallelism, which shards weights and then uses collectives to combine partial results. 2.5. Parallel Models and Collective Primitives Plaintext LLM serving relies on hybrid parallelism (Shoeybi et al., 2020; Lepikhin et al., 2020; Li et al., 2023; Pope et al., 2023) to scale model capacity and reduce time-to-first-token (TTFT). On multi-GPU systems, these approaches are built upon fundamental partitioning schemes, data parallelism (DP) (Dean et al., 2012) and tensor parallelism (TP) (Narayanan et al., 2021), as illustrated in Figure 3. Each GPU processes a shard of the tensor and periodically exchanges intermediate results, trading computation for inter-GPU communication over interconnects such as NVLink or PCIe. This communication is supported by collective primitives such as Send&Recv, AllReduce, AllGather, and AllToAll, which enable scalable synchronization without explicit point-to-point orchestration. For example, AllReduce synchronizes gradients in DP (Dean et al., 2012), while AllGather aggregates partial activations in TP (Shoeybi et al., 2020). Recent systems further reduce communication overhead by exploiting structured dependencies in model architectures (Narayanan et al., 2021; Fedus et al., 2022). Similarly, GPU-accelerated CKKS inference employs inter-device communication to coordinate encrypted polynomial fragments partitioned across coefficients or RNS limbs. Prior work uses explicit Send&Recv to convert between coefficient- and limb-parallel layouts (Al Badawi et al., 2021b), or enforces hard synchronization via AllGather to reconcile modulus-dependent ciphertext components (DESILO, 2023). 2.6. Threat Model and Security Level We assume a semi-honest threat model, in which the cloud server follows the prescribed inference protocol but may attempt to infer information from encrypted data. The client encrypts inputs locally, uploads ciphertexts and public keys to an untrusted MLaaS provider, and decrypts the returned results. Security under CKKS relies on the hardness of the Ring Learning With Errors (RLWE) problem at a target security level λ (typically 128 bits). The parameter pair N,Q\N,Q\ jointly determines ciphertext precision and size. A larger modulus chain Q=q0,…,qLQ=\q_0,…,q_L\ supports greater multiplicative depth, which is required by deeper models (approximately 55–88 for CNNs, 1010–1212 for GCNs, and over 2020 for Transformers), but also increases ciphertext size, as storage grows with both |Q| Q and the bit-length of each qiq_i. For Transformer, this typically results in a modulus chain exceeding 1600 bits. 3. Motivation Figure 4. Memory breakdown of encrypted BERT-Base inference across input sequence lengths. The stacked bars represent the major memory components. Common NVIDIA GPU memory capacities are marked for reference. Stacked bar chart of encrypted BERT-Base memory versus sequence length. Activation memory grows rapidly and becomes dominant at long sequences, exceeding common GPU memory capacities. 3.1. Distributing Large-Scale HE Inference Encrypted Transformer inference incurs substantial memory overhead from pre-encoded weights, ciphertext activations, and user keys. Although weights can be streamed layer-by-layer from CPU memory, even moderate-scale models expand to hundreds of gigabytes after encoding. Ciphertext activations grow rapidly with sequence length and soon dominate runtime memory usage, as shown in Figure 4. At sequence length 128, runtime memory already includes 666.89MB666.89\,MB of activations, 10.64GB10.64\,GB of encoded weights, and 20.45GB20.45\,GB of user keys. At sequence length 2048, activations alone reach 52.95GB52.95\,GB, exceeding the capacity of a single GPU even for a small model like BERT-Base (Devlin et al., 2019). Consequently, practical encrypted inference necessarily requires distributing both ciphertexts and weights across multiple devices. However, efficient distribution is challenging because encrypted computation induces communication patterns fundamentally different from plaintext inference. At the application level, cross-device reductions occur along either the token or embedding dimension, but in the encrypted domain these reductions require transmitting entire ciphertexts. Without loss of generality, under a packing layout (stok,ctok)(s_tok,c_tok), each ciphertext allocates stoks_tok slots to one token, so a dmodeld_model- dimensional embedding is first split across ctok=⌈dmodel/stok⌉c_tok= d_model/s_tok ciphertexts. Each ciphertext packs multiple tokens horizontally, up to S/stokS/s_tok tokens per ciphertext, where S is the total slot count. Transmitting T tokens therefore requires Ksend(T)=⌈T/(S/stok)⌉⋅ctokK_send(T)= T/(S/s_tok) · c_tok ciphertexts at level l, yielding a point-to-point communication volume of P2P(T,l)≈Ksend(T)⋅(2⋅l⋅2N⋅8bytes)P2P(T,l)≈ K_send(T)·(2· l· 2N· 8~bytes). Collective operations amplify this cost: AllReduce(T,l)≈(G−1)⋅P2P(T,l)AllReduce(T,l)≈(G-1)·P2P(T,l) and AllGather(T,l)≈(G−1)⋅P2P(T/G,l)AllGather(T,l)≈(G-1)·P2P(T/G,l). This highlights the severe communication amplification inherent to encrypted inference. Limb-parallel designs are constrained by RNS partitioning and frequent key switching, while tensor-parallel designs scale with both ctokc_tok and the multiplicative depth l. For example, transmitting 1024 tokens at full depth on two devices requires 1.3 GB for P2P transfer, 3.8 GB for AllReduce, and 0.96 GB for AllGather, compared to only a few megabytes in plaintext (Table 2). These costs demonstrate that while distributing HE inference is unavoidable, achieving scalability critically depends on minimizing when, where, and how collectives are invoked. Table 2. Per-device communication cost of common collectives in plaintext and encrypted inference. Example assumes T=1024T=1024, dmodel=1024d_model=1024, bf16, full slot utilization, S=32768S=32768, and bootstrappable CKKS parameters (∼ 40 MB ciphertexts at l=35l=35). Encrypted communication is 2–3 orders of magnitude more expensive than plaintext. Collective Plaintext cost Plaintext example Encrypted cost Encrypted example P2P send (T(T tokens)) T⋅dmodel⋅sizeof(bf16)T· d_model· sizeof(bf16) 1024⋅1024⋅2≈21024· 1024· 2≈ 2 MB Ksend(T)⋅(2⋅l⋅2N⋅8)K_send(T)·(2· l· 2N· 8) 32×4032× 40 MB ≈1.3≈ 1.3 GB AllReduce (T(T tokens)) (G−1)T⋅dmodel⋅sizeof(bf16)(G-1)\,T· d_model· sizeof(bf16) 3×23× 2 MB =6=6 MB (G−1)Ksend(T)⋅(2⋅l⋅2N⋅8)(G-1)\,K_send(T)·(2· l· 2N· 8) 3×1.33× 1.3 GB ≈3.8≈ 3.8 GB AllGather (T(T tokens)) (G−1)TG⋅dmodel⋅sizeof(bf16)(G-1)\, TG· d_model· sizeof(bf16) 3×0.53× 0.5 MB =1.5=1.5 MB (G−1)Ksend(TG)⋅(2⋅l⋅2N⋅8)(G-1)\,K_send\! ( TG )·(2· l· 2N· 8) 3×(8×403×(8× 40 MB)≈0.96)≈ 0.96 GB 3.2. Limitations of Existing Parallel Models 3.2.1. Data Dependency in HE-based Inference Encrypted inference applications are expressed as HE-operator control flows and compiled into polynomial instructions. Each abstraction layer—application, HE operator, and polynomial instruction—exhibits distinct yet interrelated data dependencies, together forming a complex multi-dimensional coupling structure that challenges efficient parallelization. At the application level, model semantics define structured dependencies among ciphertexts: Linear layers perform embedding-wise aggregation, attention blocks perform head-wise aggregation, and normalization layers perform token-wise aggregation. These patterns, realized through ciphertext packing, propagate coupling along both coefficient and modulus dimensions and directly influence pipeline organization. At the HE operator level, HE primitives introduce intrinsic coupling: Mult induces coefficient-wise dependencies through the NTT / INTT pipeline, while Rot introduces both coefficient- and modulus-wise dependencies due to KeySwitch within it. At the polynomial instruction level, ciphertexts exhibit structured dependencies across coefficients and modulus chains. Operations such as NTT/INTT and automorphisms create intra-prime coupling, where each output coefficient depends on all input coefficients within the same modulus, while cross-prime coupling arises during residue reconstruction and base conversion in KeySwitch, where new residues are computed from all lower-level residues in ModUp. 3.2.2. Limb Parallelism Is Communication-Bound and Depth-Limited Profiling inter-device traffic shows that key switching dominates cross-device communication in encrypted Transformer workloads. Rotations in matrix multiplications, polynomial nonlinearities, and relinearization all induce instruction-level coupling across RNS limbs of the same ciphertext component, making repeated synchronization unavoidable under limb-parallel execution. As a result, limb parallelism scales poorly, even negatively across GPUs. On an RTX A6000 system, a 512×512512× 512 ciphertext matrix multiplication takes 16.7s16.7\,s on one GPU, but increases to 37.5s37.5\,s on two GPUs and 62.3s62.3\,s on four GPUs. In the two-GPU case, communication alone accounts for 59%59\% of runtime, with key switching contributing an additional 34%34\%. Down to the computation graph, in BSGS matrix multiplication (Kim et al., 2022a; Ebel et al., 2025), each modulus-switching step triggers an AllGather to synchronize RNS limbs across devices (Figure 5(a)). The total rotation cost accumulates as trot=nrot⋅(τks+τcomm)t_rot=n_rot·( _ks+ _comm), and long-sequence inference requires tens of thousands of rotations (Kim et al., 2022a; Ran et al., 2023; Juvekar et al., 2018; Pang et al., 2023). Consequently, communication dominates latency; Figure 5(b) shows that 59%59\% of a single matrix multiplication is spent waiting on communication. Beyond communication costs, limb parallelism is fundamentally limited by modulus depth. A level-l ciphertext can be partitioned across at most l devices, since each RNS limb corresponds to one modulus prime. Thus, a ciphertext with l=6l=6 cannot scale beyond six GPUs regardless of tensor size or sequence length (Figure 1). In practice, even bootstrappable workloads with l≈14l\!≈\!14 (Zhang et al., 2025; Moon et al., 2024) are insufficient for scalable long-sequence inference (see Section 4.5). Figure 5. (a) Collective in RNS-parallel BSGS matrix multiplication down to compute graph level. (b) Latency breakdown of ciphertext matrix multiplication on a 2-GPU with NVLink. Two-panel figure. Panel a shows collectives inserted repeatedly during RNS-parallel BSGS ciphertext matrix multiplication. Panel b shows that on two GPUs with NVLink, communication and key switching dominate ciphertext matrix multiplication latency. 3.2.3. Conventional Tensor Parallelism Suffers Redundancy Plaintext-style tensor parallelism (TP) introduces severe inefficiencies when directly applied to homomorphic encrypted inference. Under typical CKKS packing, each token is represented by ctok=⌈dmodelstok⌉c_tok= d_models_tok ciphertexts. Column tensor parallelism (Figure 3(b)) replicates encrypted inputs across devices, quickly exhausting GPU memory since ciphertext tensors are orders of magnitude larger than their plaintext counterparts. Row tensor parallelism (Figure 3(c)) fares no better: partitioning along the embedding dimension still requires each device to hold all ctokc_tok ciphertexts to reconstruct complete token features, resulting in redundant storage and duplicated computation. This inefficiency is illustrated in Figure 6. Figure 6. (a) Encrypted diagonal matrix multiplication. (b) Directly applying plaintext tensor parallelism induces redundant ciphertext replication. Two-panel diagram. Panel a illustrates encrypted diagonal matrix multiplication. Panel b shows that directly reusing plaintext tensor parallelism replicates ciphertexts across devices and causes redundant encrypted work. Furthermore, because each ciphertext decomposes into per-prime polynomial fragments, transferring even a single token requires communicating all ctokc_tok ciphertexts across every modulus in QLQ_L. This inflates communication volume by orders of magnitude relative to plaintext execution, rendering plaintext-style token-aware collectives fundamentally inefficient under HE. 4. AEGIS As outlined in Section 3, supporting long input sequences on a multi-GPU system exposes two primary bottlenecks: memory usage and communication overhead. To address them, a parallelization strategy that jointly considers application-aware and encryption-aware optimizations is required, since data dependencies arise simultaneously from Transformer computation patterns and RNS-level HE operations. In this section, we introduce AEGIS , a multi-GPU execution framework designed to achieve high-throughput for long-sequence encrypted Transformer inference through coordinated optimization across the application layer and the polynomial-instruction layer. We begin by presenting a polynomial placement algorithm that assigns ciphertext components and HE operators to devices through a unified analysis of application-level data dependencies and RNS-level coupling among polynomial limbs. To further alleviate communication overhead, we introduce a polynomial–operator reordering mechanism that restructures low-level instruction schedules to overlap collective communication with computation, effectively hiding inter-device latency. 4.1. Dependency-Aware Workload Partitioning The fundamental mismatch between application-level tensor semantics and the encryption-level Residue Number System (RNS) representation in CKKS arises from how existing HE inference systems lower high-level application graphs into polynomial instructions. At runtime, however, each device observes only local polynomial fragments, either coefficient or modulus (prime) slices, without visibility into token boundaries, packing layouts, or aggregation semantics. As a result, plaintext-style partitioning and collectives require full ciphertext reconstruction, inducing excessive cross-device communication and yielding scalability that is dominated by RNS coupling rather than problem size. The key observation is that distributed CKKS execution is dominated by reduction operations arising from both RNS coupling and application-level aggregation, including embedding-wise accumulations in linear projections, sequence-wise reductions in attention, and reductions in layer normalization and softmax. To address this, AEGIS adopts a communication-efficient, encryption-aware polynomial partitioning strategy that jointly considers RNS dependencies and Transformer aggregation patterns, prioritizing co-location–aware data placement as introduced below. Modulus-coherent placement. Conventional limb-parallel schemes assign polynomials independently by global index p∣pi,imodG=j→device j\\,p p_i,\;i G=j\,\ j, treating each polynomial as an isolated unit and ignoring cross-limb dependencies induced by key switching and rotations. In contrast, AEGIS enforces modulus-coherent placement through a polynomial-level dependency analysis of the HE computation graph. After lowering the application graph into polynomial operators, AEGIS identifies groups of polynomials that must participate jointly in key switching or rotation chains. All polynomials within such a dependency group are first mapped to the same device before the next key-switching operation in the computation graph, ensuring that key switching and rotation remain local and eliminating cross-device synchronization. When the number of dependency groups is not divisible by G, complete groups are assigned first, and splitting is used only as a fallback for load balancing with communication hiding (see Section 4.4.1). Token-coherent placement. AEGIS additionally performs an application-level analysis before polynomial lowering to identify token-wise dependencies. Using the packing layout (stok,ctok)(s_tok,c_tok) introduced in Section 3.1, each token spans ctokc_tok ciphertexts. For ctok>1c_tok>1, dependency groups corresponding to different slices of the same token are co-located whenever possible. This token-coherent placement aligns computation with embedding-wise reductions and avoids synchronizing partial token results across devices. Although we do not focus on packing design in this paper, the placement rule itself is packing-agnostic: when (stok,ctok)(s_tok,c_tok) changes, AEGIS re-derives ciphertext dependency groups from the new layout and applies the same modulus-first, token-second hierarchy. Together, AEGIS follows a strict placement hierarchy in which modulus chain dependency is of first priority and token coherence is secondary priority; these principles underpin the workload partitioning strategies in Section 4.2 and Section 4.3. Figure 7. Output-slot coupling: (a) weight slices within the same output group generate results on the same slots; (b) splitting token output-coupled weight onto 2 devices leads to half slot waste in the next computation. Two-panel diagram of output-slot coupling in encrypted linear layers. Weight slices for the same output group map to the same slots, and splitting them across two devices leaves many output slots unused in the next step. Figure 8. Parallel self-attention: (a) data partition and communication insertion; (b) latency hiding via operator reordering. Two-panel wide figure for encrypted self-attention. Panel a shows how data is partitioned and where collectives are inserted. Panel b shows operator reordering that overlaps communication with computation to hide latency. 4.2. Workload Partition of FFNs For long-sequence encrypted inference, neither encrypted activations nor pre-encoded plaintext weights can be hosted on a single device. In CKKS, weight slices are encoded in the same format as encrypted inputs and participate in accumulation chains determined by the encrypted computation graph. For encrypted activations, slices within a token are accumulation-dependent, while slices across tokens are independent. In contrast, weight slices are structured by input-output groups: slices contributing to the same output dimension jointly determine the slots of the output ciphertext. Unlike plaintext inference, where splitting weights across devices leads to an AllGather, encrypted inference requires slot-wise accumulation. Weight slices contributing to the same output dimension must be accumulated into identical ciphertext slots, resulting in an AllReduce-like behavior. Since ciphertexts must be transmitted in entirety, partial results cannot be communicated independently. As shown in Figure 7, distributing weight slices from the same output group across devices leads to partial slot utilization: each device produces ciphertexts with only a fraction of valid slots. Communicating such ciphertexts doubles transfer volume without increasing useful work and may additionally introduce redundant nonlinear computation. More generally, plaintext AllGather patterns along the embedding dimension degenerate into encrypted AllReduce whenever slot utilization is partial. For example, with 1024 tokens, each linear layer incurs approximately 3.8 GB of extra inter-GPU communication under such misplacement (Table 2). Accordingly, FFN partitioning in AEGIS enforces the rule that weight slices belonging to the same output group must remain co-located. Under this placement, at the entry of each linear layer, every device holds O(T⋅l/G)O(T· l/G) polynomials corresponding to a disjoint subset of output groups. Each device initiates a single AllGather on a dedicated communication stream to fetch the required encrypted activations, while simultaneously performing local polynomial multiplication on its resident weight slices. After G rounds of overlapped computation and communication, each device produces its corresponding portion of the output ciphertexts. The intervening GELU nonlinearity between FFNs is then evaluated locally as an element-wise operation, requiring no additional communication. 4.3. Workload Partitioning of Self-Attention Following the placement rules in Section 4.1, self-attention is partitioned to distribute the scaled dot-product and softmax computation while preserving ciphertext coherence. After the QKV projections (Section 4.2), token polynomials remain logically partitioned along the embedding dimension. Using 2×2× GPU as an example, token i is split into two polynomial groups, denoted q/k/vi1q/k/v_i1 and q/k/vi2q/k/v_i2, corresponding to disjoint slot regions (Figure 8). Execution proceeds in three phases. (1) Partial attention computation: each device computes rotate–multiply–accumulate over its local Q×K⊺Q× K slices. Because attention scores aggregate across the full embedding dimension, this phase induces an AllReduce-like dependency at the application level. (2) Distributed softmax: instead of directly performing AllReduce, the intermediate attention polynomials are reshaped via ReduceScatter so that each device holds the data required for local softmax evaluation. Polynomial approximation of exe^x is then applied independently using the numerically stable online softmax procedure (Milakov and Gimelshein, 2018), as adopted in prior encrypted attention workloads (Zhang et al., 2025; Lu et al., 2023), without further communication. (3) Output aggregation: the normalized attention scores are AllGathered so that each device obtains the weights needed to multiply with its local V slices. The final A×VA× V computation is then entirely local, producing token-slice outputs without additional synchronization. During execution, some polynomial slices fully reconstruct tokens while others form partial ciphertexts (see Section 4.1). The communication operators required to preserve ciphertext correctness in such cases are handled by the rules in Section 4.4. 4.4. Communication Operator Insertion Although the RNS representation causes the system to view all data as uniformly sized polynomials, effective communication placement must respect both application-level aggregation semantics and underlying RNS polynomial dependencies. We therefore propose a communication-operator insertion strategy guided jointly by token-wise aggregation patterns and RNS-level data placement. Following the placement strategy in Section 4.1, if a set of polynomials residing on the same GPU corresponds to the same token representation, local reduction is first applied before any cross-device communication during token-wise aggregation. If all ctok⋅lc_tok· l polynomials corresponding to a token are colocated on a single device, no communication is required for embedding-wise aggregation. Communication is only inserted when the polynomials corresponding to the same token are distributed across devices; in this case, collectives are introduced exclusively among devices holding the relevant data. These collectives follow layer-specific application patterns as described in Section 4.2 and Section 4.3. Additional rules govern communication placement during layer execution. For each GPU, if its local polynomial set constitutes a complete ciphertext and the subsequent operation in the computation graph permits reordering, communication is scheduled according to the following rules: (1) Send before bootstrapping: although the logical workload remains unchanged, communication volume is reduced since bootstrapping restores multiplicative depth and increases the number of limbs l. (2) Rescale before send: rescaling eliminates one limb, directly reducing transfer size. (3) Local reduction before send: partial reductions are performed locally to minimize communication volume. Violating these rules leads to redundant computation and communication: delaying communication until after bootstrapping or rescaling forces devices to transmit enlarged ciphertexts and to recompute equivalent polynomial transformations independently, while skipping local reduction causes multiple devices to perform identical aggregation work on overlapping data. These rules therefore apply uniformly across different layers and applications to avoid redundant computation and unnecessary data movement. Figure 9. The AEGIS compile pass. Block diagram of the AEGIS compile pass, showing lowering from a high-level application representation to encrypted operators, then to polynomial-level instructions, followed by scheduling and multi-GPU execution planning. 4.4.1. Communication Hiding via Operator Reordering AEGIS eliminates GPU stalls caused by embedding-wise aggregation through fine-grained operator reordering. When an operation induces embedding wise dependencies (e.g., attention score aggregation), each device maintains two concurrent streams: a compute stream and a communication stream. Naively executing the computation in dependency order leads to idle communication streams, since collective operations (e.g., ReduceScatter or AllGather) cannot be issued until all ciphertext components corresponding to a token are ready, causing compute stall. Figure 8(b) illustrates this inefficiency in parallel self-attention. In the original execution order, each device must finish computing all rotated components of a token before communication can begin, leaving the communication stream idle during QK⊺QK computation. As a result, communication is serialized after computation, increasing end-to-end latency. AEGIS resolves this inefficiency by reordering polynomial operators across rotation offsets. Instead of computing all rotations for a token contiguously, each device interleaves the computation of different rotation offsets such that partial results for aggregation become available earlier. Specifically, as shown in Figure 8(b), GPU1GPU_1 first computes the second half of rotation offsets while GPU2GPU_2 computes the first half. This staggered schedule enables the subsequent ReduceScatter for softmax normalization to overlap with ongoing computation on both devices. The same reordering strategy applies to the AllGather following softmax. By exposing communication opportunities earlier in the execution, AEGIS overlaps collective operations with computation, effectively hiding communication latency without introducing additional transfers. 4.4.2. AEGIS Compile Pass HE-based inference systems span multiple abstraction layers, ranging from high-level application graphs to low-level polynomial instructions. AEGIS operates at the intermediate layer between the HE operator control-flow graph (CFG) and the polynomial-instruction level, serving as an orchestration layer for encrypted Transformer inference. As illustrated in Figure 9, the AEGIS compile pass proceeds in three stages. In Stage 1, a Python ML application annotated with a logical data-layout DSL is lowered into an encrypted application IR. At this stage, AEGIS analyzes token- and sequence-level dependencies in the application graph and produces an encrypted IR consisting of HE operator invocations while preserving high-level application semantics. In Stage 2, the encrypted application is further lowered into a polynomial-level IR. AEGIS analyzes limb- and RNS-level dependencies induced by HE operators and inserts encrypted collective communication primitives to support tensor sharding and encrypted reductions. Using a hardware-aware cost model that accounts for operator latency and inter-device communication costs, AEGIS performs fine-grained operator scheduling and reordering to overlap computation with communication. Notably, compile-time dependency analysis and collective insertion together take less than 0.7 s, making this orchestration pass lightweight in practice. In Stage 3, AEGIS generates a deterministic multi-GPU execution plan. At runtime, each inference request loads its cryptographic keys onto the assigned GPUs, partitions encrypted inputs accordingly, and executes the precomputed schedule in parallel on the target compute backends. This design enables deterministic communication hiding without runtime heuristics or dynamic synchronization. 4.5. Weak Scaling Analysis Figure 10. Weak scaling: (a) upper bound on the number of devices as a function of input sequence length (assuming full slot utilization); (b) per-device memory consumption under the maximal feasible device count for each sequence length. Two-panel weak-scaling figure. Panel a plots the maximum usable device count against input sequence length under full slot utilization. Panel b plots per-device memory at that maximal device count. The weak-scaling behavior of existing parallel designs differs fundamentally from that of AEGIS . In this section, we analyze weak scaling using the baselines described in Section 5.1.1. State-of-the-art limb-parallel approaches (Al Badawi et al., 2021b; Wang et al., 2023) are inherently constrained by the number of available RNS limbs at the bootstrapping level as shown in Figure 10(a). In practice, this limit is typically l=14+2=16l=14+2=16, including two special primes from P; since limbs are partitioned across devices, the number of usable GPUs cannot exceed the number of limbs. Cinnamon (Jayashankar et al., 2025a) is further constrained by both limb count and ciphertext count. Although it combines limb-parallel execution with phases in which each device must temporarily hold all components of a ciphertext (e.g., during FFNs), this hybrid strategy restricts scalability to the minimum of the bootstrapping level and the number of concurrently active ciphertexts. Hydra’s tensor-parallel design (Yang et al., 2025) scales with sequence length only until memory becomes the bottleneck. As model dimension and sequence length increase, Hydra replicates inputs and stores all QKV ciphertext components on every device. This replication induces substantial memory overhead and leads to per-device memory exhaustion, as shown in Figure 10(b). In contrast, AEGIS scales with sequence length without inheriting limb-count limits or requiring ciphertext replication. As illustrated in Figure 10(a,b), AEGIS maintains both scalability and memory efficiency, approaching the memory footprint of limb-parallel methods at large sequence lengths while supporting a substantially larger number of devices. 5. Experiments 5.1. Implementation We use LibTorch (Paszke et al., 2019) as the tensor container and memory allocator, and employ the CUDA backend from LiberateFHE (DESILO, 2023) for all RNS computations on GPU. Frontend compile pass (see Section 4.4.2) is implemented via torch.fx symbolic tracing and operator lowering transformations, which injects the required encrypted operations for a given computation flow. We ensure full slot utilization with stok=64s_tok=64 and apply BSGS for matrix multiplications. While the Cinnamon baseline relies on hoisting, AEGIS disables it by default because hoisting creates intermediate copies proportional to the baby-step size, substantially increasing peak memory usage. Our GPU profiling shows that ciphertext polynomials already saturate the SMs and expose sufficient parallelism, so hoisting provides only marginal throughput gains and does not reduce per-request latency relative to a one-by-one rotate-multiply-accumulate strategy. Therefore, similar to Min-KS in (Agrawal et al., 2023a), AEGIS adopts the one-by-one approach, saving per-user keys without sacrificing latency. Model. The end-to-end results in this section are obtained with a full BERT-Base Transformer on the SST-2 task. Under encrypted inference, this model reaches 0.923 accuracy and 0.929 F1 on SST-2. While AEGIS ’s placement and scheduling strategy applies generally to both encoder-style and decoder-style Transformer architectures, we focus the current empirical evaluation on BERT-Base. Encryption Parameters. We adopt a bootstrappable setting at 128-bit security. The number of slots is S=215S=2^15, with polynomial degree N=216N=2^16. The RNS modulus is sized at 16611661 bits, providing |QL|=35|Q_L|=35 levels and |P|=4|P|=4 special primes for bootstrapping. Following the NEXUS (Zhang et al., 2025) bootstrapping schedule, we set the bootstrapping level to lboot=14l_boot=14, leaving a usable depth of L=|QL|−lboot=21L=|Q_L|-l_boot=21. Environment. The 2-GPU experiments are conducted on a platform equipped with an AMD Ryzen Threadripper PRO 7975WX CPU (32 cores), 512 GB of system memory, and two NVIDIA RTX A6000 GPUs (48 GB each) with NVLink P2P connectivity. The 4-GPU experiments are conducted on a server equipped with an AMD EPYC 7763 CPU (64 cores), 2.0 TiB of system memory, and four NVIDIA A100 GPUs (40 GB each). 5.1.1. Baselines We compare AEGIS against four state-of-the-art baselines representing distinct parallelization strategies for encrypted and plaintext inference. HEBooster (Al Badawi et al., 2021b) implements limb-parallel execution following prior GPU-based encrypted inference designs. Within each GPU, polynomial coefficients are processed in SIMD fashion at the SM level, while ciphertext limbs are partitioned across devices according to the RNS model shown in Figure 1. For Cinnamon (Jayashankar et al., 2025a), we implement its output-aggregation key-switching strategy for FFNs. Other components of Cinnamon rely on custom hardware modules and specialized instruction sets and are therefore not implemented on GPUs. As a plaintext tensor-parallel baseline, we implement Megatron-LM–style row-wise tensor parallelism (Shoeybi et al., 2020) across G GPUs, followed by encrypted inference executed independently on each device. For Hydra (Yang et al., 2025), which adopts column-wise tensor parallelism, we replicate the full encrypted input on every GPU, partition FFN weights across devices, and perform all-device aggregation at each layer boundary. We further incorporate Hydra’s tree-based task-distribution strategy for nonlinear operators, while their ASIC-specific, CPU-free execution model and switch-based interconnect are not reproducible on GPGPU platforms and are excluded. These baselines capture complementary pros and cons: Limb-parallel methods are memory-efficient by avoiding replication, but incur heavy communication during key switching. Tensor-parallel methods reduce communication rounds by duplicating inputs or weights, at the cost of increased memory usage. We show that AEGIS combines the strengths of both, achieving low communication overhead without sacrificing memory efficiency. Implementation Scope. Both Cinnamon and Hydra target ASIC/FPGA platforms with specialized interconnects and unpublished implementations. Since their hardware cannot be replicated on commodity GPUs, our comparison intentionally isolates only the parallelization and task-mapping strategies that can be reproduced on a common GPGPU platform, and does not include ASIC-, ISA-, or interconnect-specific optimizations. This yields an apples-to-apples GPU strategy comparison across baselines and AEGIS . All baselines and AEGIS incorporate orthogonal and identical compiler optimizations, including late relinearization and late rescaling (Dathathri et al., 2020). Table 3. Scaling efficiency in terms of latency with different parallel strategies for a fixed 128 input tokens on 2 GPUs. Layer Latency Proportion is a reference value on single device execution. Layer Depth on Entry Layer Latency Proportion (%) Plaintext Tensor Parallel NVLink HEBooster Limb Parallel NVLink Cinnamon Limb Parallel NVLink Hydra Tensor Parallel NVLink AEGIS Parallel NVLink Q,K,VQ,K,V Linear Projection 0 28.78% 49.52% 31.71% 81.71% 89.53% 93.29% A←Q×K⊺/dA← Q× K / d 1 16.20% 50.00% 30.75% 30.75% 91.13% 92.37% A←Softmax(A)A← Softmax(A) 2 3.48% 50.00% 30.43% 30.43% 40.52% 89.32% X←A×VX← A× V 18 3.29% 50.00% 30.75% 30.75% 91.13% 94.21% X←X×WoX← X× W_o 19 1.20% 49.87% 34.63% 84.63% 88.32% 94.66% Bootstrapping 20 1.14% 50.00% 31.50% 61.23% 48.63% 91.32% X←LayerNorm(X)X← LayerNorm(X) (PostNorm) 4 2.80% 49.83% 30.11% 30.11% 33.65% 92.32% Bootstrapping 20 1.14% 50.00% 31.50% 61.23% 48.63% 91.32% FFN1:X←X×Wo1FFN_1:X← X× W_o1 4 24.59% 50.00% 34.17% 84.17% 89.33% 93.76% X←GeLU(X)X← GeLU(X) 5 3.04% 50.00% 30.11% 30.11% 47.65% 88.23% FFN2:X←X×Wo2FFN_2:X← X× W_o2 19 9.00% 50.00% 31.27% 82.27% 88.35% 93.82% Bootstrapping 20 1.14% 50.00% 31.50% 61.23% 48.63% 91.32% X←LayerNorm(X)X← LayerNorm(X) (PreNorm) 4 3.07% 49.83% 30.11% 30.11% 43.65% 92.32% Bootstrapping 20 1.14% 50.00% 31.50% 61.23% 48.63% 91.32% End to end scaling efficiency 49.85% 31.86% 53.01% 75.59% 92.98% Note: An efficiency of 1.01.0 indicates ideal scaling; values around 0.90.9 represent high efficiency. For reference, in plaintext settings, SOTA work (Shoeybi et al., 2020) indicates that tensor parallelism achieves about 95%95\% on 2 GPUs, 82%82\% on 4 GPUs, and 77%77\% on 8 GPUs. Note that while directly applying plaintext parallel outperforms limb parallel in terms of latency, the efficiency in terms of throughput is the worst, with over half of the computation wasted. 5.2. Communication Overhead Figure 11. Communication volume on two GPUs with 128 input tokens on BERT. Bar chart comparing communication volume in megabytes for different methods on two GPUs with 128-token BERT inference. The figure shows much higher communication for limb-parallel baselines than for AEGIS . As shown in Figure 11, HEBooster exhibits extremely high communication overhead due to frequent key switching. Both FFNs and self-attention incur communication on the order of 10510^5 MB, reaching 61,735.77 MB and 63,395.76 MB, respectively. Cinnamon partially reduces communication in FFNs (to 60,550.07 MB) and during bootstrapping, but still incurs substantial overhead in nonlinear layers and self-attention due to repeated CMult operations. Directly applying tensor parallelism significantly reduces communication, requiring only three rounds of collectives in FFNs (991.71 MB). However, Hydra’s tensor-parallel design, while avoiding limb-level partitioning, replicates encrypted inputs across devices and performs sequence-wise collectives during self-attention, resulting in 13,143.96 MB of communication. In contrast, AEGIS tightly bounds communication by application-level dependencies. Feed-forward layers incur communication that scales as Θ(⌈T/(S/stok)⌉⋅ctok) ( T/(S/s_tok) · c_tok), while self-attention requires only a single round of sequence-wise collectives with cost Θ((G−1)⋅T2/S) ((G-1)· T^2/S). Consequently, AEGIS reduces communication to 788.65 MB in FFNs and 2,459.66 MB in self-attention. It isolates the first source of improvement: dependency-aware placement reduces the communicated data itself by matching transfers to application-level dependencies. The remaining latency gains come from deterministic operator reordering, which overlaps the now-smaller communication with computation rather than further reducing communication volume. An ablation on two GPUs shows that placement contributes 76.27% of the speedup at 128 tokens and 88.26% at 2048 tokens, while reordering contributes the remaining 23.73% and 11.74%, respectively. As sequence length increases, self-attention dominates execution and leaves less latency to hide via overlap, making placement the larger contributor at long sequences. We quantify these latency-side effects in Section 5.3 and Section 5.4. 5.3. Scaling Efficiency of Layer Components To analyze the efficiency of task-mapping design in terms of latency, we compare AEGIS against plaintext parallelism, HEBooster limb parallelism (Al Badawi et al., 2021b; Wang et al., 2023), Cinnamon limb parallelism (Jayashankar et al., 2025a), and Hydra tensor parallelism (Yang et al., 2025). We evaluate both layer-wise and end-to-end scaling efficiency, computed as T1/TGT_1/T_G where T1T_1 is the single-GPU latency and TGT_G is the latency on G GPUs. Following prior benchmarks (Park et al., 2024; Zhang et al., 2025; Moon et al., 2024), we fix the input length to 128 tokens, the maximum length supported by single-device inference, and report results in Table 3. Limb-parallel schemes distribute ciphertext memory evenly across devices, as analyzed in Section 4.5, but nevertheless exhibit the worst scaling efficiency. In Table 3, HEBooster achieves only 31.24% end-to-end efficiency: the 1617.17s single-GPU latency increases to 2530.62s when distributed, becoming slower than the single-GPU baseline. This degradation arises from (1) the massive communication volume during intensive rotation chains, and (2) operation stalls caused by waiting on these transfers. We conclude that limb parallelism is suitable for statistical workloads with low key-switching intensity, but fundamentally mismatched to long-sequence encrypted Transformer inference. Cinnamon improves limb parallelism during PMult matrix multiplications via switching between tensor parallel and limb parallel, yielding higher efficiency for FFNs (about 80%) and moderate gains for bootstrapping (61%). However, it still suffers from poor performance on self-attention, where CMult and non-linearities dominate and communication becomes the bottleneck. As a result, Cinnamon reaches only 53.01% model-wise efficiency. Hydra tensor parallelism achieves roughly 90% efficiency on FFNs and the matrix-multiplication segments of self-attention, as these components benefit from replicated inputs and favorable parallel structure. However, Hydra’s strategy for nonlinear operations requires transferring ciphertexts based on polynomial-degree ownership, resulting in 40–50% efficiency in nonlinearities and 48.63% for bootstrapping. Its full-model efficiency reaches 75.87%. AEGIS achieves the highest overall efficiency across all layers. Compared to limb parallelism, AEGIS eliminates the dominant communication during key switching via locality-preserving RNS placement, and hides the remaining communication through polynomial-operator reordering. Compared to Hydra, AEGIS inserts only a single ReduceScatter and a single AllGather per self-attention block, as described in Section 4.3. Consequently, except for softmax (89.32%), all layers consistently exceed 90% efficiency, yielding a 92.98% end-to-end efficiency, which is 43.13% higher than HEBooster, 39.97% higher than Cinnamon, and 17.11% higher than Hydra. We also observe that layers with identical input sizes may still exhibit different efficiencies due to depth-dependent workload variation. For example, although FFN1FFN_1 and FFN2FFN_2 consume the same input shape, they operate at bootstrapping depths 4 and 19, respectively, in Table 3. Limb-parallel methods are highly sensitive to multiplicative depth: HEBooster exhibits a 2.90% discrepancy, and Cinnamon a 1.90% discrepancy, because communication scales linearly with available depth. In contrast, AEGIS is largely insensitive to such variation: its communication-to-compute ratio remains negligible due to depth-invariant polynomial placement and reduced collective frequency. 5.4. End-to-end Evaluation Figure 12. End-to-end inference (a) per-device memory consumption; (b) full-model speed-up, where a baseline 1.01.0 corresponds to the latency of processing a full request on a single device. Hatched bar segments and hollow markers denote OOM on 48GB GPU. Two-panel figure for two-GPU end-to-end inference. Panel a shows per-device memory across input lengths for each method. Panel b shows full-model speed-up relative to single-GPU execution, with hatched or hollow marks indicating out-of-memory cases on a 48 GB GPU. Beyond layer-level efficiency, we evaluate end-to-end inference performance of the encrypted BERT model on both 2×2× and 4×4× GPU systems in Figure 12 and Figure 13. In this section, we report per-device memory footprint and end-to-end speedup relative to single-GPU execution as the input length increases from 128 to 2048 tokens. 5.4.1. Memory Efficiency Figure 12(a) and Figure 13(a) show the same memory ordering across two and four GPUs. HEBooster limb parallelism and AEGIS consistently use the least memory because both shard weights and input ciphertexts rather than replicating full activations, whereas tensor-parallel and hoisting-heavy baselines retain a much larger per-device footprint. The main difference between the two settings is the OOM boundary, not the trend. Moving from two 48 GB GPUs to four 40 GB GPUs reduces AEGIS ’s per-device memory from 17.14–47.15 GB to 14.55–30.82 GB, allowing AEGIS to remain feasible across the full 2048-token range in both settings. The additional sharding postpones OOM for Hydra and Cinnamon to 512 tokens on four GPUs, but tensor parallelism still overflows at 1024 tokens and both Hydra and Cinnamon still fail at 1024–2048 tokens once replicated activations or hoisted intermediates dominate memory. Compared to HEBooster limb parallelism, AEGIS uses similar memory with 1.67%-4.51% differences along all sequence length. This difference arises because both designs evenly distribute weights and input ciphertexts, while AEGIS additionally retaining certain user-specific keys locally on each device. As sequence length increases, the memory footprint of a few key switching keys becomes negligible relative to encrypted activations, causing the memory proportion of the two approaches to converge. Compared to Cinnamon limb parallelism, AEGIS exhibits substantially lower memory consumption. Cinnamon performs output-aggregation key switching via hoisting at each layer entry, causing d rotated intermediates derived from the same ciphertext to coexist on every device. This design significantly increases memory pressure without providing latency benefits on our platform (see Section 5.1). Although Cinnamon remains memory-efficient for short sequences (25.32 GB at 128 tokens, compared to Hydra’s 29.50 GB), it exhausts device memory at 512 tokens. In contrast, AEGIS reduces memory usage by 31.59% at 128 tokens and 50.86% at 256 tokens. As sequence length increases, approaches that replicate intermediate ciphertexts scale poorly, since encrypted activations become the dominant memory component. Hydra tensor parallelism exhibits a similar limitation. At 128 tokens, Hydra consumes 41.27% more per-device memory than AEGIS for a single request, increasing to 45.60% at 512 tokens. This overhead arises because Hydra fully gathers layer outputs at each layer boundary, forcing every device to hold a complete copy of the encrypted activations before the next layer executes. As shown in Figure 4, encrypted activations increasingly dominate memory as sequence length grows. These results indicate that efficient long-sequence encrypted inference fundamentally requires distributing activation memory across GPUs, rather than replicating it. 5.4.2. Performance Figure 12(b) and Figure 13(b) report the end-to-end speed-up across different sequence lengths. For configurations that exceed device memory, we reuse the same input ciphertexts and weight plaintext slices so the measured speed-up reflects execution behavior rather than repeated data materialization. AEGIS shows the same trend on two and four GPUs: its speed-up curve remains nearly flat as sequence length grows, reaching 1.850–1.860× on two GPUs and 3.823–3.865× on four GPUs. These correspond to scaling factors of about 0.93 and 0.95, indicating that locality-preserving placement removes the dominant communication cost while operator reordering continues to hide the remaining collectives even for long contexts. We emphasize scaling factor as the primary metric because it normalizes hardware-specific throughput differences, but the corresponding end-to-end latencies on the 4×4×A100 platform are 192.8 s at 128 tokens, 544.2 s at 512 tokens, and 5036.1 s at 2048 tokens, following the same overall trend. The four-GPU curve is also slightly higher because those measurements use A100s with faster NVLink than the two-GPU A6000 setup. AEGIS consistently outperforms all baselines and achieves the highest and most stable speed-up across input lengths. On two GPUs, it reaches a 1.858×1.858× speed-up, which is 1.86×1.86× faster than directly applying plaintext tensor parallelism, 2.92×2.92× faster than HEBooster limb parallelism, 1.75×1.75× faster than Cinnamon limb parallelism, and 1.23×1.23× faster than Hydra tensor parallelism. Directly applying plaintext tensor parallelism incurs little communication, but severe slot underutilization and redundant computation reduce its speed-up to only 0.997×0.997×. This redundancy also causes substantial memory waste, consuming over 2×2× more per-device memory than HEBooster limb parallelism. HEBooster remains communication-bound and achieves only 0.637×0.637× speed-up due to repeated synchronization. Cinnamon improves linear projections and FFNs, yielding a 1.060×1.060× speed-up, but continues to lose efficiency once self-attention and nonlinear layers dominate, so its advantage diminishes as FFNs become less dominant at longer sequence lengths. Hydra comes closest to AEGIS with a 1.512×1.512× speed-up at 128 tokens, but still lags because it replicates the full K and V and handles nonlinear layers inefficiently. Although Hydra is comparable in the attention score computation (91.13%91.13\% vs. 92.37%92.37\% efficiency for A←QK⊺/dA← QK / d in Table 3), its nonlinearity strategy performs poorly on GPUs, especially in softmax. In particular, Hydra reaches only 40.52% softmax efficiency compared with 89.32% for AEGIS , so AEGIS still delivers an additional 0.346×0.346× speed-up while also reducing memory consumption. The same qualitative ordering holds on four GPUs, but the gap widens once longer sequences stress communication and memory. Tensor parallelism rises only to about 2.0×2.0× before OOM at 1024 tokens, Hydra reaches 2.10–2.23× and Cinnamon 1.08–1.48× for short inputs but both fail beyond 512 tokens, and HEBooster remains below 1×1× throughout. Figure 13. End-to-end inference on four GPUs: (a) per-device memory consumption; (b) full-model speed-up. Hatched entries denote OOM configurations with inferred values used for visualization. Two-panel figure for four-GPU end-to-end inference. Panel a shows per-device memory across input lengths for each method. Panel b shows full-model speed-up, with hatched entries marking out-of-memory configurations whose values are inferred for visualization. 6. Related Works Ciphertext Packing and Application Level Optimizations. Packing schemes have been extensively studied across diverse HE-based encrypted inference workloads, including CNNs (Al Badawi et al., 2021a; Kim et al., 2022a; Aharoni et al., 2023), GNNs (Ran et al., 2022, 2023; Kan et al., 2025), and recent works on Transformers (Park et al., 2024; Zhang et al., 2025). These approaches optimize encrypted Transformer inference at the application level, including polynomial approximation for nonlinear functions and algorithmic improvements to encrypted matrix multiplication. These techniques are orthogonal to AEGIS , which instead targets cross-operator multi-GPU orchestration under ciphertext/RNS coupling. Meanwhile, recent tensor compilers (Viand et al., 2023; Ebel et al., 2025; Krastev et al., 2024) further automate the mapping of tensor operations to HE primitives, simplifying program development and layout management. Compiler Optimization. Several compiler frameworks focus on parameter selection (Dathathri et al., 2019; Reagen et al., 2021), managing multiplicative depth on-the-fly (Dathathri et al., 2020; Lee et al., 2022, 2024b) and optimizing bootstrapping placement through analytical or learned cost models (Cowan et al., 2021; Liu et al., 2025; Cheon et al., 2024). These methods are orthogonal to packing-based approaches and primarily target latency reduction and noise management. Not to mention, the resulting dynamic adjustment of modulus-chain length can affect the scalability of RNS under limb-parallel execution, as it changes the number of remaining moduli available for partitioned evaluation. Hardware Acceleration. Prior work has explored accelerating individual HE primitives on GPUs (Dai and Sunar, 2015; Benaissa et al., 2021; DESILO, 2023) and on dedicated ASIC and FPGA platforms (Samardzic et al., 2021; Kim et al., 2022b; Agrawal et al., 2023b; Samardzic et al., 2022; Yang et al., 2023), demonstrating the feasibility of deploying RLWE-based HE schemes on commodity and specialized hardware. At the system level, early multi-device designs (Al Badawi et al., 2021b; Wang et al., 2023) exploit limb- and coefficient-parallelism but incur costly format conversions and scale poorly for rotation-intensive workloads. Application-aware FPGA/ASIC architectures such as Cinnamon and Hydra (Jayashankar et al., 2025a; Yang et al., 2025) improve scalability through specialized hardware and interconnects, and are orthogonal to our GPU-focused design. Cerium (Jayashankar et al., 2025b), also explores GPU acceleration for encrypted inference. Cerium primarily emphasizes kernel- and memory-level optimization within device execution, whereas AEGIS focuses on cross-device parallel planning and coherent placement derived from data dependencies across the application and encryption domains. 7. Conclusion This paper presents AEGIS , an application- and encryption-aware multi-GPU execution framework for long-context encrypted Transformer inference. By deriving placement from ciphertext dependencies across both the application and encryption domains, and by reordering operators to overlap the remaining collectives with computation, AEGIS reduces communication while avoiding memory overflow and communication-induced stalls. AEGIS consistently outperforms prior state-of-the-art designs in both latency and memory efficiency. Our accuracy-validated end-to-end evaluation shows up to 96.62% scaling efficiency, a 3.86× end-to-end speedup, and 69.1% per-device memory reduction on four GPUs. These results indicate that scalable encrypted Transformer inference requires coordinated application-level and encryption-level orchestration, rather than direct adaptation of plaintext parallelism. They also suggest that future HE systems should treat packing, placement, and communication scheduling as a cross-layer optimization problem. Acknowledgements.This work is partially supported by the National Science Foundation (NSF) under Grants No. CNS-2348733, No. CNS-2349538, and No. CNS-2340777. References R. Agrawal, L. De Castro, C. Juvekar, A. Chandrakasan, V. Vaikuntanathan, and A. Joshi (2023a) MAD: Memory-Aware Design Techniques for Accelerating Fully Homomorphic Encryption. In Proceedings of the 56th Annual IEEE/ACM International Symposium on Microarchitecture, MICRO ’23, New York, NY, USA, p. 685–697. External Links: ISBN 979-8-4007-0329-4, Link, Document Cited by: §5.1. R. Agrawal, L. De Castro, G. Yang, C. Juvekar, R. Yazicigil, A. Chandrakasan, V. Vaikuntanathan, and A. Joshi (2023b) FAB: An FPGA-based Accelerator for Bootstrappable Fully Homomorphic Encryption. In 2023 IEEE International Symposium on High-Performance Computer Architecture (HPCA), Montreal, QC, Canada, p. 882–895 (en). External Links: ISBN 978-1-6654-7652-2, Link, Document Cited by: §6. E. Aharoni, A. Adir, M. Baruch, N. Drucker, G. Ezov, A. Farkash, L. Greenberg, R. Masalha, G. Moshkowich, D. Murik, H. Shaul, and O. Soceanu (2023) HeLayers: A Tile Tensors Framework for Large Neural Networks on Encrypted Data. Proceedings on Privacy Enhancing Technologies 2023 (1), p. 325–342. External Links: ISSN 2299-0984, Link, Document Cited by: §6. A. Al Badawi, C. Jin, J. Lin, C. F. Mun, S. J. Jie, B. H. M. Tan, X. Nan, K. M. M. Aung, and V. R. Chandrasekhar (2021a) Towards the AlexNet Moment for Homomorphic Encryption: HCNN, the First Homomorphic CNN on Encrypted Data With GPUs. IEEE Transactions on Emerging Topics in Computing 9 (3), p. 1330–1343. External Links: ISSN 2168-6750, Link, Document Cited by: §6. A. Al Badawi, B. Veeravalli, J. Lin, N. Xiao, M. Kazuaki, and A. Khin Mi Mi (2021b) Multi-GPU Design and Performance Evaluation of Homomorphic Encryption on GPU Clusters. IEEE Transactions on Parallel and Distributed Systems 32 (2), p. 379–391. External Links: ISSN 1558-2183, Link, Document Cited by: §1, §2.3, §2.5, §4.5, §5.1.1, §5.3, §6. A. A. Badawi, A. Alexandru, J. Bates, F. Bergamaschi, D. B. Cousins, S. Erabelli, N. Genise, S. Halevi, H. Hunt, A. Kim, Y. Lee, Z. Liu, D. Micciancio, C. Pascoe, Y. Polyakov, I. Quah, S. R.V., K. Rohloff, J. Saylor, D. Suponitsky, M. Triplett, V. Vaikuntanathan, and V. Zucca (2022) OpenFHE: open-source fully homomorphic encryption library. Note: Cryptology ePrint Archive, Paper 2022/915 External Links: Link Cited by: §2.2. A. Benaissa, B. Retiat, B. Cebere, and A. E. Belfedhal (2021) TenSEAL: A Library for Encrypted Tensor Operations Using Homomorphic Encryption. arXiv. External Links: Link, Document Cited by: §6. J. Bossuat, C. Mouchet, J. Troncoso-Pastoriza, and J. Hubaux (2021) Efficient Bootstrapping for Approximate Homomorphic Encryption with Non-sparse Keys. In Advances in Cryptology – EUROCRYPT 2021: 40th Annual International Conference on the Theory and Applications of Cryptographic Techniques, Zagreb, Croatia, October 17–21, 2021, Proceedings, Part I, Berlin, Heidelberg, p. 587–617. External Links: ISBN 978-3-030-77869-9, Link, Document Cited by: §2.1. Z. Brakerski, C. Gentry, and V. Vaikuntanathan (2014) (Leveled) Fully Homomorphic Encryption without Bootstrapping. ACM Trans. Comput. Theory 6 (3), p. 13:1–13:36. External Links: ISSN 1942-3454, Link, Document Cited by: §1. H. Chen, I. Chillotti, and Y. Song (2018) Improved Bootstrapping for Approximate Homomorphic Encryption. External Links: Link Cited by: §2.1. J. H. Cheon, A. Kim, M. Kim, and Y. Song (2016) Homomorphic Encryption for Arithmetic of Approximate Numbers. External Links: Link Cited by: §1. S. Cheon, Y. Lee, D. Kim, J. M. Lee, S. Jung, T. Kim, D. Lee, and H. Kim (2024) DaCapo: Automatic Bootstrapping Management for Efficient Fully Homomorphic Encryption. p. 6993–7010 (en). External Links: ISBN 978-1-939133-44-1, Link Cited by: §1, §6. M. Cowan, D. Dangwal, A. Alaghi, C. Trippel, V. T. Lee, and B. Reagen (2021) Porcupine: a synthesizing compiler for vectorized homomorphic encryption. In Proceedings of the 42nd ACM SIGPLAN International Conference on Programming Language Design and Implementation, PLDI 2021, New York, NY, USA, p. 375–389. External Links: ISBN 978-1-4503-8391-2, Link, Document Cited by: §1, §6. W. Dai and B. Sunar (2015) cuHE: A Homomorphic Encryption Accelerator Library. External Links: Link Cited by: §6. R. Dathathri, B. Kostova, O. Saarikivi, W. Dai, K. Laine, and M. Musuvathi (2020) EVA: An Encrypted Vector Arithmetic Language and Compiler for Efficient Homomorphic Computation. In Proceedings of the 41st ACM SIGPLAN Conference on Programming Language Design and Implementation, p. 546–561. External Links: Link, Document Cited by: §1, §5.1.1, §6. R. Dathathri, O. Saarikivi, H. Chen, K. Laine, K. Lauter, S. Maleki, M. Musuvathi, and T. Mytkowicz (2019) CHET: an optimizing compiler for fully-homomorphic neural-network inferencing. In Proceedings of the 40th ACM SIGPLAN Conference on Programming Language Design and Implementation, PLDI 2019, New York, NY, USA, p. 142–156. External Links: ISBN 978-1-4503-6712-7, Link, Document Cited by: §1, §6. J. Dean, G. Corrado, R. Monga, K. Chen, M. Devin, M. Mao, M. a. Ranzato, A. Senior, P. Tucker, K. Yang, Q. Le, and A. Ng (2012) Large Scale Distributed Deep Networks. In Advances in Neural Information Processing Systems, Vol. 25. External Links: Link Cited by: §1, §2.5, §2.5. X. Deng, S. Fan, Z. Hu, Z. Tian, Z. Yang, J. Yu, D. Cao, D. Meng, R. Hou, M. Li, Q. Lou, and M. Zhang (2024) Trinity: A General Purpose FHE Accelerator. p. 338–351 (English). External Links: ISBN 979-8-3503-5057-9, Link, Document Cited by: §2.3. DESILO (2023) Liberate.FHE: A New FHE Library for Bridging the Gap between Theory and Practice with a Focus on Performance and Accuracy. Cited by: §2.2, §2.5, §5.1, §6. J. Devlin, M. Chang, K. Lee, and K. Toutanova (2019) BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), J. Burstein, C. Doran, and T. Solorio (Eds.), Minneapolis, Minnesota, p. 4171–4186. External Links: Link, Document Cited by: §3.1. A. Ebel, K. Garimella, and B. Reagen (2025) Orion: A Fully Homomorphic Encryption Framework for Deep Learning. In Proceedings of the 30th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2, ASPLOS ’25, New York, NY, USA, p. 734–749. External Links: ISBN 979-8-4007-1079-7, Link, Document Cited by: §3.2.2, §6. J. Fan and F. Vercauteren (2012) Somewhat Practical Fully Homomorphic Encryption. External Links: Link Cited by: §1. S. Fan, Z. Wang, W. Xu, R. Hou, D. Meng, and M. Zhang (2023) TensorFHE: Achieving Practical Computation on Encrypted Data Using GPGPU. In 2023 IEEE International Symposium on High-Performance Computer Architecture (HPCA), p. 922–934. External Links: Link, Document Cited by: §2.3. W. Fedus, B. Zoph, and N. Shazeer (2022) Switch transformers: scaling to trillion parameter models with simple and efficient sparsity. Journal of Machine Learning Research 23 (120), p. 1–39. Cited by: §2.5. C. Gentry, S. Halevi, and N. P. Smart (2012) Homomorphic Evaluation of the AES Circuit. In Advances in Cryptology – CRYPTO 2012, R. Safavi-Naini and R. Canetti (Eds.), Vol. 7417, p. 850–867 (en). External Links: Link, Document Cited by: §1. C. Gentry (2009) Fully homomorphic encryption using ideal lattices. In Proceedings of the forty-first annual ACM symposium on Theory of computing, Bethesda MD USA, p. 169–178 (en). External Links: ISBN 978-1-60558-506-2, Link, Document Cited by: §1. W. Gropp, E. Lusk, and A. Skjellum (1994) Using mpi: portable parallel programming with the message-passing interface. MIT Press, Cambridge, MA, USA. External Links: ISBN 0262571048 Cited by: §1. S. Jayashankar, E. Chen, T. Tang, W. Zheng, and D. Skarlatos (2025a) Cinnamon: A Framework for Scale-Out Encrypted AI. In Proceedings of the 30th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 1, Rotterdam Netherlands, p. 133–150 (en). External Links: Link, Document Cited by: §1, §2.3, §4.5, §5.1.1, §5.3, §6. S. Jayashankar, J. Kim, M. B. Sullivan, W. Zheng, and D. Skarlatos (2025b) A Scalable Multi-GPU Framework for Encrypted Large-Model Inference. arXiv. Note: arXiv:2512.11269 [cs] External Links: Link, Document Cited by: §6. W. Jung, S. Kim, J. H. Ahn, J. H. Cheon, and Y. Lee (2021) Over 100x Faster Bootstrapping in Fully Homomorphic Encryption through Memory-centric Optimization with GPUs. IACR Transactions on Cryptographic Hardware and Embedded Systems, p. 114–148 (en). External Links: ISSN 2569-2925, Link, Document Cited by: §2.1. C. Juvekar, V. Vaikuntanathan, and A. Chandrakasan (2018) GAZELLE: A Low Latency Framework for Secure Neural Network Inference. External Links: Link Cited by: §3.2.2. Z. Kan, H. Han, S. Shi, T. Hua, H. Lu, X. Li, J. Mu, and X. Hu (2025) FicGCN: Unveiling the Homomorphic Encryption Efficiency from Irregular Graph Convolutional Networks. (en). External Links: Link Cited by: §6. M. Kim, X. Jiang, K. Lauter, E. Ismayilzada, and S. Shams (2022a) Secure Human Action Recognition by Encrypted Neural Network Inference. Nature Communications 13 (1), p. 4799 (en). External Links: ISSN 2041-1723, Link, Document Cited by: §1, §3.2.2, §6. M. Kim, D. Lee, J. Seo, and Y. Song (2023) Accelerating HE Operations from Key Decomposition Technique. External Links: Link Cited by: §1. S. Kim, J. Kim, M. J. Kim, W. Jung, J. Kim, M. Rhu, and J. H. Ahn (2022b) BTS: an accelerator for bootstrappable fully homomorphic encryption. In Proceedings of the 49th Annual International Symposium on Computer Architecture, ISCA ’22, New York, NY, USA, p. 711–725. External Links: ISBN 978-1-4503-8610-4, Link, Document Cited by: §2.3, §6. A. Krastev, N. Samardzic, S. Langowski, S. Devadas, and D. Sanchez (2024) A Tensor Compiler with Automatic Data Packing for Simple and Efficient Fully Homomorphic Encryption. Proceedings of the ACM on Programming Languages 8 (PLDI), p. 126–150 (en). External Links: ISSN 2475-1421, Link, Document Cited by: §6. [37] (2024-08) Lattigo v6. Note: Online: https://github.com/tuneinsight/lattigo Cited by: §2.2. S. Lee, G. Lee, J. W. Kim, J. Shin, and M. Lee (2024a) HETAL: Efficient Privacy-preserving Transfer Learning with Homomorphic Encryption. arXiv. External Links: Link, Document Cited by: §1. Y. Lee, S. Cheon, D. Kim, D. Lee, and H. Kim (2024b) Performance-aware Scale Analysis with Reserve for Homomorphic Encryption. In Proceedings of the 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 1, ASPLOS ’24, Vol. 1, New York, NY, USA, p. 302–317. External Links: ISBN 979-8-4007-0372-0, Link, Document Cited by: §1, §6. Y. Lee, S. Heo, S. Cheon, S. Jeong, C. Kim, E. Kim, D. Lee, and H. Kim (2022) HECATE: Performance-Aware Scale Optimization for Homomorphic Encryption Compiler. In 2022 IEEE/ACM International Symposium on Code Generation and Optimization (CGO), p. 193–204. External Links: Link, Document Cited by: §1, §6. D. Lepikhin, H. Lee, Y. Xu, D. Chen, O. Firat, Y. Huang, M. Krikun, N. Shazeer, and Z. Chen (2020) GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding. (en). External Links: Link Cited by: §2.5. S. Li, F. Xue, C. Baranwal, Y. Li, and Y. You (2023) Sequence Parallelism: Long Sequence Training from System Perspective. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), A. Rogers, J. Boyd-Graber, and N. Okazaki (Eds.), Toronto, Canada, p. 2391–2404. External Links: Link, Document Cited by: §2.5. Y. Liu, J. Lai, L. Li, T. Sui, L. Xiao, P. Yuan, X. Zhang, Q. Zhu, W. Chen, and J. Xue (2025) ReSBM: Region-based Scale and Minimal-Level Bootstrapping Management for FHE via Min-Cut. In Proceedings of the 30th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 1, Rotterdam Netherlands, p. 924–939. External Links: Link, Document Cited by: §6. W. Lu, Z. Huang, Z. Gu, J. Li, J. Liu, K. Ren, C. Hong, T. Wei, and W. Chen (2023) BumbleBee: Secure Two-party Inference Framework for Large Transformers. External Links: Link Cited by: §1, §4.3. M. Milakov and N. Gimelshein (2018) Online normalizer calculation for softmax. arXiv preprint arXiv:1805.02867. Cited by: §4.3. J. Moon, D. Yoo, X. Jiang, and M. Kim (2024) THOR: Secure Transformer Inference with Homomorphic Encryption. External Links: Link Cited by: §1, §3.2.2, §5.3. D. Narayanan, M. Shoeybi, J. Casper, P. LeGresley, M. Patwary, V. A. Korthikanti, D. Vainbrand, P. Kashinkunti, J. Bernauer, B. Catanzaro, A. Phanishayee, and M. Zaharia (2021) Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LM. arXiv. External Links: Link, Document Cited by: §1, §2.5, §2.5. Q. Pang, J. Zhu, H. Möllering, W. Zheng, and T. Schneider (2023) BOLT: Privacy-Preserving, Accurate and Efficient Inference for Transformers. External Links: Link Cited by: §1, §3.2.2. D. Park, E. Lee, and J. Lee (2024) Powerformer: Efficient Privacy-Preserving Transformer with Batch Rectifier-Power Max Function and Optimized Homomorphic Attention. External Links: Link Cited by: §1, §5.3, §6. A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, A. Desmaison, A. Köpf, E. Yang, Z. DeVito, M. Raison, A. Tejani, S. Chilamkurthy, B. Steiner, L. Fang, J. Bai, and S. Chintala (2019) PyTorch: an imperative style, high-performance deep learning library. External Links: 1912.01703, Link Cited by: §5.1. R. Pope, S. Douglas, A. Chowdhery, J. Devlin, J. Bradbury, J. Heek, K. Xiao, S. Agrawal, and J. Dean (2023) Efficiently Scaling Transformer Inference. Proceedings of Machine Learning and Systems 5, p. 606–624 (en). External Links: Link Cited by: §2.5. L. Qin, J. Cui, W. Cai, and J. Huang (2025) Chimera: Communication Fusion for Hybrid Parallelism in Large Language Models. In Proceedings of the 52nd Annual International Symposium on Computer Architecture, ISCA ’25, New York, NY, USA, p. 498–513. External Links: ISBN 979-8-4007-1261-6, Link, Document Cited by: §1. R. Ran, W. Wang, Q. Gang, J. Yin, N. Xu, and W. Wen (2022) CryptoGCN: Fast and Scalable Homomorphically Encrypted Graph Convolutional Network Inference. Advances in Neural Information Processing Systems 35, p. 37676–37689 (en). External Links: Link Cited by: §1, §6. R. Ran, N. Xu, T. Liu, W. Wang, G. Quan, and W. Wen (2023) Penguin: Parallel-Packed Homomorphic Encryption for Fast Graph Convolutional Network Inference. Advances in Neural Information Processing Systems 36, p. 19104–19116 (en). External Links: Link Cited by: §3.2.2, §6. B. Reagen, W. Choi, Y. Ko, V. T. Lee, H. S. Lee, G. Wei, and D. Brooks (2021) Cheetah: Optimizing and Accelerating Homomorphic Encryption for Private Inference. In 2021 IEEE International Symposium on High-Performance Computer Architecture (HPCA), p. 26–39. External Links: Link, Document Cited by: §1, §6. N. Samardzic, A. Feldmann, A. Krastev, S. Devadas, R. Dreslinski, C. Peikert, and D. Sanchez (2021) F1: A Fast and Programmable Accelerator for Fully Homomorphic Encryption. In MICRO-54: 54th Annual IEEE/ACM International Symposium on Microarchitecture, MICRO ’21, New York, NY, USA, p. 238–252. External Links: ISBN 978-1-4503-8557-2, Link, Document Cited by: §6. N. Samardzic, A. Feldmann, A. Krastev, N. Manohar, N. Genise, S. Devadas, K. Eldefrawy, C. Peikert, and D. Sanchez (2022) CraterLake: a hardware accelerator for efficient unbounded computation on encrypted data. In Proceedings of the 49th Annual International Symposium on Computer Architecture, ISCA ’22, New York, NY, USA, p. 173–187. External Links: ISBN 978-1-4503-8610-4, Link, Document Cited by: §6. SEAL (2023) Microsoft SEAL (release 4.1). Note: https://github.com/Microsoft/SEAL Cited by: §2.2. M. Shoeybi, M. Patwary, R. Puri, P. LeGresley, J. Casper, and B. Catanzaro (2020) Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism. arXiv. External Links: Link, Document Cited by: §2.5, §2.5, §5.1.1, Table 3. A. Viand, P. Jattke, M. Haller, and A. Hithnawi (2023) HECO: Fully Homomorphic Encryption Compiler. p. 4715–4732 (en). External Links: ISBN 978-1-939133-37-3, Link Cited by: §6. Z. Wang, P. Li, R. Hou, Z. Li, J. Cao, X. Wang, and D. Meng (2023) HE-Booster: An Efficient Polynomial Arithmetic Acceleration on GPUs for Fully Homomorphic Encryption. IEEE Transactions on Parallel and Distributed Systems 34 (4), p. 1067–1081. External Links: ISSN 1558-2183, Link, Document Cited by: §1, §2.3, §4.5, §5.3, §6. Y. Yang, X. Xu, H. Zhang, J. Song, X. Tang, H. Lu, and X. Li (2025) Hydra: Scale-out FHE Accelerator Architecture for Secure Deep Learning on FPGA. In 2025 IEEE International Symposium on High Performance Computer Architecture (HPCA), p. 1174–1186. External Links: Link, Document Cited by: §4.5, §5.1.1, §5.3, §6. Y. Yang, H. Zhang, S. Fan, H. Lu, M. Zhang, and X. Li (2023) Poseidon: Practical Homomorphic Encryption Accelerator. In 2023 IEEE International Symposium on High-Performance Computer Architecture (HPCA), p. 870–881. External Links: Link, Document Cited by: §6. J. Zhang, X. Yang, L. He, K. Chen, W. Lu, Y. Wang, X. Hou, J. Liu, K. Ren, and X. Yang (2025) Secure Transformer Inference Made Non-interactive. In Proceedings 2025 Network and Distributed System Security Symposium, San Diego, CA, USA (en). External Links: ISBN 979-8-9894372-8-3, Link, Document Cited by: §1, §1, Figure 2, Figure 2, §3.2.2, §4.3, §5.1, §5.3, §6.