Paper deep dive
MOSAIC: Masked Outsourcing of Secure AI Computations
James Hsin-yu Chiang, Sheila Zingg, Kari Kostiainen, Srdjan Capkun
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 91%
Last extracted: 8/3/2026, 2:47:05 AM
Summary
The paper introduces MOSAIC, a protocol for securely outsourcing AI computations from a trusted client to an untrusted server. It utilizes a novel matrix-multiplication masking protocol based on LWE and LPN assumptions, allowing the client to keep model weights and inputs private while the server performs the heavy computation. MOSAIC achieves optimal asymptotic client overhead and significantly faster runtimes compared to prior work, enabling confidential inference for large transformer models like LLaMA-3-70B with perplexity comparable to full-precision inference.
Entities (10)
Relation Signals (8)
MOSAIC โ reliesonsecurityassumption โ LPN
confidence 95% ยท Its security reduces to the decisional LWE and LPN assumptions.
MOSAIC โ reliesonsecurityassumption โ LWE
confidence 95% ยท Its security reduces to the decisional LWE and LPN assumptions.
MOSAIC โ evaluatedon โ Llama 3 70B
confidence 92% ยท On large 70B transformer models, MOSAIC's perplexity is comparable... LLaMA-3-70B
MOSAIC โ evaluatedon โ HumanEval
confidence 90% ยท even matches full-precision BF16 inference on HumanEval
MOSAIC โ outperforms โ Slalom
confidence 90% ยท MOSAIC promises a practical pathway... compared to prior work... Slalom... does not provide privacy for the model weights
MOSAIC โ outperforms โ Trapdoor Matrices
confidence 90% ยท We report runtimes showing orders of magnitude runtime improvement compared to the most efficient prior work [6]... [6] is the closest to our work
MOSAIC โ usestechnique โ Random Hadamard Rotations
confidence 88% ยท MOSAIC addresses this with an error-scaling mechanism based on random Hadamard rotations.
MOSAIC โ breakssecurityof โ ArrowCloak
confidence 85% ยท we show that the security of their reduction is broken in Appendix F.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:We address the challenge of securely and efficiently outsourcing AI computations from a trusted but computationally weak client to an untrusted but powerful server, in the setting where the client holds both the input and the model, and the server must learn neither. We present MOSAIC, whose core is a novel matrix-multiplication masking protocol that scales to far larger matrices than prior work, enabling the safe outsourcing of modern workloads such as large transformer inference. By introducing small amounts of noise to the multiplication result and thereby relaxing correctness, MOSAIC achieves optimal asymptotic client overhead and concrete runtimes orders of magnitude faster than prior work. Its security reduces to the decisional LWE and LPN assumptions. Because this noise accumulates across the many layers of a transformer, a key technical challenge is bounding error growth; MOSAIC addresses this with an error-scaling mechanism based on random Hadamard rotations. On large 70B transformer models, MOSAIC's perplexity is comparable to popular quantization approaches and even matches full-precision BF16 inference on HumanEval. Finally, we present an end-to-end implementation showing how ideas like MOSAIC can promise a path towards large-scale confidential AI in modern data centers. Non-confidential inference is already distributed across phase (prefill/decode), layer, and time to maximize utilization of heterogeneous hardware, using RDMA-like networking to move activations, cached KV values, and weights across nodes. MOSAIC enables scaling of confidential compute by keeping the trusted computing base (TCB) small and outsourcing the bulk of the AI computation to untrusted accelerators.
Tags
Links
- Source: https://arxiv.org/abs/2607.29221v1
- Canonical: https://arxiv.org/abs/2607.29221v1
Trouble viewing inline? Open PDF directly โ
Full Text
101,316 characters extracted from source content.
Expand or collapse full text
MOSAIC: Masked Outsourcing of Secure AI Computations James Hsin-yu Chiang ETH Zurich Switzerland Sheila Zingg ETH Zurich Switzerland Kari Kostiainen ETH Zurich Switzerland Srdjan Capkun ETH Zurich Switzerland Abstract We address the challenge of securely and efficiently outsourcing AI computations from a trusted but computationally weak client to an untrusted but powerful server, in the setting where the client holds both the input and the model, and the server must learn neither. We present MOSAIC, whose core is a novel matrix-multiplication masking protocol that scales to far larger matrices than prior work, enabling the safe outsourcing of modern workloads such as large transformer inference. By introducing small amounts of noise to the multiplication result and thereby relaxing correctness, MOSAIC achieves optimal asymptotic client overhead and concrete runtimes orders of magnitude faster than prior work. Its security reduces to the decisional LWE and LPN assumptions. Because this noise accumulates across the many layers of a trans- former, a key technical challenge is bounding error growth; MO- SAIC addresses this with an error-scaling mechanism based on random Hadamard rotations. On large 70B transformer models, MOSAICโs perplexity is comparable to popular quantization ap- proaches and even matches full-precision BF16 inference on Hu- manEval. Finally, we present an end-to-end implementation 1 showing how ideas like MOSAIC can promise a path towards large-scale confidential AI in modern data centers. Non-confidential inference is already distributed across phase (prefill/decode), layer, and time to maximize utilization of heterogeneous hardware, using RDMA- like networking to move activations, cached KV values, and weights across nodes. MOSAIC enables scaling of confidential compute by keeping the trusted computing base (TCB) small and outsourcing the bulk of the AI computation to untrusted accelerators. 1 Introduction We investigate the problem of outsourcing AI computations, which are dominated by linear matrix multiplications over billions of model weights. The large dimensions of model weight matrices have given rise to highly parallelized GPU accelerators to speed up computations over billions of model weight values. Our contributions in this work enable practical outsourcing of large AI computations to untrusted accelerators, which are not permitted to learn anything about the AI computation other than approximate topology of the model architecture. Prior state-of-the- art solutions do not scale to larger model dimensions [5,6], or offer concretely impractical runtimes [7,12,14,15,18,20,21,34] or no formal security [17,25,29,32]. MOSAIC promises a practical pathway towards confidential AI computations in modern AI data 1 Implementation available at https://github.com/jachiang/mosaic centers, where a standard (non-confidential) inference request can- not be practically served by a single GPU node and is distributed across a heterogeneous accelerator pool, with each node contribut- ing to different model layers (attention vs. MLP) and computation phase (prefill vs. decode) [26,27,31]. Here, MOSAIC permits a small, user-dedicated TCB (e.g. confidential compute) to be scaled by out- sourcing the bulk of the AI computation to untrusted accelerators. Alternatively, extending the TCB would imply secure attestation of the entire accelerator pool and networking stack. Layer-by-layer outsourcing. We consider a path towards practical confidential AI computation that considers the total computational cost of a forward-pass across a transformer-style model, and focuses on outsourcing the dominating, linear computations at each model layer to an untrusted acceleratorG. In this approach, the non-linear computation is performed by the trusted clientC, inducing a slalom- like [28] execution between a trusted clientCand untrusted server (accelerator)G, as the forward-pass computation of a transformer- style model over hundreds of alternating linear and non-linear layers, with expensive linear layers outsourced to computationally powerfulG (as shown in Figure 1). Linear computation consists of matrix multiplications of model weightsํ(ํรํ) and activationsํ(ํรํ). Given that repeated lin- ear matrix arithmetic dominates computation during each forward- pass, there has been recent effort to develop cryptographic protocols to securely outsource matrix-matrix multiplications [5,6]. How- ever, such solutions have significant limitations that prevent their deployment in practice. (1) State-of-the-art in secure outsourcing of matrix multiplication ofํ(ํร ํ) andํ(ํรํ) inducesํ(ํํ ํ ํ)overhead for the client [6], whereํmust be parameterized for givenํto satisfy LPN security. This prevents its application to larger modern models, where inner dimensions are very large (e.g. an MLP projection of dim 29 568 in Qwen2.5-72B). An optimal protocol achievesํ((ํ+ ํ)ํ)trusted client complexity, and practical concrete runtimes. (2)Cryptographic proposals generally operate in the discrete, in- teger ring domain. This requires fixed-point translation to em- ulate the underlying computation over floating point. Limited dynamic range in standard fixed-point arithmetic has a cata- strophic effect on large, modern LLM architectures (Figure 8). No prior work scales AI outsourcing up to modern (70B) models and consequently does not address this problem. (3)Prior work does not investigate practical applications amenable to AI outsourcing that relies on layer-by-layer communication. Our contributions address these open challenges as follows. 1 arXiv:2607.29221v1 [cs.CR] 31 Jul 2026 Chiang et al. Secure Linear Layer (Section 5) Secure Approx. MatMul (Section 4) Secure Forward Pass (Section 5) Trusted client C Untrusted GPU G Input tokens RMSNorm Attention (softmax, scores) Residual + RMSNorm SiLU gating Residual Next layer ํ ํ ,ํ ํพ ,ํ ํ MatMul ํ ํ MatMul ํ gate ,ํ up MatMul ํ down MatMul ฮ Sec-Linear ฮ Sec-Linear ฮ Sec-Linear ฮ Sec-Linear รํฟ layers ํ (BF16) Rotate ํปํ , cast to 16-bit fixed-pt ฮ Sec-Approx-MatMul Cast to BF16, add biasํ ํ โํํ +ํ Client masks weights: ํ ++ ํธ ํค Client masks activations: ํ ++ ํธ ํฅ GPU:ํ โฒ =ํ โฒ ํ โฒ Client unmasks: ํ โํํ Figure 1: Secure, confidential forward pass for a single transformer layer. The trusted clientC(left) performs all non-linear operations (attention scores, activations, normalization) locally. All weight matrix multiplications are outsourced to the untrusted GPUG (right) viaฮ Sec-Linear , which exchanges only masked activations and results at runtime. SecurePrivateClient layerPracticalSecurityComputational outsourcinginferencecomplexityruntimesInputModelerrorPerplexity (70B) FHE [12, 15, 34]โํ((ํ+ํ)ํ) #โfixed-point 1.827 3 MPC/FSS [7, 14, 18, 20, 21] รโํ(ํํํ)#โfixed-point Slalom [28]โรํ((ํ+ํ)ํ) G#โ รfixed-point Obfuscation [17, 25, 29, 32]โรํ((ํ+ํ)ํ) heuristic / broken 4 fixed-point Trapdoor Matrices [6]โรํ(ํํ ํ ํ)G#โfixed-point MOSAIC (our work)โรํ((ํ+ํ)ํ) โfixed-point+ ํ 1 1.834 2 Table 1: Comparison against related work on confidential outsourcing of linear-layer computation. Client complexity is per multiplication of model weights(ํรํ)and activation(ํรํ). Perplexity values are reported for LLaMA-3-70B on WikiText-2. 1 ํdenotes a low-norm Gaussian induced by our protocol. 2 Our protocol perplexity is considerably lower (better) than standard NF4 4-bit (2.247) quantization baselines and very close to the BF16 reference (1.821). 3 Idealized perplexity: prior-work perplexity entries assume our proposed random-rotation preconditioning that those works do not specify but which could in principle adopt for improved accuracy. 4 Obfuscation-based security is heuristic; we break security of ArrowCloak [29] (USENIXโ25). 1) Inner protocol: Outsourcing of approximate MatMul. We pro- pose a cryptographic protocol which permits the trusted client to efficiently outsource large matrix multiplications to an untrusted acceleratorG. Our simplified protocol (described in Section 4.1 and illustrated in Figure 1) masks both model weight and input matrices by padding with low-rank uniform matrices and a full-rank, small- norm gaussian matrix. The security of our full protocol (Section 4.2) can be reduced to LWE and LPN decisional hardness assumptions. Crucially, the matrix multiplication of the encrypted model weights and encrypted inputs can be recovered by the client with a small approximation error, which only induces a minor compromise in model accuracy in practice (Section 5.4). The underlying LWE security and noisy recovery is critical in the scalability of this approach. The low-rank of the uniform padding matrices remains constant across all matrix dimensions polynomial in the security parameterํ. For an inner rankํ, fixed solely by the security parameter and gaussian noise norm, the optimal client overhead is thusํ(ํ(ํ+ํ)ํ) โ ํ((ํ+ํ)ํ). We report runtimes showing orders of magnitude runtime improvement compared to the most efficient prior work [6], and defer the discussion to the related work in Section 2. Matrix (nร n)ProtocolServer (ms)Client (ms)Total (ms)Client / Total 36864ร 36864MOSAIC8.570.298.863.3% 16384ร 16384MOSAIC2.040.162.207.3% [6]127047.513183.6% 8192ร 8192MOSAIC0.900.121.0211.8% [6]270.014.5284.55.1% [5] โ โโผ500โ 4096ร 4096MOSAIC0.480.120.6020.0% [6]57.66.0563.659.5% [5] โก โโผ130โ Figure 2: Wall-clock runtime of a single secure matrix-vector multiplication for MOSAIC versus prior state-of-the-art [5,6]. Entries for [5] are placed in the nearest row by dimension: โ 8192ร 10000 and โก 2048ร 10000. 2) Secure linear layer and noise mitigation. From our secure ap- proximate MatMul protocol, we construct a full linear layer protocol 2 MOSAIC: Masked Outsourcing of Secure AI Computations which is applied as a drop-in-replacement for the General Matrix Multiply (GEMM) or ํผํํ + ํฝํถ. We highlight two challenges. Our secure linear layer (ฮ Sec-linear ), shown in Figure 1, intro- duces both fixed-point quantization (16-bit) and protocol noise (ํํธ ํฅ + ํธ ํค ํ + ํธ ํค ํธ ํฅ ). We note that the fixed-point quantization error is inherent to all cryptographic protocols [5,6,29] that se- curely outsource linear computation. However, fixed-point impact on model accuracy has not been evaluated in prior, related cryp- tographic techniques. Let the private inputsํand model weights (ํ) be cast to 16-bit integers (with appropriate fixed-point scales) andํํaggregated over a 32-bit integer ring. To mitigate the quan- tization error, we propose the application of an efficient, random Hadamard rotation [4] to activation and weights matrices. With a series of error accumulation studies on 70B LLM models, we show that even with our protocol noise, the accuracy of the internal ac- tivations across all internal layers is close to full precision (BF16) and can exceed common NF4 and INT8 quantization. Further, we report accuracy studies (Perplexity, HumanEval) with our protocol to demonstrate that our (noise) parameter range required for 140- bit security is practical even for models with known quantization challenges (e.g., LLaMA3-70B [22]). 3) Case studies for layer-by-layer outsourcing. We evaluate two application scenarios for our solution. In our primary use-case in Section 6, we report on prefill and decoding runtimes in a data- center setting, where a trusted computing base is extended with un- trusted GPUs, connected with fast interconnects. Here, we highlight two important ongoing trends in AI. Firstly, the leading open and closed sourced frontier models now exceed 1 trillion parameters; larger model dimensions permit ever more efficient outsourcing of AI computations in MOSAIC (Figures 6 and 13), which induces a client overhead that scales optimally with larger matrices, un- like prior work. Secondly, whilst MOSAIC requires layer-by-layer communication of masked activations, a standard, non-confidential inference in practice already induces layer-by-layer communication in todayโs AI data centers, as a single inference task for frontier models is no longer feasible on a single cluster, necessitating fast remote direct memory access (RDMA) networking across the data center [26,31]. Solutions such as MOSAIC can leverage existing ultra-low latency interconnect technology for scalable, confiden- tial AI computation. Indeed our experiments provide evidence that modern GPU-to-GPU interconnects are not the bottle-neck for MO- SAIC and that outsourcing AI computations in the data-center is efficient in practice (Figure 11). In a secondary use-case illustrated in Appendix H, we implement a remote decisional inference application, where the expensive AI computation is outsourced to the cloud over the public internet. In this application, only a single forward pass may be sufficient to produce a complicated decision, limiting the number of round-trips. We show that such a decision could be produced in under 5 seconds for large 70B models. Current hardware limitations. Cryptographic protocols such as MOSAIC operate over integer rings, necessitating 32-bit integer arithmetic, which is not natively supported in the fastest cores of modern AI accelerators. We implement an emulation thereof over 8-bit Tensor cores, which results in aโผ3ร(decode) to 5โ11ร (prefill) slow-down versus a baseline, non-confidential inference on 70B models. We think much of this gap can be explained by the emulation overhead, in which each 32-bit integer MatMul is realized with 10x underlying 8-bit integer MatMul launches. We hope that works such as MOSAIC can inspire AI hardware manufacturers to consider native 32-bit integer support in future accelerator designs. 2 Related work We outline three categories of related work. (1) Private inference refers to the task of computing inference over clientโs private inputs and serverโs private model. (2) Secure outsourcing of matrix mul- tiplications assumes a client with both inputs and model weights that wishes to offload the computation to an untrusted server. (3) Model obfuscation refers to the task of keeping proprietary model weights private from an untrusted server or accelerator. Table 1 summarizes our discussion and comparison to related work. Private inference. Private inference is typically realized with cryptographic techniques such as secure multi-party computation (MPC) or fully homomorphic encryption (FHE). MPC techniques focus on secret-sharing (S) [7,18,21] and functional secret-sharing (FSS) [14,20]. The latter still requires 40s per autoregressive de- coding step for a small 13B transformer model, thus not enabling practical runtimes for most applications. Our proposal demonstrates autoregressive decoding runtimes below 1s for a 70B model (Fig- ure 11). Approaches with fully homomorphic encryption [12,15,34] employ a powerful, general cryptographic primitive that incurs a very high computational overhead that is multiple orders of magni- tude slower than dedicated matrix multiplication protocols. Gazelle proposes a hybrid between MPC and FHE [16]. We note that FHE schemes like CKKS [8] exist which enable approximate arithmetic. These are distinctly different than our approximate secure MatMul protocol, which does not incur the heavy overhead of FHE. Secure outsourcing of matrix multiplications. Slalom [28] out- sources the computationally expensive matrix multiplication op- erations layer-by-layer to a remote GPU, an outsourcing pattern adopted by this work as well. However, it does not provide privacy for the model weights and requires expensive pre-processing mate- rial consumed with each inference run. Our work protects both the model and client inputs. A recent work [5] introduces a matrix encryption technique from pairs of secret dual codes, whose orthogonality cancels the dominant masking cross-term during recovery. The construction focuses on matrix-vector multiplication and operates exclusively over prime fields. This approach can be efficient for models with small dimensions but (comparable) runtimes are superseded by [6]. The authors of [6] introduce a matrix outsourcing technique (trap- door matrices) that relies on recursive masking with both low-rank, dense and sparse full-rank masking elements - the security of their scheme reduces to decisional LPN. The trusted, outsourcing client incurs aํ(ํํ ํ ํ)cost and scales poorly for concreteํand larger matrix dimensions required in modern LLM models. Moreover, the paper introduces a conjecture that LPN security holds for smallํ in order for concrete efficiency with large matrix dimensionํ, and no concrete protocol parameters are provided to support this in practice. Still, [6] is the closest to our work in concrete performance 3 Chiang et al. and technique; their recursive masking technique permits an ele- gant trade-off between memory and online computation, which we show can be adapted as an optimization to our approximate matrix multiplication scheme (section 4.2) whilst retaining optimal client overhead. Our key cryptographic contribution is to introduce a secure approximate matrix multiplication scheme with optimal computa- tional complexity for the client (ํ((ํ+ ํ)ํ)) and the untrusted accelerator (ํ(ํํํ)), providing accuracy comparable to full pre- cision in many practical AI computation tasks. In Figure 2, we highlight client and server run-times in MOSAIC that are orders of magnitude faster than those reported by [6], which in turn improves on all prior work for matrix dimensionํ โฅ2048. Note that [6] only reports CPU runtimes and provides no public protocol parameters for reproduction. Client/Total walltime ratios are skewed in MO- SAIC by slower Nvidia CUDA core implementation of the client, whilst the server is implemented on the faster Nvidia Tensor cores (See client vs. server implementations in Section 6). We show the theoretical client/total computation complexity ratio in fig. 13 for a large range of matrix dimensions. Model obfuscation. A line of work [17,25,29,32] intends to shield private LLM models from users, by obfuscating weights and hosting de-obfuscation material inside the userโs trusted execution environment (TEE). These works use efficient scaling and permu- tation techniques to obfuscate the weights, but such approaches lack security guarantees; [29] introduces a deobfuscation attack which exploits similarity between pre-trained open-source models and fine-tuned, private weights. Further, the authors introduce an improved model weights masking scheme called ArrowCloak, for which they claim a reduction to LWE hardness. However, we show that the security of their reduction is broken in Appendix F. Whilst such techniques could in principle be related to matrix multiplica- tion outsourcing, they rely on heuristic security arguments. 3 Background Learning with Errors (LWE) and Learning Parity with Noise (LPN). The security of our solution reduces to two standard post-quantum distinguishing assumptions. Decisional LWE [23]: distinguish sam- ples(ํ,โจํ,ํ โฉ+ ํ)from uniform, whereํ โ ํ ํ is uniform,ํ โ ํ ํ secret, andํis small Gaussian noise; hardness is conjectured for polynomially many samples. Decisional LPN [3]: the same shape butํis sparse uniform with noise rateํabove a secret-rank and security level determined threshold. In this work, we mask weights and activations with a low-rank dense term plus small Gaussian noise (LWE, Section 4.1), then nest the masking by recursively replacing the dense term with an LPN instance of strictly smaller rank (Section 4.2). Transformer architecture. A decoder-only transformer maps to- kens to logits via an embedding,ํฟtransformer blocks (or model layers), and a final language-model head. Each model layer follows the residual + pre-norm pattern โ โ =โ โโ1 + MLP ( โ โโ1 + Attn(RMSNorm(โ โโ1 )) ) ,(1) combining attention and a position-wise MLP, each preceded by RMSNorm(โ)=โ/( โฅ โ โฅ 2 / โ ํ)โ ํ, which projectsโonto a sphere of radius โฅ ํ โฅ 2 , collapsing any radial disagreement. Attention projectsํ โ R ํ รํ model intoโquery/key/value heads viaํ ํ ,ํ ํพ ,ํ ํ , and computes per headsoftmax(ํ ํ ํพ โค ํ / โ ํ ํ )ํ ํ , then re-projects throughํ ํ . The position-wise MLP is a SiLU- gated two-layer feed-forwardํ down (SiLU(ํ gate ํ)โํ up ํ)with ํ f > ํ model . The seven dense projections (ํ ํ ,ํ ํพ ,ํ ํ ,ํ ํ ,ํ gate ,ํ up ,ํ down ) dominate compute and memory; softmax, RMSNorm, and SiLU are cheap elementwise ops. Our protocol outsources only those seven dense matmuls, leaving non-linear and per-head work on the trusted client. GPU compute and quantisation. Modern NVIDIA GPUs expose two compute paths: CUDA cores (general purpose cores) and tensor cores (matrix-multiply-accumulate units in FP16/BF16/INT8/FP8 with FP32 or INT32 accumulation). Tensor-core throughput exceeds CUDA-core matmul by an order of magnitude; LLM inference is consequently bottlenecked by tensor-core matmul and memory bandwidth. Quantisation casts weights and activations from BF16 to lower precision to shrink the model and accelerate compute; we use NF4, INT8 (and BF16) model accuracy baselines in Section 5.3. 4 Secure approximate matrix multiplication A key task in our confidential AI computation is the outsourc- ing of linear matrix multiplications in the floating-point domain. We first focus on the outsourcing of matrix multiplications in the the integer ring domain of chosen bit-width, in which our crypto- graphic protocol operates. Our solution approximately outsources a matrix multiplicationํํover the integer ring domain of chosen bit-width - whereํ โ Z ํรํ 2 ํ is a quantized model weight matrix andํ โ Z ํรํ 2 ํ an activation matrix - to an untrusted accelerator G while hiding bothํ and ํ . We present an asymptotically opti- mal, approximate protocol withํ((ํ+ํ)ํ)client online cost, and prove post-quantum security under LWE and LPN assumptions. The optimality claim follows directly from the inputํํand output ํํ dimensions of the outsourcing protocol. In later Section 5, we bridge the integer ring and floating point domains with a secure linear protocol which mitigates the effects of fixed-point quantization and induced protocol error in the float- ing point domain, in which the interleaved non-linear layers are performed on the trusted client. 4.1 Simple protocol with noisy unmasking We illustrate the core idea with a simplified construction to out- source the computation ofํํ , whereํ (ํรํ) and ํ (ํรํ ). To mask the quantized weight matrix, the trusted client samples ํฟ โU Z ํรํ 2 ํ , ํ โU Z ํรํ 2 ํ , ํธ ํค โN 0,ํ 2 ํรํ and sets: ํ โฒ =ํ + ํฟํ+ ํธ ํค (2) Let us denoteํฟthe left andํthe middle low-rank matrices in the mask, andํธ ํค the low-norm, full-rank Gaussian error term, parameterized by standard-deviationํ. We note thatํcan be a public matrix whilst retaining LWE security. Here, rankํdenotes the rank of both uniform, low-rank matricesํฟandํ. Critically,ํis solely a function of security parameterํand must be strictly lower than matrix dimensionsํ,ํfor client efficiency. Givenํ โ140, inner rankํcan be set to 1536 (Example 4.1). For a modern model dimension of 16 384โซ1536, the efficiency condition can easily be 4 MOSAIC: Masked Outsourcing of Secure AI Computations ํ ํรํ + ํฟ ํรํ ํ ํ รํ + ํธ ํค ํรํ = ํ โฒ ํรํ Figure 3: Structure of the masking scheme, consisting of private weights matrixํ, left low-rank matrixํฟ, middle low-rank (r) matrix ํ and full-rank gaussian ํธ ํค . The activation matrix ํ is masked analogously. ํ โฒ ํ โฒ ํรํ = ํํ ํรํ + ํฟ ํรํ ํ(ํ+ํธ ํฅ ) ํ รํ + ํ โฒ ํ ํรํ ํ ํ รํ + ํธ ํค ํ + ํํธ ํฅ + ํธ ํค ํธ ํฅ ํรํ Figure 4: Unmasking the outsourced matrix product. The GPU returnsํ โฒ ํ โฒ , which expands into the desired productํํ(blue), low-rank correction terms (green), and small-norm noise (red). The client efficiently removes the green terms inํ((ํ+ํ)ํ) using precomputed material, recoveringํํ up to a small approximation error. met (Figure 6). For smaller model dimensions, we can further reduce the inner masking rank in our full protocol, detailed in Section 4.2. Note that the masked matrixํ โฒ is now indistinguishable from a uniformly random matrix from the view of the untrusted ac- celeratorG. The security of the encryption reduces to Learning With Errors (LWE) (lemma 1). Whilst the cost of masking model weights isํ(ํํํ), it only has to be performed once for all matrix multiplications with fixedํ . The input matrix is masked similarly. For all inference runs, we sample a singleํ โU Z ํรํ 2 ํ . Then, for each individual inference pass, we sample ํ โU Z ํรํ 2 ํ and ํธ ํฅ โN 0,ํ 2 ํรํ and set: ํ โฒ = ํ + ํํ + ํธ ํฅ Again, letํbe the middle andํ denote the right low-rank masking matrices, whereํcan be a public matrix. The clientCsendsํ โฒ ,ํ โฒ to the untrusted acceleratorG, which computesํ โฒ = ํ โฒ ํ โฒ and returns the encrypted resultํ โฒ back to the client. In terms of the individual matrix components, this gives ํ โฒ =(ํ + ํฟํ+ ํธ ํค )(ํ + ํํ + ํธ ํฅ )(3) Expanding: ํ โฒ =(ํ+ํฟํ+ํธ ํค )ํ+(ํ+ํฟํ+ํธ ํค )ํํ +(ํ+ํฟํ+ํธ ํค )ํธ ํฅ (4) =(ํ + ํฟํ+ ํธ ํค )ํ +ํ โฒ ํํ +(ํ + ํฟํ+ ํธ ํค )ํธ ํฅ (5) =ํํ + ํฟํ(ํ + ํธ ํฅ )+ํ โฒ ํํ + ํธ ํค ํ +ํํธ ํฅ + ํธ ํค ํธ ํฅ (6) The client removes two correction terms: Firstly,ํฟํ(ํ + ํธ ํฅ ), which costsํ(ํ(ํ+ํ)ํ);ํ(ํ+ํธ ํฅ )induces client computational complexity ํ(ํํํ), and subsequent left-multiplication ํ(ํํํ). The second correction term(ํ โฒ ํ)ํ is also a rankํcomputation ifํ โฒ ํ โ F ํรํ ํ is precomputed once and reused, costingํ(ํํํ) per subsequent matmul. Since bothํ โฒ and public matrixํcan be given to untrustedG,ํ โฒ ํcan also be outsourced. This process is illustrated in Figures 3 and 4, and yields ํ โ ํ โฒ โํฟํ(ํ+ํธ ํฅ )โ(ํ โฒ ํ)ํ =ํํ+ํธ ํค ํ+ํํธ ํฅ +ํธ ํค ํธ ํฅ (7) The three trailing error terms remain as an error in the result. For fixed security level ํ, the client complexity is ํ((ํ+ํ)ํ). Optimal scalability for larger dimensions. Crucially, LWE secu- rity is robust against polynomially many adversarial samples. This means the LWE parameters (rankํand noise of standard deviation ํ) are fixed by the security level and do not need to grow with the matrix dimensionsํ,ํ,ํ. Concretely, for 140-bit security, we can fixํ=1536 (a GPU-friendly dimension) and gaussian noise with stddevํ=0.5 ([2]) regardless of whether the matrix has rank 4096 or 29568. This constitutes a very small norm Gaussian relative to the full 32-bit integer ring we instantiate our protocol in. Whilst the rankํis fixed by the choice of security parameter ํ, the pure LWE unmasking for clientCstill costsํ(ํ(ํ+ ํ)ํ) and is linear inํ. Our full protocol in Section 4.2 improves concrete efficiency further by replacing the uniform, low-rank components with structured, sparse masks that are cheaper to undo. Indistinguishability. It is easy to see that the masked weights (and analogously activation) matrices are computationally indis- tinguishable from uniform. We provide the proof in Appendix A.3. Lemma 1. The LWE mask is computationally indistinguishable from uniform: ํฟํ+ ํธ ํค ํ โ ํ. As a consequence, for anyํthe masked weightํ โฒ =ํ+ํฟํ+ ํธ ํค is computationally indistinguishable from uniform. Proof sketch in Appendix A.3. Analogously for ํํ + ํธ ํฅ ํ โ ํ . Bounded protocol error. We first analytically bound the proto- col error term in Theorem 1 with proof and supporting lemmas in Appendix A.1. In subsequent Section 5, we illustrate how to effectively mitigate approximation errors in the context of full AI computations via an efficient random rotation of bothํ and ํ . Unmasking ofํ โฒ ํ โฒ leaves the following error terms in the result:ํํธ ํ + ํธ ํ ํ + ํธ ํ ํธ ํ , whereํธ ํ โ Z ํรํ 2 ํ andํธ ํ โ Z ํรํ 2 ํ are discrete Gaussian noise matrices with parameterํ(distribution N Z 2 ํ ; 0,ํ 2 over Z 2 ํ , with Pr[ํ= ํ] โ exp โํ 2 /(2ํ 2 ) ). Theorem 1. Letํ โ Z ํรํ 2 ํ ,ํ โ Z ํรํ 2 ํ , and letํธ ํ โ Z ํรํ 2 ํ , ํธ ํ โ Z ํรํ 2 ํ have independent entries drawn fromN Z 2 ํ ; 0,ํ 2 . Then 5 Chiang et al. Protocolฮ Sec-Approx-MatMul : Secure Approximate MatMul. Parties: Trusted client C and untrusted accelerator G. Init W: Compute one-time masking of weights matrix. (1) C samples: ํฟ โ Z ํรํ ํ 2 ํ , ํ ํโ[ํ] โ Z ํ ํ รํ ํโ1 2 ํ , ํ ํโ[ํ] โ Z ํ ํโ1 รํ ํ 2 ํ , ํธ ํค โ N(Z ํรํ 2 ํ ,ํ 2 ) , ํ ํโ[2:ํ] โ S Z ํ ํ รํ ํโ1 2 ํ ;ํ ํ (2) C computes:ํ ํค = ํฟ(ํ ํ ํ ํโ1 ...ํ 1 )+ ร ํโ1 ํ=1 ํ ํ+1 (ํ ํ ํ ํโ1 ...ํ 1 )+ ํธ ํค . (3) C stores: ํ 1 ํ 2 ...ํ ํ and ํฝ ํ = (ํ + ํธ ํค )ํ 1 ํ 2 ...ํ ํ forํ โ [ํ]. (4) C stores: ํ โ Z ํ 2 ํ ,ํ chk โ ํ ํ ํ โฒ . (5) C storesํ โฒ =ํ +ํ ํค and ํ ํ ํ ํโ1 ...ํ 1 forํ โ [ํ] on untrusted G. Online phase: On fresh inputํ, securely outsourceํํtoGand recover result. (1) Init X: C computes and stores masking material consumed by X. (a) ํ โ Z ํ ํ รํ 2 ํ , ํธ ํฅ โ N(Z ํรํ 2 ํ ,ํ 2 ) andํ ํโ[2:ํ] โ S Z ํ ํโ1 รํ ํ 2 ํ ;ํ ํ . (b) ํ ํฅ = ํ 1 ํ 2 ...ํ ํ ํ + ร ํโ1 ํ=1 ํ 1 ํ 2 ...ํ ํ ํ ํ+1 + ํธ ํฅ . (c) ํพ= ํฝ ํ ํ + ร ํโ1 ํ=1 ํฝ ํ ํ ํ+1 (Unmasking pre-computation). (2) C forwards to G: ํ โฒ = ํ +ํ ํฅ . (3) G returnsํ โฒ =ํ โฒ ํ โฒ andํ ํ = (ํ ํ ํ ํโ1 ...ํ 1 )ํ โฒ forํ โ [ํ]. (4) C assertsํ chk ํ=ํ ํ ํ โฒ and aborts otherwise. (5) C outputsํ โ ํ โฒ โ ํพ โ ํฟํ ํ โ ร ํโ1 ํ=1 ํ ํ+1 ํ ํ (Unmask Y). Figure 5: Secure approximate matrix multiplication outsourc- ing protocol. entry(ํ,ํ)of the total errorํธ ํ ํ +ํํธ ํ +ํธ ํ ํธ ํ is approximately ํ โ๏ธ โฅ ํฅ ํ โฅ 2 2 + ํค ํ 2 2 -sub-Gaussian: for all ํก> 0, Pr |(ํธ ํ ํ +ํํธ ํ + ํธ ํ ํธ ํ ) ํํ |> ํก โค 2exp โ ํก 2 2ํ 2 ( โฅ ํฅ ํ โฅ 2 2 + ํค ํ 2 2 ) ! up toํ โฮฉ(ํ) , whereํค ํ is theํ-th row ofํandํฅ ํ is theํ-th column of ํ. Proof in Appendix A.2. Leakage & integrity. The untrusted acceleratorGclearly learns the matrix dimensions and invocation of each secure MatMul out- sourcing. Without padding matrices, the model architecture can be inferred. With padding, an upper-bound on the number of model layers and matrix dimensions is leaked, at the cost of additional computation. A naive padding scheme could add (1) dummy out- sourced linear layers or (2) padded outputํor inputํdimensions. Option (1) incurs additional communication and client masking per dummy layer, whilst option (2) increases the per-query client complexity byํ(ํ โฒ ํ)(output paddingํ โฒ ) orํ(ํ โฒ ํ)(input padding ํ โฒ ), whereํis the batch (token) dimension; padding the batch byํ โฒ analogously adds ํ((ํ+ํ)ํ โฒ ). Integrity can be straightforwardly achieved by applying well- known Freivaldsโ algorithm for asserting correctness ofํํ= ํ computed by the untrusted GPU. We reproduce this correctness check in Appendix B for the readerโs convenience. 4.2Full protocol using noisy recursive masking Next, we describe a full protocol, specified in Figure 5, that takes advantage of a recursive masking technique from [6] by recur- sively replacing dense, uniform matrices of low-rank with LPN instances of ever decreasing rank. In contrast to [6], the outer LWE instance (as described in Section 4) has a fixed rank determined by security parameterํ, and is independent of matrix dimensions. The protocol [6] in original form has a mask with rank that must scale with matrix dimensions. We obtain trusted client efficiency of ํ((ํ+ํ)ํ), rather than ํ(ํํ ํ ํ) in [6]. For intuition, we replace the left privateํฟmatrix from our naive LWE-only protocol (Equation (2)) with an LPN instanceํฟ 2 ํ 2 + ํ 2 , that is computationally indistinguishable from uniform ํฟ. ํ โฒ =ํ + ํฟ 1 ํ 1 + ํธ ํค ํ โํ +(ํฟ 2 ํ 2 + ํ 2 )ํ 1 + ํธ ํค whereํฟ 2 ,ํ 2 are dense and uniform andํ 2 a sparse matrix (sparse LPN error). The rank ofํฟ 2 ํ 2 is strictly smaller than the originalํฟ (ํ 2 < ํ 1 ), allowing us to decrease the internal rank of the non-error masking term. We continue to apply the substitution to ํฟ 2 ํ โฒ ํ โํ +((ํฟ 3 ํ 3 + ํ 3 )ํ 2 + ํ 2 )ํ 1 + ํธ ํค (8) More generally over multiple substitution steps ํ โฒ ํ โํ +(((ํฟ ํ ํ ํ + ํ ํ )ํ ํโ1 + ํ ํโ1 )...ํ 2 )ํ 1 + ํธ ํค =ํ + ํฟ(ํ ํ ํ ํโ1 ...ํ 1 )+ ํโ1 โ๏ธ ํ=1 ํ ํ+1 (ํ ํ ํ ํโ1 ...ํ 1 )+ ํธ ํค (9) Note the rank of the dense termํฟ(ํ ํ ํ ํโ1 ...ํ 1 )has been reduced toํ ํ < ํ ํโ1 < ...< ํ 1 forํฟas a denseํรํ ํ matrix. This ultimately permits a trade-off between storage and client runtime, by reducing the inner rank of matrix computations. We illustrate this trade-off concretely in Example 4.1 and Table 2. We emphasize that inner dimensionsํ ํ are solely a function of security parameterํ, and independent of the matrix dimensions. Further, the termsํ ํ+1 (ํ ํ ํ ํโ1 ...ํ 1 )forํ โ [1 :ํโ1]are strictly over sparse matrices, with row sparsity also independent of the matrix sizes. During the online phase, the input matrix of dimensionํรํis masked analogously, naturally imposing ํ(ํํ) online runtime. ํ โฒ = ํ + ํ 1 ํ 2 ...ํ ํ ํ + ํโ1 โ๏ธ ํ=1 ํ 1 ํ 2 ...ํ ํ ํ ํ+1 + ํธ ํฅ (10) The (public) matrix product chains can be precomputed during initialization and outsourced. The masking operation has ํ(ํํ) โ ํ(ํํ +ํํ ํ ํ +ํ 1 ํ 2 ํ 2 ํ + ...+ํ ํโ1 ํ ํ ํ ํ ํ)complexity, where sparsity ํ ํ andํ ํ are fixed byํand independent ofํ,ํ,ํ. We note that the masks for quantized activationํcan be computed independently from ํ . Unmasking. Upon receivingํ โฒ ํ โฒ from the untrusted accelera- torG, the client removes all cross terms with low-rank dense and sparse matrices inํ(ํํ), whilst retrieving a result with the same error as the simplified LWE-only protocol in Equation (7). Letํ โฒ ํ โฒ = (ํ +ํ ํค )(ํ +ํ ํฅ )=ํํ +ํํ ํฅ +ํ ํค ํ +ํ ํค ํ ํฅ , whereํ ํค andํ ํฅ are the matrix masks from Equation (9) and Equation (10) respectively. To recoverํํ, the trusted client must remove the cross-terms, or a close approximation thereof. Firstly, consider termํํ ํฅ , ํํ ํฅ =(ํํ 1 ํ 2 ...ํ ํ )ํ + ํโ1 โ๏ธ ํ=1 (ํํ 1 ํ 2 ...ํ ํ )ํ ํ+1 +ํํธ ํฅ (11) 6 MOSAIC: Masked Outsourcing of Secure AI Computations This term can be efficiently computed up toํํธ ํฅ by the client during the online phase, given pre-computation of the chain prod- uctsํํ 1 ํ 2 ...ํ ํ , amortized over all multiplications over the same weight matrix. The client complexity to compute this term isํ(ํํ)= ํ(ํํ ํ ํ) + ํ(ํํ 2 ํ 2 ํ) + ...+ ํ(ํํ ํ ํ ํ ํ) asํ 2 ,ํ 3 ...,ํ ํ ,ํ 2 , ํ 3 , ...,ํ ํ are fixed by the security parameter and not by matrix dimension ํ,ํ,ํ . The cross-termํ ํค ํ โฒ = ํ ํค (ํ + ํ ํฅ )is computed up to error ํธ ํค ํ + ํธ ํค ํธ ํฅ . To illustrate this, consider the following expansion. ํ ํค ํ โฒ = ํฟ(ํ ํ ํ ํโ1 ...ํ 1 ํ โฒ )+ ํโ1 โ๏ธ ํ=1 ํ ํ+1 (ํ ํ ํ ํโ1 ...ํ 1 ํ โฒ )+ ํธ ํค ํ โฒ (12) Here, the multiplicative terms ํ ํ ํ ํโ1 ...ํ 1 ํ โฒ can be securely out- sourced to the untrusted acceleratorG. Thus, the first and second terms above can be computed by the client inํ(ํํ)=ํ(ํํ ํ ํ)+ ํ(ํํ 2 ํ 2 ํ)+...+ํ(ํํ ํ ํ ํ ํ)time. The error term expands toํธ ํค ํ โฒ = ํธ ํค ํ+ํธ ํค ํ ํฅ , of whichํธ ํค ํis left as protocol error. We expand the trailing term ํธ ํค ํ ํฅ further; ํธ ํค ํ ํฅ =(ํธ ํค ํ 1 ํ 2 ...ํ ํ )ํ + ํโ1 โ๏ธ ํ=1 (ํธ ํค ํ 1 ํ 2 ...ํ ํ )ํ ํ+1 + ํธ ํค ํธ ํฅ (13) Here, termsํธ ํค ํ 1 ...ํ ํ can be preprocessed by the client and amor- tized over all matrix multiplications over the quantized model weights, whilstํธ ํค ํธ ํฅ is left as protocol error. Thus the trusted client complexity of computing this isํ(ํํ)=ํ(ํํ ํ ํ)+ํ(ํํ 2 ํ 2 ํ)+...+ ํ(ํํ ํ ํ ํ ํ): again, rankํ ํ and sparsityํ ํ are fixed and independent of matrix dimensions. Overall, this represents an optimization to reduce the rank of dense matrix multiplications at runtime compared to the simplified protocol in Section 4.1. The final resulting protocol error isํํธ ํฅ + ํธ ํค ํ + ํธ ํค ํธ ํฅ , identical to the naive protocol in Section 4. Example 4.1. We provide an example parameterization for security levelํ=140. Security of the LWE and LPN instances were parame- terized with the LWE [2] and LPN [33] estimators by Albrecht et al. and Yu et al. respectively. We assumeํ=3 for the number of nested LWE/LPN instances. โข LWE: ํ 1 = 1536,ํ= 0.5 โข LPN-1: ํ 2 = 1024,ํ 2 = 1536,ํก 2 = 80 โข LPN-2: ํ 3 = 652,ํ 3 = 1024,ํก 3 = 80 Letํ ํ be the secret dimension andํ ํ the number of permitted adver- sarial samples (polynomially many for LWE). For LWE,ํdenotes the standard deviation of the dense, Gaussian noise term. For LPN instances, ํก denotes the hamming weight of the noise vector. Compute-storage tradeoff of the LPN nesting. Each additional LPN level shrinks the dense rank that the client multiplies through (ํ 1 โ ํ 2 โ ํ 3 ) at the cost of one extra chain productํ 1 ยทํ ํ to cache and one extra sparse maskํ ํ+1 of Hamming weightํก ํ+1 . Table 2 makes this concrete for the parameters above, omitting (i) the small sparse-mask client state ร ํ ํ=2 ํ ํ ํก ํ and (i) the GPU-side chain products(ํ 1 +ยท+ํ ํ )ํ. 246810121416 0.1 0.2 0.3 0.4 Rank of Model Weight Matrix (ร10 3 ) Trusted C runtime: outsourced / local ํํ Measured efficiency of outsourcing (ํ= 1) attn proj MLP down MLP up LLaMA-3.2 1B Gemma-2-27B LLaMA-4 Maverick 400B Kimi-K2.5 DeepSeek-V3 LLaMA-3.1 70B GPT-3 175B LLaMA-3.1 405B Figure 6: Outsourcing efficiency: Measured client runtime vs. localํํruntime (ํas vector,ํ=1) for various model weight dimensions. The mask initialization is batched for 512 invocations and amortized over each X. One-time initial- ization of W is excluded. ํ C compute/(ํ+ํ)ํ C state LWE onlyํ 1 = 1536 (ํ+ํ)ํ 1 +ํํ 1 = 1536(2ํ+ํ) + LPN-1ํ 2 +ํก 2 = 1104 (ํ+ํ)(ํ 1 +ํ 2 )+ํํ 2 = 2560(ํ+ํ)+ 1024ํ + LPN-2 ํ 3 +ํก 2 +ํก 3 = 812 (ํ+ํ)(ํ 1 +ํ 2 +ํ 3 )+ํํ 3 = 3212(ํ+ํ)+ 652ํ Table 2: Client compute/storage tradeoff of mask nesting. Under decisional LWE and LPN, the nested LWE+LPN mask is computationally indistinguishable from uniform (Lemma 4); hence for anyํthe masked matrixํ โฒ is indistinguishable from uniform. Theorem 2 (Security ofฮ Sec-Approx-MatMul ). Under decisional LWE and LPN: (1)(Privacy.) The adversaryGโs view(ํ โฒ ,ํ โฒ )is computationally indistinguishable from uniform over Z ํรํ 2 ํ ร Z ํรํ 2 ํ . (2) (Integrity.) See Appendix B. Proof.(Privacy.) Lemma 4 in Appendix A.4 givesํ โฒ ํ โ ํ ํรํ under decisional LWE and LPN. The activation mask in Equa- tion (10) follows the same nested LWE+LPN form (with roles of ํฟ,ํ ํ ,ํ ํ ,ํธ ํค taken byํ ,ํ ํ ,ํ ํ ,ํธ ํฅ ) and is sampled independently, so an analogous hybrid argument yields ํ โฒ ํ โ ํ ํรํ .โก 4.3 Outsourcing efficiency inฮ Sec-Approx-MatMul Efficiency of a matrix multiplication outsourcing protocol relies on a smaller ratio of (1) trusted client protocol overhead to (2) the cost of the client computingํํlocally. This depends on both the concrete size of the matrix dimensions (larger is better), as well as the ratio of matrix dimensions (balanced dims are better, given a fixed element count). For a theoretical assessment of this client 7 Chiang et al. efficiency ratio, we illustrate this in Figure 13 in the appendix for protocol parameters in Example 4.1 and raw computational work (not run-time). To experimentally evaluate the efficacy of client outsourcing for different model dimensions in practice, we show the ratio of (1) trusted client protocol runtime to (2) the runtime of computing ํํlocally in Figure 6 with our protocol implementation and hard- ware setup described in Section 6. Overall, model weight matrices with higher rank are clearly more amenable to outsourcing with our protocol than those with lower rank, validating the theoretical scalability of our protocol. We note an implementation nuance: for a given rank, the clientCefficiency in the implemented protocol can be observed to be positively correlated to the matrix output dimensionํ; MLP down projection matrices offer the best out- sourcing efficiency in our experiments. We partially attribute this artifact to our implementation and NVIDIA runtime environment, where the client masking/unmasking computation over low-rank and sparse matrix elements is implemented on the slower, yet more general CUDA cores, and the full-rankํํis optimized to run on the faster Tensor path. The trusted client implementation is likely more sensitive to increased input matrix dimensions, for which it must perform additional work on the slower CUDA path. As future model dimensions increase, we anticipate outsourcing to become significantly more worthwhile, as shown in Figure 13. Local attention computation. Efficiency of our protocol rests on a one-timeํ(ํํ)weight-masking step (Init W) that is amortized across every subsequent forward pass reusing the sameํรํmatrix. During prefill this amortization is unavailable for attention:ํ ํ ,ํพ ํ , andํ ํ are all per-query activations, so neither operand ofํํพ โค ํ nor ofscoreยท ํ ํ is a frozen matrix that can be masked once and reused. During autoregressive decoding the situation is better. The KV cache is built once per sequence and reused for every generated token, so one could in principle run Init W onํพ ํ andํ ํ at cache- build time and amortize it over all subsequent decode steps. Even with that amortization, however, the head dimension in models today is too small for outsourcing to pay off in practice. The client-side overhead ofฮ Sec-Approx-MatMul scales asํ((ํ+ํ)ํ) against a GPU cost ofํ(ํํํ), so the useful regime is one where rankํ= min(ํ,ํ)is large (Figure 13). The outsourced weight pro- jections sit comfortably in this regime withmin(ํ,ํ)= ํ model โฅ 4096, whereas per-head attention matmuls only have inner dimen- sionํ ํ = ํ model /โ โ128 for the model sizes we consider (e.g. LLaMA-3 70B and Qwen2.5-72B both with ํ model = 8192, โ= 64). 5 Secure forward pass for LLM models Our cryptographic protocolฮ Sec-Approx-MatMul works over the inte- ger ring domain. In order to leverage this to realize a secure forward pass over an entire model, we must instantiate a secure MatMul out- sourcing at each linear layer, and bridge the floating point domain to the inner, integer ring domain. A common approach in cryptographic protocols (Section 2) is to cast to the fixed point domain for the secure linear computation. These works do not consider large modern architectures, for which fixed-point quantization results in destructive error propagation across the many sequential model layers (for 70B models:โ80 blocks, each with 4 linear layers), as illustrated in Figure 8. We show this arises from a particular sensitivity of fixed-point arithmetic to large floating-point ranges. Thus, we adapt an approach from model quantization litera- ture [4,19,30], that applies a random Hadamard rotation to activa- tion and model weights, which only induces aํ(ํ log(ํ))runtime cost. We empirically demonstrate that this approach results in non- destructive error propagation across large (70B) models, and offer accuracy that is comparable with standard 4-bit and 8-bit quanti- zation schemes. Our resulting secure-linear protocolฮ Sec-Linear is shown in Figure 7. 5.1 Secure forward pass fromฮ Sec-Linear The secure forward pass replaces each linear layer with an invoca- tion ofฮ Sec-Linear and evaluates every non-linear layer locally on the trusted client. A full secure forward pass on the model can be described as follows. (1) C holds the BF16 activation ํ 1 . (2) For โ= 1, . . .,ํฟ:C andG jointly invoke ํ โ โฮ Sec-Linear-Layer (ํ โ ,ํ โ ) returning a BF16 approximation ofํ โ ํ โ +ํ โ toC; C then computes non-linear ํ โ+1 โ ํ โ (ํ โ ) locally. (3) C outputs activation ํ ํฟ+1 in BF16. Security and client efficiency follows by sequential composition of the per-layer protocol - we detail asymptotic efficiency in Ap- pendix E for a single secure forward-pass. The per-invocation error was previously bounded in Theorem 1, but is naturally dependent on distributions of actual model weights and activations. Straw-man solution: error explodes with outliers. We show naive fixed-point scaling over 16-bits with 32-bit accumulations results in catastrophic error in Figure 8 in the early layers. For the chosen Qwen2.5-72B model, both cosine similarity with the full-precision, BF16 activation is shown, as well as the relativeโ 2 error to the BF16 reference, for 20 forward-pass runs on randomly sampled 512- token windows from Wikitext-2. Clearly, the noise accumulation from fixed-point quantization alone causes accuracy to collapse in Figure 8. Whilst the cosine andโ 2 show a mild tendency for recovery in the medium layers (RMSNorm, Non-linear activations can dampen error), this does not suffice for any useful forward pass computation. We explain this phenomena analytically. Inฮ Sec-linear-layer (Fig- ure 7), each row ofํ(resp. each column ofํ) is independently scaled by its peak magnitude before quantization to aํ-bit integer range[โํ max ,ํ max ]. Concretely, for a rowํคofํ, the quantiza- tion scale isํ ํค = โฅ ํค โฅ โ /ํ max , and the quantized representation is ห ํค= round(ํค/ํ ํค ). Activation columns are scaled analogously with ํ ํฅ = โฅ ํฅ โฅ โ /ํ max . The windowํ max must also be chosen so that the inner-product accumulator does not wrap around theํ -bit ring: each entry( ห ํ ห ํ) ํ,ํ has magnitude at mostํํ 2 max , so we require ํํ 2 max โค 2 ํ โ1 . After the protocol computes the integer-domain result ห ํ= ห ํ ห ํ + noise , dequantization recovers the approximate floating- point output by rescaling each entry:ํ ํ,ํ = ห ํ ํ,ํ ยท ํ ํค ํ ยท ํ ํฅ ํ . The protocol noise, which has fixed variance in the integer domain, is therefore amplified by the product of scalesํ ํค ํ ยทํ ํฅ ํ . When eitherํค 8 MOSAIC: Masked Outsourcing of Secure AI Computations Protocolฮ Sec-Linear : Secure Linear Layer. Parties: Trusted client C and untrusted accelerator G. Inputs: BF16 weight matrixํ โ R ํรํ , optional biasํ โ R ํ , and on each invocation a BF16 activation ํ โ R ํรํ . Output: BF16 activationํ โํํ +ํ. Init W: One-time pre-processing of floating-point weights (reused across inputs). (1) C samples and stores ํ = WHT ํ ยท diag(ํ) withํ โ โ1,+1 ํ . (2) C rotates the weights f ํ โํํ โค and quantizes each row of f ํto a signedํ-bit integer with per-row scaleํ ํค ํ = eํค ํ โ /ํ max , where the operand windowํ max is chosen so that the inner-product accumulator stays inside Z 2 ํ . (3) Cembeds ห ํ intoZ 2 ํ and invokes the Init W phase ofฮ Sec-Approx-MatMul to produce the masked weights ห ํ โฒ stored onGand the unmasking material stored onC. Online phase: On fresh floating point input ํ , outsourceํํ +ํ to G and return a floating point result. (1) Crotates the activation e ํ โ ํ ํand quantizes each column of e ํto a signedํ-bit integer with per-column scaleํ ํฅ ํ = โฅ eํฅ ํ โฅ โ /ํ max , whereํ max is chosen to ensure inner-product accumulator stays within Z 2 ํ . (2) C and G run the Online phase ofฮ Sec-Approx-MatMul on ( ห ํ, ห ํ) over the integer ring Z 2 ํ . (3) C recovers ห ํ โ ห ํ ห ํ โ Z ํรํ 2 ํ up to the protocol noise of Equation (7), and dequantizes entry-wise back to FP fromํ ํ,ํ โ ํ ํค ํ ํ ํฅ ํ ห ํ ํ,ํ , and adds the bias to output ํ โ ํ +ํ. Figure 7: Secure linear layer with input/outputs in the floating-point domain. 0204060 โ0.1 0 0.5 1 Layer index (โ ) Cosine similarity 0204060 0 1 2 3 4 5 6 Layer index (โ ) Relative โ 2 error Figure 8: Per-layer error accumulation for Qwen2.5-72B with- out Hadamard rotation, INT16 quantization atํ=0 (no pro- tocol noise). Cosine similarity/relativeโ 2 error vs BF16 refer- ence collapses after 2 layers motivating random Hadamard rotation in our protocol. orํฅcontains outlier entries, โฅ ยท โฅ โ is large relative to โฅ ยท โฅ 2 , inflating the scales and thus the float-domain noise. In particular, activation vectors in transformer models are inher- ently concentrated: a small number of entries carry disproportion- ately large magnitude while the majority remain close to zero. This is not an artefact but a structural necessity - nonlinearities such as softmax concentrate probability mass on a few channels, ReLU and its variants zero out negative entries. Activation outliers of 100ร the median magnitude are common in large language models [9]. Random Hadamard rotation. Model quantization research [4,19, 30] has proposed techniques to rotate the activation and spread the activation energy of individual channels more uniformly across all internal model dimensions. We adopt this technique forฮ sec-linear-layer , and show this minimizes the resulting error in the floating point output domain, even if intermediary computation is emulated in fixed-point. Letํป โ R ํรํ denote the normalized WalshโHadamard matrix (ํป โค = ํผ) and letํท= diag(ํ)where eachํ ํ โ +1,โ1is drawn uniformly at random. Then, we denote the randomized rotation ํ = ํปํท. Both factors are orthogonal:ํท โค ํท= ํผbecause each diagonal entry squares to+1, andํป โค ํป= ํผby the normalization Qwen2.5-32BQwen2.5-72BLLaMA-3-70B Configurationmeanmed.maxmeanmed.maxmeanmed.max INT16-rot-ํ= 0.00.470.500.960.480.441.650.530.4746.4 INT16-rot-ํ= 0.50.660.661.480.770.652.420.850.7178.7 INT16-rot-ํ= 1.00.950.912.481.200.983.551.321.07122.5 INT16-rot-ํ= 1.51.291.223.551.681.364.971.851.49179.7 INT16-rot-ํ= 2.01.651.564.662.181.766.392.401.93232.4 Table 3: Error from a single outsourced matrix multiplication underฮ Sec-Linear , in relativeโ 2 errorโฅ e ํ โํโฅ 2 /โฅํโฅ 2 (%) of one outsourced instance ofํ=ํํ +ํevaluated on clean BF16 inputsํ. We report the mean, median, and maximum over all model linear projections (attention/MLP/LM head). above. Their product is therefore also orthogonal, ํ โค ํ = ํท โค ํป โค ํปํท= ํผ, which gives us two properties we rely on below. First, the matrix product is preserved exactly, ํํ=(ํํ โค )(ํ ํ)=ํ โฒ ํ โฒ (14) where ห ํ=ํํ โค and ห ํ= ํ ํ , so applying the secure matrix mul- tiplication protocol to ห ํand ห ํrecovers the same output as the original product. Second, the rotation preservesโ 2 norms ( โฅ ํ ํค โฅ 2 = โฅ ํค โฅ 2 ), so only theโ โ norm is changed by the rotation. The ro- tation can be applied inํ(ํ logํ)via the fast WalshโHadamard transform [11]. We illustrate in Appendix D how this enables the floating-point domain error variance induced by our protocol (Equation (7)) to be reduced by factor( ํค ํ 2 2 logํ)/( ํค ํ 2 โ ํ). In Section 5.2 and Section 5.3 we empirically investigate the error contribution by a single securely outsourced linear layer and the error accumulation over all secure linear layers in selected models respectively. 5.2 Error from a single secure linear layer We isolate the error introduced by a single outsourced matrix multiplication inฮ Sec-Linear , independent of how error propagates 9 Chiang et al. through the network. For each weight matrixํwe take the refer- ence activationํfrom a clean BF16 forward pass (no MOSAIC), compute the outsourced product e ํunderฮ Sec-Linear , and compare it to the exact productํ=ํํvia the relativeโ 2 errorโฅ e ํโํโฅ 2 /โฅํโฅ 2 . Because the input is the clean reference in every case, this measures per-multiplication error in isolation, with no accumulation across layers. Table 3 reports the mean, median, and maximum of this error across all weight projections of Qwen2.5-32B and -70B as well as LLaMA-370B class models at increasing protocol noiseํ. A single outsourced multiplication and additive bias is highly accurate: the typical (median) error is 1โ2% across all models, and for both Qwen models even the worst-case projection stays below 6.4% at even ํ=2.0. The large LLaMA-3 maxima are concentrated in the early- layer value projections (the layer-0 v_proj alone accounts for the reported maximum); excluding the v_proj family, the LLaMA-3 worst case is below 4% forํ โค1.0. At this noise level, these outliers do not propagate and end-to-end accuracy is ultimately preserved (Table 4); only atํ=2.0 does LLaMA-3 degrade. Still, LLaMA-3- 70B is known to be very sensitive to similar noise introduced by standard model weight quantization [22]. In Section 5.3, we illustrate that model accuracy is well-preserved, even when per-layer error contributions propagate and accumulate across the entire model. 5.3 Error accumulation across layers Formal end-to-end error accumulation analysis requires quantify- ing the Lipschitz constant of each transformer sub-layer, which determines how much a perturbation at one layer is amplified before reaching the next. Scaman and Virmaux [24] show that computing the exact Lipschitz constant of even a two-layer ReLU network is NP-hard and the situation for transformer layers is strictly harder. Thus, we rely on empirical study of error propagation in mod- els, and compare accumulated error induced by many layers of ฮ Sec-Linear-Layer with a baseline of commonly used model quantiza- tion techniques intended for efficiency rather than security. We show comparable model accuracy for protocol noise necessary for 140-bit protocol security, as parameterized in Example 4.1. Quantization baselines (4-bit NF4, INT8, INT16-rot-ํ). We com- pare our protocol against two standard post-training quantization schemes as reference points. 4-bit NF4 [10] uses a non-uniform 4-bit NormalFloat grid whose levels track the quantiles of a standard normal distribution, exploiting the approximately Gaussian distri- bution of transformer weights. LLM.int8() [9] represents weights and activations as per-channel-scaled 8-bit integers in[โ127,127], with matrix multiplications accumulated in higher precision, as well as weight and activation outliers. We refer to (1) INT4, (2) INT8 and (3) INT16-rot-ํas running model inference with 4-bit NF4, LLM.int8() and our protocol param- eterized with gaussian stdev parameterํthat induces a random hadamard rotation on activation and model weights. Note that for ํ=0, our protocol converges to the same precision as 16-bit fixed- point with the additional hadamard rotation treatment, with the latter strictly improving the precision of the computation. Thus, we use the INT16+ํ=0 baseline to refer to the optimistic precision obtainable by prior work (Section 2) that computes over integer rings or fields in cryptographic settings. Empirical error accumulation. We measure how per-layer er- ror propagates and accumulates through the full forward pass of Qwen2.5-32B with 64 model layers (Figure 18), and several 80- layer models, namely LLaMA-3-70B (Figure 20), Qwen2.5-72B (Fig- ure 19), and DeepSeek-R1-Distill-LLaMA-70B (Figure 21), with dif- ferent sensitivity to quantization error. We report well-formed, non- destructive error accumulation across the forward pass of models for our protocol (INT16+ํ) forํ<1.5, with final layer error com- parable or improving upon INT4 and INT8 quantization (without secure outsourcing). For each model, we run inference passes on 20 WikiText-2 con- texts (512 tokens each): a clean reference at BF16, and a perturbed pass in which weights and activations are INT16-quantized, rotated via a random Hadamard matrix (eq. (14)), and sub-Gaussian noise ํธ ํค ,ํธ ํฅ at standard deviation ํ is sampled and induced by our pro- tocol as described in Equation (7). After each transformer model layerโwe capture the residual streamํฅ โ , theํ model -dimensional hidden state defined by eq. (1) that is both the output of blockโ and the input to blockโ+1, from both clean and perturbed passes. We report, averaged over all token positions, the per-layer cosine similarity cos(โ)= โจํฅ bf16 โ ,ํฅ int16โrotโํ โ โฉ ํฅ bf16 โ 2 ํฅ int16โrotโํ โ 2 (15) and relativeโ 2 error ํฅ int16โrotโํ โ โ ํฅ bf16 โ 2 / ํฅ bf16 โ 2 . Each figure shows error accumulation for protocol noise std-deviationํ โ 0.5,0.7,1.0,1.5,2.0together with three post-training quantiza- tion baselines (NF4, INT8) run through the identical measurement pipeline without rotation or injected noise. At moderateํ โค1.0 the protocol curves track INT8 and substantially outperform NF4 throughout the forward pass, confirming that the per-layer protocol error accumulates non-destructively across depth. Figure 9 in the main body highlights differences in measured co- sine similarity between Qwen2.5-72B and Llama3-70B. We highlight error accumulation behaviors that differ between models below. Quantization-friendly models. The cosine similarity and relative โ 2 error propagation for chosen models Qwen2.5-72B and Qwen2.5- 32B are well-behaved, as illustrated in Figure 19 and Figure 18 respectively in the Appendix. All INT16-rot-ํruns show error propagation that match or is less than INT4 for allํ โค2.0, and improve on INT8 forํ โค1.0. The earlier layers in Qwen2.5-72B show a higher sensitivity to our protocol error, but this effect is non- destructive, as it recovers in both cosine-similarity andโ 2 errors in later model layers. We attribute this heuristically to both RMSNorm and channel-wise activation functions, that are known to contribute well-formed error accumulation. For the chosen protocol parameter ํ=0.5 (Example 4.1), this error accumulation is strictly better than INT4 or INT8 quantization schemes across all layers. Quantization unfriendly models. Llama-3-70B (and derivative DeepSeek-R1-Distill-LLaMA-70B) are known to be sensitive to quantization [22]. Their cosine similarity and relativeโ 2 error accu- mulation is shown in Figure 20 and Figure 21 respectively. Indeed, in our experiments, INT8 and INT16-rot-ํ=2.0 show destructive 10 MOSAIC: Masked Outsourcing of Secure AI Computations 0204060 0.96 0.98 1 Layer index (โ ) Cosine similarity Qwen2.5-72B 4-bit NF4 INT8 INT16-rot ํ= 0 INT16-rot ํ= 0.5 INT16-rot ํ= 0.7 INT16-rot ํ= 1.0 INT16-rot ํ= 1.5 INT16-rot ํ= 2.0 0204060 0.6 0.8 1 Layer index (โ ) Cosine similarity Llama-3-70B Figure 9: Per-layer error accumulation: Consine similarity between activations of quantized and reference forward-pass (BF16) for 70B-class models (left: Qwen2.5-72B; right: Llama-3-70B). Whilst the two models share similar architectures, Qwen2.5-70B is more resilient against error accumulation. Still, for protocol noise std-deviation rangeํ โ [0.5,1.0], our protocol can still be parametized for 140-bit security whilst resulting in well-formed error accumulation in both models. error propagation across layers for both Llama-3-70B and a deriva- tive model DeepSeek-R1-Distill-LLaMA-70B. The relativeโ 2 error is generally large for both INT4 and INT16-rot-ํforํ โฅ1.0. For Llama-3-70B (Figure 20), we propose to parameterize our protocol with gaussian samples with parameterํ=0.5 for 140-bit security (Example 4.1). For DeepSeek-R1-Distill-Llama-70B (Figure 21), the relativeโ 2 error at the final layer remains very high for all quantiza- tions, making the model a border-line or even suboptimal candidate for quantized computation. 5.4 Accuracy of outsourced inference Table 4 reports model accuracy of an end-to-end forward pass under our protocol against two reference quantization baselines (LLM.int8() and 4-bit NF4), on two 70B-class and one 32B model spanning the quantization spectrum: Qwen2.5-72B and Qwen2.5- 32B (quantization-friendly) and LLaMA-3-70B (quant.-unfriendly). Each model is evaluated on WikiText-2 perplexity (PPL, lower is better) and on HumanEval pass@1 (HE, higher is better, ํ= 1). For the quantization-friendly Qwen2.5-72B, all five INT16-rot-ํ configurations stay within 2% of the BF16 reference on PPL across the fullํ โ [0,2]range, and HE on Qwen2.5-72B tracks the BF16 baseline to withinยฑ4 points acrossํ โ [0,2]. For Qwen2.5-32B, a potential drop in HE scores may be observed forํ=0.5/1.0, that recovers at higher noise levels. For quantization-unfriendly LLaMA-3-70B, our protocol matches PPL scores of BF16 to within 3% forํ โค1.0 and HE drops gracefully from 54.6 (BF16) to 45.7 atํ=1.0; beyondํ=1.0 both metrics degrade visibly - PPL grows to 2.055 atํ=1.5 and 3.233 atํ= 2.0. Crucially, the noise parameter required for our 140-bit-secure parameterisation in Example 4.1 sits atํ=0.5; for every model in the table there exists a secure parameterization of our protocol which performs at or above NF4/INT8 baselines. The comparison with INT8 on LLaMA-3-70B is informative as a challenging reference model: INT8 fails catastrophically on this model (PPL=425.97, HE=10.06) - a sensitivity our protocol does not exhibit because the random Hadamard rotation (Equation (14)) Qwen2.5-72BLLaMA-3-70BQwen2.5-32B ConfigurationPPLHEPPLHEPPLHE BF16 (reference)2.20153.961.82154.572.72246.34 INT16-rot-ํ= 0.02.20154.571.82751.832.72346.04 INT16-rot-ํ= 0.52.20453.661.83453.962.72442.38 INT16-rot-ํ= 1.02.20550.911.86445.732.72643.29 INT16-rot-ํ= 1.52.22151.832.05535.982.73443.90 INT16-rot-ํ= 2.02.24453.353.233โ2.74247.56 INT8 (LLM.int8())2.21349.70425.97510.062.73648.48 NF4 (4-bit)2.30355.792.24745.432.89447.56 Table 4: Model accuracy under quantization baselines (INT8, NF4) and the protocolโs INT16-rot-ํnoise model at increas- ing noise levels. Two metrics per model: WikiText-2 perplex- ity (PPL, lower is better, sequence length 2048, stride 512) and HumanEval pass@1 (HE, higher is better, ํ= 1). flattens the same outlier distribution that defeats LLM.int8() quan- tization. Forํ=0.5 (140-bit security), our protocol retains the accuracy of BF16 on both PPL and HE benchmarks. 6 Confidential AI outsourcing Here, we show how a small trusted computing base (TCB) tasked with AI computations in a modern AI data-center can support larger AI computations that exceed its trusted resources by adding pow- erful, untrusted GPU accelerators. In this setting, we demonstrate that MOSAIC is not bottle-necked by communication in the modern AI data center setting where individual GPUโs are networked with fast interconnect technologies. In Appendix H we illustrate a remote application over public networks, which only requires an LLM classification, and does not require communication-intensive autoregressive decoding. Emulation approach. To explore these use-cases, we emulate a real-world implementation with current, limited hardware support. 11 Chiang et al. Ourฮ Sec-Approx-Matmul protocol is parameterised over a 32-bit inte- ger ring, which modern AI accelerators do not natively support. Concretely, on Nvidia GPUs the high-throughput Tensor cores that dominate LLM matmul performance accelerate FP16/BF16, FP8, and INT8 datatypes, but not 32-bit integer arithmetic. Native 32-bit integer is available only on the general-purpose CUDA pipeline, which delivers roughly an order of magnitude lower throughput than the Tensor cores and is therefore impractical for the matrix sizes involved in 70B-class models. To remain on the Tensor-core pipeline, we emulate each 32-bit in- teger multiplyโaccumulate as a sum of INT8 multiplyโaccumulates, splitting each 32-bit operand into four 8-bit limbs and reassembling the result in the trusted client. This emulation costs a constant over- head per ring multiplication, a slowdown relative to a hypothetical Tensor-core kernel with native INT32 support, but still markedly faster than falling back to the CUDA. A key detail is that the limbs are unsigned bytes ([0,255]) but the Tensor-core kernel accepts only signed bytes ([โ128,127]). We bridge this by subtracting 128 from each limb before the matmul, which slides every value into signed INT8 range without losing any information. After the kernel returns, we undo the bias algebraically: expanding(ํโ128)(ํโ128)= ํํโ128(ํ+ํ)+128 2 shows that the unsigned product ร ํ ํ ํ ํ ํ equals the signed product ร ํ (ํ ํ โ 128)(ํ ํ โ128)plus a correction term that depends only on the row and column sums inducingํ(ํ+ํ)to apply. The final reduction toZ 2 32 requires no extra work either, since native INT32 addition and bit-shift discard bits above position 31 automatically. The split between the two pipelines is also reflected in the proto- col split between trusted client and untrusted GPU. The untrusted GPU performs the denseํ(ํํํ)masked MatMul and runs strictly on the Tensor-core pipeline (via the INT8 emulation above). The trusted client, by contrast, performs only low-rank dense and sparse operations of inner rankํ ํ or row Hamming weightํก(Section 4.2); these more general workloads cannot benefit from Tensor-core acceleration, and thus client kernels are implemented for general- purpose CUDA pipeline at INT32 width. Datacenter Trusted GPU 1 Trusted GPU 2 Trusted GPU 3 ํ independent TCBs GPUGPU GPUGPU GPUGPU GPUGPU Untrusted GPU pool RDMA interconnects Figure 10: Datacenter outsourcing topology:ํindependent trusted clients, each a small TCB, elastically share a common pool of untrusted GPUs, secure outsourcing AI computations over fast interconnects. Secure outsourcing in an AI Datacenter. The common approach for confidential computation in datacenters is to deploy applica- tions on a trusted computing base (TCB), realized by confidential computing technologies offered by all major cloud providers today (SGX/TDX/NCC). Whilst confidential hardware architectures now BMLBMLBMLBMLBMLBML 0 200 400 600 800 Qwen2.5-72BLLaMA-3-70B 7k4k1k7k4k1k 584584584 574574 573 502502 501 491491 490 197 186 191 150 158 155 Latency (ms / token) Local GPU baseline (no protocol)Trusted GPU (mask/unmask + non-linear) Untrusted GPU (remote MatMul)Comm (trustedโ untrusted) Figure 11: Per-token decoding latency for 70B-class mod- els (ํบ=3 untrusted GPUs), across 1k/4k/7k token context lengths. Within each context: B = local-GPU baseline (no protocol), M = sharding of individual masked weight matrix, and L = sharding of masked model by layer across untrusted GPUs. BMLBMLBMLBMLBMLBML 0 10 20 30 40 Qwen2.5-72BLLaMA-3-70B 7k4k1k7k4k1k 38.7 25.8 8.6 36.9 24.2 7.9 24.3 16.0 5.7 22.3 14.8 5.1 3.9 2.7 1.3 3.5 2.4 1.1 Latency (seconds) Local GPU baseline (no protocol)Trusted GPU (mask/unmask + non-linear) Untrusted GPU (remote MatMul)Comm (trustedโ untrusted) Figure 12: System-prompt prefill runtime for 70B-class mod- els (ํบ=3 untrusted GPUs), across 1k/4k/7k system-prompt lengths, split into trusted-GPU walltime, untrusted-GPU remote MatMul, and trustedโuntrusted communication. Within each context: B = local-GPU baseline (no protocol), M = weight matrix shard, and L = model sharding by layer. extend to GPU [13] hardware, they are limited in scope, and do not offer the same scaling and performance benefits as modern datacenter architectures optimized for AI inference. Confidential computing is also more expensive due to specialized chips, dedi- cated hardware resources, attestation key management, enhanced physical security, and other factors. We emphasize that frontier AI inference needs more than a single GPU cluster. A typical inference task is distributed across phase (pre- fill/decode), model layer (sharding) and time (interleaving between accelerators) to maximize utilization [27,31]. Data-center accelera- tors are connected via remote direct memory access (RDMA) [26] technologies, featuring single-digit microsecond latency between 12 MOSAIC: Masked Outsourcing of Secure AI Computations GPUs. Such connectivity is enabled by fast intra-rack (e.g. NVLink, PCIe) and intra-data-center interconnects (e.g. Infiniband). Such distributed, AI-native architectures are not directly compat- ible with the currently available confidential computing technolo- gies. Offering AI inference at scale would imply extending the TCB to the heterogeneous pool of accelerators and their full networking stack, requiring complex attestation key management, enhanced physical security measures, and so on. Each user of secure AI com- putation would need their models and requests securely isolated from others, while cloud providers would ideally be free to assign workloads across the distributed data center to efficiently use avail- able resources. Secure outsourcing approaches like MOSAIC offer an alternative approach that combines (a) the benefits of modern (heterogeneous and distributed) AI inference architecture with (b) a small TCB for each user (Figure 10). Confidential (and potentially attested) compute represents the scarce resource, and the value of our protocol is that a trusted GPU can elastically capture addi- tional throughput from untrusted accelerators without expanding the trusted computing base. Implementation with fast GPU interconnects. We run our imple- mentation on 70B models on Nvidia H200 with 141 GB VRAM con- nected with fast NVLink interconnects, featuring 900 GB/s GPU- to-GPU transfer speeds. Whilst similar latency can be achieved with RDMA-style interconnects across the data center in prac- tice [26,27,31], on-demand GPUs offered by AI cloud providers accessible limit us to such inter-node interconnects. For the 70B models we deploy, we use 1 trusted GPU, that se- curely outsources computation to 3 untrusted GPUs, where pro- tocol communication is over the aforementioned NVLink GPU- to-GPU interconnect. We measure end-to-end prefill and decode for 70B-class models (LLaMA-3, Qwen-2.5) on a deployment with one trusted GPU acting as the trusted client and 3 untrusted GPUs collectively serving the outsourced MatMul calls via the secure forward pass (Figures 11 and 12). Protocol parameterization. We implement protocol parameters from Example 4.1 for 140-bit security, but omit the recursive LPN- mask optimization for masking activations. This maintains security, but saves memory for the client, as the layer-specific matricesํฝ ํ in step 4 of Init W (Figure 5) no longer occupies client memory; this slightly increases the cost of computing the activation mask ํ ํฅ = ํฝ ํ ํ in step 1b of Init X at (model-independent) rank of 1536 instead of 652. Exhaustive parameter optimization is left for future work as it is highly hardware dependent. Model sharding. We implement two parallelization techniques. (1) We shard each masked model matrix across the 3 GPUs, and broadcast each masked activation to all GPUs (model-matrix par- allel); the result is gathered by the trusted GPU thereafter. (2) We shard the masked model by contiguous layer ranges across the untrusted GPUs (model-layer parallel). Evaluation results. MOSAIC is roughly 3ร(decode) to 5โ11ร (prefill) slower than running inference for the same 70B models on a single local GPU of the same type, with the higher prefill factor corresponding to layer- rather than matrix-sharding. This overhead largely reflects our emulation of 32-bit integer arithmetic on 8-bit GPU cores (10 INT8 MatMuls per INT32 MatMul; see the emulation approach above); native 32-bit-integer support in AI accelerator cores (e.g. NVIDIA Tensor) would significantly reduce this gap. We highlight three observations in prefill (Figure 12) and decode (Figure 11) experiments on 70B class models. Firstly, communication between the trusted and untrusted pools is not the bottleneck under the available inter-GPU bandwidth: it accounts for onlyโผ4โ7% of decode andโผ7โ16% of prefill wall-clock, mirroring practicality of the per-layer interaction pattern as in real-world settings [26, 31]. Secondly, the wall-time differences between model-matrix shard- ing and model-layer sharding across untrusted GPUโs is more pro- nounced in prefill (Figure 12) than in autoregressive decoding (Fig- ure 11). This is expected as autoregressive decoding is memory- bounded; adding more parallel compute offers limited benefits. Third, autoregressive decoding (Figure 11) is essentially context- invariant atโผ0.5โ0.6 s per generated token across 1kโ7k context windows, because per-token cost is dominated by the dense linear projections (independent ofํ forํ=1); the KV dimensions only enters through attention, which are implemented over the native floating-point domain and negligible at this dimension. Of that per- token latency, the untrusted GPU accounts forโผ92โ93% (โผ85โ90% remote MatMul, the rest communication), leaving underโผ8% on the trusted client. The one-time init_W weight-masking cost (โ55 s per 70B model) is amortized across all subsequent queries over the frozen model weights, quickly becoming a negligible fraction of cumulative wall-clock. References [1]Nir Ailon and Bernard Chazelle. 2006. Approximate Nearest Neighbors and the Fast Johnson-Lindenstrauss Transform. In 38th Annual ACM Symposium on Theory of Computing (STOC). 557โ563. https://dl.acm.org/doi/pdf/10.1145/ 1132516.1132597. [2]Martin R Albrecht, Rachel Player, and Sam Scott. 2015. On the concrete hardness of learning with errors. Cryptology ePrint Archive (2015). https://github.com/ malb/lattice-estimator. [3]Michael Alekhnovich. 2003. More on average case vs approximation complex- ity. In 44th Annual IEEE Symposium on Foundations of Computer Science, 2003. Proceedings. IEEE, 298โ307. [4]Saleh Ashkboos, Amirkeivan Mohtashami, Maximilian L Croci, Bo Li, Pashmina Cameron, Martin Jaggi, Dan Alistarh, Torsten Hoefler, and James Hensman. 2024. Quarot: Outlier-free 4-bit inference in rotated llms. Advances in Neural Information Processing Systems 37 (2024), 100213โ100240. https://arxiv.org/pdf/ 2404.00456. [5]Fabrice Benhamouda, Caicai Chen, Shai Halevi, Yuval Ishai, Hugo Krawczyk, Tamer Mour, Tal Rabin, and Alon Rosen. 2025. Encrypted matrix-vector products from secret dual codes. In Proceedings of the 2025 ACM SIGSAC Conference on Computer and Communications Security. 394โ408. . [6]Mark Braverman and Stephen Newman. 2025. Practical Secure Delegated Linear Algebra with Trapdoored Matrices. In Theory of Cryptography Conference (TCC). Springer, 97โ118. https://arxiv.org/abs/2502.13060. [7]Hao Chen, Miran Kim, Ilya Razenshteyn, Dragos Rotaru, Yongsoo Song, and Sameer Wagh. 2020. Maliciously secure matrix multiplication with applications to private deep learning. In International Conference on the Theory and Application of Cryptology and Information Security. Springer, 31โ59. https://eprint.iacr.org/ 2020/451.pdf . [8]Jung Hee Cheon, Andrey Kim, Miran Kim, and Yongsoo Song. 2017. Homo- morphic encryption for arithmetic of approximate numbers. In International conference on the theory and application of cryptology and information security. Springer, 409โ437. https://eprint.iacr.org/2016/421.pdf . [9] Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. 2022. LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale. In Advances in Neural Information Processing Systems (NeurIPS), Vol. 35. 30318โ30332. https: //arxiv.org/abs/2208.07339. [10] Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. 2023. QLoRA: Efficient Finetuning of Quantized LLMs. In Advances in Neural Informa- tion Processing Systems (NeurIPS). https://arxiv.org/abs/2305.14314. [11]Fino and Algazi. 1976. Unified matrix treatment of the fast Walsh-Hadamard transform. IEEE Trans. Comput. 100, 11 (1976), 1142โ1146. https://ieeexplore.ieee. org/stamp/stamp.jsp?tp=&arnumber=1674569. 13 Chiang et al. [12]Yang Gao, Gang Quan, Soamar Homsi, Wujie Wen, and Liqiang Wang. 2024. Secure and efficient general matrix multiplication on cloud using homomorphic encryption: Y. Gao et al. The Journal of Supercomputing 80, 18 (2024), 26394โ 26434. [13]Zhongshu Gu, Enriquillo Valdez, Salman Ahmed, Julian James Stephen, Michael Le, Hani Jamjoom, Shixuan Zhao, and Zhiqiang Lin. 2025. NVIDIA GPU confidential computing demystified. arXiv preprint arXiv:2507.02770 (2025). https://arxiv.org/pdf/2507.02770. [14] Kanav Gupta, Neha Jawalkar, Ananta Mukherjee, Nishanth Chandran, Divya Gupta, Ashish Panwar, and Rahul Sharma. 2024. SIGMA: Secure GPT Inference with Function Secret Sharing. In 24th Privacy Enhancing Technologies Symposium (PETS). https://eprint.iacr.org/2023/1269. [15] Xiaoqian Jiang, Miran Kim, Kristin Lauter, and Yongsoo Song. 2018. Secure out- sourced matrix computation and application to neural networks. In Proceedings of the 2018 ACM SIGSAC conference on computer and communications security. 1209โ1222. https://eprint.iacr.org/2018/1041.pdf . [16]Chiraag Juvekar, Vinod Vaikuntanathan, and Anantha Chandrakasan. 2018. GAZELLE: A low latency framework for secure neural network inference. In 27th USENIX security symposium (USENIX security 18). 1651โ1669. https://w. usenix.org/system/files/conference/usenixsecurity18/sec18-juvekar.pdf . [17]Qinfeng Li, Zhiqiang Shen, Zhenghan Qin, Yangfan Xie, Xuhong Zhang, Tianyu Du, Sheng Cheng, Xun Wang, and Jianwei Yin. 2024. Translinkguard: safeguard- ing transformer models against model stealing in edge deployment. In Proceedings of the 32nd ACM international conference on multimedia. 3479โ3488. [18]Zhengyi Li, Kang Yang, Jin Tan, Wen-jie Lu, Haoqi Wu, Xiao Wang, Yu Yu, Derun Zhao, Yancheng Zheng, Minyi Guo, et al.2024. Nimbus: Secure and efficient two-party inference for transformers. Advances in Neural Information Processing Systems 37 (2024), 21572โ21600. [19]Zechun Liu, Changsheng Zhao, Igor Fedorov, Bilge Soran, Dhruv Choudhary, Raghuraman Krishnamoorthi, Vikas Chandra, Yuandong Tian, and Tijmen Blankevoort. 2024. Spinquant: Llm quantization with learned rotations. arXiv preprint arXiv:2405.16406 (2024). https://arxiv.org/pdf/2405.16406. [20]Wen-jie Lu, Zhicong Huang, Zhen Gu, Jingyu Li, Jian Liu, Cheng Hong, Kui Ren, Tao Wei, and WenGuang Chen. 2023. Bumblebee: Secure two-party inference framework for large transformers. Cryptology ePrint Archive (2023).https: //eprint.iacr.org/2023/1678.pdf . [21]Zoltรกn รdรกm Mann, Christian Weinert, Daphnee Chabal, and Joppe W Bos. 2023. Towards practical secure neural network inference: the journey so far and the road ahead. Comput. Surveys 56, 5 (2023), 1โ37. https://eprint.iacr.org/2022/1483.pdf . [22]Minghai Qin. 2024. The uniqueness of llama3-70b series with per-channel quan- tization. arXiv preprint arXiv:2408.15301 (2024). https://arxiv.org/pdf/2408.15301. [23]Oded Regev. 2009. On lattices, learning with errors, random linear codes, and cryptography. Journal of the ACM (JACM) 56, 6 (2009), 1โ40. [24]Kevin Scaman and Aladin Virmaux. 2018. Lipschitz regularity of deep neural networks: analysis and efficient estimation. In Advances in Neural Information Processing Systems (NeurIPS), Vol. 31. https://arxiv.org/pdf/1805.10965. [25]Tianxiang Shen, Ji Qi, Jianyu Jiang, Xian Wang, Siyuan Wen, Xusheng Chen, Shixiong Zhao, Sen Wang, Li Chen, Xiapu Luo, et al.2022.SOTER: Guarding black-box inference for general neural networks at the edge. In 2022 USENIX Annual Technical Conference (USENIX ATC 22). 723โ738. https://w.usenix. org/system/files/atc22-shen.pdf . [26] Spheron. 2026. GPU Networking for AI Clusters: InfiniBand vs RoCE vs Spectrum- X Decision Guide. Spheron Network Blog. https://w.spheron.network/blog/ gpu-networking-infiniband-roce-spectrum-x-guide. [27] Yijun Sun, Xudong Liao, Songrun Xie, Hao Chen, Han Tian, Wenxue Li, Yiming Zhang, and Kai Chen. 2026. Multi-stage Flow Scheduling for LLM Serving. arXiv preprint arXiv:2603.17456 (2026). https://arxiv.org/pdf/2603.17456. [28]Florian Tramรจr and Dan Boneh. 2019. Slalom: Fast, Verifiable and Private Exe- cution of Neural Networks in Trusted Hardware. In International Conference on Learning Representations (ICLR). https://arxiv.org/abs/1806.03287. [29]Pengli Wang, Bingyou Dong, Yifeng Cai, Zheng Zhang, Junlin Liu, Huanran Xue, Ye Wu, Yao Zhang, and Ziqi Zhang. 2025. Game of Arrows: On the (In- )Security of Weight Obfuscation for On-Device TEE-Shielded LLM Partition Algorithms. In 34th USENIX Security Symposium (USENIX Security 25). 279โ298. https://w.usenix.org/system/files/usenixsecurity25-wang-pengli.pdf . [30]Songhao Wu, Ang Lv, Xiao Feng, Yufei Zhang, Xun Zhang, Guojun Yin, Wei Lin, and Rui Yan. 2025. Polarquant: Leveraging polar transformation for efficient key cache quantization and decoding acceleration. arXiv preprint arXiv:2502.00527 (2025). https://arxiv.org/pdf/2502.00527. [31]Yongtong Wu, Shaoyuan Chen, Yinmin Zhong, Rilin Huang, Yixuan Tan, Wentao Zhang, Liyue Zhang, Shangyan Zhou, Yuxuan Liu, Shunfeng Zhou, et al.2026. DualPath: Breaking the Storage Bandwidth Bottleneck in Agentic LLM Inference. arXiv preprint arXiv:2602.21548 (2026). https://arxiv.org/pdf/2602.21548. [32]Rongwu Xu and Zhixuan Fang. 2024. Tempo: Confidentiality preservation in cloud-based neural network training. In 2024 International Joint Conference on Neural Networks (IJCNN). IEEE, 1โ10. [33]Yu Yu, Kang Yang, Xiao Wang, Anyu Wang, Tianrui Wang, Hanlin Liu, Xinpeng Hao, and Juanru Li. 2025. Estimator of LPN problems over any finite fields and power-of-two rings for PCG and MPC applications. https://lpnestimator.com. https://lpnestimator.com/. [34]Itamar Zimerman, Allon Adir, Ehud Aharoni, Matan Avitan, Moran Baruch, Nir Drucker, Jenny Lerner, Ramy Masalha, Reut Meiri, and Omri Soceanu. 2024. Power-Softmax: Towards Secure LLM Inference over Encrypted Data. arXiv preprint arXiv:2410.09457 (2024). https://arxiv.org/abs/2410.09457. A Proofs A.1 Supporting lemmas The two supporting lemmas referenced by Theorem 1 are stated and proved below. Corollary 1 collects the comparison between their tail bounds that we invoke in the proof of Theorem 1. Lemma 2. Letํธ โ Z ํรํ 2 ํ have independent entries drawn from N Z 2 ํ ; 0,ํ 2 , and letํ โ Z ํรํ 2 ํ be fixed. Then entry(ํ,ํ)ofํธํ satisfies: (1) E (ํธํ) ํํ = 0. (2) (ํธํ) ํํ is ํ โฅ ํฅ ํ โฅ 2 -sub-Gaussian: for all ํก> 0, Pr |(ํธํ) ํํ |> ํก โค 2exp โํก 2 /(2ํ 2 โฅ ํฅ ํ โฅ 2 2 ) where ํฅ ํ is the ํ-th column of ํ. Proof. Fix(ํ,ํ)and write(ํธํ) ํํ = ร ํ ํ=1 ํ ํ (ํฅ ํ ) ํ , whereํ ํ := (ํธ) ํํ are independent samples fromN Z 2 ํ ; 0,ํ 2 and(ํฅ ํ ) ํ are fixed. (1) By symmetry ofN Z 2 ํ ; 0,ํ 2 ,E [ ํ ํ ] =0 for eachํ. By lin- earity of expectation, E (ํธํ) ํํ = ร ํ (ํฅ ํ ) ํ E [ ํ ํ ] = 0. (2) A random variableํisํผ-sub-Gaussian ifE [ exp ( ํกํ )] โค exp ํผ 2 ํก 2 /2 for allํก โ R. We first show that each discrete gaussian sampleํ ํ โผ N Z 2 ํ ; 0,ํ 2 isํ-sub-Gaussian for anyํ>0. Let ํ ํ (ํ)= ร ํโํ exp โํ 2 /(2ํ 2 ) for any subsetํ โ Z 2 ํ . The MGF of ํ ํ is E [ exp ( ํกํ ํ )] = โ๏ธ ํ exp ( ํกํ ) ยท Pr[ํ ํ = ํ] = 1 ํ ํ (Z 2 ํ ) โ๏ธ ํโZ 2 ํ exp ํกํโํ 2 /(2ํ 2 ) (16) Let us denoteํ ํ (Z 2 ํ โํ)= ร ํโZ 2 ํ exp โ(ํโํ) 2 /(2ํ 2 ) . Further, manipulating the term in the exponent of eq. (16) to obtainํกํ โ ํ 2 /(2ํ 2 )=โ(ํโํกํ 2 ) 2 /(2ํ 2 )+ ํ 2 ํก 2 /2, gives E [ exp ( ํกํ ํ )] = exp ํ 2 ํก 2 /2 ยท ํ ํ (Z 2 ํ โํกํ 2 ) ํ ํ (Z 2 ํ ) By the Poisson summation formula, ํ ํ (Z 2 ํ โํ)= ํ โ 2ํ โ๏ธ ํโZ 2 ํ exp โ2ํ 2 ํ 2 ํ 2 cos(2ํํํ) Sincecos(2ํํํ) โค1 for allํ,ํand each coefficientexp โ2ํ 2 ํ 2 ํ 2 is non-negative, ํ ํ (Z 2 ํ โํ) โค ํ ํ (Z 2 ํ ). Thus, 1โค ํ ํ (Z 2 ํ โํกํ 2 ) ํ ํ (Z 2 ํ ) andE [ exp ( ํกํ ํ )] โค exp ํ 2 ํก 2 /2 , confirming thatํ ํ isํ-sub-Gaussian. 14 MOSAIC: Masked Outsourcing of Secure AI Computations For independentํ-sub-Gaussian variables,(ํธํ) ํํ = ร ํ (ํฅ ํ ) ํ ํ ํ is( ร ํ (ํฅ ํ ) 2 ํ ํ 2 ) 1/2 = ํ โฅ ํฅ ํ โฅ 2 -sub-Gaussian, since E " exp ํก โ๏ธ ํ (ํฅ ํ ) ํ ํ ํ !# = ร ํ E [ exp ( ํก(ํฅ ํ ) ํ ํ ํ )] โค ร ํ exp ํ 2 (ํฅ ํ ) 2 ํ ํก 2 /2 = exp ํ 2 โฅ ํฅ ํ โฅ 2 2 ํก 2 /2 . The tail bound follows from the standard Chernoff method: for any ํก>0 andํ>0,Pr[(ํธํ) ํํ > ํก] โค E exp ํ(ํธํ) ํํ /exp ( ํํก ) โค exp ํ 2 โฅ ํฅ ํ โฅ 2 2 ํ 2 /2โ ํํก . Minimizing over ํ yields Pr |(ํธํ) ํํ |> ํก โค 2exp โ ํก 2 2ํ 2 โฅ ํฅ ํ โฅ 2 2 โก Lemma 3. Letํธ ํ โ Z ํรํ 2 ํ andํธ ํ โ Z ํรํ 2 ํ have mutually indepen- dent entries drawn fromN Z 2 ํ ; 0,ํ 2 . Then entry(ํ,ํ)ofํธ ํ ํธ ํ satisfies: (1) E (ํธ ํ ํธ ํ ) ํํ = 0. (2) For all ํก> 0, Pr |(ํธ ํ ํธ ํ ) ํํ |> ํก โค 2exp โํก 2 /(4ํํ 4 ) +ํ โฮฉ(ํ) Proof. Fix indicesํ,ํand write(ํธ ํ ํธ ํ ) ํํ = ร ํ ํ=1 ํ ํ ํ ํ , where ํ ํ = (ํธ ํ ) ํํ andํ ํ = (ํธ ํ ) ํํ are mutually independent samples fromN Z 2 ํ ; 0,ํ 2 . (1) By independence andE [ ํ ํ ] = E [ ํ ํ ] =0,E [ ร ํ ํ ํ ํ ํ ] = ร ํ E [ ํ ํ ] E [ ํ ํ ] = 0. (2) Condition onํ=(ํ 1 , . . .,ํ ํ ). Givenํ, the sum ร ํ ํ ํ ํ ํ is a linear combination of independent ํ -sub-Gaussian variables with fixed coefficients. By lemma 2, the conditional tail satisfies Pr " โ๏ธ ํ ํ ํ ํ ํ > ํก ํ # โค 2exp โ ํก 2 2ํ 2 โฅ ํ โฅ 2 2 Since eachํ ํ isํ-sub-Gaussian,ํ 2 ํ is sub-exponential withE ํ 2 ํ = ํ 2 ํ โค ํ 2 . Standard sub-exponential concentration (Bernstein in- equality) yieldsPr[ โฅ ํ โฅ 2 2 >2ํํ 2 ] โค ํ โฮฉ(ํ) . On the high-probability event โฅ ํ โฅ 2 2 โค 2ํํ 2 , the conditional bound becomes 2exp โํก 2 /(4ํํ 4 ) after substitution, and the total probability law gives the result.โก Corollary 1. The cross termํธ ํ ํธ ํ is dominated by the linear error termsํธ ํ ํandํํธ ํ . Comparing the tail bounds from Lemma 2 and Lemma 3: the linear terms decay asexp โํก 2 /(2ํ 2 โฅ ํฅ โฅ 2 2 ) , while the cross term decays asexp โํก 2 /(4ํํ 4 ) . With โฅ ํฅ โฅ 2 2 = ํ ํฅ 2 (average squared entryํฅ 2 ), the linear exponent scales asํก 2 /(ํํ 2 ํฅ 2 )versus ํก 2 /(ํํ 4 )for the cross term. The cross term is therefore negligible wheneverํ 2 โชํฅ 2 , a condition required by the protocol for useful signal-to-noise ratio. A.2 Proof of Theorem 1 Theorem 1. Letํ โ Z ํรํ 2 ํ ,ํ โ Z ํรํ 2 ํ , and letํธ ํ โ Z ํรํ 2 ํ , ํธ ํ โ Z ํรํ 2 ํ have independent entries drawn fromN Z 2 ํ ; 0,ํ 2 . Then entry(ํ,ํ)of the total errorํธ ํ ํ +ํํธ ํ +ํธ ํ ํธ ํ is approximately ํ โ๏ธ โฅ ํฅ ํ โฅ 2 2 + ํค ํ 2 2 -sub-Gaussian: for all ํก> 0, Pr |(ํธ ํ ํ +ํํธ ํ + ํธ ํ ํธ ํ ) ํํ |> ํก โค 2exp โ ํก 2 2ํ 2 ( โฅ ํฅ ํ โฅ 2 2 + ํค ํ 2 2 ) ! up toํ โฮฉ(ํ) , whereํค ํ is theํ-th row ofํandํฅ ํ is theํ-th column of ํ. Proof.The linear terms(ํธ ํ ํ) ํํ and(ํํธ ํ ) ํํ are indepen- dent, since they depend on the independent noise matricesํธ ํ and ํธ ํ respectively. By lemma 2,(ํธ ํ ํ) ํํ isํ โฅ ํฅ ํ โฅ 2 -sub-Gaussian and (ํํธ ํ ) ํํ isํ ํค ํ 2 -sub-Gaussian. For independent sub-Gaussian variables, the sub-Gaussian parameters add in quadrature, so their sum isํ โ๏ธ โฅ ํฅ ํ โฅ 2 2 + ํค ํ 2 2 -sub-Gaussian. By corollary 1, the cross termํธ ํ ํธ ํ is negligible wheneverํ 2 โชํฅ 2 , contributing only an additive ํ โฮฉ(ํ) to the tail probability.โก A.3 Proof of Lemma 1 Lemma 1. The LWE mask is computationally indistinguishable from uniform: ํฟํ+ ํธ ํค ํ โ ํ. Proof sketch.We showํฟํ + ํธ ํค ํ โ ํcolumn by column. Interpret the ํ โth column of ํฟํ+ ํธ ํค as ํฟํ ํ +ํ ํค,ํ โ F ํ ํ and let ํป 0 = ํฟํ+ ํธ ํค =(ํฟํ 1 +ํ ํค,1 , . . .,ํฟํ ํ +ํ ํค,ํ ). Forํ โฅ ํ>0, defineํป ํ =(ํข 1 , . . .,ํข ํ ,ํฟํ ํ+1 +ํ ํค,ํ+1 , . . .,ํฟํ ํ +ํ ํค,ํ ), whereํข 1 , . . .,ํข ํ are independent uniform samples overF ํ ํ . The two consecutive hybrids โข ํป ํ , and โข ํป ํ+1 , which replaces the(ํ+1)-th entryํฟํ ํ+1 +ํ ํค,ํ+1 with a fresh uniform sampleํข ํ+1 , differ only in that single coordinate; distinguishing them is exactly a decisional-LWE distinguisher for the public matrixํฟand noise ํ ํค,ํ+1 . By the hybrid argument,ํป 0 ํ โ ํป ํ s โ (ํข 1 , . . .,ํข ํ ) s โ ํ, with total advantage bounded byํtimes the decisional-LWE advantage. Hence ํฟํ+ ํธ ํค ํ โ ํ .โก A.4 Nested LWE+LPN mask Lemma 4. The nested LWE+LPN mask is computationally indis- tinguishable from uniform: (((ํฟ ํ ํ ํ + ํ ํ )ํ ํโ1 + ํ ํโ1 )ยทํ 2 )ํ 1 + ํธ ํค ํ โ ํ. Proof sketch. Letํน ํ denote the nested mask of the statement andํน 1 = ํฟ 1 ํ 1 + ํธ ํค the simplified LWE mask. We showํน ํ ํ โ ํ by a hybrid argument that unwinds the nested LPN instances one level at a time until we reach ํน 1 , then invokes Lemma 1. Defineํhybridsํน ํ ,ํน ํโ1 , . . .,ํน 1 , whereํน ํ is obtained fromํน ํ+1 by replacing the innermost LPN instanceํฟ ํ+1 ํ ํ+1 + ํ ํ+1 with a freshly sampled uniform matrixํฟ ํ of the same dimensions. Here ํฟ ํ+1 โU Z ํ ํ รํ ํ+1 2 ํ ,ํ ํ+1 โU Z ํ ํ+1 รํ ํโ1 2 ํ andํ ํ+1 has row Ham- ming weight ํก ํ+1 (with ํ 0 := ํ). 15 Chiang et al. Fix a transitionํน ํ+1 โ ํน ํ . Distinguishing these two distributions reduces to distinguishing, at the position where ํฟ ํ appears, โข ํฟ ํ+1 ํ ํ+1 + ํ ํ+1 , a decisional LPN sample with secret dimen- sionํ ํ+1 , sample countํ ํโ1 , and noise Hamming weightํก ํ+1 , from โข uniform ํฟ ํ โU Z ํ ํโ1 รํ ํ 2 ํ . The reduction receives the LPN challengeํฟ โ , embeds it at the ํth position of the mask, samples all remainingํฟ ํ ,ํ ํ ,ํ ํ andํธ ํค honestly, and forwards the constructed mask to the distinguisher. Its advantage transfers without loss, so ํน ํ+1 ํ โ ํน ํ . Chaining theํโ1 LPN hybrid steps givesํน ํ ํ โ ํน 1 with distin- guishing advantage at most(ํโ1)times the best decisional-LPN advantage. By Lemma 1, ํน 1 ํ โ ํ . Composing yields ํน ํ ํ โ ํ . โก B Computational integrity We refer to a well-known randomized check (Freivaldsโ algorithm) to ensure correctness ofํํ=ํcomputed by the GPU at runtime, at the cost of additional input-independent precomputation. For verification, the client samples a random vector(ํ)and performs the following assertion; ํ ํ ํํ= ? ํ ํ ํ Observe thatํ ํ ํis input independent and can be preprocessed and reused for each inference run; at runtime, the assertion above can be evaluated in ํ((ํ+ํ)ํ). To violate soundness, the cheating prover must returnํsuch that the following holds ํ ํ ํํ= ํ ํ ํ โโ ํ ํ (ํํ โํ)= 0 whereํํ โํโ 0. This occurs with probability 1/|F|(Schwartz- Zippel Lemma). C Theoretical outsourcing efficiency For our proposed protocol, we show the ratio of (1) trusted client protocol overhead to (2) the cost of the client computingํํlocally in Figure 13. We consider multiplicative operations for (1) and (2). D Protocol error after rotation Reduction of the infinity norm. For any fixed vectorํฅ โ R ํ , each entry ofํ ํฅis a normalized sum ofํrandom-sign terms. By sub-Gaussian concentration [1], with high probability, โฅ ํ ํฅ โฅ โ โฒ โฅ ํฅ โฅ 2 ยท โ๏ธ logํ ํ (17) In contrast, without rotation โฅ ํฅ โฅ โ can be as large as โฅ ํฅ โฅ 2 when a single entry dominates. Therefore the quantization scale after rotation satisfies ํ โฒ ํฅ = โฅ ํ ํฅ โฅ โ ํ max โฒ โฅ ํฅ โฅ 2 ํ max ยท โ๏ธ logํ ํ (18) Float-domain noise variance. Consider the dominant error term ํธ ํค ํfrom eq. (7) in the floating-point domain. Entry(ํ,ํ)ofํธ ํค ห ํ in the integer domain is the inner product ร ํ (ํธ ํค ) ํํ ห ํฅ ํํ , where each (ํธ ํค ) ํํ is an independent Gaussian with varianceํ 2 . By lemma 2, the variance of this entry isํ 2 โฅ ห ํฅ ํ โฅ 2 2 , where ห ํฅ ํ is the quantized 50100150200 10 โ2 10 โ1 10 0 Rank of Model Weight Matrix ํ (ร10 3 ) Trusted C Work: outsourced / local ํํ Theoretical efficiency of outsourcing (ํ= 1) ํ=ํ/4 ํ=ํ ํ= 4ํ Figure 13: Theoretical trusted-client work vs. localํํwork (ํ=1; X as a vector) with protocol parameterization in Exam- ple 4.1. activation column. Dequantization rescales the integer result by both scales ํ ํค ํ ยท ํ ํฅ ํ , giving a full (floating-point) variance of Var(ํธ ํค ํ) ํ,ํ = ํ 2 ยท โฅ ห ํฅ ํ โฅ 2 2 ยท ํ 2 ํค ํ ยท ํ 2 ํฅ ํ (19) Note that ห ํฅ ํ = round(ํฅ ํ /ํ ํฅ ํ ) , so โฅ ห ํฅ ํ โฅ 2 2 โ โฅ ํฅ ํ โฅ 2 2 /ํ 2 ํฅ ํ . Substitut- ing into eq. (19), Var(ํธ ํค ํ) ํ,ํ โ ํ 2 ยท โฅ ํฅ ํ โฅ 2 2 ยท ํ 2 ํค ํ . Substitutingํ ํค ํ = ํค ํ โ /ํ max , the dequantized variance simpli- fies to Var(ํธ ํค ํ) ํ,ํ = ํ 2 ยท โฅ ํฅ ํ โฅ 2 2 ยท ํค ํ 2 โ ํ 2 max (20) With Hadamard rotation, substituting eq. (17) into eq. (20), this becomes Var(ํธ ํค ํ ํ) ํ,ํ โฒ ํ 2 ยท โฅ ํฅ ํ โฅ 2 2 ยท ํค ํ 2 2 ํ 2 max ยท logํ ํ (21) By symmetry, the noise fromํํธ ํฅ is reduced analogously via the activation rotation. The ratio of eq. (21) to eq. (20) gives the variance reduction factor for the ํธ ํค ํ term: Var(ํธ ํ ํ ํ) ํ,ํ Var(ํธ ํ ํ) ํ,ํ โ ํค ํ 2 2 ยท logํ ํค ํ 2 โ ยท ํ = ํ(ํค ํ ) 2 ยท log(ํ) ํ (22) and analogously forํํธ ํฅ . This ratio equalslog(ํ)/ํin the worst case of a single dominant outlier ( โฅ ํค โฅ โ โ โฅ ํค โฅ 2 ). For a transformer dimensionํ=4096, this yields an approximateํ/logํ โ340ร reduction in noise variance. The cross termํธ ํค ํธ ํฅ from eq. (7) is unaffected by the Hadamard rotation:ํธ ํ andํธ ํ are sampled in the integer domain indepen- dently of bothํandํ, so their product distribution is invariant 16 MOSAIC: Masked Outsourcing of Secure AI Computations Asymptotic cost Component / RegimeClient CGPU G Per-layer breakdown 7 weight projections (ํ,ํพ,ํ,ํ , gate, up, down)โํ(ํ 2 ํ) Per-MatMul outsourcing overhead (ร 7)ํ(ํํ)โ Fast WalshโHadamard rotation (per call)ํ(ํํ logํ)โ Per-head attention (softmax,ํํพ โค ,ยทํ )ํ(ํ ํ ํ)โ Element-wise non-linearities (RMSNorm, etc.)ํ(ํํ)โ Per-layer total (ํ โฅ logํ )O(d s l)O(d 2 l) Full forward pass over ํฟ layers General (ํ new tokens, seq. lengthํ )ํ(ํฟํ ํ ํ) ํ(ํฟํ 2 ํ) Prefill (ํ=ํ )ํ(ํฟํ ํ 2 ) ํ(ํฟํ 2 ํ ) Decode (ํ= 1)ํ(ํฟํ ํ ) ํ(ํฟํ 2 ) Init W (one-time, amortised)ํ(ํฟํ 2 logํ)โ Table 5: Asymptotic per-layer and full-forward-pass com- plexity ofฮ Sec-FwdPass . Notation:ํ= ํ model ,ํฟtransformer layers, total sequence length ํ , ํ new tokens per query. to the rotation. By corollary 1, this term remains negligible relative to the linear error terms both before and after rotation. E Secure forward-pass complexity Table 5 summarises the per-layer and full-forward-pass complexity for anํฟ-layer transformer with model dimensionํ model , interme- diate (MLP) dimensionํ f โ3.5ํ model , total sequence lengthํ , and ํnew tokens per query. Per-layer GPU work is dominated by the seven weight projections atํ(ํ 2 model ํ), while per-layer trusted- client work is dominated by per-head attention atํ(ํ model ยท ํ ยทํ) for any context ํ โฅ logํ model . Theํฟ-layer forward pass and the prefill (ํ= ํ ) and autoregressive decode (ํ=1) cases follow directly. A one-time per-weight Init W cost ofํ(ํ 2 model logํ model )totalsํ(ํฟํ 2 model logํ model )across the model and is amortised across every subsequent forward pass. F Broken obfuscation security We show a formal break in the security reduction of ArrowCloak [29] (USENIXโ25). The scheme protects the privacy of a fine-tuned model ํ vic derived from a pre-trained modelํ pre , where the pre-trained model is explicitly public and static in the adversaryโs view. The obfuscated weight isํ obf = ํ vic ํปfor a secret โencryptionโ key matrixํป. The authors claim a reduction to search-LWE hardness and, as part of that reduction, introduce the following hybrid step (reproducing eq. 12, section 6.2 of [29]): H 0 : ํ=ํ obf ํป โ1 +(ํ pre โํ vic ) H 1 : ํ=ํ obf ํป โ1 + ํธ, whereํธis a fresh discrete-Gaussian LWE sample. Computational indistinguishability ofH 0 andH 1 is the essential step in their reduction; the authors support it with empirical evidence thatํ pre โ ํ vic is Gaussian-like. Trivial distinguisher. However, the two hybrids are trivially dis- tinguishable becauseํ obf andํ pre are part of the adversaryโs public view per the ArrowCloak threat model. Define the deterministic distinguisherD(ํ) :=[ํ= ? ํ pre ]. Then: NVLink 0.001ms/400Gb Datacenter 0.05ms/100Gb Regional 1ms/100Gb WAN 10ms/100Gb WAN (far) 20ms/10Gb Internet 20ms/1Gb 10 0 10 1 10 2 10 3 Network scenario Latency (seconds) Qwen2.5-72B latency: sys-prompt setup (1k/4k/7k tokens) vs. 35-token runtime query Sys 1k tokens Sys 4k tokens Sys 7k tokens Rt (per query, 35 tokens) Figure 14: Network sensitivity: system-prompt prefill at 1k / 4k / 7k tokens (dashed) and per-query runtime prefill (solid) at 35-token across various simulated network scenar- ios. While the (one-time) 7k-token sys prefill approaches 45 minutes on a 1 Gbps Internet link, the per-query runtime stays close to 20 s. Table 6: Communication volume per phase (1k-token system prompt). init W is a one-time model setup cost; sys-prefill runs once and is amortized over subsequent runtime prompts. rt-prefill is a query of 35 tokens. โSendโ isCtoG; โRecvโ is G toC. Llama-3-70BQwen2.5-72B SendRecvSendRecv init_W (one-time)278.0 GBโ285.9 GBโ sys_prefill (one-time)17.1 GB31.6 GB17.4 GB32.2 GB rt_prefill (per query)0.60 GB1.11 GB0.61 GB1.13 GB โขInH 0 ,ํ=ํ obf ํป โ1 +(ํ pre โํ vic )=ํ vic +ํ pre โํ vic =ํ pre deterministically, so Pr[D(ํ)= 1 | H 0 ]= 1. โขInH 1 ,ํ=ํ vic +ํธequalsํ pre only when the fresh Gaussian ํธlands on the single fixed pointํ pre โํ vic , which happens with probability negligible in ํ. ThusDachieves distinguishing advantage 1โ negl(ํ), contradict- ing the indistinguishability stepH 0 ํ โ H 1 their reduction relies on. G Supplementary deployment figures We complement the main body figures with per-query prefill laten- cies across different network environments in the remote decisional setting (Figure 14) for 1/4/7k context sizes. We show communication volume for a 1k prefill required for our proposed protocol (Table 6) applied to 70B models. H Outsourcing remote classifications We also consider a case-study of an application that is not latency critical, and thus permits interactive outsourcing of computation across public networks, between a local trusted client and a remote 17 Chiang et al. init_W Sys 1kSys 4kSys 7k Rt (Sys 1/4/7k) 0 20 40 60 80 100 4.7 66 45 17 87 Latency (seconds) Trusted GPUUntrusted GPU Comm (10 ms / 100 Gbps) Figure 15: System prefill and runtime latency under simu- lated WAN (10 ms RTT, 100 Gbps,ํบ=3 untrusted GPUs) for Qwen2.5-72B. init_W Sys 1kSys 4kSys 7k Rt (Sys 1/4/7k) 0 20 40 60 80 100 4.7 63 43 16 85 Latency (seconds) Trusted GPUUntrusted GPU Comm (10 ms / 100 Gbps) Figure 16: System prefill and runtime latency under simu- lated WAN (10 ms RTT, 100 Gbps,ํบ=3 untrusted GPUs) for LLaMA-3-70B. cloud accelerator (Figure 17). In addition to emulating the ring integer arithmetic (as mentioned at the beginning of this section) to measure client and remote GPU computation runtimes, we simulate network latency induced by our protocol for different network settings (Figures 15 and 16). TCB Local site Trusted clientCloud GPU Public network e.g. 10 ms RTT, 100 Gbps 1 forward pass / query Field report โผ 35 tok Decision (severity, action) Cloud Figure 17: Remote outsourcing topology: a trusted client at the local site outsources a single forward pass per query to an untrusted cloud GPU over a public network (e.g. 10 ms RTT, 100 Gbps). Industrial diagnostics. Consider an industrial predictive mainte- nance scenario in which a company operates a fleet of equipmentโ pumps, compressors, motors, gearboxesโand wishes to outsource diagnostic reasoning to a large language model hosted on an un- trusted cloud accelerator. The system prompt encodes the full diag- nostic knowledge base: equipment-specific failure mode signatures, vibration and temperature thresholds, oil-analysis limits, multi- symptom escalation rules, and historical case studies. This domain context is lengthy (1kโ7k tokens in our benchmarks) but changes infrequentlyโat most when maintenance procedures are revised. The runtime prompt, by contrast, is a short field report (โผ35 tokens) written by a technician on the factory floor: a handful of sensor readings and a one-line observation such as โCoupling very hot, faint burnt smell from motor end.โ The modelโs task is to return a bounded decision, e.g. a severity classification and recommended action, not an open-ended generation. We choose this scenario precisely because it is not latency- critical: a maintenance classification that completes in seconds is well within operational requirements, unlike interactive chat or real-time control. Importantly, the task requires only a single for- ward pass at runtime over a bounded number of tokens, avoiding the compounding communication cost of autoregressive decode rounds. The expensive yet stable system-prompt setup is performed once and amortized over many lightweight runtime queries. Evaluation results. Figures 15 and 16 show that latency for deci- sional inference can remain practical. Despiteโ80ร4 communica- tion rounds, a single decisional inference query can be completed in under 5 seconds over a fast public network (10ms, 100Gbps). One can observe greatly reduced transfer volumes for the short runtime prompt compared to the system prompt, shown in Table 6. Overall run-time is clearly affected by communication latency; still for the short, real-time diagnostic prompt, the system prompt length does not have a measurable effect. The system prompt prefill of 7k to- kens can be completed in about a minute, implying that frequent background, prefill updates are permissible. Simulated runtimes for a wider range of realistic network settings are detailed in Figure 14. I Layer-by-layer error accumulation We supplement the main body with error accumulation metrics on final model layers in Table 7 and error accumulation (cosine relative โ 2 error) across layers for Qwen2.5-32B (Figure 18), Qwen2.5-72B (Figure 19), LLaMA3-70B (Figure 20), DeepSeek-R1-Distill-LLaMA- 70B (Figure 21). 18 MOSAIC: Masked Outsourcing of Secure AI Computations Noisy vs. refBF16 reference Stepcosrel-โ 2 max|ํฅ| ํ ํฅ |ํฅ| max|ํฅ|/ํ ํฅ Quantization-friendly models Qwen2.5-32B, protocol noise ํ= 1.0 โ= 620.99910.0391548023.411.98236 โ= 630.99840.0532434024.113.95177 post-RMSNorm0.99870.04481732.861.3761 Qwen2.5-72B, protocol noise ํ= 1.0 โ= 780.99600.0724160214.27.75116 โ= 790.99410.0938285015.78.90183 post-RMSNorm0.99490.08123353.61.6793 Quantization-unfriendly models LLaMA-3-70B, protocol noise ํ= 1.0 โ= 780.97950.17742330.660.46356 โ= 790.98130.1771860.880.6095 post-RMSNorm0.98220.1654942.031.4246 DeepSeek-R1-Distill-LLaMA-70B, protocol noise ํ= 1.0 โ= 780.82890.54644000.700.45572 โ= 790.83870.5399600.770.5578 post-RMSNorm0.84660.5098872.001.4644 Table 7: Reference (BF16, noise-free) residual-stream am- plitude and matched perturbed-pass metrics at the last two transformer model layers and after the final RMSNorm. 19 Chiang et al. 0204060 0.96 0.97 0.98 0.99 1 Layer index (โ ) Cosine similarity 4-bit NF4 INT8 INT16-rot ํ= 0 INT16-rot ํ= 0.5 INT16-rot ํ= 0.7 INT16-rot ํ= 1.0 INT16-rot ํ= 1.5 INT16-rot ํ= 2.0 0204060 0 0.1 0.2 0.3 Layer index (โ ) Relative โ 2 error Figure 18: Per-layer error accumulation: Qwen2.5-32B (64 model layers). For 20 recorded runs, the 1-standard deviation band is shown for selected quantization schemes. 0204060 0.96 0.97 0.98 0.99 1 Layer index (โ ) Cosine similarity 4-bit NF4 INT8 INT16-rot ํ= 0 INT16-rot ํ= 0.5 INT16-rot ํ= 0.7 INT16-rot ํ= 1.0 INT16-rot ํ= 1.5 INT16-rot ํ= 2.0 0204060 0 5ยท 10 โ2 0.1 0.15 0.2 0.25 Layer index (โ ) Relative โ 2 error Figure 19: Per-layer error accumulation: Qwen2.5-72B (80 model layers). 20 MOSAIC: Masked Outsourcing of Secure AI Computations 0204060 0.5 0.6 0.7 0.8 0.9 1 Layer index (โ ) Cosine similarity 4-bit NF4 INT8 INT16-rot ํ= 0 INT16-rot ํ= 0.5 INT16-rot ํ= 0.7 INT16-rot ํ= 1.0 INT16-rot ํ= 1.5 INT16-rot ํ= 2.0 0204060 0 0.5 1 1.5 Layer index (โ ) Relative โ 2 error Figure 20: Per-layer error accumulation: LLaMA-3-70B (80 layers). INT8 baseline collapses at โ= 79 and goes off-scale. 0204060 0.4 0.6 0.8 1 Layer index (โ ) Cosine similarity 4-bit NF4 INT8 INT16-rot ํ= 0 INT16-rot ํ= 0.5 INT16-rot ํ= 0.7 INT16-rot ํ= 1.0 INT16-rot ํ= 1.5 INT16-rot ํ= 2.0 0204060 0 0.5 1 1.5 2 Layer index (โ ) Relative โ 2 error Figure 21: Per-layer error accumulation: DeepSeek-R1-Distill-LLaMA-70B (80 layers). 21