Paper deep dive
Modernizing Amdahl's Law: How AI Scaling Laws Shape Computer Architecture
Chien-Ping Lu
Intelligence
Status: succeeded | Model: anthropic/claude-sonnet-4.6 | Prompt: intel-v1 | Confidence: 93%
Last extracted: 3/24/2026, 3:54:03 AM
Summary
This paper by Chien-Ping Lu reformulates Amdahl's Law for modern heterogeneous computing systems driven by AI scaling laws. The author introduces a resource allocation model replacing the classical serial/parallel split with a 'scalable fraction' S and efficiency ratio R for specialized vs. programmable hardware. The key result is a finite collapse threshold S_c = 1 - 1/R: beyond this critical scalable fraction, optimal investment in specialized hardware falls to zero—a phase transition. The framework explains why GPUs keep becoming more programmable and why domain-specific AI accelerators have not displaced GPUs.
Entities (27)
Relation Signals (23)
Resource Allocation Model → extends → Amdahl's Law
confidence 98% · We reformulate Amdahl's Law for modern heterogeneous systems with scalable workloads
Chien-Ping Lu → authored → Resource Allocation Model
confidence 97% · We reformulate Amdahl's Law for modern heterogeneous systems with scalable workloads... We model total execution time as T(x)
Scalable Fraction S → determines → Collapse Threshold
confidence 97% · S_c = 1 - 1/R... beyond a critical scalable fraction, specialization becomes suboptimal
Efficiency Ratio R → determines → Collapse Threshold
confidence 97% · S_c = 1 - 1/R... the minimum efficiency ratio required to justify nonzero specialization is R_c = 1/(1-S)
Gustafson's Law → extends → Amdahl's Law
confidence 97% · Gustafson's Law relaxes this by allowing the parallel portion of the workload to scale independently with the number of compute units
Resource Allocation Model → introduces → Scalable Fraction S
confidence 97% · We replace the classical parallel fraction P with the scalable fraction S
Collapse Threshold → implies → Phase Transition
confidence 96% · a phase transition rather than an asymptotic tail... The transition occurs at a finite threshold S_c = 1 - 1/R
Resource Allocation Model → →
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Classical Amdahl's Law assumes a fixed decomposition between serial and parallel work and homogeneous replication; historically, it bounds how much parallel speedup is attainable. Modern systems instead combine specialized accelerators with programmable compute, tensor datapaths, and evolving pipelines, while empirical scaling laws shift which stages absorb marginal compute. The central tension is therefore not the serial-versus-parallel split alone, but resource allocation across heterogeneous hardware, given efficiency differences, and workload structures that determine how effectively additional compute can be converted into value. We reformulate Amdahl's Law for modern heterogeneous systems with scalable workloads. The analysis yields a finite collapse threshold: beyond a critical scalable fraction, specialization becomes suboptimal for any efficiency advantage of specialized hardware over programmable compute, and optimal specialized investment falls to zero, a phase transition rather than an asymptotic tail. We use this framework to interpret increasing GPU programmability and why domain-specific AI accelerators have not displaced GPUs.
Tags
Links
- Source: https://arxiv.org/abs/2603.20654v1
- Canonical: https://arxiv.org/abs/2603.20654v1
Trouble viewing inline? Open PDF directly →
Full Text
27,783 characters extracted from source content.
Expand or collapse full text
Modernizing Amdahl’s Law How AI Scaling Laws Shape Computer Architecture Chien-Ping Lu Abstract Classical Amdahl’s Law assumes a fixed decomposition between serial and parallel work and homogeneous replication; historically, it bounds how much parallel speedup is attainable. Modern systems instead combine specialized accelerators with programmable compute, tensor datap- aths, and evolving pipelines, while empirical scaling laws shift which stages absorb marginal compute. The central tension is therefore not the serial-versus-parallel split alone, but resource allocation across heterogeneous hardware, given efficiency differences, and workload structures that determine how effectively additional compute can be converted into value. We reformulate Amdahl’s Law for modern heterogeneous systems with scalable workloads. The analysis yields a finite collapse threshold: beyond a critical scalable fraction, specialization becomes suboptimal for any efficiency advantage of specialized hardware over programmable compute, and optimal specialized investment falls to zero—a phase transition rather than an asymptotic tail. We use this framework to interpret increasing GPU programmability and why domain-specific AI accelerators have not displaced GPUs. 1 Introduction Amdahl’s Law [2] has long served as a foundational model for reasoning about performance scaling: Speedup = 1 (1− P) + P N (1) wherePdenotes the fraction of work assumed to be parallelizable andNdenotes the number of processors. This formulation assumes: • homogeneous compute units, • replication-based scaling, and • a fixed decomposition between serial and parallel components. These assumptions no longer reflect modern systems. Modern platforms combine dedicated units, programmable compute, tensor engines, mixed precision arithmetic, and hierarchical memory systems within a single device. Hennessy and Patterson described the present period as a “new golden age for computer architecture” [6], emphasizing domain-specific efficiency optimization. In the AI era, however, the central architectural driver is not core count or core efficiency alone, but the way scaling laws reshape workload structure and, through it, resource allocation. Today’s architectures are heterogeneous, dominated by programmable compute, augmented by specialized units, and driven by workloads whose structure evolves with scale. As a result, 1 arXiv:2603.20654v1 [cs.DC] 21 Mar 2026 “speedup” and “core count” lose their physical meaning. Even the term “core” has become an unstable abstraction spanning wide SIMD lanes, tensor datapaths, and fixed-function assist blocks. We therefore return to a more fundamental quantity: Total normalized execution time under constrained resources. 2 Limitations of Classical Formulations Amdahl’s Law assumes a fixed workload decomposition, which in normalized time form can be written as T A (N) = (1− P) + P N (2) with speedup given by 1/T A (N). The key point is not the notation, but the assumption that the decomposition parameter remains fixed as the system scales. Gustafson’s Law [5] relaxes this by allowing the parallel portion of the workload to scale independently with the number of compute units: Speedup G = (1− P) + P · N(3) The motivation is that the parallel portion of the workload is allowed to grow with system size, effectively scaling with processor countN, while the non-scaling portion is treated as bounded in practice. Passing to normalized time, T G (N) = 1 Speedup G = 1 (1− P) + PN .(4) This can be written exactly in Amdahl form, T G (N) = (1− P G (N)) + P G (N) N ,(5) with an effective decomposition ratio P G (N) = NP (1− P) + PN (6) In that exact sense, Gustafson is Amdahl with a workload ratio that varies with scale. The distinction is therefore not between two different physical laws, but between fixed and scale-dependent decomposition. Figure 1 situates both laws in their historical setting. Both are expressed in terms of speedup versus processor count, which is precisely the language this paper moves beyond. Thus, the key limitation is not parallelism, but: The assumption of a fixed relationship between workload components. Historically, however, workload scaling was still largely a modeling assumption rather than an empirically established property of real systems. Not every problem admits arbitrarily scalable instances, and classical formulations do not explicitly represent communication, synchronization, or memory hierarchy. Modern workloads violate this assumption. The more useful distinction for current systems is between effectively bounded workload, for which additional compute no longer produces meaningful value, and workload whose scalability is justified not merely by physical possibility, but by continued marginal value, as evidenced by AI scaling laws. 2 204060 0 20 40 60 N (processor count) speedup Speedup Form Amdahl, P = 0.5 Amdahl, P = 0.9 Amdahl, P = 0.99 Gustafson, P = 0.5 Gustafson, P = 0.9 Gustafson, P = 0.99 204060 0 0.2 0.4 0.6 0.8 1 N (processor count) normalized time Time Form Figure 1: Historical legacy of classical scaling laws. Amdahl’s law (solid) and Gustafson’s law (dashed) shown side by side in speedup form and normalized-time form. The left panel reproduces their classical presentation in terms of speedup versus processor countN. The right panel shows the corresponding time-domain view. Together they illustrate the historical legacy of speedup-centric analysis while highlighting the variable choices that motivate the present reformulation. 3 A Resource Allocation Model The present reformulation is motivated by a change in physical description. Classical models are written in terms of serial versus parallel work, processor countN, and speedup under replication. That language was appropriate in an era when parallel computing still had to be justified and the central question was how much speedup could be obtained despite the serial fraction. In current systems, parallelism is largely taken for granted. What we observe instead is a recurring architectural phenomenon: the tension between specialized accelerators and more programmable compute fabrics, often manifested in practice as NPU-like specialization versus GPU-like programmability. This motivates the variable changes used in the new formula: •We drop the serial-versus-parallel framing because it no longer captures the dominant distinction in modern workloads. • We do not use processor or core countNbecause modern performance is not determined by a meaningful count of homogeneous cores. • We introducexto represent the fraction of constrained hardware resource allocated to specialized logic. This makes the model about allocation rather than replication. • We introduceRto represent the efficiency advantage of specialized hardware relative to programmable compute. This makes the efficiency trade-off explicit, rather than treating programmability as inherently noncompetitive. •We replace the classical parallel fractionPwith the scalable fractionS. The point is not whether work is parallel in the old sense, but whether its growth continues to create value, for 3 example through scaling laws. Although the model is written with a single allocation variablex, it is not restricted to one chip. The constrained budget represented byxmay be interpreted at the level of a chip, a computing node, a cluster, or a larger AI infrastructure, provided that it denotes the fraction of the relevant constrained resource allocated to specialized versus programmable computation. We model total execution time as T(x) = 1− S 1 + (R− 1)x + S 1− x (7) wherexdenotes the constrained budget allocated to specialized hardware,Rthe efficiency ratio of specialized relative to programmable compute, andSthe dynamically scaling fraction of execution time. This formulation does not depend on core count, processor count, or an explicit parallel-versus- serial split. It directly captures heterogeneous resource allocation through two physically meaningful quantities: the efficiency advantageRand the scalable workload ratioS. Figure 3 visualizes this execution-time surface for representative values of S at fixed R = 10. InterpretingS as a Value-Scalable Fraction In the present model,Sshould not be read merely as “the work that can run in parallel.” A more useful interpretation is thatSmeasures the fraction of computation for which additional compute still produces meaningful utility gain. Informally, one may think of value-scalable stages as those satisfying d Value d Compute > ε,(8) for some minimal utility threshold ε. Under this interpretation, scalability is governed by marginal value rather than by parallelizability alone. A stage belongs to the bounded portion (1− S) if additional compute mainly increases throughput without materially improving fidelity, capability, or output quality. This boundedness is practical rather than absolute: scaling may remain physically possible, but it no longer adds enough value to justify continued investment. A stage belongs to the scalable portionSif more compute continues to produce meaningful gains. This also means thatSis not imposed by hardware alone. It is partly a human and system-design choice about where additional compute continues to create value. In modern AI systems, however, that choice is not arbitrary: empirically observed scaling laws show that larger models, richer post-training, and more inference-time compute often continue to produce measurable gains. In that sense, S is partly a design factor, but one grounded in observed value scaling. This distinction is especially useful in rendering systems. Classical rasterization is highly parallel in implementation, but it is not necessarily value-scalable once screen-space resolution and visibility have been fixed: beyond that point, additional compute does not substantially increase scene information [1]. By contrast, ray tracing and path tracing without learned reconstruction exhibit classical Monte Carlo improvement, with error decreasing roughly as 1/ √ Nas more samples are taken. In that regime, additional compute continues to buy photometric fidelity, so the workload is value-scalable. Neural denoising and reconstruction modify this picture. Once learned reconstruction can recover high perceptual quality from relatively low sample counts, the classical high-sample regime is compressed: beyond a modest acquisition threshold, additional rays may add little utility, while neural inference becomes the stage in which marginal compute still produces visible value [9]. In 4 that sense, modern reconstruction systems do not simply accelerate rendering; they reallocate which stages belong toSand which stages become effectively bounded. Figure 2 summarizes this transition with a noisy low-sample Monte Carlo renderer and its denoised counterpart, illustrating how learned reconstruction can shift useful computation away from brute-force sampling. Monte Carlo, 16 spp (noisy)Denoised / reconstructed Figure 2: Example rendered images illustrating how neural denoising and reconstruction shift graphics workload structure. Low-sample Monte Carlo rendering provides a noisy acquisition signal, while learned denoising recovers useful image quality from that input; as reconstruction quality improves, brute-force Monte Carlo rendering becomes effectively value-bounded and a larger share of the scalable workload shifts into learned post-processing. The example shown is the Crytek Sponza scene at 16 samples per pixel from the Intel Open Image Denoise gallery. 4 Collapse Threshold The optimal allocation satisfiesdT/dx= 0, and a collapse occurs when the optimum reachesx= 0. Evaluating the derivative at x = 0: dT dx x=0 =−(1− S)(R− 1) + S(9) Setting this to zero yields the critical threshold: S c = 1− 1 R (10) Equivalently, for a fixed scalable fractionS, the minimum efficiency ratio required to justify nonzero specialization is R c = 1 1− S (11) This expression is useful because it gives an immediate quantitative interpretation. IfS= 0.9, then specialization is justified only if the dedicated hardware is at least 10×as efficient as generic programmable compute. IfS= 0.95, the required advantage rises to 20×. As the scalable fraction grows, the efficiency bar for dedicated hardware rises rapidly. Interpretation When S ≥ S c , the optimal allocation becomes x ∗ = 0. 5 Beyond this threshold, specialization is no longer beneficial. It is important to emphasize thatx ∗ = 0 is not an asymptotic result. The transition occurs at a finite threshold,S c = 1−1/R. Beyond this point, specialization is no longer part of the optimal design for any efficiency advantage of specialized hardware over programmable compute. This is a regime change rather than a gradual degradation. Equivalently, once the threshold is crossed, any allocation to specialized hardware increases total execution time, makingx ∗ = 0 the globally optimal solution. Notably, this transition depends only on the workload ratioSand the efficiency ratioR; it is independent of absolute performance or system scale. Figure 3 illustrates the transition. For low values ofSthe execution-time curve exhibits a clear minimum atx ∗ >0; asSincreases the minimum shifts toward the origin and eventually disappears, producing a monotonically increasing curve. This is the architectural gravity of scalable workloads: asSgrows, the optimum is pulled away from rigid specialization and toward programmable compute. 0 5· 10 −2 0.10.150.20.250.30.350.40.450.50.550.60.65 0.5 1 1.5 2 2.5 3 x ∗ = 0 x (specialization fraction) T ( x ) (normalized execution time) S = 0.2 S = 0.5 S = 0.8 S = 0.9 S = 0.95 optimal locus x ∗ (S) Figure 3: Normalized execution timeT(x) versus specialization fractionxforR= 10 and varyingS. Dashed markers indicate the optimal allocationx ∗ . For lowS, the curves are U-shaped and specialization is beneficial; asSapproachesS c = 0.9, the optimum collapses toward the origin. The dashed black curve traces the optimal locus, terminating at the collapse pointx ∗ = 0. Above the threshold (S= 0.95), the curve is monotonically increasing and no investment in dedicated hardware is optimal. Bandwidth-Limited Extension The baseline model treats the specialization advantageRas a constant. That is appropriate for the first-order allocation law, but one obvious objection is that memory bandwidth can prevent specialized compute from realizing its full theoretical efficiency. In Roofline-style terms [8], increasing compute allocation without a corresponding increase in data supply eventually makes performance bandwidth-limited. One simple way to capture this effect is to replace the constant efficiency ratio by an effective 6 24681012141618202224262830 0 0.2 0.4 0.6 0.8 1 Collapse: x ∗ = 0 Specialization: x ∗ > 0 R (efficiency ratio) S (dynamic fraction) S c = 1− 1/R Figure 4: Phase boundary in (R, S) space. Above the curveS c = 1−1/R, the optimal specialization fraction collapses to zero: no finite efficiency advantage justifies dedicating silicon to fixed-function hardware. Below the curve, a nonzero allocation to specialized hardware reduces total execution time. This phase boundary is the allocation diagram of modern compute architecture. ratio that decays with increasing specialized allocation: R eff (x) = R max 1 + γR max x ,(12) whereR max is the peak theoretical efficiency ratio andγis a memory-friction coefficient summarizing the workload’s arithmetic intensity relative to available memory bandwidth. The execution-time surface then becomes T mem (x) = 1− S 1 + (R eff (x)− 1)x + S 1− x .(13) This extension has two useful implications. First, memory friction pulls the right-hand side of the surface upward, so the interior optimumx ∗ moves closer to zero even before the collapse threshold is reached. Bandwidth limitations therefore strengthen the pressure toward programmable hardware. Second, for the friction model in Equations 12–13, the collapse threshold itself is unchanged. Evaluating the derivative at the origin gives dT mem dx x=0 =−(1− S)(R max − 1) + S,(14) which yields the same boundary condition S c = 1− 1 R max .(15) For this bandwidth-friction extension, memory hierarchy effects deform the interior of the allocation surface without altering the first-order phase boundary. The finite collapse remains governed by workload structure and peak specialization advantage, while bandwidth limitations flatten the approach to that threshold. 7 5 Architectural Interpretation This model implies a structural shift in architecture design: asSincreases, the contribution of early-stage, fixed-function computation declines and the dominant portion of execution time shifts toward dynamically scaling computation. This leads to: x ∗ → 0(16) Specialized hardware transitions from essential to non-optimal. The interpretation is not that bottlenecks disappear, but that bounded stages occupy a shrinking fraction of total execution time. Once the scalable portion dominates, the architecture is forced toward more general and more reconfigurable compute fabrics. 6 Evidence from Modern Systems In modern AI systems, workload scaling is no longer merely hypothetical. It has become an empirically grounded structural property of dominant workloads. AI scaling laws provide empirical evidence that additional effective compute can continue to improve model quality in a structured way. Pre-training scaling laws show that increasing model size, dataset size, and training compute yields systematic improvements in capability and loss behavior [7], making scaling a design principle for large-scale AI systems rather than a speculative assumption. Post-training exhibits related scaling behavior as well: in reinforcement learning, preference optimization, and reasoning-oriented fine-tuning, additional compute applied after pre-training can continue to improve performance in a structured way. Test-time computation has also become a genuine scaling axis: longer reasoning chains, multiple candidate generation, verification stages, and richer search procedures can improve output quality at inference time, making inference compute expandable rather than fixed. Workload growth is therefore no longer simply an analytical convenience. As computational capacity increases, it is frequently absorbed into larger models, richer training procedures, and more compute-intensive inference, rather than solely reducing execution time. In the present model, the parameterScaptures the fraction of computation associated with these dynamically scaling workload components. Across application domains, the growing influence of AI and learned models is driving shifts such as: •Reconstruction replacing direct computation: systems infer a full result from partial, noisy, or cheaply sampled inputs (e.g. denoising, super-resolution, neural codecs, and inpainting), with close analogues in non-visual signal recovery and imaging, not only in rendering pipelines. •Late-stage post-processing dominating execution time: the expensive work migrates to stages after a cheaper front end—for example neural passes after rasterization in graphics, large rerankers after retrieval, long or speculative decoding in language models, or heavy fusion layers after lightweight encoders. •Model-driven output generation: learned models substantially produce or shape the delivered artifact, as in large language or code assistants, speech synthesis, and machine translation, not only in display-centric workloads. These shifts limit what can be achieved by local optimization of a classical fixed pipeline within a single vertical: as the scalable ratioSrises, more wall-clock time and value-producing work shift into stages that scale with model capacity and inference compute, so end-to-end scaling depends less on domain-specific front-end tuning and more on the shared scalable path. 8 6.1 Why GPUs Keep Becoming More Programmable Given the empirical trends summarized in Section 6, the model predicts a gradual erosion of fixed-function pipelines and an expansion of programmable domains as the scalable fractionSrises. Graphics is especially illustrative. The shift of visual quality generation toward later-stage compute predates AI. A major milestone was deferred shading, which made front-end visibility and rasterization stages serve increasingly as data-preparation steps for richer downstream computation rather than as the sole locus of image quality generation [1]. AI-generated pixels, frame reconstruc- tion, denoising, and neural upscaling intensify that trend by further reducing brute-force sampling and shifting value-producing work into learned stages whose cost and capability scale with model complexity and inference compute [9]. In the language of the present model, graphics increasingly inherits scaling-law behavior through these AI-driven stages, while front-end visibility and rasteriza- tion act more like bounded acquisition stages. These trends increaseS, pushing workloads toward the collapse region of Figure 4. This is precisely the long-run trajectory of GPUs: • fixed-function graphics pipelines gave way to shaders—for example, hardware transform-and- lighting moved into programmable vertex shaders, and fixed texture combiners and per-pixel lighting moved into programmable pixel (fragment) shaders, • shaders evolved into unified programmable compute, • graphics hardware absorbed tensor acceleration, and •tensor acceleration is increasingly presented as programmable matrix machinery rather than rigid frontends. As rendering quality depends more on neural reconstruction than on fixed multi-pass logic, graph- ics processors are pulled toward more programmable, matrix-oriented substrates. Programmability is not itself equivalent to AI scaling; rather, it is the architectural response to workloads whose value-producing stages keep moving and whose scalable work increasingly lands on shared tensor- and compute-intensive machinery. Figure 5 shows this shift at the level of rendering passes: early visibility stages become bounded acquisition passes, while later stages absorb more of the scalable, compute-intensive workload. The same logic motivates the next question: if scalable work keeps migrating, why do AI domain-specific accelerators not simply displace the GPU? 6.2 Why AI Domain-Specific Accelerators Have Not Displaced the GPU AI domain-specific accelerators win whenSis small, the workload is stable, and computational boundaries remain fixed. In that regime, the bounded portion of the workload is large enough for specialization to matter. However, when workload structure evolves, their relative contribution diminishes and their functionality is absorbed into programmable systems. This is precisely the regime above the boundary in Figure 4, where specialization collapses to x ∗ = 0. This explains why AI domain-specific accelerators have not simply displaced GPUs. The issue is not that application-specific accelerators or other dedicated designs are intrinsically weak; it is that, once the workload mix described in Section 6 shifts toward larger values ofS, the efficiency advantage of fixed hardware is overwhelmed by the fact that scalable work dominates the runtime budget. The key point is value rather than physical possibility. It is often possible in principle to scale hardware around a fixed mechanism, but its value declines if software and model evolution move the 9 Graphics pipeline under rising S Pass 1 Primary visibility bounded / low-res Pass 2 Secondary visibility bounded / low-sample Passes 3+ anti-aliasing, denoising, reconstruction, frame synthesis, post-processing becoming dominant (S ↑) Bounded acquisition stagesScalable, compute- and tensor-intensive stages DLSS-5-style limit: Later passes collapse toward one learned pass Figure 5: Shift of graphics workload structure under risingS. Neural denoising and reconstruction compress the classical high-sample rendering regime: once useful image quality can be recovered from low-resolution or low-sample acquisition, brute-force Monte Carlo rendering becomes effectively value-bounded, primary and secondary visibility increasingly behave as bounded acquisition stages, and passes 3+ absorb a larger share of the scalable workload through anti-aliasing, denoising, reconstruction, frame synthesis, and related post-processing. In the extreme limit, many of these later passes collapse into a single learned reconstruction stage. frontier of useful computation elsewhere. In AI, sparsity, routing, cache compression, quantization, and inference-time system optimization can reduce cost per token without requiring a corresponding fixed-function redesign [3,4]. In that regime, software optimization can outrun narrow hardware advantage, and the contribution of any one dedicated mechanism becomes effectively bounded relative to the evolving programmable workload. More broadly, this is a statement about AI domain-specific architectures: specialization wins only while the targeted mechanism remains a large and stable share of value-producing computation. Efficiency advantage alone is insufficient when workload composition shifts. 7 Conclusion We presented a reformulation of Amdahl’s Law in the framing of modern heterogeneous systems rather than in that of late-1960s parallel computing. We showed that: • modern workloads require dynamic decomposition, • specialization has a collapse threshold, and • architectural convergence is a predictable outcome. This model applies across domains—graphics, AI, simulation, and inference—and replaces parallel-versus-serial framing, core count, and fixed bottleneck assumptions with resource allocation, hardware heterogeneity, and workload structure as the primary variables governing performance. If current scaling trends persist, investment in fixed-function silicon should continue to shrink, formerly dedicated logic should be absorbed into broader compute domains, and specialization should survive mainly as lightweight assist layers rather than dominant frontends. The disappearance of specialization is not gradual; it is triggered by a finite threshold in workload structure. 10 References [1]Tomas Akenine-Möller, Eric Haines, Naty Hoffman, Angelo Pesce, Michał Iwanicki, and Sébastien Hillaire. Real-Time Rendering. A K Peters/CRC Press, 4 edition, 2018. [2] Gene M. Amdahl. Validity of the single processor approach to achieving large scale computing capabilities. In Proceedings of the April 18–20, 1967, Spring Joint Computer Conference, AFIPS ’67 (Spring), pages 483–485. ACM, 1967. [3]DeepSeek-AI. Deepseek-v2: A strong, economical, and efficient mixture-of-experts language model. arXiv preprint arXiv:2405.04434, 2024. [4] William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. Journal of Machine Learning Research, 23(120):1–39, 2022. [5] John L. Gustafson. Reevaluating Amdahl’s law. Communications of the ACM, 31(5):532–533, 1988. [6]John L. Hennessy and David A. Patterson. A new golden age for computer architecture. Communications of the ACM, 62(2):48–60, 2019. [7] Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361, 2020. [8]Samuel Williams, Andrew Waterman, and David Patterson. Roofline: An insightful visual performance model for multicore architectures. Communications of the ACM, 52(4):65–76, 2009. [9]Lei Xiao, Salah Nouri, Matt Chapman, Alexander Fix, Douglas Lanman, and Anton Kaplanyan. Neural supersampling for real-time rendering. ACM Transactions on Graphics, 39(4):142:1–142:12, 2020. 11