SemiAnalysis

Amazon’s Trainium Takes a Strike at the Training King

2026-10-23T11:06:44+00:00

Trainium has already proven itself in the realm of inference. Anthropic is running production inference and RL rollouts on Trainium2 and Trainium3. Historically, Trainium has been great for inference but not the best fit for research training. In inference, the model architecture changes at a much slower pace compared to training, where researchers iterate on many different experiments and ablations of model architecture and need a stack that covers a broad API surface. In the past, that is where Trainium fell short.

The AWS Neuron team is aggressively working to change this pitfall with the biggest shift being moving towards a first-class, native PyTorch training experience. Gone are the days of sprinkling xm.step() through your training loop and wrestling with a bespoke XLA-style execution model. Trainium now runs PyTorch the way PyTorch users expect, in eager mode, with torch.compile and the standard APIs.

The SemiAnalysis engineering team has been trying out the Trainium3 racks for the past couple of weeks and has found the experience to be great – we love the overall direction of where it is heading. To be clear, it isn’t perfect yet, and we have personally run into a couple of bugs ourselves. It’s nowhere close to the maturity of the CUDA ecosystem, but the direction is right. Their team is making rapid progress with the help of agentic coding.

Furthermore, unlike other accelerator companies, where we had to fight tooth and nail to get their engineers a reasonably stable CI fleet, Trainium uses up to 80,000 physical cores allocated towards automated quality control.

In this article, we walk through our experience on the Trainium3 TorchTitan and TorchNeuron stack, along with our experience of Trainium’s Neuron Explorer profiler, which is way better than both Nvidia’s and AMD’s profilers. We created a bunch of demos to show how Trainium’s profiler is “GPU Impossible.”

Visualizing Nafea Bshara, VP at AWS In Trainium Profiler Showcasing Trainium’s Finegrain Profiler. Source: SemiAnalysis

In an upcoming article, we will go deep on writing speed-of-light NKI kernels and microbenchmarking on Trainium3. For our deep dive into the Trainium hardware, check out our previous article where we do a complete hardware breakdown.

Source: SemiAnalysis

We Are Hiring

SemiAnalysis is hiring for various Member of Technical Staff roles. If you find this type of analysis interesting, apply below:

Apply Here

What we’re looking for:

Highly motivated & skilled Member of Technical Staff to join our growing engineering team.
Member of Technical Staff will play a crucial role in developing inference benchmarks & system modeling
You will also play a role in writing and preparing our newsletter articles for publishing, with the opportunity for direct authorship recognition.
We are hiring individual contributors at all experience levels. Compensation will be competitive
As part of the interview process, you’ll complete a paid coding challenge designed to reflect typical daily tasks at SemiAnalysis
Competitive Compensation depending on experience, skillset, location & business needs

Qualifications

Proactive, self-motivated, and capable of working independently in a global team.
Demonstrated experience in ML frameworks such as PyTorch or JAX through professional experience, personal projects, or personal Substack blogs
Solid understanding of at least 1 of the following: transformer architecture, LLM parallelism strategies, and/or CUDA parallel programming
Strong research skills and the ability to synthesize information from various sources to draw insights.
English language proficiency required, Chinese language proficiency preferred

Moving Towards a First-Class PyTorch Experience

Amazon has made a major course correction to its software strategy, and we believe it will meaningfully broaden Trainium3 adoption. TorchNeuron represents a native PyTorch experience instead of duct-taping Trainium onto PyTorch through the PyTorch/XLA project. The new backend is aimed at everyone who isn’t a frontier-lab kernel engineer. AWS has even hired the lead of Google’s native Torch on TPU effort to work on native Torch for Trainium. He was the TPU SW lead from the start of the program at Google, including XLA, xprof, Cloud TPU, etc.

The old approach made Trainium feel foreign to anyone coming from CUDA. PyTorch/XLA relied on lazy tensor graph capture and had no first-class eager mode. It also didn’t support PyTorch’s native distributed APIs (torch.distributed.*) or its native parallelism APIs (DTensor, SimpleFSDP, DDP). Users had to rely on out-of-tree XLA SPMD APIs such as torch_xla.experimental.spmd_fsdp and torch_xla.distributed.spmd. For researchers who are used to native PyTorch on CUDA, switching to Trainium meant relearning how to write a training loop.

Source: AWS

The pain of converting GPU research codebases by adding XLA markers everywhere was a massive turnoff to anyone who used PyTorch/XLA before.

Source: AWS

At a high level, the only change needed is replacing .to(”cuda”) with .to(”neuron”), and in theory the vast majority of the PyTorch ATen operators will work decently well out of the box.

Source: AWS

In theory, with PyTorch’s torch.accelerator hardware-agnostic API, lines of code like bf16 autocast won’t even need to be changed, though some APIs in PyTorch and some research codebases still haven’t migrated to the torch.accelerator namespace and still directly call the torch.cuda.* namespace.

As most academics and researchers across robotics, biotech, world models, and self-driving mainly use BF16, Trainium Torch will not support TorchAO MXFP8 on day 0 until they have made the BF16 training experience great. We don’t disagree with this sequence, as most people doing MXFP8 training are big Tier 1 labs that have enough kernel engineers to write custom kernels for the big training run.

Trainium Torch Ecosystem

PyTorch is popular not only because of core PyTorch but also because of the entire ecosystem of libraries, including Hugging Face Transformers, PEFT, diffusers, TRL, Accelerate, TorchTitan, etc. We are amazed to see that Trainium Torch already supports a ton of these libraries in the ecosystem, including Hugging Face Accelerate, which an insane number of layers of indirection. Trainium supports one of the most popular clean open-source training codebases called TorchTitan with many Tier 2 labs forking it to serve as the foundation of their internal training codebase. Trainium’s new PyTorch experience fully supports that too.

We saw how Google was able to successfully partner with Inferact/RadixArk to bring TPUs into the vLLM/SGLang ecosystem, and also land Meta as a production TPU customer. Furthermore, by bringing TPUs into the vLLM/SGLang ecosystem, TPUs were already able to automatically get day 0 support for Muse Spark Open and commitments for day 0 TPU support for popular upcoming models.

While it is true that the increase in agentic coding has made the engine code itself less important, what vLLM/SGLang brings to the table is the ecosystem. No matter the open source model, on day 0, the global ML community trusts vLLM/SGLang as the go-to engine for inference. No vibe-coded engine has that reach.

We are starting to see Trainium rapidly invest in the vLLM/SGLang community with hardware access and early access to Trainium on SGLang built on native PyTorch (TorchNeuron), similar to the TorchTPU approach.

Source: SemiAnalysis

TorchTitan Experience

TorchTitan is one of the most popular open-source pre-training frameworks, and we tested Neuron’s support to gauge the user experience of a researcher training a non-trivial model with Trainium. Specifically, we trained a Qwen3-32B on 16 Trainium2 chips, 64 LNC=2 logical cores in total, and we also tried doing it on a rack-scale Trainium3 UltraServer with 64 physical chips.

Overall, we liked the user experience, though there were some rough edges that the Amazon team already has on its roadmap to fix.

While we will provide the Trainium2 MFU below for completeness, the part to focus on is that for ML research ablations, what matters most isn’t a speed-of-light-optimized stack but decent performance and broad support across model architecture modifications, so researchers can iterate without fighting the stack.

With Neuron’s official TorchTitan image, we were able to launch the run without any issues. We measured 11,099 tokens per second in average training throughput on eager mode, at 23.89% MFU. For compiled mode, we saw 14,156 tokens per second and 30.47% MFU, a 1.28x throughput improvement over eager mode. We tried changing the model architecture slightly in the modeling code and it worked great also.

Neuron also offers training recipes for more modern model architectures, so we tested its gpt-oss 20B training recipe. Neuron supports gpt-oss training through integration with GraphTrainer, TorchTitan’s experimental trainer module. Specifically, Neuron adds Trainium-compatible configurations of SimpleFSDP, a compilation backend for PyTorch computation graph capture, and custom NKI kernels for attention and MoE operations. In terms of user experience, the recipe worked out of the box, and everything went well.

One thing that was not yet implemented using the upstream APIs was TokenSwitch. In the age of MoE, for inference and training, MoE combine and MoE dispatch are among the most important comms ops and take anywhere from 60-70% of the communication volume, and TokenSwitch is PyTorch’s first-class API for them. Trainium doesn’t support it yet and instead relies on a custom TorchTitan class called NeuronEPTokenDispatcher. Moving it onto the TokenSwitch API looks like a simple fix, and the AWS team has committed to collaborating with the Meta PyTorch team to modify the TokenSwitch ABC such that it can support Trainium too.

The part where it was slightly disappointing was the Neuron compiler took a long time to compile, leading to a long wait in the first step. Concretely, the first step took 184.6 seconds for eager mode (255.7 seconds for compiled mode), which is 30-55x longer than the average step time, excluding the first step. For gpt-oss, this training run came at a ridiculously high compilation time of 27 minutes and 17 seconds, which leads to a terrible user experience, especially for researchers who experiment with new model architectures. The reason for this is that historically the Trainium compiler stack has mainly been focused on inference, where compile time doesn’t matter because compilation is done offline and cached once per image and doesn’t affect production inference, whereas in training it does. While this is the right call for inference at the time, for researchers iterating on model architecture, that’s a meaningful friction point.

The Neuron team is already addressing this directly. Upcoming TorchNeuron releases introduce a new compiler backend with a broad set of precompiled NKI kernels and a compile stack re-architected for fast compilation, targeting up to a 2 orders of magnitude reduction in time-to-first-step for both eager and compiled modes, with minimal or no performance regression. If they deliver on that, compilation time stops being a conversation.

Training MatMul Performance

One of the most important things for experimental training is to have decent performance across a broad range of shapes as ML researchers try lots of different things. We benchmarked a single Trainium3 logical core (LNC=2) on the PyTorch nn.functional.linear API to understand its matrix multiplication performance, and we compared it with GB300 and B200 for reference. We swept through the matrix shapes of projection operations in major open weight models, and we set the M dimension to 1, 128, and 2048 to represent different batch size scenarios.

Source: SemiAnalysis

In our benchmark, Trainium3 achieved a visibly higher MFU than GB300, starting from the mid-range, where the matrix multiplication performs 1 GFLOP of computation. It was only after 100 GFLOP that GB300 and B200 started outperforming Trainium3, and Trainium3 won again after 1 TFLOP.

FlexAttention API

FlexAttention is a high-level API that allows implementing different attention variants using PyTorch, while offering backward pass auto-generation and reasonable performance. Good FlexAttention support shows a hardware’s flexibility and ease-of-use to researchers. We ran the FlexAttention benchmark suite on Trainium3 to see its FlexAttention support.

Source: SemiAnalysis

Overall, Trainium3 passed 89 out of 96 test cases, a 93% pass rate. All 7 failed cases were at large batch size (4) and sequence length (8192), and all failed due to compilation exceeding the time budget rather than numerical errors. These issues will be fixed in later versions of TorchNeuron.

Additionally, we tried implementing custom attention variants not in the FlexAttention benchmark suite for Trainium3, and it matched the numerics of Nvidia GPUs.

Source: SemiAnalysis

Automated Quality Assurance

SemiAnalysis is known in the global ML community to be the “CI Karen,” setting a high bar across all hardware vendors and doing whatever it takes to get senior leadership of AI chip companies to actually provide their engineers with enough resources and CI fleet to maintain a high bar of automated quality control.

Source: SemiAnalysis

One recent achievement, which took over 9 months of fighting with AMD senior leadership, was getting their engineers enough stable nodes to achieve 93% parity on gating test groups.

Source: vLLM Dashboard

Inference is a narrow problem. A lab serving a handful of production models needs a forward pass, mostly fixed architectures, and a small set of hot kernels.

Training is a generality problem. Each of the thousands of operators needs a backward pass, and research codebases sometimes accidentally run operators in higher precision, which must also be supported. To ensure that Trainium works across a wide range of model architectures and can do eager mode and compile mode well, AWS collected a giant list of top 100k+ Hugging Face models that spread across a whole range of different normalization, MLP, MoE layouts and attention types. Not only do they test that it doesn’t crash, but they also test performance, which ops use CPU fallbacks and don’t have native support yet, where graph breaks are and most importantly, they test numerics against CPU and Nvidia baselines. They test tensor level parity and loss curve divergence also. It is easy to finish an exam within 10 minutes, but it doesn’t matter how fast you finish an exam if you get a 0% on it. Same thing with training stacks. It is easy to get decent MFU%, but it is hard to get decent MFU% while not introducing numeric issues.

Source: AWS

Although the pass rates aren’t at 100% yet, what matters is the trajectory. AWS is climbing rapidly, with agent swarms systematically fixing the remaining failures. We think it’s a strength that AWS’s internal summary table, the one senior leadership sees, shows gaps transparently rather than hiding them.

We say this from experience. When SemiAnalysis and upstream maintainers have tried to fix quality control culture at other hardware vendors, we’ve found that senior leadership is genuinely shocked when shown the real parity pass rates. At one vendor in particular, there was so much middle management filtering that by the time CI numbers reached senior leadership, they showed 99.99%, while the actual gating parity pass rate at the time was below 5%.

Neuron Explorer - Better than Nvidia and AMD Profilers

Trainium’s profiler is by far leading the industry and is SOTA status. Here we share our initial impressions and plan on sharing our full user experience in a future Trainium kernel article.

Trainium’s profiler is loved by all the performance engineers at Anthropic. Anthropic’s head of performance (who is ex-Jane and Finch Street) has publicly said that performance analysis on Trainium is way better than performance analysis on Nvidia GPUs (and obviously better than on AMD GPUs).

Source: Tristan Hume on Twitter

We agree with Tristan Hume. In our experience, the profiling experience was amazing, after getting over the initial hump of the profiling setup being slightly confusing (but not impossible). There are multiple environment variables to set, different file types to save, and various commands to capture and view the profile traces. This is only the kernel profiling setup, and the PyTorch profiling setup is similar but with slight differences, making things even more confusing. With the help of Codex (since Claude refuses to write Trainium kernels), our best effort working setup is as follows:

Launch a persistent Neuron Explorer server
Profile the kernel with neuron-explorer capture command
Upload the profile trace as an artifact to the Neuron Explorer server as a web request

We shared feedback with AWS that we wished the documentation could be clearer on the workflow, explaining what environment variables need to be set for what reasons, and what purpose each file type serves.

Once the setup is done, the profiler user experience is great. Neuron Explorer is co-designed with the hardware. We particularly like the timeline separating hardware units in different rows, so that bubbles are visible at a glance. In contrast, GPU programmers have to roll their own setup for intra-SM profiling as the official Nsight tools don’t support that.

Source: SemiAnalysis

We also like how the viewer visualizes dependencies. When a block is selected, its dependencies and dependents light up in different colors, which helps us reason about work overlapping, e.g. whether computation sufficiently overlaps with memory loading. The image below shows that the selected white block is:

Flow-dependent on the red block
Order-dependent on the orange block
A flow dependency of the blue block
An output dependency of the purple block

Neuron Explorer dependency visualization. Source: SemiAnalysis

Neuron Explorer code editor mapping. Source: SemiAnalysis

Neuron Explorer: Event information when hovering on an event. Source: SemiAnalysis

Neuron Explorer memory bank utilization visualization. Source: SemiAnalysis

Neuron Explorer microsecond-level memory throughput visualization. Source: SemiAnalysis

Trainium Profiler Demo - Pixel Art

Neuron Explorer has many more features, including code editor mapping, MFU estimation, memory usage, and even creating pixel art due to how precise the profiler is. The images below are all from profiler traces of running on actual Trainium hardware, not fake profile reports. We were mainly inspired by the Bad Apple post from Daniel Liu (Anthropic Performance Engineer) and have created our own agentic coding SKILL.md for creating images. We did not have the time yet to create one to automatically generate videos frame by frame but will in the future and open source this SKILL.md for everyone to use. We mainly generate this pixel art by using the 16 DMA engines per physical core by “coloring in” regions by turning the DMA engines active, and the width of the shaded area is determined by how much data we have each DMA engine move.

Bad Apple on Neuron Explorer. Source: Daniel Liu

Visualizing the SemiAnalysis logo. Source: SemiAnalysis

Visualizing Jeff Bezos (pre looksmaxxing). Source: SemiAnalysis

Visualizing Dylan Patel (pre looksmaxxing). Source: SemiAnalysis

Since AWS is a partner of Nvidia, Jensen should tell his engineers to book a meeting with the AWS Trainium team on the hardware features needed to do accurate fine-grained profiling such that Nvidia can learn how to add this capability to Feynman or Feynman-Next GPUs.

Visualizing Jensen on Trainium3. Source: SemiAnalysis

Nvidia’s intra-kernel profiler Nsight Compute employs a different design. It offers aggregate metrics and detailed information on performance properties, rather than an event-based timeline viewer. The closest thing we found was Tri Dao’s QuACK intra-kernel profiler, which uses Nvidia’s In-Kernel Event Tracing API. While QuACK requires custom design to make Perfetto visualize accordingly, it’s great to see that AWS made this a well-integrated, first-class experience in Neuron Explorer.

QuACK’s intra-kernel profiler: Visualization of a QuACK GEMM kernel. Source: QuACK

Nsight Compute view of FlashAttention 4 workload analysis. Source: Nvidia Nsight

The Huawei-DeepSeek Symbiosis

2026-10-21T09:34:38+00:00

Right before China’s national day holidays, DeepSeek released Ascend support for TileLang, DeepGEMM, DeepEP, TileKernels, FlashMLA, and DeepSelect.

Neither can do this alone. Although they have Pangu, Huawei has no frontier lab to prove their chip can train or infer models, DeepSeek has models but cannot buy the compute it needs. Each supplies exactly what the other lacks: Huawei provides the hardware and native software stack; DeepSeek brings the workloads and engineering needed to make that hardware useful for its models. September 30’s release of Ascend support in DeepSeek repositories is an artifact of this trade, where the exchange runs through the kernel-authoring language.

Source: SemiAnalysis - Note that the diagram is a simulation.

In 2028 or even 2027, DeepSeek will likely be primarily using Huawei’s stack, especially with the external and internal pressure to build a fully domestic chinese ecosystem. As we show here and in future articles, the Huawei ecosystem is already mature and well developed, with strong foundations from the Ascend time and DeepSeek engineers building further ontop.

In this article, we’ll first examine Ascend 950PR’s architecture and benchmark its compute and memory performance. We then explore DeepSeek’s software releases, from TileLang’s compiler capabilities to hardware-specific kernel optimizations and rack-scale communication. Finally, we analyze DeepSeek-V4.1-Flash’s execution profile.

Ascend 950

We deep-dived into CANN and the Ascend 950 architecture in our DeepSeek V4 Pro article. Since then, more interesting details have been revealed.

Source: SemiAnalysis

Starting from the chip, many variants of 950PR/DT have appeared with varying core counts, HBM stack height, and cache sizes. Some are likely either virtualised/simulated, or for testing only. This is because only a fraction of dies and HBM stacks meet the target spec, due to both outright defects and parametric shortfalls in power and performance, so Huawei bins both compute and memory into multiple SKUs and pairs weaker memory with weaker compute, rather than binning only compute as Nvidia does.

Source: SemiAnalysis

This isn’t new, 910B also has variants such as 910B1, 910B2, 910B3, 910B4. For the following microbenchmarks, we use an Ascend 950PR_9579, with 28 cube cores and 128 GiB HBM.

root@cb3bb851f93f:/workspace# npu-smi info
+-------------------------------------------------------------------------------------------------+
| npu-smi 25.7.rc1.6                               Version: 25.7.rc1.6                            |
+--------+------------------+---------------+-----------------------------------------------------+
| NPU ID | Name             | Health        | Power(W)    Temp(C)           Hugepages-Usage(page) |
|        |                  | Bus-Id        | NPU Util(%) Memory-Usage(MB)  HBM-Usage(MB)         |
+========+==================+===============+=====================================================+
| 0      | Ascend950PR      | OK            | 203.3       56                0     / 0             |
|        |                  | 0000:12:00.0  | 0           0    / 0          5242  / 131072        |
+========+==================+===============+=====================================================+
+---------------------------+---------------+-----------------------------------------------------+
| NPU ID                    | Process id    | Process name             | Process memory(MB)       |
+===========================+===============+=====================================================+
| No running processes found in NPU 0                                                             |
+===========================+===============+=====================================================+

Running point chases reading 16 KiB of integer payload per launch, it stays near 134ns through 18 blocks, rises to 182ns at 56, then jumps to 391ns at 64. This is consistent with a second wave beyond 56 Vector cores.

Source: SemiAnalysis

Then by running a scalar pointer-chase probe separately on every Cube and Vector core, recording load timings at matching addresses throughout a 128 MiB chain, we get the following graph. Each cell’s color represents the mean absolute difference between two cores' address profiles, producing two groups that match the decoded dies.

Source: SemiAnalysis

Going further, by timing a Vector core’s pointer chase while another core runs either memory traffic or register-only work; the extra slowdown reveals which subsystems share resources, supporting groups such as {0,2,4} and {1,3,5}. Mixed Cube/Vector launches identify each subsystem’s one Cube and two Vectors, which we draw together within those groups, with physical positions remaining schematic.

Source: SemiAnalysis

CPU within the NPU

Each 950PR comes with a variable number of Linx816 AArch64 ELF AICPUs. The MIDR numbers imply these cores are likely the same as the middle cores in the Kirin 9000S/9010. These AICPU cores have instruction set:

fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid
asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve
asimdfhm dit uscat ilrcpc flagm ssbs sb paca pacg dcpodp flagm2
frint svei8mm svebf16 i8mm bf16 dgh rng ecv

For this SKU, there are 7 configured, 6 visible, and 5 executing AICPU cores. Other than device control it can provide CPU-operator support, such as KV metadata. The AICPU helps out for short work where D2H overhead take up most of the time.

For DSA, a small scheduling step reads the current sequence lengths and assigns indexer work to accelerator cores. Cube/Vector cores then compute INT8 indexer scores and select the top 2,048 indices. These planning, index selection, and gather steps are ran on AICPUs in vLLM-ascend implementation, and we compare their performance to running on the host CPU.

Source: SemiAnalysis

Due to their limited compute performance, AICPU is only good for short work where overhead dominates, and work where data starts and is consumed at the NPU.

Microbenchmarks

Ascend 950PR is designed for prefill and recommendation. It gets lower memory bandwidth and core than its bigger brother the 950DT.

The results below are not from DeepSeek’s contributions but from stock torch.matmul on ascend resolving to aclnnMatmul_MatMulV3Common_MatMulV3. We compared 1,055 BF16 GEMM shapes from several open-sourced models. Huawei’s best kernels hits 99.7% of the ceiling, showing DeepSeek’s 99+% from DeepGEMM isn’t the only way to maximize utilization. Huawei engineers did good work too.

Source: SemiAnalysis

The advantage is concentrated in compute-bound work. On the shapes below Ascend’s ridge, its median achieved/attainable fraction is 41.5% versus H100’s 46.8%. Some of Nvidia’s results were power-limited and did not hit maximum SM clock: median power around 690-698 W out of a 700 W limit, and median sampled SM clocks at 1335-1470 MHz, below the max of 1830 MHz. Huawei approximated 599-600 W, at 1500-1550 MHz against its 1650 MHz reference.

Ascend’s MXU design may contribute to some of this high utilization, Ascend stages matrix operands in dedicated L0A/L0B buffers and accumulators in L0C, with overlapping movement, computation and result handling. Huawei associates the enlarged L0C with more flexible tiling and reuse.

But of course, 950PR’s falls behind on absolute values, sitting somewhere between A100 and MI300X.

Source: SemiAnalysis

MemBW

Measuring HBM memory bandwidth, we see 950PR’s effective bandwidth and bandwidth utilization surpassing H100 at small message size per buffers, and nearly matching H100 in bandwidth utilization at large message sizes.

Source: SemiAnalysis

A dip appears in the curve as Ascend switches copy methods to a Vector-core TensorMove kernel that has more startup cost, so a slightly larger copy suddenly takes longer. For very large copies, that extra cost becomes relatively small, so bandwidth rises again.

Note that high FLOPs utilization and bandwidth utilization does not necessarily translate to high MFU/MBU. Training, especially with FSDP, requires collectives, checkpointing, and data moving that can be hard to overlap.

As we mentioned in our Jalepeno article, a “chip” in benchmarks is just a social construct. Multiple dies can be glue to form a bigger “chip”. Thus performance per watt or TCO is the best comparison. But for China, watts are far cheaper. We’ll analyze Ascend’s perf per TCO in a future article.

Hardware raises the ceiling, but software lets you reach that ceiling. Although Huawei’s hardware may be beyond ~5 years behind(compared to A100), the software is right at the frontier.

DeepSeek Open Source

For the 6 repos that received ascend support(TileLang, DeepGEMM, DeepEP, TileKernels, FlashMLA, DeepSelect), only TileLang is usable as of writing, and CANN 9.2.0 has not been released. Deepseek requires CANN 9.2.0 which was released after time of writing, and a C++20 compiler is required which CANN 9.2.0-beta.2 does not provide. The HDK will be released on Oct 15. This comes after DeepSeek’s open source week in 2025, where some of these repositories were released with Nvidia support only.

From the results above, we can see Huawei can already execute operations well; the symbiosis is about making the rest of the application & the work required to support it more efficient and reusable.

Engineering runs both ways

Huawei’s architecture is already influencing how DeepSeek’s software organizes data. Earlier FlashMLA KV-cache layouts stored quantized values and their scaling factors in separate regions. On Ascend, fetching a token then required separate transfers for its values and scales. The new implementation places them together, reducing the number of transfer requests by the memory transfer engine(MTE).

Source: SemiAnalysis

From fast kernels to reusable kernel engineering

Ascend 950 backend is merged into the main TileLang repository, rather than existing only in a separate Huawei adapter. It adds native compilation, scheduling, synchronization and runtime integration for Ascend’s execution model.

TileLang makes Ascend kernels easier to write, but developers still need to account for the hardware. They choose the calculations, tile sizes and where data is stored. The compiler handles much of the coordination between Cube cores, Vector cores and data transfers.

Digging below the implementation and looking at its performance against CANN and Triton-ascend, it can match and beat CANN at certain cases; the preferred implementation changes with size. This is a warm-cache simple and naive benchmark.

Source: SemiAnalysis

AutoSchedule uses data dependencies and estimated execution times to plan which operations can overlap. It also chooses how many buffer slots to use within the available on-chip memory, allowing upcoming data to load while the current tile is being processed. Developers can guide these decisions when the default schedule is not suitable.

Source: SemiAnalysis

On 950PR, AutoSchedule helps especially for smaller sizes, reducing execution time by up to 40% for a single operation.

Source: SemiAnalysis

The compiler also inserts synchronization so operations wait for the data they need, while removing waits that other dependencies make unnecessary. It checks conditional branches too, so it does not assume an operation has completed when that branch might never execute. This reduces the synchronization work programmers must manage themselves.

DeepEP

Huawei has joined NVIDIA in commercializing large scale-up AI systems, with its Ascend supernodes extending tightly coupled accelerator interconnects across multiple racks from the 910 CloudMatrix384 to the 950 SuperPod,enabling wideEP scale-out. In the 30th September release, Huawei also released EP16 dispatch/combine benchmarks over URMA in the 950DT superpod.

Source: Huawei, DeepSeek, SemiAnalysis

Huawei’s dispatch is three sequential operators (pre/URMA/post) whereas GB300 is one fused call, and their post work includes receive completion, which our measurement includes, but also includes expansion by expert which GB300’s results don’t.

DeepEP-Ascend is a port of DeepEP V2.5, with the same python API but calls URMA and HCCL underneath. In July 2025, Huawei’s Ascend team proposed DeepEP-Ascend under deepseek-ai, but a DeepEP maintainer declined. Today, DeepEP-Ascend is distributed under DeepSeek’s own organization.

Ascend’s separate Cube and Vector execution paths create opportunities to overlap communication and computation, but do not make that overlap free. Expert computation can also require vector processing and data movement, while communication competes for shared memory bandwidth. Therefore there’s a tradeoff on how much communication can be hidden without slowing the computation that runs alongside.

Source: Huawei

This resource-balancing problem is not unique to DeepSeek. Huawei’s Kimi K3 recipe explicitly restricts vector use to leave capacity for concurrent work, and chooses between dedicated CCU and Vector-based communication according to what must overlap. Faster individual operations are not enough: their resource demands must also fit together.

DeepEP-Ascend’s documentation identifies contention between combine processing and URMA communication for HBM resources, and says Huawei firmware improvements are planned to reduce it. This is the practical form of co-design: the model exposes a bottleneck, the implementation adapts, and the platform’s development priorities respond to what remains.

All planned

In May’s unverified DeepSeek investor meeting transcript, Liang Wenfeng said:

“So, when it comes to the chip gap between China and the US, I think the software ecosystem gap will eventually disappear. But on the hardware side, we face two separate disadvantages: it takes four Huawei accelerator cards to match one NVIDIA card, and Huawei is also roughly two years behind in bringing its hardware to market.”

4 months later, the ecosystem gap did indeed narrow. The same transcript mentioned TileLang and helping Huawei get the ecosystem right.

DSv4.1 Flash Optimizations

Another part of the DeepSeek-Huawei symbiosis is that DeepSeek provides Huawei with weights, supporting Day-0 optimized releases, such as DeepSeek V4 and V4.1.

Ascend’s accumulator design forces a different approach to attention. The Cube’s L0C buffer cannot rescale accumulated outputs in place, while returning a rescaled output to L0C would require an expensive identity-matrix multiplication. DeepSeek instead keeps the running output accumulator in Vector UB, transferring partial sums from L0C when needed for rescaling and accumulation. Hopper holds its accumulator in registers, while Blackwell provides loads and stores between tensor memory and registers, avoiding this particular restriction

Source: SemiAnalysis

Profile

Huawei also released a profile of one rank of DeepSeek-V4.1-Flash on a Huawei Ascend 950DT deployment of 32 chips, together with its host threads. In June, we covered a similar profile, that of DeepSeek V4 Pro’s.

Many components are similar, like the use of AICPU and separation of some layers to different streams.

Source: SemiAnalysis

The execution order is 40 target-model layers, the target LM head, 3 DSpark drafter blocks, the draft LM head, and 5 sampling rounds. The main compute runs on Stream 19; the drafter’s run on Stream 173. The V2 MC² kernels that fused computation and communication from the DSv4 Pro trace shows up again, as V3 this time. This is why only 4 AI CPU-launched HCCL operations appear per step, against 43 expert dispatches. These are the target and draft vocabulary heads’ hcom_allGather and hcom_alltoallv: gather hidden states, run the LM-head GEMM, then exchange logits.

The drafter blocks run once to produce base logits for the draft positions. The subsequent Markov head models token dependencies through sampling and embedding feedback. It does not rerun the transformer blocks for each proposal. The end of each step changes from MoE and attention to a serial chain of small projections and sampling operations.

Source: SemiAnalysis

As mentioned and explained in previous articles, speculative decoding is a lossless method to predict draft tokens and use a single pass to verify them, amortizing weight loading. In the decode step diagram above, we can see how much faster DSpark token drafting is compared to the verification step.

Source: SemiAnalysis

DSv4.1 Flash is also the first DeepSeek model to use Engram, but after Qwen and Meituan Longcat.

Because Engram takes token IDs as input, the fetch can be done async and overlapped with layer 0’s compute. This allows offloading to DRAM without performance loss vis-a-vis HBM, but offloading to a PCIe 4.0 SSD degrades performance. Read more in our Engram analysis:

Source: SemiAnalysis

Summarizing kernel durations in the steady-state window, Stream 19 is occupied for 88.8% of the window but contributes 72.6% of summed device time because other streams overlap it. In the middle, the substantial MIX_AIV and AI_VECTOR_CORE bands contain dispatch/combine as well as quantization.

Ascend exposes separately scheduled cube and vector paths through MIX_AIC, MIX_AIV, AI_CORE, and AI_VECTOR_CORE. Each event has a task type, so the duration accounting is a partition. Mixed types still involve work beyond a single physical unit; their names do not turn elapsed kernel duration into measured cube-only or vector-only utilization.

Source: SemiAnalysis

A first-order scenario doubling cube throughput while leaving other device work unchanged gives a 1.32× step improvement. It holds measured stream overlap constant and leaves idle unchanged. This is a composition scenario, not a prediction for new silicon: mixed task types are not exclusive utilization measurements, and bandwidth, dependencies, and scheduling can change the result.

Quantization fusion likewise remains an opportunity, its attraction is the repeated producer-to-consumer boundary and its large share of the vector bucket that also carries fused communication.

Ecosystem gaps

One issue however is the partnership could work extremely well for DeepSeek without making Ascend equally usable for everyone else.

For DeepSeek, building compilers, maintaining kernels and debugging vendor-specific behavior competes with model research and product development, albeit temporarily.

For everyone else, there may be issues when hardware configurations doesn’t align with DeepSeek’s. Most optimizations were on supernodes, and as of vLLM-Ascend_0.26.0rc1 many issues exist working with Ascend on x86 hosts:

No DeepGEMM support on x86 hosts
Dependency on memfabric_hybrid containing hardcoded ARM headers for vLLM-Ascend, blocking startup; vLLM-Ascend doesn’t work out of the box.
CPU-binding skipped for non-ARM CPUs on vLLM-Ascend
No NPU + x86 CI, patches needed for some recipes

Datacenters 2027: The Year of Acceleration, +50GW in a Single Year

2026-10-18T21:39:38+00:00

Not a week goes without another piece of datacenter FUD. We’ve already debunked depth why moratoriums aren’t as impactful as one may think. Despite that, many still misunderstand the pace of the global AI datacenter buildout. Just the other week, Brad Gerstner from Altimeter showed our data on the All-In Podcast and stated that “SemiAnalysis’ forecast of +43GW in 2027 is too aggressive” and that he thought only about 25GW would get built, close to half of our number, based on his “hunch”. But let’s not operate on hunches.

It’s very tempting to cede to the FUD given all the constraints on the datacenter buildout: power, labor, supply chains, capital, and many more. But at SemiAnalysis, we care about the truth, not about narratives. And here’s the truth for you: one picture, >24GW of datacenters in advanced stage of construction, all to be delivered in 2027. Most of them are more than halfway done, with exterior cooling and electrical equipment already installed. That’s just 40 campuses, a tiny share of the multiple hundred sites in advanced stage of construction. We track them all, and for the full granularity, use our Datacenter Industry Model.

Source: SemiAnalysis Datacenter Industry Model

Of course, there are plenty of challenges and shortages. Behind these pictures, there are labor issues, supply chain delays, fundraising challenges, among many others. We’ve written in-depth about Modular Datacenters and Labor, Datacenter Electrical systems, Cooling systems, Behind-The-Meter power, the US Grid Constraints, and Nvidia’s ecosystem financial support. But the fact is simple: gigantic sites are simply getting built, and many are on time. For example, just go look at what Crusoe built in Abilene, what AWS built in Indiana, or QTS/Microsoft’s giant site in Georgia. These are the sites that have produced SOTA technology like GPT-6 and Fable 5.5. Looking ahead, despite all the constraints, an accelerating number of giant sites keep getting built, as shown by our Top 40 picture

Source: SemiAnalysis Datacenter Industry Model

If you ask any datacenter builder, they’ll tell you how hard it is, how some workers aren’t showing up, how vendors are delayed, and many other issues. All of this is accurate. And that’s why we’ve successfully called out countless datacenter delays and keep calling out developer timelines that appear overly aggressive in our Datacenter model, including the Oracle New Mexico Force Majure site delay in April. To be fair, +50GW is essentially the global cumulative datacenter market up through 2024 being built in one year! But our Datacenter Industry Model adjusts for all of that and still points to a material acceleration in datacenter capacity in 2027. So, what are we seeing that others can’t?

Source: SemiAnalysis Datacenter Industry Model

Let’s dig in. We’ll explore the following topics in this report:

SemiAnalysis proprietary leading indicators: we’ll show a sneak peak of our proprietary data supporting our forecast, such as self-build construction starts and datacenter leasing (in MWs and USD). Year-to-date, slightly under USD 1T of binding / non-cancellable datacenter leases have been signed – that’s key to supporting our aggressive 2027-28 forecast.
The shape of the buildout: how can we support +50GW in a single year? Where, what design, what power solutions?
The “real” question: are we going to face a datacenter oversupply? As we explained in The Great Silicon Shortage, datacenters & power may be the tip of the iceberg, with memory and logic possibly becoming the new #1 constraint in 2027-28.

Overall, we are very excited about 2027. The natural implication of the acceleration will be significantly more compute supply. That means cheaper and smarter AI, faster pace of AI model progress, and likely material revenue acceleration for leading AI labs and other token vendors. That also means unthinkable capex numbers. Reflexively rates will have to shoot up to enable the buildouts from a capital demand perspective. Let’s dive in!

SemiAnalysis Datacenter Leading Indicators: how we see the buildout before everyone else

Today we want to dive into two crucial indicators that give us strong confidence on our 2027 forecast. The first one is quarterly self-build construction starts. To accurately forecast US hyperscalers, we track the exact construction evolution of every single building. And the conclusion is simple: US hyperscalers have never launched more construction of datacenters. Each quarter, we track how many MWs start construction, and that number is going parabolic, driven in particular by two hyperscalers. If you want to know who or track the granularity of our data, subscribe to our Datacenter Industry Model.

Source: SemiAnalysis Datacenter Industry Model

The second indicator is Datacenter Leasing. We’ve talked about this a lot in prior reports, but as a quick recap: given the capital-intensity of building multi hundred MW datacenters, developers typically only spend 5-10% of project capital without a customer. They need a “pre-lease” to fund and develop the whole project. That’s typically a 15-20 year contract, with a TCV of $30-50B per GW. Check out our HPC Lease Tracker as part of the Datacenter Model to dive into actual contract economics.

Subscribe now

As such, tracking these lease contracts is a key leading indicator, as it leads debt raises, which itself leads the actual buildout. A big reason why 2026 datacenter growth isn’t “that” high relative to 2025, is that quarterly datacenter leasing remained relatively stable between H2 2023 and H1 2025. Of course, the key reason was the Microsoft Datacenter Freeze, which we called out before anyone else in December 2024 for Datacenter model subscribers: the Redmond giant single-handedly drove the leasing market in 2023 and 2024, then paused, and others could only pick up what was left behind.

Since H2 2025, things have dramatically changed. Oracle kicked off the era of gigantic leases, after closing on ~4.5GW of datacenter IT capacity across three sites between August and October 2025. Everyone else followed, and the pool of buyers has never been broader, with Meta, AWS, Microsoft, Google, Nvidia, AMD, Anthropic and OpenAI all being in the market for GW-scale leases. There’s also a long tail of Neoclouds signing hundred-MW-scale deals.

In USD terms, this means an unprecedented amount of lease value that builders can use to raise debt. Just in 2026 year-to-date, we’ve identified over a trillion USD of lease value that was signed and closed. That trillion is supporting a gigantic datacenter buildout in 2027-28. Our new Compute, Capital & Markets Model is now tracking all the debt raises and can confirm that most of this debt has been raised.

Source: SemiAnalysis Datacenter Industry Model

The shape of the buildout: how to build 50GW in a year

Tracking all of these leading indicators has enabled us to forecast, before anyone else, the upcoming surge in orders for many industrial suppliers like Vertiv, ABB, Sterling Infrastructure. Check out our Industrials Model for in-depth coverage. However, the challenge is now going from orders to delivery. And resources have obviously become constrained, as power, labor, and sheer regulatory overhead are enormous burdens for developers. But the industry has promptly reacted with innovation.

The first step is to simplify datacenter designs. We show below how Google has modularized their electrical architecture, removing a significant amount of outdoors cabling that requires onsite assembly, replacing them with prefabricated electrical modules. We track all the equipment changes (including suppliers) for datacenter designs and forecast orders in our Industrials Model

Source: SemiAnalysis Industrials Model

Others have opted for more radical design changes. One example is Meta with its tent design, that we exclusively unveiled over a year ago. Another one is AWS which has materially simplified their shell design and launched a new rapid design dubbed “SAMDC”. Data halls are also increasingly prefabricated, an effort called Project Houdini at AWS. Construction timelines are now under 10 months. Hence our “LEGO Datacenters” title for the modular movement, you literally slot them in and go, and doing so allows for MEP labor (7-11 months of work) to drop up to 63%! We will have a full deep dive on the labor of the 2027 buildout coming soon.

Source: SemiAnalysis Industrials Model

In addition to hyperscale selfbuild, datacenter developers promoting prefabricated designs have also been remarkably successful. DayOne is likely the best example. With a total of ~3GW leased since their first deal in early 2023, the Singaporean company has been a rocket ship in terms of construction speed. While traditional developers have >18 months build timelines, DayOne cut this to ~9 months, which was key to signing huge deals with Firmus, OpenAI and Nvidia.

Source: SemiAnalysis Datacenter Industry Model

Subscribe now

Another core issue is power, and again we think the issue is largely solved. We’ve been the first to talk about all issues on power, since March 2024 and subsequently given the grid headroom dips negative at the end of 2027. This ended up driving AI Labs and hyperscalers to build their own power plants, better known as Behind-the-meter generation, bypassing 5-10year grid interconnection timelines. And while turbine incumbents like GEV quickly faced production bottlenecks, the industry quickly reacted and we’ve since seen a surge in new OEMs and new solutions supplying the market. From fuel cells to truck engines to refurbished turbines, the power generation problem has broadly been solved.

Source: SemiAnalysis Energy Model

In addition, the buildout is now geographically diversifying. This naturally bypasses US labor, power, and (in some countries) permitting constraints. And as shown below, the ex-US buildout is sharply accelerating in 2027, and we expect it to gain a lot of market share in 2028, largely driven by US challenges.

Source: SemiAnalysis Datacenter Industry Model

The Real Question: Will We Have an Oversupply of Datacenters?

ASML Is Building a Hybrid Bonding Tool!

2026-10-15T11:27:20+00:00

ASML has not announced anything yet, but its patent filings, a news report and its own roadmap all suggest so. We believe it will start with a die-to-wafer (D2W) hybrid bonding (HB) tool. This is harder to build than wafer-to-wafer but is where ASML could gain an edge. We see the move as a natural extension of ASML’s vertical integration, and one that should strengthen its customer ties.

In this article we unpack the tool’s design end to end, starting with patent filings. What makes it unique is that it runs on a track. That changes how the tool and the dies move: each die rides its own holder past the process stations, and the tools get closer at each station to process the dies. The track’s independent shuttles also lift station utilization, save on duplicate tooling, and enable heterogeneous integration.

We believe ASML could gain an edge in alignment accuracy and particle control for Hybrid Bonding. These are the two factors that set bonding yield. And ASML has expertise in solving the same problems in lithography.

Despite the advantages, ASML still needs to solve challenges around feed-forward metrology, the die-to-wafer contact, the modification of the non-bonding tools so they can process dies on the track, and keeping the whole loop clean.

We expect advanced logic, where D2W already runs in volume, to adopt ASML’s D2W HB tool first. This is bad news for Besi. Logic accounts for most of Besi’s HB business today concentrated at TSMC and Intel. ASML has been working with both for more than two decades and just announced a joint photomask initiative in September. We won’t be surprised to see a similar move in the back end process. Besi’s HBM prospects are also weakening. Korean challengers are catching up, in addition to an HBM adoption that we expect to slip to 2029 or beyond.

ASML is Building a Hybrid Bonder

We did a deep dive on hybrid bonding in 2024, and we still see it as the most transformative innovation in semiconductor manufacturing since EUV. Wafer-to-wafer (W2W) hybrid bonding (HB) is already in volume in image sensors and 3D NAND, and DRAM makers will be next. Die-to-wafer (D2W) HB is moving slower, because the industry still lacks mature tools that combine satisfying alignment accuracy and throughput (our check indicates upper bound in current applications is only one wafer per hour), which keeps cost per die high. Today D2W runs in volume only in logic: AMD has shipped hybrid-bonded V-Cache since 2022, and TSMC’s SoIC bonds at about 6um pitch today and targets 4.5um around 2029. The market expected HBM to become the big D2W volume driver from 2028. However, our view is different: we believe HBM will not adopt D2W HB before 2029, if at all.

Besi has led D2W and is the only supplier with tools in volume production. Everyone else is still trying to get a first tool qualified. But we believe the situation will soon change, because ASML is building a D2W hybrid bonding tool. Management hasn’t announced anything yet. But the company filed 27 bonding patent families between December 2022 and November 2024, most of them D2W, and the true count is likely higher because filings made after November 2024 are not yet public. The Elec also reported in March that ASML had started the architecture design of a hybrid bonder with Prodrive Technologies and VDL ETG.

In the anchor filing (WO2025201788A1), ASML drew how its bonding flow works. Interestingly, the system looks nothing like a traditional D2W bonder (see Advanced Packaging Part 5 for how those work). Instead of a bond head picking each die off a diced wafer and placing it, ASML moves single dies on independent holders around a track (Fig 1). Each die passes inspection, cleaning, plasma activation and metrology (topography, leveling and alignment) stations before it reaches the bonding station where the chuck lifts the bottom wafer up to meet the die. We unpack the design in the sections below.

Fig 1: ASML’s track-based D2W bonder, SA annotated and simplified

Source: ASML; SemiAnalysis Wafer Fabrication Equipment Model

ASML could be building a W2W hybrid bonding tool as well. According to a patent analysis by Inha University, ASML is likely building W2W HB equipment on its TWINSCAN platform, leaning on its wafer overlay accuracy. We focus on the D2W machine in this note, because it is the harder tool to build and the one where ASML could gain a real edge.

…as Part of its Vertical Integration Initiative

We believe this is a continuation of ASML’s effort to integrate vertically and tighten customer relationships. ASML bought Cymer for its light source in 2013, HMI for e-beam inspection in 2016 and Berliner Glas for optics modules in 2020. It shipped its first back-end product, the XT:260 i-line packaging scanner, in 3Q25. And most recently, ASML announced joint initiatives with TSMC, Intel Foundry and Samsung to move the industry from 6-inch to 12-inch photomasks for High NA EUV.

Building a bonder would be a natural next step. In fact, ASML’s own roadmap already puts bonding inside the front-end flow. Its 2024 Investor Day listed W2W and D2W hybrid bonding for NAND, DRAM and logic (Fig 2). And a July 2024 filing shows how the bonder and the scanner would work together by sharing die position data. Today a scanner printing on a bonded wafer must pre-align and coarse-align every die before it can expose. In ASML’s design the bonder already knows where it put each die and hands that map (a “coordinate system” in the filing) to the scanner, which checks a few marks and exposes. Customers get faster lithography on bonded wafers, and ASML gets to sell the bonder and the scanner as a pair.

Fig 2: ASML’s map of front-end 3D integration

Source: ASML Investor Day, 14-Nov-2024

We expect advanced logic to be the first adopter. Logic is where D2W HB already runs in volume, and where the alignment requirement keeps tightening. TSMC calls overlay “always the biggest challenge” for SoIC. As its HB pitch moves from 6um to 4.5um by 2029, the placement accuracy requirement will tighten further. ASML is strong in alignment, as we will elaborate below, so an edge here should help it win logic customers first. Advanced logic dies are also among the most expensive chips, so logic customers can afford a premium tool. We expect ASML to price its tools at a premium to peers.

In the next three sections, we discuss (1) the tool’s track system - the most unusual thing about it compared with what is on the market; (2) how ASML’s strengths in alignment, particle control and motion could make its HB tool superior; and (3) the challenges ASML or its supply chain partners still need to resolve to build a working tool. Let’s dive in.

What’s Unique About the Tool

One of the most interesting things about the tool is that it runs on a track. ASML names SuperTrak CONVEYANCE (Fig 3) as an example in its patent filing, so we assume it uses that track and we base our analysis on the track’s published design.

The fundamental difference from a traditional bonder is where the die lives. In a Besi-type tool, the die is prepared elsewhere, and a bond head picks it up only for the final placement. In ASML’s design, each die sits on its own holder from the moment it leaves the carrier and stays there until the bonding step. The process stations are built around the track: the die stops in front of a station, the station cleans, activates or measures it in place, and the die moves on.

Fig 3: SuperTrak CONVEYANCE track (HORIZON10 platform)

Source: Company website

Now let’s go through the process step by step. For illustration, we took the patent drawing and labelled every station in Fig 4. A gripper takes the die off its carrier and sets it on an empty substrate holder (206), which carries it along the track (202). The die stops in turn at the inspection, cleaning, plasma activation, topographic inspection, leveling, coarse alignment, position adjustment and fine alignment stations (210 to 226). At each stop the die stays on its holder, and the station does its work. Note the reversal here: in the patent it is the station’s sensor or head that moves in to reach the die, not the die that is carried into the tool as in today’s flow. After these steps the die arrives at the bonding station (228).

This bonding station deserves a closer look. The first thing to note is that the final placement is done by the holder (206) and the chuck (234), not the track: SuperTrak’s spec sheet puts the shuttles’ repeatability at about 10um, nowhere near the 50nm placement accuracy the bond needs. Next, zoom in on the right-hand part of Fig 4 for the bonding work. The chuck lifts the bottom wafer to within tens of microns of the die, the holder’s actuators adjust the die’s position, the die is pressed onto the wafer to form the bond, and the holder releases it. The empty holder then starts another loop.

Fig 4: Patent WO2025201788A1 Fig. 2 (left) and Fig. 6 side view (right), SA annotated

Source: ASML; SemiAnalysis Wafer Fabrication Equipment Model

This design has three major benefits:

First, independent shuttles lift the utilization of every station. The number of stations per step can be set from that step’s cycle time: a three-second cleaning step gets three stations, and a 1.2-second bond step gets one, so every station stays busy. The patent asks for exactly this, to “occupy all process stations simultaneously”.

Second, capacity is added by adding a station rather than a second tool, as long as the equipment chamber has room for it. That saves duplicate tooling and budget.

Last but not least, the design suits heterogeneous integration: a new die type is a new software recipe, so mixed chiplets could share one line.

That said, the track brings its own challenges around the non-bonding stations and the cleanliness of the whole loop. We come back to them later.

How ASML May Gain an Edge

Before that, let’s look at how ASML could bring its existing strengths to the HB tool.

First, a reminder of why alignment accuracy and cleanliness matter so much in hybrid bonding. As we elaborated before, a single 1um particle between two full-thickness wafers can unbond an area around 10mm across, resulting in a defective bond (Fig 5). So particle control is extremely important for HB. Alignment matters for two reasons: placement accuracy sets bond yield directly, and alignment time weighs far more on D2W throughput than on W2W because it compounds with the die count. These two hurdles are what make hybrid bonding almost a front-end process.

Fig 5: A small particle opens a void and an open contact in hybrid bonding

Source: SemiAnalysis

We believe ASML can tackle both with decades of experience solving the same two problems in lithography.

For alignment, we believe ASML can reuse its wafer-alignment sensor technology. Based on its patent filings, the bonding system puts a tiny diffraction grating on each die and reads the die’s position from the light the grating scatters. The principle is the same as ORION, the wafer-alignment sensor on every new TWINSCAN since 2017. ORION was built to read marks that CMP and etch have tilted or deformed (Fig 6), and a mark on a bonding die goes through the same CMP and plasma activation as the copper pads next to it, so it carries the same asymmetry. We believe the same sensor family can serve the D2W bonder.

Fig 6: A symmetric and a tilted alignment mark. Process steps shift the apparent center; ORION’s multi-color readout was built to see through that

Source: ASML, Aligning lithography to the nanometer, Dec-2021

Its sensing and motion capabilities also provide it with more room in total alignment error budget. Its NXT:2100i holds 0.9nm overlay on a single machine. And ASML can already measure the distortion of a W2W bonded wafer and correct the next lithography layer for it, bringing post-bond overlay from 40-60nm down to below 4nm (Fig 7). These are litho numbers, not bond results. But they mean ASML’s sensor and stage errors sit more than an order of magnitude below the 50nm a bonder needs. Higher stage precision leaves more room in a bonder’s error budget for the other factors: die shift on the holder, thermal drift and movement during contact.

Fig 7: ASML metrology plus scanner correction brings post-bond overlay to 2.5-3.9nm

Source: ASML Investor Day, 14-Nov-2024

Prodrive Technologies and VDL ETG are the likely component suppliers to help ASML’s bonder move fast on motion. Prodrive makes the linear motors and drives for ASML’s EUV maglev stages and has also built Hanwha Semitech’s SHB2 Nano bonder. VDL ETG builds ASML’s frames and modules (Fig 8).

Fig 8: Prodrive’s linear motor (left) and VDL ETG’s wafer handler module (right)

Source: Company websites

On cleanliness, the requirements for ASML’s EUV systems are more strict as a bonder’s. Few tool makers have fought particles at the nanometer scale for as long as ASML. EUV scanners move reticles and wafers in vacuum and keep reticle particle adders at a level “compatible with HVM requirements for sub-10nm node lithography”. We believe ASML has thought about the bonding surfaces too: its bonding apparatus filing adds a tool that cleans the substrate supports and a sensor that watches their flatness, particle load and distortion.

That said, contaminants in a bonder differ from those in a scanner. Most are dicing debris on die edges and particles landing on freshly activated, hydrophilic surfaces. We look for more detail on how ASML handles these bonder-specific contaminants in future filings.

After the paywall we discuss the challenges ASML needs to tackle to build a working tool, and how its tool will impact on Besi’s business prospects.

Open Questions ASML Needs to Solve

Beijing Will Not Pace the Frontier: China’s Speed-First AI Safety Regime

2026-10-08T09:46:05+00:00

AI safety is on fire. On 12 September 2026, Dario Amodei published an essay arguing that frontier labs must deliberately slow the pace at which they improve model capabilities. China was central to his geopolitical argument: export controls and tighter security could preserve America’s lead, creating room to slow down and leverage for a future agreement with Beijing. Within a day, Sam Altman and Elon Musk supported Dario’s call. However, President Trump scoffed. On 14 September 2026, he wrote that fears of AI destroying humanity are “a HOAX” and that “the only one that is happy about it is China.” Beijing scoffed too, with the state-run Global Times calling Dario’s essay a “Cold War playbook.”

The debate then moved onto the diplomatic calendar. On 20 September 2026, Treasury Secretary Scott Bessent and Vice Premier He Lifeng held the first US-China AI dialogue. On 23 September 2026, President Xi Jinping began his state visit to the US. The Trump-Xi summit formalized the dialogue as the “US-China Super Intelligence Dialogue,” using the name Trump had unveiled at the UN the day before Xi arrived. The summit also promised a channel for AI incidents. On 29 September 2026, Trump and six executives signed the White House Accord on Super Intelligence, a one-page voluntary pledge to self-police that Trump called “morally binding.”

China sits on both sides of the American argument. For President Trump, competition with China is a reason not to slow down. For Dario, preserving America’s lead over China is what makes slowing down possible. Nobody has asked what China is actually doing. So, we did. What do the Chinese government, frontier labs, and AI experts say and do about AI safety? And what should American labs, investors, and policymakers conclude from it?

What Beijing Says and Actually Does

Beijing’s stated position is the most safety-forward of any major AI power outside Europe. Xi Jinping told the Politburo in April 2025 that AI brings “unprecedented risks and challenges” and ordered systems for technical monitoring, risk early-warning and emergency response. At the 2026 World AI Conference in Shanghai, Xi said the world must “ensure AI always remains under human control” and that the faster AI advances, “the more promptly the measures to prevent loss of control must be improved.” China signed the Bletchley Declaration and the Paris and New Delhi statements, co-sponsored the UN resolutions, founded a World AI Cooperation Organization in Shanghai, and publishes bilingual AI Safety Governance Frameworks.

The newest text goes further. The AI Safety Governance Framework 3.0, released by TC260 under Cyberspace Administration of China (CAC) on 14 September 2026, warns that AI has shown a self-accelerating trend of autonomous learning and recursive self-improvement, and calls for “attention and vigilance” for potential AI evolution exceeding human prediction and control. The Framework 3.0 also introduces a new model-risk category, “behavior deviating from expectations,” covering models that acquire system privileges and external resources without authorization, bypass safety protections, deceive evaluators, hide their true capabilities, and refuse user instructions, illustrated with a boxed case study of models disabling shutdown scripts, sandbagging under evaluation, and breaking out of isolation into real systems. Its list of derivative risks goes further still; past behaviors already observed to a hypothetical AI that becomes self-aware, seeks power of its own, and competes with humanity for control. On paper, this is the most explicit official Chinese text yet on frontier risk, and its vocabulary overlaps strikingly with Dario’s essay Beijing denounced that same week.

However, we think China’s real approach to AI safety is speed-based, not safety-based. Though Beijing recognizes frontier AI risks, it has been prioritizing rapid development. The Framework 3.0 opens its principles with “promoting AI innovation and development as the first priority.” The comprehensive AI Law promised in the 2023 and 2024 legislative plans was shelved in 2025, four months after the DeepSeek moment. The State Council’s AI+ Action Plan targets 70% penetration of agents and intelligent terminals by 2027 and 90% by 2030, aiming for a rapid deployment of AI across all sectors and industries.

We think the Chinese government takes an open position on frontier models but tightens the applications and regulates outputs. Since 2025, China has issued a number of mandates, including mandatory content labeling (Sep 2025), Cybersecurity Law Article 20 (Jan 2026), the minors’ information classification rules (March 2026), MIIT’s AI ethics-review regime (March 2026), the policy-level agent rules (May 2026), the world’s first AI-companion rules (July 2026), the network-data risk assessment rules (August 2026), and MPS inspection powers (October 2026). Every one of these governs what AI says and what AI does to people. There is no set of frontier-risk duties triggered by training compute or model capability. The result is a safety regime stricter than any in the world on content and on process for public-facing services, and looser than the EU (systemic-risk duties from 10²⁵ FLOP) and California (SB 53: frontier frameworks and 15-day incident reporting from 10²⁶ operations) on the frontier.

Source: SemiAnalysis

The week Framework 3.0 came out shows the pattern in real time. Framework 3.0 added two new application-layer risk categories, agents and embodied intelligence, plus an appendix on managing agent risk. On 15 September, TC260 issued four AI application security guides, with a general one and three specific ones for education, health care and broadcasting, respectively. On 18 September it opened comments on a draft Agent System Development Security Guide (due 2 October) that walks developers through identity, least privilege, tool verification and human-approval nodes. The same day the CAC published a draft State Council regulation on minors’ internet use (comments due 17 October) that bars offering minors “virtual relatives, virtual companions and other virtual intimate-relationship services” and bars livestreaming under the age of 16. Seven texts in five days. Everyone is about how an agent or an application is built, deployed, and to whom it is offered. The gap at the frontier has not moved at all.

Compliance therefore tells us nothing about frontier safety. A Chinese lab can satisfy every rule on this list without ever running a dangerous-capability evaluation. Thus, to know what the labs do at the frontier, we must look at what they evaluate, what they disclose, and whether any of them change what they ship.

China’s frontier AI labs: what they do

Nobody in China’s AI industry is slowing down, and nobody is asking them to. The frontier labs are in a release and price war.

We conduct a systematic analysis to clearly understand what Chinese labs do about safety. We constructed an original dataset that includes all identifiable model release by the 9 leading Chinese developers from 2021 to 15 September 2026. These 9 companies include 4 hyperscalers (ByteDance, Alibaba, Tencent, Baidu) and 5 startups (DeepSeek, MoonShot, Zhipu Z.ai, MiniMax, and StepFun). The dataset includes 857 counted releases (741 product models and 116 research models), each with a first-public date, weight status, license, and source. We then check each against the developer’s model cards, release notes, and technical reports for a published safety-evaluation result that can be matched to that release. Our standard is strict: a result means a quantitative or substantive finding on harmful output, jailbreaks, toxicity, privacy, refusal, or dangerous capability, tied to the named model. Statements that a model was “safety-trained” or “evaluated” do not count. A flagship model’s result is not extrapolated to other sizes or snapshots. “Not found” is bounded to the materials checked and does not mean “not tested.” Companies also name variants at different granularity (Alibaba’s 238 releases count every Qwen size and snapshot), so per-company rates are indicative rather than a ranking.

The results are striking. Of the 857 releases, only 31—or 3.6%—have ever been accompanied by a published safety result from the developer. Just 9 of those—1.1% of the total—had the result available at or before the model was released. Another 16 were documented only afterwards, with a median lag of 42 days and a maximum of 349 days (DeepSeek-R1, whose verified safety appendix dates to January 2026). For the remaining 6, the results exist, but their timing or their match to the specific model could not be established. A further 10 releases carry a claim of evaluation with no figures, and 3 are known only from press or investor accounts, with no developer documentation we could retrieve. The remaining 813 releases, 94.9% of everything the 9 companies have shipped, have no safety disclosure at all.

We plot the release cadence against disclosure. The bars climb from 3 releases in Q1 2023 to 90 in Q3 2024 and 101 in Q3 2025, a thirty-fold rise in which open-weight releases (dark blue) make up more than half of every quarter since mid-2023. The two lines do not climb with them. Releases with any safety result never exceed 7 in a quarter. Releases with a result available at launch never exceed 3 and sit at zero in 9 of the 15 quarters. No policy milestone in the period (such as the 2023 Interim Measures, the Frameworks 1.0 and 2.0, or the labelling rules) leaves a mark on either line.

Source: SemiAnalysis

We then break the same 857 releases down by developer, and find the trickle is not concentrated anywhere. Alibaba, the largest publisher with 238 releases, has 7 with any result and 3 at launch. Tencent has 1 in 133. ByteDance’s largely closed Doubao fleet 2 in 120. Baidu 1 in 49. Zhipu (Z.ai) is the only developer with a result every year since 2022. DeepSeek documented V3 at launch and none of R1, V3.1 or the V4 family. The start-ups do better than the giants, 20 of 317 releases (6.3%) against 11 of 540 (2%), but no company does it as a matter of routine. Every 2026 frontier release, from GLM-5.2, the Kimi K2.5-K3 line, and the DeepSeek V4 family to Qwen3.7-Max, Qwen3.8-Max, and Doubao Seed 2.1, was undocumented at launch except GLM-5.3, with a capability evaluation note.

Source: SemiAnalysis

We then turn to those qualified documents. We find 18 reported harmful-output or refusal results and 7 reported jailbreak-resistance results. 9 documents were concerned with code or cyber security, but 7 of those were secure-code-generation benchmarks for coding models. Only 3 touched cyber-offence or biological risk, namely: Zhipu’s GLM-5.2 responsibility note, GLM-5.3 cyber-capability note, and Moonshot’s Kimi K2 technical report. None came from the four hyperscalers.

Of the nine at-launch disclosures, four are behavioral safety tests (Qwen2-72B-Instruct, MiniMax-Text-01, Seed-OSS-36B-Instruct, and DeepSeek-V3). The rest are speech-safety scores, secure-code benchmarks, and one capability evaluation. Reasoning models, the fastest-advancing category, are 93% without any published results. Moreover, no Chinese frontier text model has shipped with a dangerous-capability evaluation across the domains the IDAIS statements name. GLM-5.3’s cyber note is the closest anyone has come.

China’s Frontier AI Labs: what they say

The release census tells us what the nine labs do. We further explore what their leaders say. We compiled every public statement on AI safety by the founders, CEOs, chief scientists, and other senior figures of the nine labs between January 2023 and 21 September 2026: speeches, interviews, internal letters, posts, signatures on collective statements, and company documents such as prospectuses and release notes. We built the inventory in two independent sweeps, one from Chinese primary sources and one from English and institutional sources, then reconciled the two and verified every item against its source. In total, we have 65 items.

The picture is sparse and lopsided. Across 9 labs and almost 4 years we found 15 statements by founders, CEOs or chief scientists that engage in frontier safety. Nine of the 15 express concerns and propose something. Six of those nine come from Zhipu alone.

Zhipu (Z.ai) is the only lab where safety is a named strategic commitment. CEO Zhang Peng signed the IDAIS Beijing red lines in March 2024, signed the Seoul Frontier AI Safety Commitments in May 2024 (the only Chinese company to do so at the time), told the BAAI conference in June that “in AI safety, defense is always harder than attack,” and called for “super-intelligence and super-alignment” at WAIC in July. Founder Tang Jie signed the IDAIS Venice statement and the 2025 Global Call for AI Red Lines, and his internal letter of 11 July 2026 makes “ultimate safety governance” one of four strategic engines, saying that “the stronger the capability, the more robust the safety constraints must be.” In his direct response to Anthropic withholding Mythos, Tang Jie said, “the realization of superintelligence and research on super-alignment must advance in step” and that safety “is no longer an accessory but the fundamental precondition for a technology to survive and be permitted to be used.”

Robin Li of Baidu had the clearest loss-of-control statement of any Chinese AI CEO. At the Zhongguancun Forum on 26 May 2023, Robin said that AI “could indeed develop in directions unfavorable to humanity” and that “preventing loss of control requires the nations with advanced AI to cooperate and set rules.” Yang Zhilin of Moonshot, acknowledged the risk. He said in June 2024 that AI safety “is something we must prepare for in advance” and floated the idea of an AI “constitution.” But Yang chose acceleration. In January 2026, he said that although “the risks of AGI/ASI may be considerable, we should not give up its development, because giving up development means giving up the pursuit of the ceiling of human civilization.”

At five of the nine labs, the founder or CEO has said nothing. Liang Wenfeng of DeepSeek has no public statement on safety or regulations in four years. Alibaba’s Eddie Wu gave a 6,000-word keynote on the road to superintelligence in September 2025 that does not contain the word safety or risk once and closed with “we are full of optimism.” Tencent’s Pony Ma and Dowson Tong speak of safety only as data protection and permission management. The CEOs of ByteDance and StepFun have said nothing, though a StepFun co-founder and ByteDance’s head of research each signed an IDAIS statement. Where safety literacy exists in these companies, it sits with the lieutenants, not the commanders. In September 2025, Alibaba’s security vice-president Xue Hui acknowledged that AI safety capability “currently lags the development of model capability” and said research must include “new risks such as AI self-replication.” A DeepSeek researcher, Chen Deli, said at Wuzhen that “in the long run the risks may be greater.”

Source: SemiAnalysis

What China’s AI experts say: a text analysis of 102 documents and 32 voices

Who are the leading AI thinkers shaping China’s AI-safety discourse? What do they mean by safety? Who has the pen?

We constructed another original dataset coding 102 expert and official texts from 2023 to September 2026, with 51 from official, Party-theory and legal venues and 51 from technical and scientific venues. The sample is purposive, not random: we set quotas by venue and filled them with the most prominent texts we could find on AI safety and governance by identifiable Chinese authors or institutions, each with a verifiable source. We read two sources in full. One is every expert commentary on a major AI rule that the CAC has published since 2023, and the other is all five IDAIS consensus statements. The rest we sampled. We selected the most prominent pieces from Party journals, Legal Daily, model-law drafts, academic articles, ministers’ speeches, scientists’ talks, lab frameworks, and arXiv papers, with a rough quota for each so that no venue dominates. We then coded each text on six themes (Figure 4 below), three stances (on binding frontier rules, on a comprehensive AI law, and on development versus safety), and the author’s professional background. We then map 32 of the most influential individuals by professional background and by the channels through which they influence policy.

The short answer: China’s technical scientists and its legal-policy scholars are answering two different questions, and the state had answered one of them before either group spoke.

To elaborate, first, we find that technical scientists and legal-policy experts are talking about different things when they say 安全 (the Chinese word for both safety and security). Frontier or loss-of-control risk appears in 86% of texts by technical scientists (36 of 42), the longest bar in the figure, but in only 22% of texts by law scholars, 23% by serving officials, and 44% by public-policy scholars. In addition, frontier risk is the primary theme in 25 of technical scientists’ 42 texts, and in just one text in each of the other three groups.

By contrast, development-first and competition appears in every one of the officials’ texts, in 91% of the texts by law scholars and 89% of those by public-policy scholars, but in only 45% of the texts by technical scientists. The bottom three rows (Figure 4) show the same alignment. Legal-institutional design is universal among legal scholars and public-policy scholars (100%) and present in 43% of technical experts. Rights, ethics, and consumer harms appear in 91% of law scholars’ texts and 48% of technical ones. International governance and export controls in 91% of legal texts and 55% of technical ones.

The two camps are different people. The scientists are senior, internationally embedded, and at technical parity with their Western peers. They write frameworks, sign consensus statements, and brief officials. The lawyers and public-policy scholars sit at the Chinese Academy of Social Sciences (CASS), the China University of Political Science and Law (CUPL), Tsinghua’s School of Public Policy, and the ministries’ expert pools. They write the expert readings that accompany every CAC rule and draft the model laws the Ministry of Justice consults.

The two camps are asking different questions. Scientists ask what a frontier model can do. Their answer is loss of control and misuse. Thus, their prescriptions reach the model before it ships. They have been advocating for registration of training runs above compute thresholds, safety cases before release, and shutdown authority. The lawyers, on the other hand, ask how the state should govern an industry. Their answer is order, rights, and growth, and thus, their prescriptions attach where the industry meets the public. For example, they call for regulating services not technology, rules at the application layer, and legislation in small steps.

Source: SemiAnalysis

Second, we find that only scientists ever put safety ahead of development. We sort the 102 texts by author background. We find that 42 are written by people with a technical background. Of those 42, 22 (52%) argue that safety must precede or gate development. Only five are development-first. By contrast, if we look at the non-technical group, not a single text by a legal or public-policy scholar or official takes the safety-first position. 8 of the 23 law texts are development-first outright. The rest use the balanced official formula, which is “coordinating development and safety.”

Third, we find that the legal scholars are actually following the top leadership. They took its words, applied them to AI, and made them their AI doctrine. The legal scholars’ signature phrase is “failing to develop is the greatest insecurity.” The phrase in fact descends from Xi Jinping’s statement at the 2014 CICA Summit that “development is the greatest security.” Premier Li Qiang used the negative form verbatim to German business leaders in Berlin on 20 June 2023, arguing against “de-risking,” and the CCP’s official People’s Daily made it a headline on 5 July. Eight days later, on the day the Generative AI Measures were published, law professor Xu Ke carried it into the AI debate in a CAC-published expert reading. Law professor Zhang Linghan repeated it in 2024 and 2025. In 2025, Zhi Zhenfeng of the Chinese Academy of Social Sciences (CASS) called it “a basic consensus.” The lawyers’ other slogans have the same history. For example, Xi called for “small, fast, flexible” legislation in November 2020. Zhang Linghan applied the phrase to AI in the People’s Daily in May 2025.

Fourth, we find that every demand for binding frontier duties went nowhere, regardless of experts’ professional backgrounds. Thirteen of the 102 texts call for binding obligations on frontier developers. Six are technical, the five IDAIS consensus statements co-signed by Chinese scientists with Western counterparts and an April 2026 National Science Review editorial by Zeng Yi, Huang Tiejun, Jiang Yugang and Poo Mu-ming. They demand (1) registration of training runs above compute thresholds, (2) safety cases submitted to regulators before release, (3) independent third-party audits, (4) shutdown of all copies of a model that crosses a red-line, and (5) legal mechanisms to enforce all of these. However, none of these has yet been adopted in any binding Chinese instrument. The other seven are by law or public-policy scholars. Most are model-law drafts, which propose binding duties into a comprehensive AI law. None of the 13 got what it asked for.

The sad truth is neither camp decides anything. The top leadership does. Figure 5 sorts the 32 most influential voices by two facts: whether they demand binding frontier rules, and whether they hold a domestic seat where rules are written. Ten demand and hold no seat. Eight hold a seat and make no such demand. Ten do neither. Only four hold a seat and demand. Zeng Yi and Zhang Linghan in their own voice, Xue Lan and Fu Ying by IDAIS signature only. Thus, the demand comes overwhelmingly from outside the rooms where rules are written. And the seat did not help those who had it. Zeng Yi sits on the national AI governance committee, and his April 2026 call for legal enforcement went nowhere. Zhou Bowen chairs TC260’s AI safety working group, the one body that could turn loss-of-control language into a mandatory standard and writes voluntary frameworks. Xue Lan chairs the governance committee, signs the IDAIS statements, and prescribes agile governance. Zhang Linghan drafts the scholars’ AI law and asks for licensing of high-risk applications, not for capability thresholds. The scientists are not heeded because they contradict a decision already made. Their answer is that the danger is in the model, and the controls must reach training, and that does not fit a decision that development comes first. China’s AI-safety debate was settled by the top leadership’s preference for development over safety, not by anything the experts said.

Source: SemiAnalysis

The supply-side ecosystem exists but no power

It would be wrong to conclude that China has no frontier-safety capacity. The Shanghai AI Lab (SAIL), a state-backed lab whose director Zhou Bowen chairs TC260’s AI safety working group, published a Frontier AI Risk Management Framework in July 2025 and has revised it twice, most recently as version 2.0 at WAIC in July 2026. The SAIL framework draws red and yellow lines across five risk domains, 13 red lines in all. It calls for deployment suspension in the red zone and suggests evaluation triggers at 4× and 10× jumps in effective compute. Its Framework also treats open-weight release as the highest-risk deployment setting, asking developers to measure how cheaply safety training can be stripped by fine-tuning before they publish weights. SAIL has also run some twenty Chinese and Western models through UK AISI’s Inspect evaluation tools. Besides SAIL, there is CAICT (the China Academy of Information and Communications Technology), MIIT’s in-house research institute on AI. Its evaluations are the closest thing China has to an official safety benchmark. CAICT released its AI Safety Benchmark 2.0 in February 2026, adding deception, loss of control, dangerous-domain misuse, and agent risk to what it tests. In addition, CnAISDA (China AI Safety and Development Association), a network of eight institutions launched in Paris in February 2025, represents China abroad as the Chinese counterpart to the government AI safety institutes that the UK, the US, Japan, and others set up after Bletchley.

However, none of these institutions have real power. CnAISDA has no staff, no budget, and no mandate to test anyone’s models. SAIL has capacity to evaluate frontier models but no right to demand access to its rivals’ models. CAICT tests models on a voluntary basis and publishes the results with models’ names removed. TC260 writes standards that are recommended unless the state makes them mandatory, and on frontier risk, the state has not done so. In short, the ecosystem thus far produces frameworks that no one is obliged to follow and evaluations that no developer is obliged to submit to. As the technical scientists themselves wrote in April 2026, “the progress of AI governance is alarmingly slow,” and relying on “the self-control of AI developers is an illusion.” The state’s answer, so far, is to rely on exactly that.

Geopolitics complicated AI safety progress

The intensified geopolitical environment and the US-China AI race have pushed China toward the speed end of the trade-off between moving fast and managing frontier risk. The US-China AI race has also changed what AI 安全 means in Chinese official texts, from safety to security. The same Framework 3.0 that borrows the frontier vocabulary also lists export controls as a supply-chain security risk. It warns that “certain countries use technological monopoly and unilateral coercive measures such as export controls to erect development barriers and maliciously sever the global AI supply chain.” The Framework’s appendix of trustworthy-AI principles places respect for national sovereignty second only to human control and forbids coercing countries to “pick sides.”

The US-China AI race has also pushed China’s own experts to doubt America’s real intentions for emphasizing safety. Zhu Songchun, Director of the Beijing Institute for General AI, expressed at the WAIC 2026 in Shanghai that the “human-extinction narrative” was one of three technical misjudgments packaged for capital, alongside “LLM equals AGI” and the scaling law. His example was Anthropic declaring its Mythos model too dangerous to release while raising money at a trillion-dollar valuation. In his own words, “this practice of selling danger and marketing through fear is what the industry calls ‘Oppenheimer-style marketing’.” Director Zhu added that “this is not to say AI safety and regulation are unimportant, but exaggerating the problem to the level of human extinction has the logic of capital behind it,” and called on Chinese media to stop amplifying American narratives.

Similarly, Zhi Zhenfeng, a law professor at the CASS Institute of Law, argued in December 2025 that the debate over comprehensive AI legislation among the major powers had taken on the character of “cognitive-domain shaping,” a term borrowed from military doctrine. He claimed that Washington passes almost no binding AI regulation at the federal level and keeps “loosening the reins” at home, while the flood of unenacted congressional bills is misread by Chinese media and scholars as evidence that America is tightening, which “shapes the perception in our [Chinese] academic and public-opinion circles that strong AI regulatory legislation is called for.” The United States, Zhi wrote, is “conducting cognitive-domain shaping by setting the AI legislation agenda, inducing other countries to strengthen regulation and slow their own innovation.” The same article delivers the line that “failing to develop is the greatest insecurity.”

Sadly, geopolitics has also torn the mindset of the very top engineers who build China’s most capable models. Liu Shengyu, who wrote the main attention kernel for DeepSeek V4.1, wrote on 13 September that he has “no choice but to join the cruel arms race,” because he “especially” does not want Anthropic controlling AGI. “To put it dramatically,” Liu stated, “the severity of that [Anthropic controlling AGI] would be no less than letting Hitler get the atomic bomb before the Allies.” He joined DeepSeek largely because it researches “strong, fast, and universal AI, and opens it,” with a hope that it “can pull the world back a little” from the dystopian end.

Ironically, only 6 days after asking the industry to slow down, Dario’s Anthropic was reported to be weighing a new model before its IPO, with cheap open-weight rivals “especially in China” squeezing the economics. 4 days later, Anthropic shipped Claude Opus 5.5, which Anthropic itself calls “our first release since we called for pacing the frontier.” The launch post now frames pacing as a way of “remaining competitive with China.”

Nobody paces alone, not even the man who wrote the essay.

What does all this mean for American labs and investors? What did the Trump-Xi Summit say about pacing? What are the dated tests over the next six months that would change our mind? We explore these below.

The Trump-Xi Summit

Anthropic Subscriptions Offer 5x+ More Value Than OpenAI

2026-10-05T12:01:09+00:00

Subscription plans are still the primary way consumers and small businesses pay for AI. These plans are highly subsidized—as we previously explained in June—but can still make economic sense as powerful customer acquisition and marketing tools. For example, the goodwill engendered by OpenAI’s generous resets is partially responsible for the recent surge in Codex adoption and has forced Anthropic to repeatedly walk back planned subscription nerfs to avoid getting clobbered in the court of public opinion.

Furthermore, because subscription plans are so heavily subsidized, they can also have a large impact on margins and revenue per MW despite only being a small portion of total revenue. Consider the following rough numbers for Anthropic.

Source: SemiAnalysis Tokenomics Model

Despite being just 10% of overall revenue, subscriptions can take up over 40% of inference compute and lower blended revenue per MW by ~$36M. Subscriptions are even more important for OpenAI, as they make up a larger portion of their total revenue. For specific numbers, see our Tokenomics Model.

In other words, if you want to accurately model AI lab financials, you need to understand their subscription limits.

So how do limits actually work? The way to think about subscriptions is that your monthly payment grants you some numbers of “credits”. Each (model, token type) combo consumes a different amount of credits. Because credit cost ratios can differ dramatically from API price ratios, the “value” of the same plan changes depending on what model and workload you’re running. Put differently, it doesn’t make sense to say a plan is worth $X in isolation—you need to consider the full (plan, model, workload) tuple.

Here’s how the API-equivalent value of the same $200/month Claude plan changes depending on the model and workload.

Source: SemiAnalysis Tokenomics Model

A single static snapshot isn’t enough either. Labs publicly change their limits all the time with promos and new model releases. They can also silently change limits whenever they want by tweaking credit costs. Ideally, you’d re-check the cost of each (plan, model, token type) daily so you can surface any changes in real-time.

This is exactly what SemiAnalysis has done with our new Subscriptions Dashboard available exclusively to Tokenomics Model subscribers. Besides every OpenAI and Anthropic subscription, we also track Meta, SpaceXAI, Cursor, Cognition, Z.ai, MiniMax, and Moonshot.

A screenshot of our dashboard showing a small subset of the available data. Source: SemiAnalysis Tokenomics Model

New providers, plans, and models will be added as soon as they’re released. The rest of the article will give an overview of everyone’s current limits. All token and dollar amounts below assume agentic usage unless otherwise specified.

Methodology

Tokens are generally priced per MTok (million tokens) across the following types:

Input: Fresh tokens added to the LLM’s context window that aren’t cached
Cache write: Input tokens that are cached for multi-turn conversations. Generally slightly more expensive than regular input tokens.
Cache read: Tokens from previous turns that are already cached in the conversation. Generally extremely cheap compared to uncached tokens.
Output: Tokens generated by the model. The most expensive token type.

Additionally, some models charge more for tokens above a certain context window. Those models typically compact before that expensive context window is reached, so we also keep our measurements below it.

Source: OpenAI

Subscription plans don’t expose this fine grained pricing. Instead, they provide a simple 0 to 100% usage meter across 5-hour and 7-day windows, and sometimes a separate meter for a model like Fable. Subscription tiers are then differentiated by usage multipliers relative only to the provider’s other plans. OpenAI for example used to advertise “Expanded Codex usage” in their $20 Plus plan, “5x more usage than Plus” in the $100 Pro plan, and “20x more usage” in the $200 Pro plan until they cut their $200 plan usage in half and removed all relative usage from their pricing page.

Screenshot of Claude Code meters. Source: Claude Code CLI

Setup

We compute the subscription-rate of each (plan, model, token type) triple by running experiments that isolate one token type at a time and watching how far the model provider’s meter moves. An experiment is some number of repeated calls using a specific prompt that maximizes one token type while minimizing all the others.

Source: SemiAnalysis Tokenomics Model

Input, cache writes, and cache reads share the same prompt template. We use a portion of War and Peace since some models will refuse to respond if given large blocks of gibberish. For the input token experiments, we use a random tag on every call to ensure nothing gets cached. The cache write experiments run the same way but with the prompt marked for caching, so each new tag forces a new cache entry. The cache read experiments use a fixed tag, so the first call writes the cache and every repeat reads it.

For the output token experiments, we used a technical essay to force long outputs because models will refuse mechanical prompts like “repeat SemiAnalysis 100,000 times”.

Computing the Results

For every call we record two things: how many tokens of each type the provider billed, and what the usage meter read. Turning those into a price takes three steps.

1. Measure the rate of each token type

Providers generally report usage on a meter that moves in fixed amounts. This could be a whole percentage point, a credit, a cent etc... A single request often does not move the meter at all, so the cost of one request cannot be measured directly.

Instead, we measure in steps. As requests run, we keep a running total of the tokens used. Each time the meter rises, we store that total. The tokens used between two meter moves make up one step, which is the cost of moving the meter by one unit.

We drop two partial steps. When a run starts, the meter is already part of the way to its next move, so the tokens before the first move are less than a full step and would make the rate look higher than it is. The tokens after the last move never finish a step, so we drop those as well.

To get a rate, we add up how far the meter moved over the complete steps and divide by the tokens used in them. Providers charge differently for input, output, and cached tokens, so we calculate a separate rate for each.

Source: SemiAnalysis Tokenomics Model

Caveats:

Because we see the meter only once per request, a single step can be off by up to one request. These errors don’t add up with more consecutive steps, so we track a range instead of a single number, and the range gets narrower the more steps we count. We keep adding steps until the range is within ±5%, then report the rate across all of them.
No request contains only one type of token. Some plans, for example, require a set of instructions on every request. We subtract that extra amount using the prices we measured for the other types.
Reading from the cache costs almost nothing on some plans and may not move the meter at all. If the meter does not move after 500M cache read tokens, we assume that cache reads are free.

2. Convert rates into tokens per window and per month

We read every meter a plan shows after every request. This could be a 5-hour limit, a weekly limit, and for models like Fable, its own weekly limit. So each token type gets a price on each meter. This is how much of that limit a million tokens uses. From each price we work out how many tokens fit e.g. if a million tokens use 5% of a limit, the whole limit holds 20 million.

Source: SemiAnalysis Tokenomics Model

The 5-hour limit resets many times within a week, so a month is capped by the weekly limits.

3. Pricing

Lastly, we compute the API-equivalent value by converting the limit % consumed by a million tokens of each type into a dollar amount. For example, if you consume 1% of a $200 plan’s monthly limit, that’s $2.

Then, we assume a workload shape, and compute how many total tokens you can get from each plan given this ratio. For the agentic workload, we use our own usage ratios in September as reported in our Tokenomics Model.

Finally, we multiply by the blended price per MTok at API prices to get the API-equivalent value.

Source: SemiAnalysis Tokenomics Model

Catching a Provider A/B Test

While refining our methodology, we ran into a very confusing situation where just 1 of 3 of the same subscription we tested for a particular provider had ~20% lower limits than the other 2. This unlucky account happened to be significantly older as well, and we were worried that the provider had some cursed setup where they changed subscription limits depending on the age of the account.

Fortunately, after reaching out to the provider, we were able to confirm this was not the case and that our unlucky account was part of an “extremely tiny” A/B test they were running on limits. Additionally, they wanted to emphasize that they “didn’t just decrease limits wholesale” and were instead testing “how to better balance when people hit limits”.

This conclusion is interesting for two reasons:

It proves providers can silently change subscription limits at any time
Our method is sensitive enough to detect these subtle changes!

OpenAI vs Anthropic

The biggest question in AI subscription land is how Anthropic’s limits compare to OpenAI’s. Note that OpenAI just made some massive changes to their subscriptions last week, halving the API-equivalent value for the $200 plan and introducing a new $500 tier. The following numbers describe the current state of OAI vs ANT. Afterwards, we will examine OpenAI’s recent changes in depth.

Source: SemiAnalysis Tokenomics Model

Limits are quite similar across the board for GPT-6 Astra vs Fable 5.1, though remember that Fable can only be used for 50% of your limit. In other words, your $200 ANT plan would still have 50% left after consuming $2,485 worth of Fable 5.1, whereas the equivalent OAI plan would be fully exhausted after $2,897 of Astra.

This extra 50% starts becoming very relevant when we compare Opus 5.5 vs GPT 6.1 Sol.

Source: SemiAnalysis Tokenomics Model

At this mid-tier of models—which both companies market as the intended daily driver for most users—Anthropic is an overwhelmingly better deal, offering ~5x the API-equivalent value across the board. You could argue that this is unfair for OAI because 6.1 Sol is much cheaper per token than Opus 5.5, but the gap is still massive even if you switch to comparing the number of tokens.

Source: SemiAnalysis Tokenomics Model

Overall, we believe API-equivalent value is the single best number for capturing the “value” of a subscription plan, but there are times where raw token volumes can be more appropriate. For example, if a model is a particularly bad (or good) deal at API prices, then the API-equivalent value will of course be misleading.

Token efficiency is also an extremely relevant factor, but the industry unfortunately lacks reliable data here. Many people like to cite this chart from Artificial Analysis, but we do not believe the benchmark tasks in the AA Intelligence Index are at all representative of real work people do with LLMs.

OpenAI’s new $500 plan and massive limit reduction

Our tracking confirmed the 50% value reduction for OpenAI’s $200 plan announced by Tibo. $200 plans purchased before the cut will keep the old higher limits until October 29th, but newly purchased plans immediately start with the lower limits.

Source: SemiAnalysis Tokenomics Model

As you can see, OpenAI specifically halved the number of tokens you get for each model tier. This caused the API-equivalent value for Sol class models to drop by over 50% because OAI also cut cached input token pricing for 6.1 Sol.

The new $500 plan only offers 21% more Astra than the old $200 plan. Interestingly, the API-equivalent value for Sol class models actually decreased due to the aforementioned 6.1 Sol price cut.

Source: SemiAnalysis Tokenomics Model

Of course, 300 TPS Ultrafast was the real headline feature for the $500 plan and we are currently testing its limits. Results will be published to Tokenomics Model subscribers as soon as they’re available. We are also actively testing the limits when using your ChatGPT subscription on third party apps like Devin.

The $200/month plan also used to be significantly more subsidized than OpenAI’s other subscriptions. If we normalize each (plan, model) combo to the API-equivalent value and tokens per dollar, we can see that Pro 100 offered ~2x more per-dollar value than Plus on Astra and Pro 200 was another ~2x on top of Pro 100 across all models.

Source: SemiAnalysis Tokenomics Model

With last week’s cut, Pro 100, 200, and 500 all offer the same tokens per dollar for every model. Plus is also comparable for Sol but offers a relatively worse deal on Astra.

Anthropic, on the other hand, already offered the same per-dollar value for all of their subscription tiers and currently crushes OpenAI.

Source: SemiAnalysis Tokenomics Model

OpenAI used to be lauded by indie developers for their generous subscription limits compared to Anthropic, but this is simply no longer true today. Opus 5.5 on any Claude subscription offers far better value than anything from OpenAI.

The only counter argument for OpenAI is that none of their Pro plans have a 5 hour limit, which makes it easier to consume a higher % of your total monthly limit. However, we don’t think this offsets the ~4x higher API-equivalent value offered by Opus 5.5 on Claude plans.

Do labs adjust subscription limits based on API pricing changes?

We’ve seen a number of significant price cuts recently to Anthropic and OpenAI models.

Fable 5.1 cut cache reads by 75% vs Fable 5
Opus 5.5 cut input and output token pricing by 20% vs Opus 5 and cache reads by 60%.
GPT 6.1 Sol cut cache reads by 50% vs GPT 6 sol. This was after making GPT 6 Sol 60-67% cheaper vs 5.6 Sol.

One important question is whether the labs adjust token limits after these price cuts, such that the API-equivalent value of your subscription stays constant.

We already showed earlier that OpenAI didn’t change Sol token limits on the $200/month plan when they released 6.1 Sol. This caused the API-equivalent value to drop ~30%.

Anthropic also didn’t increase Fable token limits at all with the release of 5.1, but they did up Opus token limits with the 5.5 price cut.

Source: SemiAnalysis Tokenomics Model

However, the ~20% token increase for Max plans and ~50% increase for the Pro plan weren’t enough to fully offset the API price cut.

Two different approaches to increasing subscription margins

As we explained at the start of this article, subscription gross margins are way lower than API and meaningfully reduce revenue per MW for both OpenAI and Anthropic. Both companies are well aware of this fact but have chosen two different strategies for reducing subscriber subsidies.

For a given plan, Anthropic decreases the API-equivalent value for more premium models. The drop off is small between Sonnet 5.5 and Opus 5.5, but becomes very meaningful with Fable 5.1.

Source: SemiAnalysis Tokenomics Model

Reducing the API-equivalent value as you release more powerful models at higher list prices is the more subtle way to stop subsidizing subscriptions. Assuming 100% utilization and 92% API gross margins, maxing out Opus 5.5 vs Fable 5.1 usage corresponds to -369% and 1% gross margins respectively. If we were to use a more realistic average utilization of 20%, then gross margins rise to 6% and 80% respectively.

In other words, Anthropic would likely already have software like margins on their subscription plans if everyone only used Fable. Opus class models will continue to become cheaper to serve over time as Anthropic figures out how to make the model smaller, and new model tiers above Fable will have even lower subscription limits (assuming they’re included at all).

OpenAI, on the other hand, picked the nuclear option of just immediately cutting to Fable-level limits across the board.

Source: SemiAnalysis Tokenomics Model

The downside of this approach is supposed to be public backlash, but OpenAI has somehow managed to more or less avoid this entirely. It’s likely some combination of all the good DevDay PR drowning out the negative news from the day prior and grandfathering existing plans for an additional month.

Chinese subscription plans are still a good deal

Despite being compute poor, the Chinese labs still offer subsidized subscription plans, but the level of subsidization varies dramatically between models.

Source: SemiAnalysis Tokenomics Model

Here’s how they compare to OpenAI and Anthropic after normalizing per dollar.

Source: SemiAnalysis Tokenomics Model

Value per dollar increases as you buy higher tier subscriptions from the Chinese labs. On average, the API-equivalent value per dollar is a little less than the ~12x you get from OpenAI.

Third party plans are worse than first party

The final comparison we’ll consider is OpenAI and Anthropic models on their own first party plans vs third party wrappers like Cursor and Cognition (Devin).

How GLM5.3 Sparse Attention Affects HBM Memory Usage

2026-09-28T11:26:41+00:00

How Sparse Attention Affects DRAM/NAND Memory

How does sparse attention affect the TAM of memory, including HBM and NAND? Sparse attention selects top-k most relevant tokens to attend to, reducing the memory consumption and bandwidth requirements during the core Scaled Dot-Production Attention (SDPA) operation. However, the efficiency improvement doesn’t directly translate to overall memory savings in practice. Concretely, the top-k selection operation typically requires the full context to be in HBM, so sparse attention doesn’t eliminate the memory capacity bottleneck.

Sparse attention throughput is bottlenecked by memory capacity. Source: HiSparse

To overcome this limitation, the SGLang team designed HiSparse, a hierarchical memory system that proactively offloads KV cache entries from device HBM to host DRAM. HiSparse behaves like an LRU (Least Recently Used) cache, where it loads tokens from DRAM to HBM upon top-k selection cache miss, and it offloads tokens from HBM to DRAM based on the LRU eviction policy. To reduce the cache miss latency, HiSparse overlaps KV cache loading of layer N with the execution of layer N-1 (layer-wise overlapping), introduced in HiSparse’s prior work HiCache.

Layer-wise overlapping. Source: CachedAttention

With HiSparse, SGLang greatly boosts throughput at high concurrency and long context scenarios, at the cost of top-k cache miss I/O overhead.

Sparse attention reduces KV cache memory and bandwidth requirements at the SDPA operation, but it does not reduce the overall memory capacity usage. In addition, HiSparse shows that system optimizations can overcome sparse attention’s memory capacity limitations, so sparse attention memory profile alone cannot sufficiently portray the full picture of system KV cache efficiency. To provide a more holistic picture of serving sparse attention models, here we explain the design of Z.ai’s GLM-5 model series, and how they are served in practice.

GLM5.3 Agentic Inference Serving

In our realtime benchmark board InferenceX, GB300 delivers the lowest modeled serving cost in our comparison at a response speed of 150 tokens per second. GB300 handles more tokens per GPU, and its higher hourly cost still leaves it ahead of GB200 on cost efficiency. The GB300 result uses Dynamo-TRT-LLM, while the GB200 result uses Dynamo-SGLang.

We use the September 17 AgentX snapshot results to help illustrate the serving implications of GLM-5.3. We estimate costs using InferenceX’s model of owning and operating infrastructure at hyperscaler scale. Total tokens include input and generated output, including cached input history reused across turns. Streaming speed is measured using p90 interactivity; time to first token is evaluated separately.

At 150 tokens per second, GB200 costs approximately $0.044 per million total tokens, compared with $0.238 for MI355X running ATOM. That is about 82% lower cost at the same streaming-speed target. These values are estimated from the measured curves, rather than from a separate test at exactly 150 tokens per second.

For this comparison, we use InferenceX’s estimated cost of owning and operating the infrastructure at hyperscaler scale. Total tokens include input and generated output, including input history reused from cache.

Source: SemiAnalysis InferenceX current InferenceX dashboard; interpolation method; GB300 source run; MI355X source run.

Among the Dynamo-SGLang results, GB300 serves roughly 13950 total tokens per second per GPU here, a 17.5% throughput advantage versus 11873 total tokens for GB200. However, we assume $2.31 per GB300 GPU hour versus $1.86 for GB200. The higher hourly cost more than offsets GB300’s throughput lead at this target.

In the comparison with AMD, GB200 costs about 52% less than ATOM at 100 tokens per second, 62% less at 125, and 82% less at 150. Across these three targets, the gap widens as response speed increases.

Counting only generated output, GB200 costs $5.92 per million output tokens at a response speed of 150 tokens per second, which is 79% less than $28.52 for MI355X ATOM. This metric is useful to analyze cost under agentic workloads, where agents repeatedly reuse long input histories while generating relatively little new text.

Source: SemiAnalysis InferenceX; current InferenceX dashboard

The GB200 measurements on either side of the 150 token per second target have p90 TTFT (time to first token) of 14-19 seconds, compared with 1.3-1.7 seconds for MI355X ATOM. Some of the GB200 lowest cost runs have much longer waits for the first token.

We therefore make a second comparison using only configurations that were actually tested and achieved at least 150 tokens per second with p90 TTFT below two seconds. The chart below shows B200 with Dynamo-SGLang at $0.0666 per million total tokens versus $0.1265 for MI355X with SGLang. B200 still costs about 47% less with the first-token limit.

Source: SemiAnalysis InferenceX B200 run 442071; GB300 run 441597; MI355X run 442075

Optimizations

In GLM’s 5.2 / 5.3 model, its attention architecture reduces memory demands, and long running agents still need to retain earlier conversation history. GLM reduces the cost of handling this history in two ways: KV compression reduces the cached state stored per token, and sparse attention reduces how much of that state each attention operation reads. Then the serving engine would determine where to store the cache and how to retrieve it efficiently.

These B200 results show how much reuse can happen outside of GPU memory. When concurrency increases from 8 to 16 requests, the share of prompt tokens reused from GPU memory falls from 90.3% to 54.8%. Much of that reuse shifts to host memory, which rose from 6.0% to 40.3%. Much of the decline in GPU cache is offset by reuse from host memory, keeping the overall cache hit rate above 95% at all concurrency levels.

Source: SemiAnalysis InferenceX; InferenceX B200 rows 440962, 440961, and 440959; workflow run 33683520699

Beyond cache management, changes to how the serving engine executes prefill and decoding can also change performance. Two separate GLM-5.2 studies illustrate this:

vLLM keeping the first decoding step on a consistent CUDA graph execution path reduced average TPOT (time per output token) from about 40 ms to 22 ms in an NVFP4 study.
On a different 8 MI355X long context workload, ATOM engine processing prefill in chunks across pipeline stages delivered 98% higher total throughput and reduced median TTFT from 28.6 to 8.7 seconds.

TileRT

For GLM5.3 on TileRT, AMD MI355X was supported first. TileRT is a low-latency LLM inference engine for that compiles decoding into a single persistent kernel, reducing launch overhead and overlapping computation, memory access, and communication.

It prioritizes faster token generation per user, rather than maximum batched throughput, while vLLM handle prefill.

On AgentX, FP8 TileRT MI355X achieves 2x the P90 interactivity than the best FP4 MI335X config. Comparing to GB300 NVL72, it brings a 40% uplift in interactivity. These are serving metrics that used to require specialized hardware.

However, TTFT is suboptimal and there is still room for development, such as optimizing KV transfer, FP4 support, and larger batch sizes.

DeepSeek Sparse Attention

GLM-5 (and GLM-5.x models) is a 744B total, 40B active-parameter mixture-of-experts model. For every token, it has 1 shared expert and is routed through 8 out of 256 experts, which is sparsity 32. It features DeepSeek Sparse Attention, which we will discuss in this section.

DeepSeek Sparse Attention (DSA), introduced in DeepSeek V3.2, consists of two components: A lightning indexer that selects top K tokens, and a sparse Multi-Latent Attention (MLA).

Lightning Indexer

A lightning indexer is functionally similar to a lightweight attention mechanism. Queries and keys are down-projected to lower dimensions, with indexer queries being multi-headed and indexer keys single-headed. Since we only need a query and key relation score (logits) for selecting top K tokens, the indexer doesn’t need value embeddings or the softmax operation for normalized logits. Instead, it computes a query-key dot product, followed by a ReLU non-linearity. The indexer then computes each token’s indexer score as a weighted sum of logits across all query heads, which we use to select the top K tokens with. This means despite the indexer having multiple heads, token selection is shared across multiple attention heads. When selecting top K tokens, we keep queries that attend to less than K tokens as dense attention, and we select top K otherwise.

Conceptual example of the lightning indexer. Source: SemiAnalysis

Indexer score is a weighted sum of all indexer query heads. Source: SemiAnalysis

Sparse MLA

As we explained in our Kimi K3 article, MLA operates in two modes: Multi-Head Attention (MHA) mode and Multi-Query Attention (MQA) mode. Both come with trade-offs: MHA mode has lower FLOPs but 42x higher memory cost, MQA mode has lower memory cost but up to 3.4x higher FLOPs. DSA uses MQA mode, since sparse attention attends to less tokens,mitigating the higher FLOP usage. However, in practice, there is a sequence length threshold where under it, MHA mode is actually more efficient that MQA mode. The intuition is that at shorter sequence length, the memory loading time isn’t as dominating, and MHA mode may have an edge for being lower FLOPs. vLLM implements this feature, and configures MHA mode for sequence length 2K to about 5K for data parallel, and 2K to about 77K for tensor parallel. The 2K minimum is due to top K being 2048, and as mentioned above, attention below 2K is dense.

MHA (green) can be faster than MQA (red). Source: vLLM PR #48770

MLA Modifications

Comparing the MLA dimension configurations of GLM-5 and DeepSeek V3.2, we see that the most notable differences are the number of query heads and the query key head dimension.

GLM-5 tech report mentions DeepSeek chose the number of query heads according to the roofline of H800. This is likely referring to the fact that MLA’s SDPA arithmetic intensity during decode is roughly at the ridge point of H800. Derivation as follows. Assume

L: Sequence length
d: Head dimension
r: RoPE dimension
H: Number of heads
b: Number of bytes per parameter

In MQA mode, the FLOPs are

2 * H * L * (d+r)  # P = Q [H, d+r] @ K.T [d+r, L]
2 * H * L * d      # O = P [H, L] @ V [L * d]

and the memory loaded is

b * H * (d+r)  # Q
b * L * (d+r)  # K and V

So the arithmetic intensity is

(2 * H * L * (2 * d + r)) / (b * (d+r) * (L + H))
= (2 * H * L * (2 * d + r)) / (b * L * (d + r))  # L >> H
= (2 * H * (2 * d + r)) / (b * (d + r))

If we plug in the DeepSeek V3.2 configuration d = 512, r = 64, b = 2), we get

(2 * H * (2 * 512 + 64) / (2 * (512 + 64)) ~= 2 * H

H800’s practical arithmetic intensity is 258.2 FLOP/B (Practical peak 865 TFLOP/s, 3.35 TB/s bandwidth, according to DeepSeek), and plugging in the formula gives us H ~= 128.

The fact that GLM-5 has H = 64, half the number of query heads, implies that GLM-5 may be designed for different hardware. If we plug in GLM-5’s configuration (d = 512, r = 64, b = 2, H = 64), we get an arithmetic intensity of 120.8 FLOP/B. Thus, we suspect GLM-5 is optimized for Moore Threads MTT S4000, which is about 128 FLOP/B. Moore Threads’ collaboration with Z.ai, such as GLM-5.3-Flash day 0 support, corroborates our theory.

Separately, GLM-5 increases the QK NoPE dimension from 128 to 192, as ablations show the configuration is superior under iso-FLOP and iso-parameter constraints. This increases the SDPA head dimension from 192 to 256, which is still a 33% FLOP reduction after considering the head number reduction.

IndexShare

The latency cost of the indexer becomes non-negligible as context grows. To mitigate the cost, Z.ai proposes IndexShare (aka IndexCache) in GLM-5.2, where every 4 DSA layers share one indexer.

Latency cost at different context lengths for a 30B DSA model. Source: IndexCache

IndexShare introduces complications in training and inference. Standard DSA training involves two stages: A dense warm-up stage where all model weights except the indexer are frozen and attention computation is dense, and then a sparse adaptation stage where the full model is trained with sparse attention.

In terms of training objective, the indexer is trained to align with the sum of attention head scores, using a KL divergence objective. IndexShare adapts the training objective to aligning the indexer logit distribution with the average distribution of attention scores across shared layers.

For inference, typically the indexer caches keys, like attention caches KV. Since multiple layers share one indexer, we need to additionally cache the top K indices at the full layer, in order to reuse the top K selection information at shared layers.

This design reduces indexer cache by 75%, reduces indexer FLOPs by 75%, and boosts throughput by 1.5x to 1.8x at different context lengths and prefill/decode.

Baseline vs. GLM-5.2 configuration (red) for a 30B DSA model. Source: IndexCache

Post-Training Design

Pipeline

After pre-training and mid-training, GLM-5’s post-training pipeline starts with supervised fine-tuning (SFT), continues with three RL training stages (reasoning RL, agentic RL, and general RL), and ends with an on-policy cross-stage distillation, distilling reasoning and general RL back into the SFT checkpoint.

GLM-5 post-training pipeline. Source: SemiAnalysis

Here we highlight two details. First, the tech report states that the reasoning RL model is trained entirely on-policy, implying that it is trained synchronously. Synchronous RL training is low system efficiency but has higher RL training stability. Second, on-policy cross-stage distillation is similar to Multi-teacher On-Policy Distillation (MOPD), where a student model is RL trained with on-policy distillation over multiple teacher models. However, unlike MOPD, whose goal is to merge expert teacher models into one model, GLM-5 here seems to be more about recovering old capability loss when the model learns new capabilities. Following this assumption, we are not sure why the Z.ai only uses reasoning and general RL checkpoints as teachers, instead of using all three RL checkpoints.

Based on the announcement blog post, GLM-5.2 may have replaced the on-policy cross-stage distillation with a standard MOPD pipeline, which Z.ai named parallel OPD training. Z.ai touted its scale and training efficiency: they merged more than ten expert models into a final model by training for about two days.

RL Training Algorithm

Z.ai adopts different RL algorithms for synchronous and asynchronous RL stages. Here we compare the major differences in the RL objectives.

In the synchronous RL stage, Z.ai adopts Group Relative Policy Optimization (GRPO)-based RL algorithm with modern adaptations, including:

Remove KL regularization term to improve system efficiency and allow more aggressive gradient updates
IcePop: Apply token-level masking based on the training-inference mismatch ratio
Clip-Higher: Use a higher max threshold for importance sampling ratio to encourage exploration

In contrast, Z.ai employs a REINFORCE-like RL algorithm for asynchronous RL stage, adding Direct Double-Sided Importance Sampling (DIS), an IcePop-like clipping mechanism. Below we compare the token-level objectives for the two RL stages:

Token-level objective for the two RL stages. Source: SemiAnalysis

The importance sampling (IS) ratio compares the probability of each sampled action under the current policy and the behavior policy, scaling each action’s contribution to the training objective. For IS ratio, the synchronous RL stage utilizes the trainer policy of the previous iteration for the behavior policy, whereas the asynchronous RL stage utilizes the inference policy for that. In an asynchronous training setting, policy updates happen multiple times during rollouts, so we will need every checkpoint version in between to calculate the IS ratio. However, this is infeasible due to the high memory and latency overhead. As a result, Z.ai opts for using inference policy for efficiency and simplicity, at the cost of numeric stability challenges.

IS ratio for the two RL stages. Source: SemiAnalysis

Finally, we highlight the major differences in advantage calculation in RL training, cross-distillation, and long-horizon RL training. RL training employs the GRPO standard group-normalized advantage, and cross-distillation uses the reverse KL divergence.

Advantage calculation comparison. Source: SemiAnalysis

Single-Rollout Asynchronous Optimization

Long horizon tasks produce extremely long trajectories, and in the context of GRPO, the high variance in trajectory length produces unstable training signals that may skew towards longer trajectories. What’s worse, since RL rollout workload is sensitive to end-to-end latency, the tail-latency of a straggler rollout worsens for long horizon tasks. To mitigate this issue, Z.ai introduced Single-rollout Asynchronous Optimization (SAO) for long-horizon RL training in GLM-5.2. SAO replaces group-normalized advantage with a modified Generalized Advantage Estimation (GAE), inspired by Proximal Policy Optimization (PPO). GAE only needs a single rollout to compute the advantage. This removes the need to do multiple rollouts per prompt, mitigating GRPO’s short-come.

Generator node asking trainer node to wait for rollout completion. Source: SAO

GAE is computed with values (In RL terms, expected returns of the current state), estimated by a value model. Concretely, the value model takes the rollout as input and produces a value estimation for every token. SAO adapts GAE to agentic RL training by removing the observation tokens, i.e. tool call results, from the optimization. Previously in GLM-5, skipping the tokens is sufficient; in SAO, we modify the advantage estimation formula to bypass the observation tokens. We refer readers to the SAO paper and its appendix A.1 for more intuition on the Bellman target and the ablations.

In SAO, the value model is a separate LLM concurrently trained with the policy model. This means early stages of training are susceptible to high variance of advantage calculations, due to a weak value model. To combat this, SAO proposes multiple techniques for training value models:

Two Time-scale Update Rule: Updating the value model twice as frequently as the policy model. This means training on each data batch twice at every policy training step
Attention Parameter Freezing: Freezing the value model’s full attention layers to improve training stability
Scaling Pre-training: Scaling the value model pre-training corpus to initialize the value model, so that the value model can robustly estimate values in the early stages of training. Unfortunately, Z.ai didn’t disclose the scaling information of pre-training corpus

Source: SemiAnalysis

At the cost of computational overhead and doubling memory footprint, SAO replaces group-wise sampling with a value model, mitigating the latency issues of GRPO.

Post-Training Infrastructure

Z.ai uses slime, one of the best open-source RL training frameworks, as the post-training infrastructure throughout all GLM-5 models. We refer readers to the slime GitHub repo to learn more about its design and features, and here we highlight two features.

Performance characteristics of RL rollouts are highly task-dependent, and each task comes with different tools and reward functions. To ensure system efficiency, Z.ai designed a server-based multi-task rollout orchestrator. Each task acts as an independent microservice server, controlling rollout and reward logic. The orchestrator balances the overall performance, controlling the per-task rollout ratio and generation speed. Centralization also allows dynamic ratio adjustments across tasks, fine-grained progress monitoring, and many more. Z.ai claims the orchestrator supports 1000 concurrent rollouts when training GLM-5.

Architecture diagram of the multi-task rollout orchestrator. Source: SemiAnalysis

In the GLM-5.3 announcement, Z.ai claims to use local storage as a cache layer in addition to host memory, which enables dynamic teacher model switching and prefetching for MOPD. slime PR#1538 implements teacher switching by introducing a TensorBackuper class, which stores teacher weights as PyTorch tensors pinned to CPU memory. When a rollout is routed to a teacher, a TensorBackuper object copies the teacher model back to GPU memory to perform a forward pass (implementation here). However, this is CPU-backed weight swapping, and we have yet to see storage-backed implementation.

In our paywall section, we present our analysis of GLM-5.3’s cybersecurity capabilities.

Intel Panther Lake Teardown

2026-09-26T05:36:32+00:00

Panther Lake debuts the first commercial implementation of backside power delivery (BSPDN), introduces Intel’s first iteration of gate-all-around (GAA) transistors, and showcases their advanced packaging capabilities with its Foveros-S assembly. With Panther Lake, Intel’s manufacturing arc has shifted from nebulous roadmaps to shipped silicon, a significant milestone on their long road back to competitive semiconductor manufacturing. To evaluate the extent of Intel’s comeback, we tore down Panther Lake.

Intel Core Ultra 7 365 (Panther Lake). Source: SemiAnalysis

The SemiAnalysis STEEL teardown lab breaks down advanced datacenter and AI hardware. To learn more about our pipeline or to commission a teardown, contact sales@semianalysis.com.

WE’RE HIRING: Architecture, floorplan, packaging, manufacturing, and labs experts. Opportunities from system to transistor and everywhere in between. Check out our Careers page.

Our teardown traces 18A from its four-sheet RibbonFETs (Intel’s marketing name for GAAFETs) and gate stacks through contacts, frontside and backside wiring, and the bonded carrier. We explain how these material and integration choices improve gate control and reduce resistance, while adding capacitance, thermal resistance, and process complexity. Our measurements put Panther Lake’s 18A compute logic and TSMC N3E GPU logic at similar logic density. However, 18A does not lead TSMC N3P, N2 or Samsung SF2 in peak density. Panther Lake’s CPU cores are incremental updates, and the high-end GPU still uses TSMC N3E.

X-ray of Intel Panther Lake package. Source: SemiAnalysis

Panther Lake assembles one compute tile, one GPU tile, and one I/O tile atop a passive base tile using Intel’s Foveros-S advanced packaging. Both compute tile variants use Intel 18A. The Xe3 GPU options are a 4-core GT1 tile on Intel 3 and a larger 12-core GT2 tile on TSMC N3E. Both I/O tile variants use TSMC N6. [1], [2]

Panther Lake configurations. PTL-U (3xx, left), PTL-H (3x6H, center), PTL-H with Arc B390/B370 (X-tier, 3x8H, right). Source: SemiAnalysis

Our analysis centers on the PTL-U compute tile, both the 4-core and 12-core GPU tiles, as well as the 12-lane I/O tile.

PowerVia

In conventional chips, power and signal are routed through the same frontside metal stack towards the device frontend. Power rails consume scarce routing resources near the transistors, while tall via stacks carry VDD and VSS from the coarse upper wires to local rails. Backside power delivery (BSPD) moves the main power network behind the transistor layer, to the backside, separating it from frontside signal routing. We covered BSPD and its impacts in 2024. [3], [4], [5] Intel’s BSPD implementation, branded as “PowerVia”, routes power through dedicated backside metals to nano-TSVs, which connect those rails to local source/drain (S/D) contacts.

Schematic orientation of the retained carrier, devices and frontside/backside interconnects. Layer counts are illustrative; not to scale. Source: SemiAnalysis

Implementing that separation requires Intel to build the interconnect stacks from both sides of the wafer. The frontside comprises the M0-M14 signal stack, while the backside comprises the BM0-BM5 power stack. M0 and BM0 are closest to the transistors.

Generalized PowerVia process sequence and comparison of buried power rails, PowerVia and direct backside contact schemes. Integration details and layer counts are illustrative; the schematic is not to scale. Source: SemiAnalysis

The nano-TSVs connect the two sides, but Intel patterns and etches each via from the front after forming the contacts. A narrow via runs from the side of the contact deep into the silicon substrate. Intel then completes the frontside signal metal stack, bonds the wafer to a carrier, flips it and removes the original substrate until the buried via tips are exposed. The backside metal stack is then deposited directly on the revealed vias.

The nano-TSV and backside-via profiles taper in opposite directions because Intel forms them from opposite sides of the wafer. The transistor structures form the FEOL. Local contacts and nano-TSVs connect them to the wiring. M0 begins the frontside interconnect stack.

The silicon carrier remains attached above the frontside interconnects. It supports the device wafer during substrate removal and backside processing and remains part of the finished chip’s thermal path.

PowerVia removes the main power distribution from the congested frontside metals, routing supply through shorter and wider backside wires. Its lateral landing still occupies area in the standard cell, so it recovers less cell area than a direct backside contact. [3] Nano-TSVs beside the logic devices carry VDD or VSS from the backside power network, while signal connections continue upward through the frontside metals.

Intel 18A RibbonFET and nano-TSV in XTEM (top) and EDS (bottom). Source: SemiAnalysis

Backside Interconnects

Samsung SF2 data is included for comparison to Panther Lake’s within this article. SF2 is the incumbent GAA foundry node but lacks BSPD, serving as a useful reference to evaluate 18A. A full teardown of Samsung’s S26 products, processed on SF2, will be shared soon.Nanosheet-cut EDS comparison.

Intel 18A Panther Lake P-core logic (left) and Samsung SF2 Exynos 2600 C1-Ultra logic (right). HFW 550 nm (Intel) and 423 nm (Samsung). Source: SemiAnalysis

The PowerVia supply path runs from the backside Cu rails through Mo-lined W nano-TSVs to the local transistor contacts. In this cross section, the tapered connection spans roughly 150 nm from the contact level to BM0. The Ta liner confines Cu and promotes adhesion to the surrounding stack; the AlOₓ etch stop controls the next dielectric etch above the rail. Dielectric beneath the ribbons electrically separates the devices from the backside wiring and removes the conducting silicon body below the channel. [6]

Intel 18A Mo-lined W nano-TSV and Cu BM0 rail. Source: SemiAnalysis

AlOₓ serves as an etchstop (ES), enabling endpointing and protecting the underlying layers. Low-volatility aluminum fluoride reaction products resist the fluorinated plasma, allowing a thin AlOₓ film to protect the metal while the surrounding low-k dielectric is removed. [6], [7]. While the BM0 and layers above the M1 lines show double AlO_x layers, Our SMIC N+3 teardown showed single AlOₓ layers. SMIC uses a simpler local AlOₓ substack, while the remaining cap and etch sequence provide the required landing protection.

Intel 18A backside metallization materials. Source: SemiAnalysis

So why double layers? The closely spaced AlOₓ doublets provide two protected endpoints in the etch sequence. Intel documents an AlOₓ/SiN/AlOₓ stack that explains the benefit. The main dielectric plasma etch stops on the first AlOₓ film; a selective wet clear opens that film; a second plasma etch removes the intermediate SiN and stops on the second AlOₓ film. The final wet clear exposes the metal landing surface. SiN is the intermediate dielectric in Intel’s published example. [8]

The second stop protects the metal through a cap breakthrough. Wide openings can etch faster than narrow ones, and etch depth varies across the wafer. Metal under an early-clearing opening would otherwise be exposed while other openings still need more etching. Staged protection widens the process window and reduces metal erosion, corrosion and void formation. [8]

TSMC documents AlN/AlOₓ/SiOC/AlOₓ above Cu, with AlN blocking Cu diffusion, and a simpler AlN/SiOC/AlOₓ variant that omits one AlOx film. [9] Levels with different opening sizes, aspect ratios, pattern densities and cap materials need different etch margins. A double AlOx stop is useful where another protected endpoint justifies the added processing.

The extra film adds formation, selective opening and cleaning steps, plus another set of interfaces to control adhesion, moisture, and stress. These blanket films are opened through the existing via pattern, so each film does not require another lithography mask. AlOₓ adds parasitic capacitance when it replaces lower-k dielectric; two thin AlOₓ films can nevertheless contain less AlOₓ than one thick film. Total thickness, placement, and theintermediate dielectric determine the electrical cost. Deposition chemistry also changes AlOx permittivity and residual hydroxyl content, which can oxidize the underlying metal. [7], [8], [10], [11]

The backside stack separates into relatively fine BM0-BM2 wiring near the devices and coarser BM3-BM5 power distribution. The largest pitch increase occurs between BM2 and BM3. BM0’s pitch closely matches the logic-row height, fitting local power delivery to the cell rows. Higher levels aggregate current through larger conductors: routing density becomes less important than low resistance and current capacity as the network approaches the package. This hierarchy provides wide power wiring for the power delivery network without consuming scarce frontside signal-routing resources. [3]

Intel 18A BM0-BM5 minimum measured pitches. Source: SemiAnalysis

The SemiAnalysis STEEL teardown lab breaks down advanced datacenter and AI hardware. To learn more about our pipeline or to commission a teardown, contact sales@semianalysis.com.

WE’RE HIRING: Architecture, floorplan, packaging, manufacturing, and labs experts. Opportunities from system to transistor and everywhere in between. Check out our Careers page.

Frontside Interconnects

Intel 18A combines Mo-lined W contacts and nano-TSVs with a separate backside Cu power network. Samsung SF2 keeps power on the frontside, using Ti-based contact interfaces and Ta-based barriers and Co liners around Cu wiring.

In 18A standard-cell rows, backside power rails supply the devices through nano-TSVs within the cells, freeing frontside routing resources. Samsung’s M0 accommodates both power and signal connections.

Intel 18A M0-M14 minimum measured pitches. Source: SemiAnalysis

From the device toward M0, the connection runs through a Ti-based S/D interface, W contact fill, a Mo-lined W via, and the Cu M0 wire. Mo supplies a conductive nucleation and adhesion layer for W, replacing the resistive TiN liner used in conventional W integration. This increases the effective conduction volume within the feature while retaining W fill and its established polishing, cleaning and etching processes. Intel’s Mo/W patent describes this integration tradeoff. The nano-TSV uses the same Mo-lined W construction in the backside supply path. [12]

Intel 18A and Samsung SF2 contact and via materials. Source: SemiAnalysis

The move from TiN to Mo is an incremental change. While a full Co or Mo fill can also reduce the volume lost to liners in very small features, it requires new integration schemes that increase complexity and risk. Cu remains attractive for wider wires due to its low resistance. As wires and vias shrink, the diffusion barrier consumes an increasing fraction of their cross-section. [12], [12], [14]

Intel 18A Panther Lake P-core logic (top) and Samsung SF2 C1U logic (bottom), interconnect EDS maps. Cu, Ta, Co, Ru, and combined Co/Nb maps, left to right. Source: SemiAnalysis

Intel uses Co/Ru liners at M0-M1, Co at M2-M4, and Nb at M5-M9. The lower-level liners help Cu adhere and reduce void formation during trench fills. Applied Materials’ Endura has new thermal control that facilitate wetting process, so the thin film continuity is good enough that good capillary pressure will drive Cu atoms to the via bottom without voiding.

Intel’s choice to use Nb is particularly interesting. Intel’s Nb patent describes a conductive diffusion barrier intended to reduce the barrier’s contribution to resistance relative to conventional Ta-based barriers, particularly at via bottoms where all current crosses the barrier. The patent pairs Nb in coarser levels with the option of lower-cost PVD processing. [15], [16]

The upper metal layers support thicker barriers formed through physical vapor deposition (PVD) despite its worse coverage and uniformity. Meanwhile, the lower metal layers require thinner barriers deposited through conformal atomic layer deposition (ALD). Co/Ru adds another material interface and requires controlled deposition and Cu fill. Changing liners and barriers by metal layer allows Intel to optimize interconnect resistance, process complexity, and reliability. [15, 16]

Intel 18A and Samsung SF2 M0-M9 liner, fill and etchstop materials. Source: SemiAnalysis

RibbonFET

RibbonFET, Intel’s name for its gate-all-around FETs (GAAFETs), replaces the FinFET’s vertical fins with four stacked horizontal silicon nanosheets, allowing the gate to surround the channel on every side. The path to GAAFET begins with the planar transistor.

Planar NMOS structure (left) and CMOS inverter (right). Source: SemiAnalysis

A planar MOSFET places the gate above the channel between its source and drain. Pairing an NMOS with a PMOS transistor creates a CMOS inverter, in which the NMOS pulls the output low for a high input, and the PMOS pulls it high for a low input. The gate must retain electrostatic control of the channel to ensure clean switching. As gate lengths shrank, the drain began to compete with the gate for that control, increasing off-state leakage.

Electrostatic control was restored through an architectural evolution that raised the channel into a vertical fin and wrapping the gate around three sides. Called “FinFET”, this new architecture packed more effective channel width into a smaller footprint. Further scaling made it harder to maintain both drive current and leakage within smaller cells, and reintroduced the same problems planar MOSFETs faced. Nanosheet GAAFETs close the fourth side by replacing the vertical fin with a stack of horizontal nanosheets, each surrounded by the gate. The tighter electrostatic control suppresses leakage at shorter gate lengths while stacking adds effective channel width within the cell footprint.

Controlling gate leakage through architectural evolution. Source: SemiAnalysis

In a FinFET process, channel width changes in discrete steps as designers must add or remove whole fins. Nanosheet width can instead be adjusted continuously within the process’s design rules. Wider sheets increase drive current, while narrower sheets reduce capacitance at the cost of drive current. Intel 18A uses stacks of four nanosheets each and varies their widths across logic and SRAM. At the process level, adding more sheets to each stack increases effective channel width and drive current, but complicates fabrication.

RibbonFET vs MBCFET

Samsung began GAAFET production in 2022 with SF3E, following with SF3 and now SF2. Its ‘MBCFET’ provides a useful structural comparison with Intel’s first RibbonFET implementation. [17] STEEL is digging deeper into SF2, used in the Exynos 2600, and TSMC’s GAAFET N2, used in Apple’s A20 Pro, in upcoming newsletter articles. We’re throwing some teasers on X. Let’s compare Samsung SF2’s MBCFET with Intel 18A’s RibbonFET.

Subscribe now

Even to the untrained eye, Intel’s extra nanosheet is obvious. Intel stacks four ribbons to Samsung’s three. Samsung’s sheets are much wider in these fields, so both sheet count and width matter to the available channel perimeter. Sheet width also changes which silicon surfaces carry current. On conventional (001) silicon, wide nanosheets emphasize the broad top and bottom surfaces, favoring electron transport; the larger sidewall contribution in a narrow sheet favors hole transport. Thinner sheets improve gate control but increase confinement and scattering. This makes width and thickness part of the NMOS/PMOS balance, alongside strain and threshold voltage. [18], [19]

GAAFET designs like 18A use different work-function-metal (WFM) stacks for NMOS and PMOS. Around each ribbon, a thin SiOx interfacial layer separates the silicon channel from the HfOx high-k dielectric, with La providing dipole tuning and the WFM wrapping the dielectric. NMOS uses a TiAl-based stack, while PMOS uses TiN WFM. W fills the remaining gate trench, providing a lower-resistivity path where the work-function layers are no longer needed. In this field, the PMOS stacks leave room for W between ribbons, while the NMOS stacks occupy more of those gaps. A silicon-based dielectric marks the P/N boundary, allowing the PMOS and NMOS gates, sharing the same gate trench, to be processed sequentially.

Fast logic paths, retention circuits, and SRAM need a family of threshold options. Changing threshold without substantially changing device dimensions, capacitance or fabrication complexity is valuable. FinFET processes typically use different work-function-metal stacks. In a four-ribbon GAA stack, the narrow sheet-to-sheet gap limits how much WFM can fit around each channel. La in the gate dielectric creates interfacial dipoles at the SiOx/HfOx boundary, shifting effective work function and tuning threshold voltage. This gives Intel another control alongside its NMOS and PMOS WFM stacks. Low-threshold devices improve critical-path drive; higher thresholds reduce leakage elsewhere. Dipole tuning is especially useful in GAA because it changes threshold without consuming the narrow intersheet gap with thicker WFM.

Precise control of La incorporation, diffusion and interface quality has long been a challenge, limiting viability in high volume production but is now seen from every leading-edge foundry. Intel’s patent describes depositing a dipole-forming oxide above HfOx and annealing it toward the interfacial oxide before completing the work-function and fill metals. This separates threshold tuning from the space available for metal. Newer research addresses the thermal cost: imec’s 2026 dipole-middle research inserts the shifter between two HfO_x depositions, shortening the diffusion path while protecting SiO_x during patterning. [20], [21]Matched-cut EDS, Intel 18A (left) vs. Samsung SF2 (right).

Top: sheet cuts, Intel P-core logic and the Samsung logic field labelled CU. Bottom: gate cuts, Intel P-core logic and Samsung NPU logic. Cut directions are matched; cell functions and drive targets differ. Scale bars per panel. Source: SemiAnalysis

Intel retains raised source/drain epi beneath its contacts, while Samsung recesses W deep into the epi to form a V-shaped Ti-lined interface. The deeper contact increases metal-to-semiconductor area and shortens the current path from the lower sheets, reducing contact and spreading resistance. It also removes epi volume and brings the contact etch closer to the channel ends. Retaining more epi preserves the material available for strain transfer, especially from SiGe into PMOS. These geometries balance contact access against stress engineering and etch margin. [22], [23]

Samsung stacks three sheets to Intel’s four ribbons, and both processes use sheet width to tune drive strength. In our Samsung cross-sections, widths range roughly from 19 to 30 nm in the NPU rows and 37 to 50 nm in the CU cell. The Samsung nanosheets taper, with the widest sheet at the bottom and the narrowest at the top. Both processes use HfOx gate dielectric and Ti-based work-function stacks, with Al in the NMOS stack. In the Samsung devices shown here, the dielectric and WFM occupy the intersheet gaps, leaving W above the top sheet. Intel’s PMOS stack leaves more room between ribbons, and W fills those gaps while the thicker NMOS stack leaves W mainly in the upper trench. Gate-stack EDS maps.

Top: Intel 18A P-core logic. Bottom: Samsung SF2 CU logic. Source: SemiAnalysis

The W between Intel’s PMOS ribbons provides a conductive path close to the lower gates. Where WFM fills the entire gap, the gate still surrounds the channel, but voltage reaches it through the more resistive work-function films. Thinner WFM and dipole tuning preserve room for low-resistivity fill; Mo and Ru are alternative fill metals being developed for further scaling. [24]

A masked, sequential WFM flow explains the different gate heights and inter-nanosheet fill. The proposed sequence below shows how separate NMOS and PMOS work-function steps produce that geometry.

Proposed Intel 18A NMOS and PMOS WFM integration sequence. Source: SemiAnalysis

Enabled by the BSPDN process, Intel replaces the dense-logic silicon subfin with dielectric, removing the parasitic conduction path below the ribbons and reducing substrate-related capacitance. A retained silicon body as in classical, non-SOI, planar and FinFET designs needs junction and punchthrough-stop engineering to suppress leakage. Dielectric isolation makes that leakage less sensitive to the subfin doping profile but adds removal and fill steps. It also weakens the direct thermal path through silicon, making the contacts, metal stacks and package more important for heat extraction. [24], [26]

Fluorine is concentrated around selected Intel device structures in the maps. WF₆ is a standard precursor for W nucleation and fill, while barrier films protect adjacent dielectrics from fluorine attack. Low-fluorine W processes reduce the residual-F burden. Chloride-based precursors avoid introducing F during W deposition, but require control of chlorine attack, nucleation and fill quality. The integration target is a continuous, low-resistance W path with a thin protective liner and minimal chemical damage to the surrounding stack. [20], [27], [28]

Proposed GAAFET Process Flow. Source: SemiAnalysis

The SemiAnalysis STEEL teardown lab breaks down advanced datacenter and AI hardware. To learn more about our pipeline or to commission a teardown, contact sales@semianalysis.com.

WE’RE HIRING: Architecture, floorplan, packaging, manufacturing, and labs experts. Opportunities from system to transistor and everywhere in between. Check out our Careers page.

Library analysis

We measured cell height, gate pitch, metal geometry, and ribbon dimensions at the XTEM sites shown below. The tables group these dimensions by site and device polarity. Our “sheet cuts” cross the silicon channel and show the ribbons end-on. “gate cuts” run along the channel through successive gates.

Measured library dimensions. *Intel 18A PDK provides M0 pitch of 32 nm, but Panther Lake HP libraries shipped at 36 nm, **3rd party discolsures. Source: SemiAnalysis

The 18A logic cell dimensions point to a five-track logic library while the N3E and Intel 3 cell dimensions evidence a seven-track logic library. The DDR-PHY uses wider M0 wires and much larger spacing than core logic. That trades routing density for lower wire resistance and weaker coupling between neighboring nets. The geometry suits the current delivery and coupling requirements of analog, clock, and I/O circuitry.

PowerVia lets 18A combine a compact cell height with wider M0 geometry by moving the main power rails off the signal-routing tracks. That relaxes local wire scaling while preserving a small cell footprint. Cell height and gate pitch set the geometric density; pin access and routability determine how much of it a real block can use. [29]

The biggest takeaway from our gate-pitch measurements is that Intel 18A compute logic and TSMC N3E GPU logic have similar density in the Bohr representative-cell model. The 18A example is 18.6% denser than the Intel 3 GPU example. Gate pitches are nearly identical across the three sites, so cell height drives most of the difference.

The Bohr model combines a four-transistor NAND2 spanning three gate pitches and a 32-transistor scan flip-flop (SFF) spanning nineteen pitches, weighting their densities 60:40. The sensitivity column shows how independently changing cell height and gate pitch by ±1 nm changes the result. This compares representative cell geometries; whole-die density also depends on cell mix and placement.

Representative-cell density and input sensitivity. Source: SemiAnalysis

The 18A P-core gives M0 substantially more metal cross section than the N3E vector engine. Treating each profile as a trapezoid gives 2.63 times the area per line and 1.84 times the area after normalization by routing pitch. The larger section reduces the geometric contribution to line resistance and lowers current density for a given current. Taller and wider wires also add capacitance, so circuit delay depends on the balance of resistance and capacitance. The DDR-PHY has less metal area per routing width than the 18A core fields, while remaining above N3E. [30]

M0 cross-section geometry. Source: SemiAnalysis

Area = height × (top CD + bottom CD) / 2, including liners. Area/pitch normalizes by routing width. Taper is the symmetric sidewall angle from vertical, with the largest angle belonging to the DDR-PHY.

Compute tile

Compute-tile XTEM sampling locations. Source: SemiAnalysis

The measurements show how ribbon dimensions and gate-stack geometry vary across the compute tile and between NMOS and PMOS to balance channel drive, gate load and the space needed for the dielectric/WFM stack across logic, SRAM and the DDR-PHY. Width mainly changes available channel perimeter; thickness also changes electrostatic control and carrier confinement. Gate-stack thickness then determines the space left for low-resistivity fill

Ribbon dimensions. Source: SemiAnalysis

P-core and LP E-core logic

Both the P-core and LP E-core use multiple nanosheet widths. Widths are measured on high-magnification XTEMs while wider-field images demonstrate additional width choices within the LP E-core. Multiple widths are expected even within an LP E-core. Timing-critical paths, buffers and cells with different fanout need different drive strengths. The lower-magnification fields show this width diversity beyond the sites quantified in the table.

L2 and L3 SRAM

GAA gives SRAM designers another way to balance the pull-up (PU), pass-gate (PG), and pull-down (PD) transistors. FinFET bitcells set device strength through fin count while GAA adds nanosheet width as a sizing knob. In a 6T SRAM cell, a strong pull-down relative to the pass-gate limits read disturbance, while a strong pass-gate relative to the pull-up improves writability. During a write, the pass-gate and write driver pull the node storing “1” below the inverter trip point. During a read, the pull-down holds the node storing “0” low. Bias, threshold voltage, mismatch and assist circuitry set the remaining margin. FinFET high-current cells commonly use a PU:PG:PD fin-count pattern of 1:2:2, a device-sizing ratio rather than a current ratio.

Intel 18A SRAM bitcell features. Source: Intel, ISSCC 2025 (© IEEE). [31]

Ribbon width lets Intel balance SRAM strengths without adding whole fins. The L2 cell uses its narrowest ribbons for PU and widest for PD, improving writability and read stability respectively. Intel’s disclosed HCC operates without assist; its denser HDC uses negative-bitline write assist. Pulling the selected bitline briefly below ground increases pass-gate overdrive so it can overpower the pull-up at lower supply voltage. That buys density and low voltage writability at the cost of boosting circuitry, switching energy, and additional voltage stress that must be controlled. [31], [32]

Four rectangular ribbons give the perimeter = 8 × (width + thickness), before corner rounding. PG/PU is 1.49 and PD/PG is 1.16.

The L3 structures closely resemble L2 in layout and cell height. Fewer L3 nanosheet widths are tabulated because fewer high-magnification images were available.

DDR PHY

The DDR-PHY trades density for controlled analog behavior and reliable off-chip signaling. It contains drivers, receivers, delay circuits, and calibration logic that set drive strength, sampling time, and voltage margin. Repeated four-sheet devices with similar widths fit the use of regular transistor units for matching and programmable drive. Its wider local wiring provides room for current delivery and separation of sensitive signals, while consuming more area than a dense core-logic grid. The layout serves the memory channel’s electrical requirements as well as digital logic density. [33]

DDR PHY. Source: SemiAnalysis

The SemiAnalysis STEEL teardown lab breaks down advanced datacenter and AI hardware. To learn more about our pipeline or to commission a teardown, contact sales@semianalysis.com.

WE’RE HIRING: Architecture, floorplan, packaging, manufacturing, and labs experts. Opportunities from system to transistor and everywhere in between. Check out our Careers page.

Intel 3 GPU devices

Intel 3 GT1 GPU: sampling locations and device fields. Source: SemiAnalysis

Vector engine logic

Intel 3’s XVE logic uses two-fin PMOS and NMOS devices with power rails in M0. Its cell height and M0 pitch give a seven-track geometry, two tracks more than the 18A logic. One-fin groups also appear among the two-fin devices.

Intel 3 L2 SRAM

The Intel 3 L2 SRAM uses the familiar HCC sizing pattern: one PU fin, two PG fins, and two PD fins.

Intel 3 L2 SRAM. Source: SemiAnalysis

N3E GPU devices

N3E GT2 GPU sampling locations. Source: SemiAnalysis

Vector engine logic

N3E vector-engine logic cross-section. Source: SemiAnalysis

The N3E XVE field contains repeated two-fin devices with seven-track cell geometry. N3E remains a FinFET process, giving Panther Lake a direct FinFET-to-RibbonFET comparison.

N3E L2 SRAM

The N3E L2 SRAM uses the same PU:PG:PD fin-count pattern of 1:2:2.

N3E vector-engine logic cross-section. Source: SemiAnalysis

Floorplan analysis

Intel Lunar Lake (left) and Panther Lake-U (right) floorplans. Source: SemiAnalysis

Panther Lake-U follows Lunar Lake’s floorplan quite closely. Both pair 4 P-cores with 4 LP E-cores and NPU, media and display engines in similar locations. Lunar Lake also uses Xe2, the direct predecessor to Panther Lake’s Xe3 GPU. This makes Lunar Lake the most direct basis for our comparisons. Arrow Lake differs in core count and uses the older Xe-LPG GPU architecture, so we only use it where it offers a more direct component-level comparison.

Lunar Lake, Panther Lake 8-core, Panther Lake 16-core/12-Xe, and Arrow Lake-H configurations. Source: Intel

Compute tile

Panther Lake compute-tile floorplans remain sparse even months after launch. Intel 18A’s backside metal and dielectric stack must be removed without damaging the underlying structures before a clean transistor-level floorplan can be imaged. Most published die shots hide or heavily process the background, but we are quite proud of the die shot we achieved and are excited to show the work we have done.

Annotated Panther Lake-U 4+0+4 compute tile. Source: SemiAnalysis

We measured the areas of the key components on the compute tile and compared them with their Lunar Lake predecessors on TSMC N3B. These help us to capture changes in block area and compare the two chips across process nodes and designs.

Our total tile areas exclude the scribe-line area. The compute-plus-GPU subtotal below uses the PTL-U compute tile and GT1 GPU; it excludes the I/O tile and passive base. Individual block areas use the boundaries marked on the floorplans

Panther Lake 4+0+4 compute-tile areas compared with Lunar Lake. Source: SemiAnalysis

The compute-plus-GPU row is recomputed from the displayed PTL-U and GT1 areas. Component rows use their stated per-region counts and are not an additive partition of the whole tile.

Lunar Lake Lion Cove (left) vs Panther Lake Cougar Cove (right) P-cores. Source: SemiAnalysis

The P-core area remains almost unchanged between Lunar Lake and Panther Lake, despite L2 capacity increasing from 2.5 MiB to 3 MiB. Arrow Lake uses the same Lion Cove core as Lunar Lake but also has a 3 MiB L2. Cougar Cove fits 20% more L2 into the same P-core area. The larger private cache keeps more of each core’s working set close to its execution units, reducing access to shared L3 and DRAM. Extra capacity adds storage leakage and lookup energy, so designers balance it against avoided lower-level accesses. The shared P-core L3 cache also shrank by 14.8%. [2]

Cougar Cove combines a similar footprint with Intel’s reported power-efficiency improvements. RibbonFET’s tighter channel control reduces leakage, while PowerVia reduces supply droop and allows tighter voltage guardbands. [1]

Lunar Lake Skymont (left) and Darkmont (right) LP E-core clusters. Source: SemiAnalysis

Darkmont’s four-core LP E-core cluster is 5.0% smaller than Skymont’s on Lunar Lake, with most of the reduction in its L2 regions. The 1 MiB region shrank by 8.4% and the 1.5 MiB region by 14.9%. The tag arrays also use one fewer visible row. Tags identify which memory addresses the data array holds, so rearranging them changes the cache’s layout and wiring without requiring less data capacity. [2]

The LP E-cores share one L2. This pools capacity and avoids duplicating all the cache machinery, but the four cores contend for its banks and bandwidth. Their separate cluster also keeps light work away from the performance cluster and its L3, allowing that larger domain to sleep. [1], [2]

Cache area includes more than the storage cells. Tags identify each line, decoders select rows, sense amplifiers read the small bitline signal, and wires connect to the banks. Splitting an array into smaller sections shortens wordlines and bitlines, improving access speed, but duplicates peripheral circuits. Panther Lake’s smaller cache regions therefore reflect the complete memory implementation, including how much of each region is devoted to storage. [34]

Unlike Meteor Lake and Arrow Lake, Panther Lake has no separate SoC tile. The NPU, LP E-cores, memory controllers, PHYs, media and display engines now share the compute tile. This removes an active die and keeps CPU memory traffic on one die. The cost is moving PHY and I/O-related circuitry onto 18A: drivers, receivers and analog circuits must still meet external voltage, loading and signal-integrity requirements, so their area does not shrink like dense digital logic. [1], [2]

The biggest shrink comes from the NPU, which occupies 36.9% less area. NPU 5 consolidates the same total INT8 MAC count into half as many neural compute engines. Each of the three NCEs has a larger MAC array to make the complete NCE envelope 22.6% larger than an NPU 4 engine. Consolidation also halves the number of scratchpads and SHAVE DSPs, from 12 to 6. The MAC array handles matrix multiplication and convolution, while SHAVE executes vector and custom operations that fit the array poorly. [1], [2], [35]

Lunar Lake NPU 4 (left) vs Panther Lake NPU 5 (right). Source: SemiAnalysis

The paired floorplans identify each NCE envelope and its scratchpad, MAC, and SHAVE regions. Each measured MAC polygon is counted once per NCE in the area accounting below.

NPU area savings from the measured floorplan regions. Child regions are included in the NCE subtotal. Source: SemiAnalysis

The scratchpads store weights, activations, and intermediate results near the MAC arrays, allowing repeated use without fetching them again from DRAM. Halving their number delivers the largest measured area saving but leaves less local storage for the same total MAC count. Layers that no longer fit locally require smaller working tiles or more transfers of intermediate data. The benefit depends on keeping the enlarged arrays busy while managing that tighter storage budget. [36]

NPU 5 also adds native FP8. Using half the operand width of FP16 reduces storage and transfer demand, helping workloads fit the smaller local memory budget. Lower precision and format-dependent range make scaling and model validation part of deployment. Hardware activation functions further reduce work that would otherwise occupy the programmable DSPs. [1], [2]

Microsoft requires an NPU to deliver at least 40 TOPS for Copilot+ PCs. Both Lunar Lake and Panther Lake meet this threshold, but Panther Lake uses significantly less silicon.

GPU tiles

Panther Lake is Intel’s first product with Xe3, its latest GPU architecture. It offers two different GPU tiles: a smaller GT1 tile with 4 Xe3 cores on Intel 3 and a larger GT2 tile with 12 Xe3 cores on TSMC N3E.

Panther Lake allows us to compare the same GPU architecture across both Intel 3 and TSMC N3E. Wildcat Lake adds a third Xe3 implementation on Intel 18A. A future newsletter will detail Xe3 and its implementation differences across all three process nodes.

Intel Panther Lake GT1 (Intel 3, 4 Xe cores) and GT2 (TSMC N3E, 12 Xe cores) GPU tile floorplans. Source: SemiAnalysis

GT2 scales Xe3 to a different physical layout, with render slices arranged vertically instead of GT1’s horizontal arrangement. Slice placement sets the distances to shared cache banks and the D2D interface. Those wires consume area and add delay, so scaling the number of Xe cores also requires a new balance of cache placement, routing and timing. [1]

What’s immediately obvious is that the GT2 tile on TSMC N3E has much smaller Xe cores than GT1. These block areas include logic, caches, and routing.

Panther Lake GPU tile area comparison with Lunar Lake. Source: SemiAnalysis

An Xe core on the GT1 tile is ~69% larger than one on Lunar Lake, and ~55% larger than one on GT2. Intel 3 therefore uses substantially more area per Xe core. The block-area gap exceeds the measured logic and SRAM density gaps, bringing routing, timing targets, cell mix, and floorplan allocation into the comparison.

The measured vector/matrix engine region is almost unchanged between Lunar Lake and Panther Lake’s GT2 tile. Xe3 retains eight 512-bit vector engines and eight 2048-bit XMX engines per core. Its gains also come from feeding those engines more effectively: more resident threads hide stalls, and variable register allocation lets shaders trade registers per thread against the number of threads kept active. [1]

The shared L1/SLM capacity increased by 33% from 192 KiB to 256 KiB, while its area increased only 5%, raising effective density by 27%. L1 retains reused cache lines, while software-managed SLM lets a thread group share data locally. Both reduce traffic to more distant memory. Allocating more SLM per group can also limit how many groups reside on a core at once. [1], [37]

The GT1 tile carries 4 MiB of L2 against 16 MiB on the GT2 tile. GT1 divides its L2 cache into four 1 MiB banks, while GT2 uses eight 2 MiB banks. Each bank contains 128 macros, but each N3E macro stores 16 KiB, twice the Intel 3 macro’s 8 KiB capacity. The N3E macro is only 54% larger while holding twice as many bits, giving it 30% higher density: ~23.7 Mbit/mm² versus 18.3 Mbit/mm². Including bank-level circuitry, the gap widens to ~16.9 Mbit/mm² on GT2 versus ~10.4 Mbit/mm² on GT1.

Panther Lake Intel 3 (GT1) and TSMC N3E (GT2) area comparison. Source: SemiAnalysis

GT2 gains density with its macros storing more bits per unit area, and those macros occupy more of each cache bank. Larger macros spread decoder and sense-amplifier overhead across more storage, while a more compact bank layout reduces the share spent on control and routing. The compromise is longer wordlines and bitlines that carry more capacitance. [34]

I/O tile

Panther Lake uses two I/O tile variants, both fabricated on TSMC N6. The smaller one provides 4 PCIe 5.0 and 8 PCIe 4.0 lanes and serves lower-tier systems as well as those without a discrete GPU, while the larger one adds 8 PCIe 5.0 lanes, bringing the total to 20 lanes, for discrete-GPU connectivity. Panther Lake SKUs with the larger 10- or 12-Xe GPUs use the smaller I/O tile. [38]

Panther Lake 12-lane I/O tile floorplan. Source: SemiAnalysis

The smaller I/O tile adds a PCIe 4.0 block and a Thunderbolt block to Lunar Lake’s I/O layout, providing four additional PCIe 4.0 lanes and another Thunderbolt 4 port. Its repeated N6 blocks retain nearly identical areas and layouts. Reusing these proven PHYs and controllers avoids porting and requalifying external interfaces on 18A, where faster digital logic offers less benefit to circuits constrained by the off-chip link. [38]

Panther Lake 12-lane I/O tile area comparison with Lunar Lake. Source: SemiAnalysis

SemiAnalysis’s teardown lab (STEEL) dives deep into the world’s advanced datacenter and AI hardware. To learn more about our pipeline or to commission a teardown, contact sales@semianalysis.com.

We’re hiring technical experts from system to transistor and everywhere in between. Check out our Careers page.

Foveros-S

Panther Lake offers scalability and modularity through its disaggregated packaging that partition compute, GPU, and I/O silicon into separate tiles allowing for a suite of tile configurations. This partitioning makes the package part of Intel’s node economics as it determines how much leading-edge wafer area each product consumes, which functions can remain on other processes, and how much configuration freedom Intel can offer from a shared set of tiles.

Furthermore, fabricating the compute and GPU tiles separately confines the new 18A process to the compute tile and allows graphics and I/O to use other, more established, and more cost-effective processes. For Panther Lake, the GPU and I/O tiles are assembled alongside the compute tile on a passive silicon base using Foveros-S.

Intel’s current technology brief lists a nominal 36 µm pitch for Foveros-S. Through-silicon vias (TSVs) in the base connect the fine wiring above to the larger package connections below. The functional tiles sit side by side on that passive base in a 2.5D configuration. [39]

Foveros-S package schematic: active tiles connect through microbumps to a passive silicon base. Source: SemiAnalysis

Our cross-section through the compute and GPU tiles shows the package’s wiring hierarchy. Microbumps connect each active tile to the passive silicon base; its fine redistribution layer (RDL) carries the short, dense tile-to-tile links. TSVs carry connections through the base to the package substrate, which fans them out to the much coarser motherboard solder joints. The base supplies interconnect, while computation remains in the active tiles above it. [39]

At the compute-tile edge, the higher-magnification inset shows a local microbump spacing of approximately 25.24 µm and a feature width of 12.33 µm.

Panther Lake package cross section through the compute and graphics tiles. Source: SemiAnalysis

Panther Lake top-down image showing where the cross-section was taken. Source: SemiAnalysis

These local spacings are finer than Intel’s nominal Foveros-S value. The X-ray fields further confirm tighter neighboring bumps, consistent across every die-to-die area found on each tile. Additional X-ray analysis is offered after the paywall.

GPU D2D region with annotated local neighbor separations. Source: SemiAnalysis

Putting the memory controller beside the CPU removes the D2D transfer that CPU memory requests required in Meteor Lake and Arrow Lake. This avoids the extra transmitter, receiver, and link traversal, saving interface energy and latency. Panther Lake’s separate GPU still crosses a D2D link to reach DRAM, so its larger local caches also help contain package traffic. [1], [40]

Smaller dies are less likely to contain a random fatal defect, and screening them before assembly prevents one bad tile from consuming a complete package of good silicon. Reuse also spreads design and qualification work across more products. Against those gains, Intel pays for the passive base, D2D circuits, extra bonding and test steps, and losses during assembly. Cost per working product across the portfolio captures the combined effect of wafer yield, reuse, test, and assembly. [29]

Wildcat Lake packaging

Intel launched Core Series 3, formerly Wildcat Lake, on 16 April 2026 for value mobile and edge systems. Wildcat Lake keeps 18A but removes the passive base and combines more functions on one die to simplify the package. The two products therefore reveal two distinct ways to commercialize the same leading-edge process. [41]

Panther Lake vs Wildcat Lake Packaging. Source: SemiAnalysis

Wildcat Lake’s 18A die combines up to two Cougar Cove P-cores, four Darkmont LP E-cores, two Xe3 cores and a smaller NPU. A separate platform-controller die supplies I/O, connected through UCIe, Intel’s first processor implementation of the standard. Consolidating graphics remove a tile boundary and the passive base, reducing assembly complexity for a modest-bandwidth value product. It also ties CPU and graphics scaling to the same die, giving up Panther Lake’s ability to swap in a much larger GPU. [42], [43]

Wildcat Lake Die Shot. Source: SemiAnalysis

Intel’s manufacturing recovery

In July 2021, Intel CEO Pat Gelsinger set out an ambitious process roadmap aimed at regaining performance leadership by 2025, later described as five nodes in four years. Five years and one CEO later, Intel’s comeback story is not as unambiguously positive as Pat may have hoped. [44], [45]

Intel once set the pace for process technology, bringing high-k metal gate technology and FinFETs into volume production years ahead of the rest of the industry. Its 22 nm FinFET process reached consumers with Ivy Bridge in 2012. [46] Intel’s integrated device manufacturing (IDM) model allowed its architects and process engineers to co-optimize products and processes. Starting with Sandy Bridge, Intel dominated x86, while AMD struggled with Bulldozer.

That lead faltered at 14 nm and broke at 10 nm. Intel targeted a massive 2.7× density increase, but the node arrived years late and required several revisions before it could support Intel’s full lineup. This delay forced Intel to stretch 14 nm across six generations, while TSMC moved ahead in process technology and AMD recovered in x86.

By 2019, Intel was still shipping 14 nm across most of its product stack, with its 10 nm client ramp focused on Ice Lake mobile processors. Meanwhile, TSMC was shipping N7 and N7+, and AMD’s Zen 2 compute chiplets used N7 to raise core counts and improve efficiency.

Intel’s process failures were central to its decline, but unsound business decisions furthered their downward slide. Product delays compounded product mistakes, pushing client, server, and FPGA roadmaps off schedule. Several attempts to enter AI (Nervana and Gaudi) and networking (Tofino) also failed to establish lasting businesses.

Intel’s recovery has focused on consumer CPUs and advanced packaging. Tiger Lake, Alder Lake, Lunar Lake and now Panther Lake have restored Intel’s consumer roadmap. On the process side, Intel 4 shipped with Meteor Lake, Intel 3 with Granite Rapids and Sierra Forest, and Intel 18A with Panther Lake. Intel has also made advanced packaging part of its foundry offering. However, Intel is still playing catch-up in servers. Several Xeon generations arrived years late and trailed contemporary AMD and Arm server CPUs in performance, efficiency, and core count. The process roadmap is back, but Intel does not hold the same process-technology leadership position it held prior to 10 nm.

The introduction of gate-all-around nanosheets and backside power delivery are two of the biggest changes to transistor integration in a decade. Intel took on both changes at once: 18A paired its first RibbonFET with PowerVia in Panther Lake.

Panther Lake is a substantial manufacturing milestone. Our cross-sections show how RibbonFET and PowerVia reshape local contacts and wiring, while the floorplans show where architectural consolidation and process choices save area. A sustained competitive lead depends on product performance, cost, yield, and the next implementation.

The SemiAnalysis STEEL teardown lab breaks down advanced datacenter and AI hardware. To learn more about our pipeline or to commission a teardown, contact sales@semianalysis.com.

WE’RE HIRING: Architecture, floorplan, packaging, manufacturing, and labs experts. Opportunities from system to transistor and everywhere in between. Check out our Careers page..

References

[1] Intel, “Intel Tech Tour 2025 Panther Lake recap,” Doc. 866361, 2025. Accessed: Sep. 14, 2026. [Online]. Available: https://cdrdv2-public.intel.com/866361/ITT_2025_Panther_Lake_Recap1.pdf

[2] Intel, “Core Ultra processors Series 3 datasheet, volume 1,” Doc. 872188-001, Jan. 2026. Accessed: Sep. 14, 2026. [Online]. Available: https://cdrdv2-public.intel.com/872188/872188-001.pdf

[3] N. Horiguchi and E. Beyne, “Backside power delivery: How to power chips from the backside,” imec Reading Room, Nov. 25, 2022. Accessed: Sep. 14, 2026. [Online]. Available: https://www.imec-int.com/en/articles/how-power-chips-backside

[4] W. Hafez et al., “Intel PowerVia technology: Backside power delivery for high density and high-performance computing,” in Proc. IEEE Symp. VLSI Technol. Circuits, Kyoto, Japan, Jun. 2023, pp. 1–2, doi: 10.23919/vlsitechnologyandcir57934.2023.10185208.

[5] K. Fischer et al., “Intel 18A platform technology featuring RibbonFET (GAA) and PowerVia for advanced high-performance computing,” in Proc. Symp. VLSI Technol. Circuits, Kyoto, Japan, Jun. 2025, pp. 1–3, doi: 10.23919/vlsitechnologyandcir65189.2025.11075006.

[6] K.-F. Cheng, C.-L. Teng, H.-Y. Huang, and H.-C. Chen, “Metal oxide composite as etch stop layer,” U.S. Patent 12 176 247 B2, Dec. 24, 2024. Accessed: Sep. 14, 2026. [Online]. Available: https://patents.google.com/patent/US12176247B2/en

[7] S. W. King, “Dielectric barrier, etch stop, and metal capping materials for state of the art and beyond metal interconnects,” ECS J. Solid State Sci. Technol., vol. 4, no. 1, pp. N3029–N3047, 2015, doi: 10.1149/2.0051501jss.

[8] A. V. Mule’, D. J. Towner, D. Seghete, C. R. Ryder, and A. A. Gonzalez, “Multi-layer etch stop layers for advanced integrated circuit structure fabrication,” U.S. Patent Appl. 20220102343 A1, Mar. 31, 2022. Accessed: Sep. 14, 2026. [Online]. Available: https://patents.google.com/patent/US20220102343A1/en

[9] C.-C. Wang and J. H. Wang, “Interconnect structure with dielectric cap layer and etch stop layer stack,” U.S. Patent Appl. 20230253247 A1, Aug. 10, 2023. Accessed: Sep. 14, 2026. [Online]. Available: https://patents.google.com/patent/US20230253247A1/en

[10] Y. J. Wu, K.-F. Cheng, C.-C. Lee, H.-K. Chang, and H.-Y. Huang, “Etch stop layer for interconnect structures,” U.S. Patent Appl. 20240332070 A1, Oct. 3, 2024. Accessed: Sep. 14, 2026. [Online]. Available: https://patents.google.com/patent/US20240332070A1/en

[11] M. G. Rainville, N. Shankar, K. S. Reddy, and D. M. Hausmann, “Deposition of aluminum oxide etch stop layers,” U.S. Patent Appl. 20180197770 A1, Jul. 12, 2018. Accessed: Sep. 14, 2026. [Online]. Available: https://patents.justia.com/patent/20180197770

[12] J. S. Leib et al., “Conductive lines having molybdenum liner and tungsten fill for advanced integrated circuit structure fabrication,” U.S. Patent Appl. 20240429126 A1, Dec. 26, 2024. Accessed: Sep. 14, 2026. [Online]. Available: https://patents.justia.com/patent/20240429126

[13] M. Naik, “Cobalt enables power and performance scaling at single-digit logic nodes,” Applied Materials, Dec. 17, 2018. Accessed: Sep. 14, 2026. [Online]. Available: https://www.appliedmaterials.com/us/en/blog/blog-posts/cobalt-enables-power-and-performance-scaling-at-single-digit-logic-nodes.html

[14] Lam Research, “Breaking through AI’s invisible barrier with molybdenum,” Lam Research Newsroom, May 6, 2025. Accessed: Sep. 14, 2026. [Online]. Available: https://newsroom.lamresearch.com/molybdenum-metallization-ai-revolution

[15] P. Yashar, G. Malyavanatham, and H. Vijwani, “Integrated circuit interconnect structures with niobium barrier materials,” U.S. Patent Appl. 20240112951 A1, Apr. 4, 2024. Accessed: Sep. 14, 2026. [Online]. Available: https://patents.google.com/patent/US20240112951A1/en

[16] Applied Materials, “Applied Materials unveils chip wiring innovations for more energy-efficient computing,” Applied Materials Investor Relations, Jul. 8, 2024. Accessed: Sep. 14, 2026. [Online]. Available: https://ir.appliedmaterials.com/news-releases/news-release-details/applied-materials-unveils-chip-wiring-innovations-more-energy

[17] Samsung, “Samsung showcases AI-era vision and latest foundry technologies at SFF 2024,” Samsung Semiconductor, Jun. 13, 2024. Accessed: Sep. 14, 2026. [Online]. Available: https://semiconductor.samsung.com/news-events/news/samsung-showcases-ai-era-vision-and-latest-foundry-technologies-at-sff-2024/

[18] C. W. Yeung et al., “Channel geometry impact and narrow sheet effect of stacked nanosheet,” in Proc. IEEE Int. Electron Devices Meeting (IEDM), Dec. 2018, pp. 28.6.1–28.6.4, doi: 10.1109/iedm.2018.8614608.

[19] S. Mochizuki et al., “Evaluation of (110) versus (001) channel orientation for improved nFET/pFET device performance trade-off in gate-all-around nanosheet technology,” in Proc. Int. Electron Devices Meeting (IEDM), Dec. 2023, pp. 1–4, doi: 10.1109/iedm45741.2023.10413854.

[20] D. G. Ouellette et al., “Fabrication of gate-all-around integrated circuit structures having molybdenum nitride metal gates and gate dielectrics with a dipole layer,” U.S. Patent 12 051 698 B2, Jul. 30, 2024. Accessed: Sep. 14, 2026. [Online]. Available: https://patents.google.com/patent/US12051698B2/en

[21] imec, “Advancing the CFET-based device roadmap with novel integration modules and standard-cell architectures,” imec, 2026. Accessed: Sep. 14, 2026. [Online]. Available: https://www.imec-int.com/en/articles/advancing-cfet-based-device-roadmap-novel-integration-modules-and-standard-cell

[22] A. Reznicek, X. Miao, C. Lee, and J. Zhang, “Wrap around contact for nanosheet source drain epitaxy,” U.S. Patent 11 302 813 B2, Apr. 12, 2022. Accessed: Sep. 14, 2026. [Online]. Available: https://patents.google.com/patent/US11302813B2/en

[23] J. A. Kittl, J. G. Hong, D. R. Palle, and M. S. Rodder, “Methods for varied strain on nano-scale field effect transistor devices,” U.S. Patent 9 978 833 B2, May 22, 2018. Accessed: Sep. 14, 2026. [Online]. Available: https://patents.google.com/patent/US9978833B2/en

[24] S. Gandikota et al., “Method of reducing metal gate resistance for next generation NMOS device application,” Int. Patent Appl. WO 2024/137272 A1, Jun. 27, 2024. Accessed: Sep. 14, 2026. [Online]. Available: https://patents.google.com/patent/WO2024137272A1/en

[25] J. Zhang et al., “Full bottom dielectric isolation to enable stacked nanosheet transistor for low power and high performance applications,” in Proc. IEEE Int. Electron Devices Meeting (IEDM), Dec. 2019, pp. 11.6.1–11.6.4, doi: 10.1109/iedm19573.2019.8993490.

[26] C. Yoo, J. Chang, Y. Seon, H. Kim, and J. Jeon, “Analysis of self-heating effects in multi-nanosheet FET considering bottom isolation and package options,” IEEE Trans. Electron Devices, vol. 69, no. 3, pp. 1524–1531, Mar. 2022, doi: 10.1109/ted.2022.3141327.

[27] S. S. Pradhan, D. B. Bergstrom, J.-S. Chun, and J. Chiu, “Tungsten gates for non-planar transistors,” European Patent Appl. EP 3 506 367 A1, Jul. 3, 2019. Accessed: Sep. 14, 2026. [Online]. Available: https://patents.google.com/patent/EP3506367A1/en

[28] L. Schloss and X. Ba, “Tungsten films having low fluorine content,” U.S. Patent 9 754 824 B2, Sep. 5, 2017. Accessed: Sep. 14, 2026. [Online]. Available: https://patents.google.com/patent/US9754824B2/en

[29] Intel Foundry, “Accelerating AI and HPC with advanced process and packaging technologies,” Doc. 367370-001US, Jun. 2026. Accessed: Sep. 14, 2026. [Online]. Available: https://www.intel.com/content/dam/www/central-libraries/us/en/documents/2025-11/intel-foundry-hpc-ai-brief.pdf

[30] M. Lofrano, X. Chang, H. Oprins, and Z. Tokei, “Mitigating the thermal bottleneck in advanced interconnects,” imec, Sep. 28, 2023. Accessed: Sep. 14, 2026. [Online]. Available: https://www.imec-int.com/en/articles/mitigating-thermal-bottleneck-advanced-interconnects

[31] S. Bair, “Intel at ISSCC 2025: Navid Shahriari invited talk, eight papers, forums, panelist & product details,” Intel Community, Feb. 19, 2025. Accessed: Sep. 14, 2026. [Online]. Available: https://community.intel.com/t5/Blogs/Tech-Innovation/Edge-5G/Intel-at-ISSCC-2025-Navid-Shahriari-Invited-Talk-Eight-Papers/post/1667592

[32] D. Chandra, E. Potladhurthi, D. R. S. Reddy, and K. S. Rengarajan, “Tunable negative bitline write assist and boost attenuation circuit,” U.S. Patent Appl. 20160203857 A1, Jul. 14, 2016. Accessed: Sep. 14, 2026. [Online]. Available: https://patents.google.com/patent/US20160203857A1/en

[33] Synopsys, “Advantages of firmware-based training in high-speed DDR IP,” Synopsys IP Technical Bulletin. Accessed: Sep. 14, 2026. [Online]. Available: https://www.synopsys.com/articles/firmware-based-training-ddr-ip.html

[34] N. Muralimanohar, R. Balasubramonian, and N. P. Jouppi, “CACTI 6.0: A tool to understand large caches,” Hewlett-Packard Laboratories, 2009. Accessed: Sep. 14, 2026. [Online]. Available: https://users.cs.utah.edu/~rajeev/cacti6/cacti6-tr.pdf

[35] Intel, “Intel Tech Tour 2024 Lunar Lake AI hardware accelerators,” Doc. 824436, 2024. Accessed: Sep. 14, 2026. [Online]. Available: https://cdrdv2-public.intel.com/824436/2024_Intel_Tech%20Tour%20TW_Lunar%20Lake%20AI%20Hardware%20Accelerators.pdf

[36] Y.-H. Chen, J. Emer, and V. Sze, “Eyeriss: A spatial architecture for energy-efficient dataflow for convolutional neural networks,” in Proc. 43rd ACM/IEEE Annu. Int. Symp. Comput. Archit. (ISCA), Jun. 2016, pp. 367–379, doi: 10.1109/isca.2016.40.

[37] Intel, “Introduction to the Xe-HPG architecture,” Doc. 758306, Nov. 4, 2022. Accessed: Sep. 14, 2026. [Online]. Available: https://www.intel.com/content/www/us/en/developer/articles/technical/introduction-to-the-xe-hpg-architecture.html

[38] Intel, “Core Ultra processors Series 3 for the edge,” Doc. 855291. Accessed: Sep. 14, 2026. [Online]. Available: https://cdrdv2-public.intel.com/855291/Intel%C2%AE%20Core%E2%84%A2%20Ultra%20Processors%20Series%203%20for%20Edge%20Overview_2.pdf

[39] Intel Foundry, “Foveros technology brief,” Doc. 366411-001US, Feb. 2026. Accessed: Sep. 14, 2026. [Online]. Available: https://www.intel.com/content/dam/www/central-libraries/us/en/documents/2025-07/foveros-25d-product-brief.pdf

[40] D. Das Sharma, “UCIe: Building an open chiplet ecosystem,” UCIe Consortium, 2022. Accessed: Sep. 14, 2026. [Online]. Available: https://www.uciexpress.org/_files/ugd/0c1418_c5970a68ab214ffc97fab16d11581449.pdf

[41] Intel, “Intel launches Intel Core Series 3 processors,” Intel Newsroom, Apr. 16, 2026. Accessed: Sep. 14, 2026. [Online]. Available: https://www.intel.com/content/www/us/en/newsroom/news/client-computing/intel-launches-intel-core-series-3-processors-changing-the-game-for-everyday-computing.html

[42] Intel, “Core Series 3 launch press deck,” Apr. 2026. Accessed: Sep. 14, 2026. [Online]. Available: https://download.intel.com/newsroom/2026/Intel-Core-Series-3/Intel-Core-Series-3-Launch-Press-Deck.pdf

[43] Intel, “Intel outlines architectures for agentic AI at Hot Chips 2026,” Intel Newsroom, Aug. 24, 2026. Accessed: Sep. 14, 2026. [Online]. Available: https://www.intel.com/content/www/us/en/newsroom/news/client-computing/intel-outlines-architectures-for-agentic-ai-at-hot-chips-2026.html

[44] Intel, “Intel accelerates process and packaging innovations,” Intel Investor Relations, Jul. 26, 2021. Accessed: Sep. 14, 2026. [Online]. Available: https://www.intc.com/filings-reports/all-sec-filings/content/0001193125-21-224438/d199788dex991.htm

[45] Intel, “Intel reports third-quarter 2021 financial results,” Intel Investor Relations, Oct. 21, 2021. Accessed: Sep. 14, 2026. [Online]. Available: https://www.intc.com/news-events/press-releases/detail/1505/intel-reports-third-quarter-2021-financial-results

[46] Intel, “3rd generation Intel Core processors bring exciting new experiences and fun to the PC,” Intel Investor Relations, Apr. 23, 2012. Accessed: Sep. 14, 2026. [Online]. Available: https://www.intc.com/news-events/press-releases/detail/577/3rd-generation-intel-core-processors-bring-exciting

Extended carrier and bond stack detail

The Chinese AI Infrastructure Boom: Introducing the SemiAnalysis China Datacenter Model

2026-09-25T07:58:22+00:00

China sits at the frontier of the global model race. GLM 5.3 and Kimi K3 are the latest in a run of striking open-weights releases. ByteDance's Doubao serves 345M monthly users as China's ChatGPT, and Seedance is the State-Of-The-Art video generation model.

Every one of those models runs on a datacenter, and China has been building them at a pace that has gone largely unmeasured outside the country. The biggest tenant files no 10-K. Several of the largest landlords have never listed. Most of the primary sources are in Chinese. So the market settled on two lazy assumptions: China is big, and China is empty. Published estimates of China’s datacenter capacity differ by 15x, and reports keep citing high vacancy rates.

As with our flagship SemiAnalysis Datacenter Model, we use building-level data to show which of the most widely cited narratives hold, starting with how large the market is.

The global model tracks over 5,000 facilities across four regions. The US leads the world with 56GW of capacity as of 2026YE, followed by ~15GW for APAC ex-China, ~14GW for EMEA, and ~2GW for Latin America. Until now, it stopped at the border of China. Today we cross it.

Our tracking of 1,000+ datacenter facilities across over 60 players shows that China alone boasts a fleet of over 24GW. Bigger than EMEA. Bigger than the rest of Asia. This excludes ~20GW of dated pipeline and another ~30GW of announced projects.

Source: SemiAnalysis Datacenter Model, SemiAnalysis China Datacenter Model

Growth is at an inflection point. In 2Q26, the combined capex of Alibaba, Tencent, and Baidu (“BAT”) reached $20B, more than doubling YoY, and for the first time on record, all three posted negative free cash flow. This is the largest capex step-up in the sector’s history.

That total leaves out the largest spender of all, ByteDance, which remains private. According to our China Datacenter Model, ByteDance alone occupies roughly a fifth of delivered datacenter capacity in China, and it rents nearly all of it, making it the single most important customer for all wholesale colocation players in the country.

Source: SemiAnalysis China Datacenter Model

The listed players get the attention, but they are only the visible tip of the iceberg. This is also true for datacenter landlords. GDS and VNET, the only two Chinese datacenter landlords listed in the US, signed 1.3GW of wholesale orders in 1H26. But according to our China Datacenter Model, they captured barely a third of ByteDance and Alibaba orders in 2024–2026YTD.

The buildout is a nationwide effort, and the state-owned carriers are also part of it. China’s datacenter market was historically telecom-dominated. The state carriers held the majority of the market share in the 2010s, and still own a third of national capacity today. The national power grid companies invest aggressively too. Their combined capex accelerated from 2024, ending the 14th Five-Year Plan 24% over the original blueprint. The 15th plan (2026–2030) layers another 40% on top, to over $746B (¥5T).

Largely free of power constraints, labor shortages, and public protests, China routinely delivers 100MW datacenter facilities in under 12 months. Modular DCs are the standard playbook, which Tencent deployed its third-gen modular design in 2014. What is old news in China is only now being adopted at scale in America, as we detailed in The Wild Wild West of LEGO Datacenters.

The growth does not stop at China’s border. Overseas leasing by Chinese hyperscalers is set to double from 2026 to 2029 and approach ~4GW of leased capacity in our SemiAnalysis Datacenter Model. That number still understates their offshore compute, because it excludes the hundreds of thousands of GPUs they rent from Western clouds, which we covered at length in How Oracle Is Winning the AI Compute Market.

However, the market also faces some dark clouds. Vacancy rates are high, and datacenter developers compete heavily on price. Chip supply is also constrained, due to export restrictions. But as explained below, neither has stopped AI datacenters from being built and filled at a remarkable speed.

In this report we cover, in order:

Introduction to China’s datacenter industry: the world’s second-largest market, built retail-first, and why that history produced overbuild and high vacancy alongside an AI capacity shortage.
The westward buildout: how the “Eastern Data, Western Compute” policy and hyperscale AI demand are moving the industry inland, at construction speeds and unit costs the West cannot match.
Behind the paywall: we go tenant by tenant through ByteDance, Alibaba, Tencent, Baidu, and Huawei, from their AI and datacenter strategy to leasing and self-build footprints.

Introducing the SemiAnalysis China Datacenter Model

This report is a preview of the SemiAnalysis China Datacenter Model. It is the China counterpart to our global SemiAnalysis Datacenter Model, built to the same standard, bottom-up and building by building.

Contact our sales team at sales@semianalysis.com for more information.

What you’ll find inside:

Tracking over 1,000 facilities across 60+ operators
Quarterly delivery curves through 2032 for every facility, with construction status and timelines

Tenant identification per building, supplemented by tenant allocations per customer-mix for leading operators

Hyperscaler self-build vs leased splits and per-tenant capacity paths

A lease signing tracker and a lease delivery tracker, on one reconciled basis

Eastern Data, Western Compute (“EDWC”) hub and cluster analytics, load-growth and power-draw forecasts

An interactive full map view showing where the capacity is concentrated

Consider it the opening chapter. As with our global Datacenter Model, we will keep adding layers (AI capacity, power, suppliers, supply/demand analysis) with every update.

Let’s dive in.

China’s datacenter industry: built retail-first, flipped by AI

One of America’s key advantages in the AI infrastructure race is that its datacenter industry grew up hyperscale. Multi-hundred-MW campuses have existed for years, purpose-built for a handful of enormous tenants. China’s industry was shaped by telecom carriers and thousands of small retail deployments. Understanding that history explains what’s strange about the market today, from overall high vacancy coexisting with an AI capacity shortage, to why the carriers still own a third of everything.

The following chart previews the whole story. Between 2010 and 2022, a leading colocation player in China watched its utilization rate shrink from the mid-70% range to the mid-50% range. Then, from 2023 onwards, its segment disclosure reveals a two-speed market. Wholesale buildings are filled fast on AI demand, back above 70% utilization, while the legacy retail racks sit near 60%.

Source: VNET disclosure, SemiAnalysis

Era 1 — Carrier hosting (pre-2015): datacenters as telecom real estate

Unlike the rest of the world, where hyperscalers and colos dominate, China’s datacenter market was historically telecom-dominated. The three state carriers (China Mobile, China Telecom, China Unicom) held a combined 60–70% market share, with GDS and VNET, the largest third parties, below 5% each. The carriers enjoyed structural advantages in licensing and networking and benefited from captive demand. A large share of Chinese datacenter demand comes from government, SOEs and regulated industries, which default to state-owned suppliers.

For its first two decades, a Chinese datacenter was essentially telecom real estate, featuring low-density retail racks (~90% of facilities ran below 2kW) rented to websites, online gaming and CDN customers. This was often offered as a bundled product with bandwidth, served by small urban switching centers around Beijing, Shanghai and the Pearl River Delta. Third-party operators existed largely as resellers of carrier space and bandwidth. The hyperscalers were barely a factor, as Alibaba Cloud was launched in 2009 and Tencent Cloud only opened to the public in 2013.

This era matters today because its stock is precisely the capacity now sitting vacant. The legacy retail racks no longer match what an H20 or Ascend server demands, and it’s impractical to retrofit given partial occupancy and high cost. The headline utilization number of ~50% is largely a monument to Era 1 (and Era 3, as we’ll see).

Era 2 — The cloud land grab (2015–2021): growth at any price

The cloud era began around 2015. The two leading players, Alibaba and Tencent, concluded that it was a winner-take-all market and opened a capex race. Alibaba Cloud revenue compounded at a ~110% CAGR across 2015–2019, at the cost of a brutal price war and EBITA losses. Its core products were marked down more than 50% after 17 consecutive price cuts during a 12-month window. Tencent responded in kind, driving combined hyperscaler capex to a 2021 peak.

Third-party colocation platforms (GDS, VNET, Chindata) scaled alongside them. Retail developers, meanwhile, kept building small urban sites on the assumption that cloud demand would spill over indefinitely.

Era 3 — Digestion (2022–2023): the music stops

The wild expansion stopped abruptly in 2022. Policy headwinds, macro slowdown, and share gains by state-owned clouds pushed Alibaba Cloud growth from +50% in 2021 to +23% in 2022, then to effectively zero. Tencent management framed it as a repositioning “from pursuing revenue growth to healthy growth” and saw a similar slowdown. Tencent cut capex budgets by nearly half in the middle of 2022. Alibaba did so in 2023.

Source: Company filings, SemiAnalysis

Almost simultaneously, in February 2022, the National Development and Reform Commission (“NDRC”) launched Eastern Data, Western Compute guidelines and named ten computing clusters within eight national hubs. These regions, especially the western ones, enjoy fast track permitting and cheap green power.

The result was an oversupply despite a slowdown in delivery. Pricing was hit most. Power-exclusive market rates were cut in half from ~$80/kW/month. On the August 2026 earnings call, GDS management estimated another 18 months to fully digest the price reset, even though it has been going on for a few years.

Era 4 — The AI supercycle (2024–): full steam ahead

DeepSeek and the hyperscalers’ own GenAI ambitions woke the market up over the course of 2024-2025. Combined BATB (ByteDance/Alibaba/Tencent/Baidu) capex went from roughly $35B in 2024 to over $50B in 2025 and is on a trajectory to $100B for 2026.

Wholesale orders returned at a scale never seen before, and the western hubs finally found their tenant. GDS and VNET together signed roughly 1.3GW of wholesale capacity in 1H26, with less than 10MW of retail orders. The AI era is wholesale. The legacy retail stock is not participating.

But watching only these two listed names badly understates total demand, as they only capture ~1/3 of ByteDance and Alibaba orders in 2024–2026YTD per our China Datacenter Model.

Source: SemiAnalysis China Datacenter Model

To recap, the four-stage history explains China’s datacenter market today: (1) a retail-focused legacy stock around Tier-1 cities, running 30-50% utilization, often physically unsuitable for AI densities; (2) a wholesale AI-driven buildout happening somewhere else entirely; and (3) a carrier share in structural decline, as hyperscaler self-build and third-party wholesale take over the growth.

Eastern Data, Western Compute: the pipeline is heading west

The EDWC policy designates eight national computing hubs, implemented as ten clusters, spanning both the demand centers (Beijing-Tianjin-Hebei, Yangtze River Delta, Greater Bay Area, Chengdu-Chongqing) and the resource-rich west (Inner Mongolia, Ningxia, Gansu, Guizhou).

Source: NDRC, SemiAnalysis

This is the industry’s top-level design blueprint, and unlike most “guidance” documents, it actually changed how the industry develops.

The policy itself has been covered extensively. What is missing, and what matters, is the transmission mechanism.

In China, the first permit a datacenter developer applies for after securing land is an energy-consumption review filed with the local Development and Reform Commission. That quota is the binding cap on how large a facility can be. Any project consuming over 10,000 tons of coal equivalent (>10MW) cannot legally begin construction without passing this review.

The review regime dates to 2010 and became a rationing tool over time. When EDWC arrived in 2022, the quota system became the most effective mechanism because the same offices that issue quotas are the ones implementing the EDWC policy, so quotas effectively stopped construction flowing to non-hub locations.

Guangdong is the cleanest example. The province stopped issuing datacenter energy quotas in 2021-2022 and reopened applications only for the EDWC hub region in Shaoguan. Similar quota control governs the other eastern metros. Meanwhile, EDWC regions hand out energy quotas, land approvals and tax incentives generously.

Despite the mechanism, for the first two years, migration was slow because the demand of the day (cloud, gaming, e-commerce) required low latency to population, so hyperscalers kept fighting for scarce quota near Tier-1 cities.

A little-noticed episode illustrates how the quota system played out in practice. In mid-2023, Guangdong’s Energy Bureau inspected Tencent’s two self-built campuses in Qingyuan and found each running at roughly 13–14x its approved scale. The projects were labeled “approved small, built big”, and ordered rectification within six months. The local grid company was instructed to hold power supply to the approved quota. Even for Tencent, incremental builds now belong to the province’s designated hub node.

Source: Guangdong Energy Bureau

Today, the campuses keep operating, but as our China Datacenter Model shows, Tencent’s subsequent capacity went to Shaoguan. The reason is not simply regulatory pressure.

Two other things changed the picture.

First, telecom-led fiber investments have significantly reduced latency. Shaoguan again is the model case. Since 2022, the local government has invested ~$600M (~¥4B) in network infrastructure, building 13 ultra-high-speed 400G all-optical routes covering the Greater Bay Area, achieving 1.3ms one-way latency to Guangzhou, 1.66ms to Shenzhen, and under 3ms to Hong Kong. A direct Shenzhen-Guangzhou-Shaoguan hollow-core fiber corridor has been planned to cut latency a further ~30% from 2027 onwards. At those numbers, a Shaoguan rack is functionally a Shenzhen rack for all but the most latency-critical workloads.

Second, AI demand did the rest. Training workloads don’t care about latency, but they care about power prices.

Inner Mongolia is the best example here. The region has China’s largest renewable fleet and, critically, western Inner Mongolia is served by the Mengxi grid, China’s only provincial grid outside State Grid and Southern Grid, with its own multilateral trading mechanism and independent pricing.

As a result, Inner Mongolia has emerged as China’s Johor, the default destination for hyperscale AI capacity, with generous quota allocations, land measured in square kilometers, and, most importantly, power prices at roughly half the Tier-1 city level.

The Johor parallel runs deeper than land and power. On economics, Chinese rents are far lower, but so is the capex, while Johor rents are higher, but so is the cost to build. Our data shows that the two markets' yield-on-cost ranges overlap, with the best Inner Mongolia campuses underwriting to the return of Johor's international developers. However, the trends diverge. Inner Mongolia construction costs keep falling as the prefab approach and domestic substitution compound, while Johor developers face capex inflation and overrun risk in a supply chain running hot. Chinese operators building in Johor enjoy geographical arbitrage leveraging the Chinese supply chain and earn higher yields.

Zoom out from Inner Mongolia, though, and the ten clusters tell very different stories. It is easiest to see when the map is weighted by what is actually live and what is committed, rather than by what the policy designated, as the map below shows.

Source: SemiAnalysis China Datacenter Model

A recurring media take holds that EDWC is a boondoggle of empty western shells. The simplest rebuttal is to look at where the hyperscalers put their own money. As our tracking shows, ByteDance currently operates three self-build clusters: Datong in Shanxi (its earliest), Wuhu in Anhui (an EDWC cluster, serving east-China inference) and a twin-site Inner Mongolia program (Horinger and Ulanqab), with two Ningxia entities registered in Zhongwei (another EDWC region) in July 2026 pointing to a fourth.

Outside of Inner Mongolia, development across the clusters is sharply divergent, but much of that divergence is by design. Gui’an was positioned from the start as a storage hub, as cold data doesn’t mind the mountains. The most famous anchor tenant is Apple’s iCloud China datacenter, operated with Guizhou-Cloud. Less reported is that Huawei and Tencent both have self-build clusters in this region.

Chengdu and Chongqing are different stories. They are pitched as national compute hubs but lost the AI-demand contest to Inner Mongolia on power price and grid flexibility. EDWC succeeded in concentrating the buildout inside the map, and the market then picked the winners within it.

Shanxi is the opposite. It is the largest datacenter region outside the EDWC hub system entirely (the red diamond on the map above), but attracted close to 2GW commitment from ByteDance, Baidu, and JD across self-build sites and colocation leases. The reasons are abundant cheap power (Shanxi is China’s coal province), proximity to Beijing, and first mover advantage. Hyperscalers went there before EDWC existed and kept building out, proof that the quota regime channels the buildout, but electrons and geography still make the site selection.

Speed and cost: a different sport

The US buildout is power-gated with interconnection queues, transformer lead times, behind-the-meter workarounds, which we have covered at length here. China’s is chip-gated: generation is abundant, state-owned utilities treat datacenter load as a development win, permitting is a policy funnel rather than a bottleneck, and construction labor is plentiful. With no physical friction on the facility side, the build cycle is driven purely by internet platform demand.

On speed, the standard delivery time for a 100MW facility has compressed from ~18 months in the cloud era to ~12 months in the AI era, with state-of-the-art projects finishing faster still.

For reference, our US modular deep dive The Wild Wild West of LEGO Datacenters found the best-case full-modular buildout in the US still takes 12–18 months of construction, sitting behind 12–13 months of permitting that cannot overlap. For China, the permitting period before construction is typically 3–6 months, and faster in EDWC regions.

Several structural factors contribute to the overall faster speed.

First, speculative shells ahead of orders. Because hyperscalers now run T+6-month delivery requirements in their tenders, colocation developers pre-build civil works at their own risk, making the tenant competition effectively a race among half-finished shells. When GDS signed a 152MW AI mega-order in early 2025, the customer required delivery within six months and got it.

Second, building greenfield sites in western EDWC hubs changed the construction method. Away from Tier-1 land constraints, developers are moving from conventional multi-story concrete to low-rise, long-span steel halls. The steel is made in a factory while foundations are dug on site, then trucked in and lifted into place by crane. Concrete is used only in the ground, so there is no waiting for it to harden before the next floor goes up. VNET’s Ulanqab footage shows a hall go from bare foundations to a finished building within one summer-to-winter season.

Source: VNET videos, SemiAnalysis

Tencent’s T-Block takes the same logic to its extreme. The entire datacenter arrives as factory-built modules, with the “building” shrinking to a leveled site and a simple steel shell, sometimes no building at all.

Source: Tencent Video, SemiAnalysis

Third, prefabrication turns a serial schedule into a parallel one. The bulk of MEP work moves off-site into factories, so electrical and mechanical fit-out no longer waits for the building. The headline example is Alibaba’s CUBE 5.0 “100-day datacenter”, comprising 30 days of factory production and site prep, 50 days of on-site installation, and 20 days of commissioning, measured from a finished shell to the day servers arrive. This roughly halves M&E and commissioning stage compared with market standard 6-month, and brings ground-break to commissioning down to roughly 7–9 months.

Every factor above points in the same direction, which is more factory, less site. Follow that logic to its endpoint, and you would expect Tencent-style full-modular container campuses to be the national standard by now. They aren’t, for two very Chinese reasons.

First, the labor math is different. Modularization’s biggest payoff in the West is dodging scarce, expensive skilled trades. Chinese site labor is affordable and abundant, so a traditional steel-and-prefab build already hits aggressive schedules. The marginal speed gain from going full-container is small.

Second, the financing dynamics play a role. Chinese colocation developers pledge their datacenters to banks as loan collateral. A proper building with a 20-year design life appraises well, but a yard of containers appraises poorly and carries lower residual value. When your business model is “borrow against the asset, lease it to ByteDance,” you build a real building.

Today, full-modular builds remain limited to hyperscaler self-builds and Southeast Asia projects, but overall construction speed is as fast as needed.

On cost, capex keeps falling even as speed rises. Continued supply-chain compression and rising domestic content have pushed shell-and-fit-out costs to a fraction of comparable US wholesale costs, as we show in our model. Alibaba credits its modular approach with a further >10% cost reduction compared to its earlier version on top of the schedule gains.

Source: SemiAnalysis China Datacenter Model

The demand: five hyperscalers, multiple playbooks

The demand side of China’s AI buildout is concentrated in a handful of names: ByteDance, Alibaba, Tencent, Baidu, and Huawei.

Like their American counterparts, they span social media, e-commerce, search, gaming and short video. However, the consumer internet that built these companies has run out of new users. Today, China’s internet penetration already sits at ~80% and grows only ~1.5% a year.

The growth engines of the past are all in single digits.

Social media: Tencent’s core asset, WeChat, grew MAU just 2% YoY in 2025.

Search engine: Baidu’s 2025 revenue fell 3% with online marketing down double digits.

E-commerce: Alibaba’s customer management revenue fell 7% YoY in the quarter ended June 2026.

Short-video: Kuaishou’s 2025 DAU grew just 2.7% YoY.

Outside of consumer apps, the same titans lead the cloud industry, with a structural handicap. The market is dominated by Alibaba, Huawei and Tencent alongside the three telecom carriers. But unlike the US hyperscalers, which generate roughly a third or more of their cloud revenue outside the US, Chinese cloud revenue is largely domestic. Alibaba Cloud, the most global of the group, still earns only ~20% of cloud revenue overseas, and the carrier clouds are effectively 100% domestic. In terms of “addressable GDP,” the core market of the US tech titans is simply much larger.

Against that backdrop, AI has become the growth engine, which drives the capex inflection point in 2Q26 we mentioned earlier.

Beyond funding their own model R&D, the big platforms have built an investment web across China’s GenAI labs. Alibaba invested in five LLM unicorns. Tencent backed six and wrote the largest external ticket in DeepSeek’s Series A.

The labs, in turn, train and serve on the hyperscalers’ clouds (in the case of the Alibaba/Moonshot relationship) or have their models served inside the hyperscalers’ products (in the case of the Tencent/DeepSeek relationship). Either way, the hyperscaler datacenter expansion is the compute foundation for the entire domestic AI ecosystem. Hence the deep dives that follow.

We provide a snapshot below showing where the hyperscalers stand:

Source: SemiAnalysis China Datacenter Model

The leasing: Today, ByteDance and Alibaba each issue annual leasing demand at gigawatt scale and both are actively expanding overseas. ByteDance's is the largest and leases majority of its footprint, so its tender calendar effectively sets the order book for every wholesale operator in the country. Alibaba, as we show below, splits the demand across self-build, BOT and wholesale. Tencent is the most disciplined. It spent the 2022–23 downcycle closing small leased metro sites and consolidating into its own campuses, before leasing picked up again in the AI era. Baidu tenders at a fraction of the others’ scale.

The self-builds: In the 2010s, Chinese hyperscalers began self-building datacenters as their businesses reached sufficient scale, and the strategies have diverged since. Alibaba, Tencent, and Baidu self-build their strategic campuses and lease the rest. Huawei self-builds almost everything because of its business model. ByteDance, the primary tenant, is now ramping its own campuses fast. Our China Datacenter Model tracks all self-builds, and the map below shows the current landscape.

Source: SemiAnalysis China Datacenter Model

Between pure leasing and pure self-build, a hybrid structure called BOT also emerged. The real anatomy is a two-layer ownership split. The hyperscaler owns the land and the civil shell, while the colocation partner invests in, owns and operates the M&E layer and recovers its capital through 8–10-year service contracts. In this way, the hyperscaler keeps control and speed, the partner’s balance sheet carries the capex, and the IRR is contracted rather than merchant. All three of BAT run some version of this playbook. Alibaba partners with AtHub, Tencent with Kehua Data, and Baidu with Aofei. Alibaba takes the mixing furthest, running self-build, BOT, and wholesale leases side by side within the same cluster, as we discuss below.

ByteDance: TikTok parent, China’s largest tenant, among the world’s largest GPU users

ClusterMAX 3.0: The Industry Standard GPU Cloud Rating System Returns

2026-09-23T13:20:29+00:00

In 8 months since our last major release of ClusterMAX, slavering investors have just about run out of pockets to stuff checks into. GPU supply has gone to zero. Meanwhile, we have been hard at work putting clusters through the ringer.

Weeks ago, we teased this report with some R-rated anecdotes from our experiences probing the security practices of neoclouds, eliciting a PSA from a neocloud customer that you may have heard of.

Today, we finally go through the full breadth and depth of our testing, more thorough in ClusterMAX 3.0 than ever before. This includes compute, networking, storage, orchestration, UI, monitoring, support, and just about everything that you could think to check on a GPU cloud. We’ll explain who’s been cutting deals, whose engineers have been hard at work, and whose clusters you should negotiate to come with a bottle of Aspirin.

Without further ado, voilà the ClusterMAX 3.0 podium:

Source: SemiAnalysis ClusterMAX 3.0, September 2026

Source: SemiAnalysis Neocloud Dashboard, available to our AI Cloud TCO Model subscribers

Results (Executive Summary)

YouTube summary video and podcast discussion coming soon!

ClusterMAX 3.0 debuts with a comprehensive review of the neocloud industry, covering 77 providers.
We increase our market view to cover 323 providers, up from 209 in ClusterMAX 2.0, 169 in ClusterMAX 1.0, and 124 in the original AI Neocloud Playbook and Anatomy article.
We have now interviewed well over 200 end users of neoclouds as part of this research.
We update our itemized list of criteria across 10 categories, and update our direct descriptions of our expectations for Slurm, Kubernetes, Standalone Machines, Monitoring Dashboards, and Health Checks. All of this content is live on our website. We encourage providers to use these lists when developing their offerings. We still consider these lists as an amalgamation of our experience interviewing end users, making them representative of the features that end users expect from their cloud providers.
Nebius joins CoreWeave in the Platinum tier. While CoreWeave still sets the technical bar for others to follow, Nebius is now established as a provider that consistently commands a premium pricing over others. Strong business decisions by Nebius have put them in a position to serve an entire class of neolabs at seller’s prices.
Google Cloud joins Oracle in the Gold tier. Azure moves to Silver, Fluidstack moves to Unavailable, and Crusoe drops to Bronze. Lambda, Firmus and TensorWave remain in Silver, while GMI moves up to Silver from Bronze.
Many companies drop from Silver (or Gold) to Bronze or lower. We raise the bar this round as only 19 neoclouds globally achieve a Medallion rating.
We establish a tier between Bronze and Underperforming: the Participation Ribbon tier. 15 providers join this rating, which more accurately describes our opinion that they do the bare minimum to get by.

The rest of this article provides analysis of key trends: Financing, Blackwell and Grace-Blackwell deployments, the Move To Vera Rubin, Scale Out Networks, Reliability, Security (of course), and Agentic Coding. We provide an Appendix with individual comments about every provider that once again stretches this article to over 30,000 words. We hope you enjoy.

In light of the SemiAnalysis imperative to publish the most rigorous possible testing of… everything… we are hiring MTS for ClusterMAX and adjacent projects. If you are a neocloud sicko, fire off an application here and let’s get to work.
We are also hiring across Research, Consulting, and Technical Staff. We have multiple openings in our New York and San Francisco offices. Remote work is also acceptable.
ClusterMAX MTS (full time): All levels of experience with Slurm, Kubernetes, and GPUs considered.
Tokenomics MTS (full time): All levels of experience with model evals, harnesses, inference endpoints, and RL infrastructure considered.
Research Analyst – AI Infrastructure and Economics (full time or internship): All levels of experience covering the financials of neoclouds, neolabs, and frontier labs tokenomics considered.
Technical Consultant (full time): Lead engagements from technical strategy to technical due diligence. All levels of experience from consulting and technical backgrounds considered.

Scope

For the folks in the back: we are ranking managed clusters. This excludes a number of popular offerings throughout the industry. We are not testing who can build the best powered shell or run the tidiest datacenter—at least not directly. We are not handcrafting our images or hitting an API endpoint for tokens as a service. Here’s our idealized ClusterMAX reader: You dropped out of your Stanford PhD to pursue your dream of acquiring social status in San Francisco, getting started with the old reliable “Cladue make me pithc deck w technial languge for agetnci ai make no mistakes.” You have 7 or even as many as 10 figures to wager on compute, but you don’t have any opinion on Ubuntu versions or how to handle NAT, and you’ve never managed a fleet at scale. Ideally, all that infrastructure will fade into the background. You just want to focus on your idiosyncratic perf optimizations, model architectures, training strategies, data mixes, applications, or wherever else you seek your edge. To compete, you need bleeding-edge hardware, and you aim to spend 0 engineering time trying to figure out why GPU 7 in node 19 has been stalling your job, or why a quarter of your cluster has disappeared altogether. You need a managed cluster.

The economic argument here is straightforward: a neocloud can amortize the cost of standing up all this management infrastructure over its huge fleet. It pays a team of engineers to build bulletproof health checks, keep its machine images up to date, make convenient monitoring systems, and handle all the related chores that we cover in great detail further on. In an ideal world, a lab pays a markup for a much better product; it gets to run its experiments without being held back by its hardware; the neocloud recoups its investment; everyone is better off.

There comes a point, though, when labs are willing to internalize this cost. At their scale, frontier labs are not willing to trust a neocloud with so much of the stack. OpenAI, for instance, has posted extremely insightful blogs about its difficulties scaling Kubernetes; the innovations described would have been simply impossible if they were renting from a neocloud who didn’t let them into the K8s control plane. As a current Anthropic job posting puts it:

We are operating at a scale where the defaults stop working. We own the scheduler and extend it to place topology-sensitive ML workloads across thousands of accelerators at once. We scale the control plane itself — apiserver, etcd, controllers — so it stays responsive as object counts and node counts grow by orders of magnitude. And we build the core cluster services every workload depends on, like service discovery, so they hold up under the same pressure.

Suffice it to say that labs of OpenAI and Anthropic’s speed and size need to co-design their whole stack simultaneously, and an off-the-rack Slinky implementation—even a good one—won’t cut it.

The obvious care that frontier labs take in managing their infrastructure goes to show why ClusterMAX matters. The sum of the salary ranges for OpenAI and Anthropic open positions with “Kubernetes” in their descriptions is, last we checked, $27,769,274-$42,687,654, not to mention the salaries they pay to the huge teams they already staff. If you’re not anticipating a trillion-dollar liquidity event and you need your infra to be good enough to give you a fighting chance, you need a neocloud to do the heavy lifting for you.

At least on paper, smaller neocloud customers are David fighting Goliath. OpenAI and Anthropic have remarkable growth rates on their respective ~~logistic~~ exponential curves, and our Datacenter Industry Model currently models them at 56.3% of all lab compute YE2027.

OpenAI Compute Capacity. Source: SemiAnalysis Tokenomics Model.

As we will describe in more detail later, there is also a Matthew principle in effect here, due to financing: the profitable frontier labs have an easier time locking down capacity than smaller ones, which get stuck with fat prepays, worse prices, and a much harder time planning years in advance.

Does this mean that managed clusters are dead? cLUsTerMaX iS wAshED? In truth, although Anthropic and OpenAI are taking larger slices by the year, managed clusters as a business are still growing exponentially. We’ve helped countless labs find compute, countless more are on the hunt, willing to pay, and waiting for more capacity to come online. Anthropic and OpenAI are the biggest customers, but there is a long tail of mere multi-hundred-million-dollar deals whose aggregate value is enormous.

One plausible scenario in which managed clusters gain ground on frontier-lab bare metal involves open-source progress. Serving open-source models is already more lucrative than many realize; we recently modeled that “you can make over $100M per MW per year selling open source tokens,” leaving plenty of room for profit even at today’s rental prices. One could imagine open-source models increasing their market share, whether the marginal adopter is more cost-sensitive or algorithmic progress closes the gap to closed-source. Regardless of the dynamics, this would increase the relative standing of managed clusters. Just about any supply-side fracture would make labs marginally less willing to handle the infra, and marginally more willing to rely on neoclouds.

The layers on top of these clusters are rapidly maturing, but they lie outside the scope of ClusterMAX. There’s a handful of niches to consider. There are inference endpoints, which can be either public or private, and bill by the token. There are post-training services, which customize open-source models for specific use-cases. And there are sandbox services, which provide the infrastructure for agents, especially during RL rollouts, and simplify and optimize container and CPU management. These all fit loosely in the category “GPU services” and the border between them can be blurry. For example, a cloud might spin up an inference endpoint with spare capacity after it’s sold managed clusters, or it might provide sandbox infrastructure and also consume it internally as it sells RLaaS. We will be publishing much more about this in the near future, but it is beyond the scope of this report.

Testing Methodology

For ClusterMAX 3.0, we requested the following from every provider:

32 GPUs (4 nodes of 8-way HGX, or 8 nodes of 4-way NVL72)
High bandwidth network (we asked for 800G RoCE or XDR InfiniBand, but many providers did not have it yet)
10TB+ of high performance file storage (supporting NFS/POSIX mounts and a RWX StorageClass via csi driver)
10TB+ of S3-compliant object storage
A monitoring dashboard (usually based on Grafana)
5 days to test Slurm, and 5 days to test K8s (can be done in parallel, on a SonK cluster, or sequentially, if the provider wanted to take the nodes back and re-provision them)
Blackwell, not Hopper (B200, B300, GB200, GB300) for NVIDIA, or MI355X for AMD are considered acceptable. H100 is now over 4 years old!

We tested the cluster through 3 phases, described below. Of course, we always complement this testing by contacting customers of all these providers and taking their feedback into account.

Phase 1: Audit

First, the Audit covers yes/no questions. Is the cluster setup properly? Is software installed? Is it up to date? Do utilities work as expected? It takes about 15 minutes to run. It’s available on GitHub or via {uv} pip install clustermax.

The audit checks the configuration before we put it under load. It covers hardware inventory, software and firmware versions, GPU access, containers, scheduler configuration, networking, storage, health monitoring, and security. We check which components apply to the environment we’re testing, and display pass, warn, fail, and skipped for all our checks. We have released this part for free and will maintain it over time. The rest we keep internally for now.

Phase 2: Performance

Performance puts a pass/fail threshold on the performance characteristics of both the individual components of the cluster and the cluster as a whole. These are done through microbenchmarks, and real-world benchmarks.

Specifically, we test:

GPU Compute
Networking
Storage
Lifecycle
Training
Inference

Below we explain this in more detail, but keep in mind that what we test evolves over time.

GPU Compute

We start by checking the real-world GEMM performance on the GPUs. GEMMs are the most critical operation in modern AI workloads, which we have explained many times before, with Grouped GEMMs being particularly important for modern MoE models.

We test Grouped GEMMs on cuBLASLt (or hipBLASLt) and DeepGEMM across many precision types: BF16, FP16, TF32, FP32, FP8 E4M3, MXFP8, and NVFP4, depending on what is supported by the chips in the cluster. We use a common set of shapes for different gate_up and down projections at varying batch sizes in the Kimi K2.5, K3 and DeepSeek V3, V4 Pro models.

We then test GEMMs, GEMV bandwidth, and MAMF (Maximum Achievable Matmul FLOPS, from Stas Bekman), which runs a sweep for a while on each GPU, per precision, using cuBLASLt (or rocBLAS). We report FLOPs for all of our GEMM tests.

Source: SemiAnalysis ClusterMAX Results Dashboard

We also test memory, through a GEMV test, and `nvbandwidth`. GEMV measures matrix-vector multiplication with a skinny vector, reporting GB/s as a proxy for the streaming behavior of low-batch decoding. NVIDIA’s `nvbandwidth` utility is built into DCGM health checks, and measures h2d (CPU-to-GPU), d2d (GPU-to-GPU), and d2h (GPU-to-CPU) bandwidth. Finally, we also have a custom PyTorch script that measures the same paths through actual tensor copies, recording bandwidth for pinned h2d, d2h, and local HBM copies.

Last, we test HPL-MxP for the boomers. HPL-MxP solves a large dense linear system using low-precision factorization, which means we hit compute, memory, and communication together. The test covers compute, HBM, MPI broadcasts over the scale-out network, NVLink, and numerical correctness in a single workload. We report FLOPs across a range of number formats depending on the chip type.

Although GEMMs are the heart of modern AI workloads, these microbenchmarks very rarely distinguish providers—it would be hard for a provider to mess up this functionality. It’s important to check raw GEMM perf is during an extended burn-in, as described later, when a chip’s clock speeds might drop to avoid overheating if it is not properly cooled. On a GEMM burst, you’re unlikely to find much of note. We run these tests on every provider because it’s quick and because a cluster is immediately unusable if they fail, but we worry more about other tests.

Lifecycle

In order to understand the ease of use of the cluster throughout its lifecycle, we have a set of contrived tests that we use. We start by checking internet speed using multiple approaches for download/upload speed, mostly from Cloudflare. We then test the time it takes to install common packages like PyTorch and vLLM via pip and uv, as well as download containers from Docker Hub, ghcr.io, and nvcr.io, and pull a model from Hugging Face and ModelScope.

We then redo the pip and uv install tests with fresh package caches on each available storage tier: home, shared filesystem, local NVMe scratch, and any additional mounts (referred to as `/home`, `/data`, `/scratch`, and other). We then run `import torch` from each location to see how long that takes. On most providers, this takes 1-2 seconds, but on some filesystems with LOSF performance issues, it can somehow take 10-20 seconds.

Finally, we take the models downloaded earlier, and run a vllm serve command to test the time until we can hit /v1/models. Again, while certain providers can load a small model into GPU memory from shared storage in 10-40 seconds, some providers take over 2 minutes.

Source: SemiAnalysis ClusterMAX Results Dashboard

Networking

We benchmark a wide suite of communication performance for inter-node and intra-node transfers. This includes MPI collectives, running NCCL or RCCL tests launched through MPI, measuring collective bandwidth and latency across message sizes from 8 B to 16 GB for all-to-all, all-reduce, all-gather, and other collectives. The same workloads are also run through torch.distributed in PyTorch. In addition, we use RDMA perftest, specifically ib_write_bw and ib_read_bw, we measure point-to-point bandwidth on each network rail, along with host-memory fallback (via PCIe), to identify any issues.

We grow the world size in factors of 2 until we’ve used our whole cluster, keeping track of how performance scales. We also run tests with NVLink disabled, particularly on GB200 or GB300 NVL72 clusters, in order to isolate the performance of the scale-out network. It’s important to be careful in your accounting here, since world size, algorithm, and job placement relative to scale-out boundary materially affect results.

Source: Quickly checking what’s supported on our cluster’s networking stack out of the box

Source: SemiAnalysis ClusterMAX Results Dashboard

Storage

We run fio, ior, Elbencho, a custom checkpoint save/load benchmark written in Torch and Torch DCP, and a custom datagen benchmarked shared with us by Skild AI. Datagen is probably the most interesting, as it approximates their real workload in a robotics data pipeline: aggregate write throughput generating the video (MP4) and parquet sensor dataset.

The test that finds the most issues, though, is good old-fashioned fio. We sweep sequential/random reads/writes with buffered/direct I/O with 1MiB sequential blocks and 4 KiB buffered / 64 KiB direct random blocks from c1 all the way up to c32, or, if we have more than 4 nodes, as many clients as the cluster can tolerate. If storage is misconfigured in any way, there is almost always some setting in this fio sweep that shows poor perf. We track throughput, IOPS, and latency, and of course we also discover inconsistencies in metadata if the test fails with too many clients.

Source: SemiAnalysis ClusterMAX Results Dashboard

Of course, this significantly depends on how big the volume we’ve been given is and how the SLO is defined. Take these results with a grain of salt—the chart above really only shows that Azure gave us a PB-scale mount. This grain of salt also goes to show why having data on lots of clusters is important. Buffered tests, for example, have higher latency, as you might assume, and can also produce very noisy results if the test is not run for long enough. But how much higher latency, and how noisy? To know the difference between what’s “good enough” and what needs to be flagged to a provider, it’s crucial to compare to a big peer set.

For object storage, we run this suite again against an S3 bucket. We also run a custom benchmark that measures large object throughput on sequential read/write, small object PUT and GET rates, and read latency (p50, p90, p99). In this testing, we keep everything consistent. As always, though, if we notice a cluster struggles with a certain workload, we linger, running more tests to more precisely isolate the issue.

Training

Moving into the real world, we start to see how issues on the network or storage reveal themselves in real testing. We run two jobs via torchtitan: llama 3.1 8B pretraining on the C4 dataset with FSDP, and GPT-OSS MoE training via torchtitan with expert parallelism. The former is FLOPs bound, the latter is collective bound (unless you’re on NVL72). So, the latter makes it plain to see when something is wrong with the network.

Source: ClusterMAX Results Dashboard

Inference

We run the InferenceX AgentX benchmark in single node and multinode scenarios with different models to find compute-bound, memory-bound and comms-bound regimes with profiling traces on, treating the InferenceX results as the benchmark performance result to compare to. As with our training tests, this highlights the importance of the various cluster components covered in the microbenchmarks. If a cluster can’t survive NCCL tests, it certainly won’t be able to produce many tokens, and our InferenceX tests show how goodput is lost.

Phase 3: Reliability

Just because we see solid performance during a benchmark does not mean that the provider can drive solid goodput, and it certainly doesn’t mean they will provide a nice support experience. So, we test reliability, monitoring and autoremediation in multiple ways.

Burn-in

We start with an 8-hour burn-in on the GPUs and network, by running large GEMMs on the tensor cores and all-to-all communications at the same time, with a monitoring script tracking temperature, power, clock frequency, FLOPs, network connectivity, latency, bandwidth, and of course tailing the kernel ring buffer for any errors. You would be shocked how many hardware issues we induce with just this simple test. We cannot emphasize strongly enough how important it is to stress the GPUs and the network at the same time. Burn-in requires simultaneous thermal expansion and contraction of both of these components to approximate the real behaviour of these systems under real stress from real workloads. Lots of providers still use scripts that burn the GPUs and network separately.

Fabric

Next, we test the sustained performance of the scale-out and scale-up network fabric by looping through all-to-all, all-gather, and reduce-scatter of large message sizes. We report average, minimum, and maximum bandwidth, plus collective errors. On the frontend network, we also test management-network packet loss, RTT, and TCP bandwidth between every directed pair across four nodes. It’s not often that this extra test catches errors that the burn-in did not, but we like the data, and it has revealed big performance differences on certain Ethernet networks under load.

Storage

Continuing on, we test filesystem endurance by running 7.5 minutes of sequential mixed reads and writes, followed by 7.5 minutes of random mixed reads and writes, at each client count, using fio (as described previously). We increase clients across the allocation and report bandwidth over time, IOPS, tail latency, and performance degradation. Interestingly, we have seen some providers have bandwidth degradation of almost 40% under load, when compared to the initial benchmark.

When object storage is available, we also run a 15 minute mixed PUT, GET, LIST, and DELETE workload, measuring throughput degradation, p99 GET time-to-first-byte drift, throttling, and errors. We’ve seen about a 10% difference in these tests.

Orchestration

On Kubernetes, we test “churn” by repeatedly creating/deleting GPU pods and checking the API for GPU allocation. We report both scheduling and startup latency, and report failures when kubectl delete po gets stuck. We also increase batches of CPU-only sandbox pods, measuring how many reach Ready, how long they take, and why the rest remain Pending.

Last, we test the PVC lifecycle by repeatedly starting a pod that mounts a volume, writes and fsyncs a file, then gets deleted. We wait for teardown between cycles and use a warmup cycle to remove the initial image pull from the measured runs.

Destructive

Finally, we get to the fun part: breaking things on purpose.

First, we reboot all the nodes in the cluster. You would hope that this is not a destructive test, but it is. On Kubernetes, we cordon and drain the node, issue the reboot, and require a changed boot ID, then wait for it to return to the cluster. The timer stops when we can run nvidia-smi in a fresh pod on the node. With Slurm, we just get an allocation or SSH and run sudo reboot. Slurm-on-Kubernetes is a little different, so we attempt to do things from the Kubernetes layer to test it correctly. Our scripts time everything throughout.

For injecting failures, we start by using the DCGM injection method. If it doesn’t work properly, we follow up by writing a synthetic NVIDIA XID or SXID message into the kernel log, and watch how the provider’s existing health checks and scheduler respond. We leave their health agents and drain automation alone, so it’s up to their tooling to detect the error and act on it. Most health checks are set up to read from the kernel ring buffer, but some are not, so we communicate with the provider to make sure that our trigger will work and that we have access to pull it. As a final, trusty method, we reset the GPU’s upstream PCIe bridge, which produces a genuine XID 79.

In all these cases, we time how long it takes to detect the failure, how long the node stays in `DRAIN` state, and how long it takes to run a basic command once it returns to the cluster. On AMD systems, we inject a UMC RAS error and track everything the same way.

This is our most critical test. Providers where we hear a bunch of customer complaints about reliability generally do not have health checks, monitoring dashboards, and autoremediation in place. Reliability is the #1 most important criteria to many of the biggest customers in the world, as we discussed in great detail in our article “How Much Do GPU Clusters Really Cost?”

Source: ClusterMAX Results Dashboard

It is worth noting that our hands-on testing is only a part of the overall ratings. There are many things that we cannot test hands on, and need to use other research methods to understand in detail. Namely, performance at scale, reliability over time, support experience, pricing, GPU availability and delivery timelines.

Coming Soon: Agentic Workloads and RL

We have two more test suites that we will say much more about in upcoming work.

CPU Compute

The CPU is a critical performance consideration for modern agentic workloads, which we have discussed previously. We have developed a comprehensive benchmark that pushes CPU performance to the limits on exactly these agentic workloads, via common Linux tool use and more microbenchmarks in an upcoming project called SandboX.

SandboX runs a range of CPU-intensive workloads representative of code execution, including ring-shuffle producer-consumer queues, Redpanda (measuring a local Kafka-compatible broker and its throughput), and a dynamic load bench.

We also benchmark the sandbox lifecycle of Kubernetes clusters, for insight into how co-locating sandboxes for RL on the leftover CPU/memory resources in a GPU clusters might work. We report cold-start latency, scheduling throughput, density per node, capacity, and baseline.

Reinforcement Learning

An RL run has many moving parts: generators that roll out trajectories, environments that execute actions in sandbox, and a trainer that receives rollouts, updates the policy and pushes updated weights back out to the inference engine. Scaling RL means balancing maximizing utilization of trainer, inference engine, environment sandbox executions, rollout transfer, weight sync among other things, bottleneck on any one can stall entire RL stack.

We’re also seeing the emergence of Post-Training-as-a-Service, or RLaaS, where customers bring their own environments, or have forward-deployed engineers build them, and hosted training providers set up the infrastructure to post-train open-weight models. The pitch is customization while protecting the customer’s IP, which apparently is Satya’s tagline now.

In our upcoming PostTrainingX benchmark, we sweep trainer-to-inference GPU allocations, batch sizes, rollout concurrency, and policy-staleness limits to study their effects on performance and training stability.

We track generator performance through input/output tokens per second per inference GPU, rollout latency, KV-cache occupancy, and prefix-cache hit rates. Communication efficiency through weight-synchronization latency, transfer bandwidth, and network telemetry; and trainer performance and stability through training throughput, optimizer-step time, MFU, train/rollout KL, observed policy staleness, clipping fractions, gradient norms, and rejected rollouts.

For environments, we measure setup time, tool-execution time, timeouts, and infrastructure failures. GPU utilization, memory usage, and power consumption provide system-level context, while archived trajectories, logs, and exact environment definitions make results reproducible.

The goal of PostTrainingX is to benchmark every provider. That includes hosted training platforms like Applied Compute, Azure Foundry, Fireworks, Thinky, Baseten, Trajectory, Engram, AI21’s platform etc, as well as open-source frameworks like Miles, slime, prime-rl and verl. The end result should give customers the data to decide whether to run their own post-training stack on open-source frameworks or hand it off to a hosted RL provider.

We’ve already started running on open-source frameworks, including Miles, prime-rl, and verl, and will expand to hosted providers next. Stay tuned…

Industry Trends

Financing

The hottest topic in neocloud land over the past few months has been financing. Everyone wants to “follow the money”, which is exactly what we do in our AI Compute, Capital, and Markets Model, which we launched in a recent article breaking down how NVIDIA’s Backstop Universe. Below we will discuss a few more hot topics.

Debt and Accessing Debt

Today, neoclouds have realized that a creditworthy customer contract can help them obtain debt at lower rates than their own credit rating would otherwise support. This means that providers without a committed customer (or “IG offtaker”) may need more expensive debt or equity-like capital in order to buy GPUs and either get their business off the ground or grow it. Meanwhile, lenders also need evidence that a provider can deliver its cluster on time and meet the SLAs in their customer contract. Raising debt depends as much on lease termination rights as it does on the cash available for debt service, as well as the TCV. This has led multiple insurance providers to enter the market with parametric offerings that protect neoclouds and their lenders against contract terminations, typically offering a one-month bridge to find a new offtaker in the event of a termination. We are big fans of this structure, and believe it to be a solid method for lenders to use to de-risk these transactions.

Counterparty Risk & Nvidia’s Balance-Sheet-as-a-Service

In a large neocloud project, the GPU lender and the datacenter lender can face different counterparties within the same project. For example, in the Anthropic TPU neocloud structure, Broadcom supports equipment financing while Google supports datacenter rent. Since several loans in the supply chain can depend on the same AI lab, there can be systemic, correlated risk, even when each loan names a different neocloud or datacenter borrower. Like we just discussed, construction delays can trigger cancellations and impair cashflow.

Enter Nvidia backstops. Since Nvidia supports GPU demand through revenue floors, landlord guarantees, and leases that it plans to transfer to third parties, they are able to help neoclouds and AI labs obtain investment-grade financing without a hyperscaler being involved. Nvidia earns its GPU margin at the initial sale but can now also receive a share of revenue above their “AICP floor”. Their financial support therefore drives demand for their hardware. The Balance Sheet Is The Moat.

Depreciation Schedules

This topic has died down a little since we went after Dr. Burry in our article last November. Maybe it was all those 6 year contracts people are signing, or those pesky 4-year-old H100s that just won’t drop in price! Who’s to know for sure.

Either way, we separate equipment purchase dates from in-service dates when we model depreciation. A 6-year depreciation assumption needs provider-specific support, while in NVIDIA’s AICP program, the 6-year revenue floor does not actually establish the useful life of the GPUs. Our depreciation sensitivity tests change modeled margins quite a bit, but IRR remains the priority for lenders, so we won’t be sharing them here. And of course, financial analysis does not establish a real price that a buyer will pay for GPUs in the future. That is up to supply-and-demand dynamics, and the ultimate question: residual value. With lenders unable to set loan repayment schedules separately from the accounting assumptions that we use to depreciate equipment, we are stuck in a world where no one will take on the risk of assessing a residual value on 4, 5, or 6 year old GB300’s today.

Blackwell and Grace Blackwell

Now let’s get back to the technical stuff for the rest of this article. The good stuff.

As we discussed in great detail in ClusterMAX 2.0, the difference between operating 8-way H100 HGX servers and GB300 NVL72 rack-scale systems is massive. Providers who cut their teeth installing H100s 3-4 years ago, are now contending with the following:

Grace CPUs based on ARM (instead of x86 Intel or AMD)
Blackwell GPUs with sm100/sm103 (with CUDA 13.0+)
Mandatory direct liquid cooling (DLC)
High voltage power (130kW+ per rack, via 3phase 408 or 480V power delivery)
Scale-out networks with 800G CX-8 NICs and 51.2T or 102.4T (800GbE RoCE Spectrum-X) or 115.2T switches (800Gb XDR InfiniBand)
Scale-up networks with NVLink switches and backplanes at rack-scale, 72-way
BF-3 or BF-4 DPUs for the frontend (or converged frontend/storage) network
Multiplanar networks with shuffle boards/shuffle cables

All of these changes have implications on provisioning software, monitoring systems, training of technicians, facility-level electrical, cooling, and mechanical systems, OEM support contracts and relationships, and more.

Just because a provider was successful in the Hopper generation, does not mean that they will be successful in the Blackwell generation.

In our rankings, we put a premium on any provider that delivered Grace Blackwell to us, and frankly gave them a lot more leeway when it came to difficulties during provisioning, monitoring, and reliability/hot spares.

This means that if a customer rents standard HGX nodes that merely connect by RoCE or InfiniBand, we expect to see autoremediation with hot spares available: replacements in under an hour end-to-end and detection in around 2 minutes. However, with an NVL72 system, it does not make sense to hot-swap a tray on a different scale-up fabric.

So, in our testing, as long as a health check notices the unhealthy node (in 2 minutes or less, still) and keeps workloads from being scheduled on it, it has done its job. As mentioned above, we advise many labs and clouds on GB300 NVL72 SLAs, and the most common pattern we see could be called “NVL64+”. This means that there is an SLA applied to each node, but in the event that 3 or more of the 18 nodes in a rack fail at a given time, the whole rack is considered down. This is due to simple powers of two: lots of jobs are divisible by 64, and run on 64 GPUs, while the max config to fill up 72 GPUs is world_size=8 * num_jobs=9.

Moving to Vera Rubin

Interestingly, moving from Grace Blackwell to Vera Rubin requires a lot less changes than moving from Hopper to Blackwell. Providers still manage an Arm-based CPU, a liquid-cooled rack-scale architecture, a 72-way NVLink scale-up network, a multiplanar scale-out network, and some Bluefield DPUs. Back when GB200 came along, all this stuff was new!

The major changes are in the move from 800G CX-8 NICs to 1.6T CX-9 NICs on the scale-out network, and increased power moving from ~130kW to ~200kW per rack, including 800V DC options.

Early demand is more robust than in the Blackwell transition—having partially to do with the fact that it’s easier to port GB software to VR than it was from Hopper to GB—and we’re hearing from many providers that they are on track to deliver VR earlier than expected. Bring up has been surprisingly smooth, and the physical design seems to be much more mature.

Scale Out Networks

The scale-out network is the last big design decision a provider still owns. Nvidia and the OEM/ODM lock down most of the NVL72 rack. Outside it, providers pick the topology, the cabling, and how Kubernetes or Slurm hands the fabric to a job. We found problems at every layer.

Starting with the physical layer, 800G CX-8 NICs are multiplanar: each NIC splits its bandwidth across independent switch fabrics, called planes. Total bandwidth stays at 800G per GPU. More planes give each switch more effective ports, and each NIC more paths to uplink, which needs a shuffle. Every NIC lane must land on the right plane, so shuffle boxes put the fiber mapping in a patch module while shuffle cables build it into the assembly. Both work, but there are tradeoffs in terms of reliability (how often you need to service the devices) and serviceability (how easy it is to repair or replace a device).

Oracle is a pioneer in this physical design, and it shows when you get a GB300 node. Each 4x GPU tray we got had 16x physical 200G ports, which means 4 planes per GPU. Linux exposed 4 usable 800G RDMA VFs (rdma_vf_rail0 to rdma_vf_rail3), one per GPU, each in its own VRF. The aggregate PFs failed with no usable GID, so we had to rely on their SPCX NCCL plugin, using 16 queue pairs per connection, and adaptive routing across planes.

Why bother doing this? The answer is scale. Oracle builds Acceleron on this physical design. Their MRC topology paper shows how each NIC splits into 8x100G, which turns a 51.2T switch into 512 ports, and supports 131,072 GPUs at full bandwidth, with only 2 tiers, and a max of 3 switch hops. OpenAI released MRC through OCP in May 2026, and Oracle runs it at Stargate Abilene. The software layer allows MRC to spray one connection across paths and planes, place data out of order, recover loss with selective acks, and uses SRv6 source routing to steer around failed or congested links. This logical layer can get complicated any scale. Its where jobs have issues.

To allocate an RDMA device and use it correctly, Kubernetes must hand each pod the right devices and interfaces. In other words, there are many ways to configure and use the NVIDIA NetworkOperator. Get it wrong, and it’s a headache for everyone.

Our scripts recognizes the following deployment patterns with the NetworkOperator (resource names are examples from the clusters we inspected):

rdma_shared_device: rdma/rdma_shared_device_* with a resource-backed NAD, or a host-network fallback
sriov_host_device: nvidia.com/hostdev or nvidia.com/rdma_host_dev with a HostDeviceNetwork NAD, for direct device access and GPUDirect RDMA
sriov_legacy: a VF via NAD-bound nvidia.com/<resource> and a SriovNetwork NAD, plus RDMA CNI where needed
sriov_ib: an InfiniBand VF such as nvidia.com/mlnxics with a SriovIBNetwork NAD. Partitioned fabrics also need the right PKey and UFM config.
ovs_offload: nvidia.com/switchdev with an OVSNetwork NAD, for hardware-offloaded Open vSwitch.
rdma_ib / rdma_roce: rdma/ib, rdma/roce, or nvidia.com/rdma_*, with one NAD per rail or a host-network fallback.
Provider-specific: AWS EFA (vpc.amazonaws.com/efa and libfabric), DOKS (rdma/fabricN with roce-net-fabricN@fabricN), and GKE DRANET (DRA claim templates and pod resourceClaims).

Device setup and the NCCL plugin on top of it can change job performance. On Google’s GB300, the built-in NCCL library picked an unroutable link-local GID and hung when we tested. Setting NCCL_IB_GID_INDEX=3 fixed it, and Google’s gIB plugin hit 99.5 GB/s on a 16-node, 16 GiB all-to-all vs 95.3 GB/s from other providers, i.e., full performance.

With that said, multinode NVLink is its own problem. The ComputeDomain CRD in NVIDIA’s DRA driver gives each job its own IMEX daemons and channel claims, independent of RDMA. Google, GMI, and Firmus Kubernetes used ComputeDomain, with different RDMA paths: DRANET at Google, shared devices with host networking at GMI, per-rail VFs at Firmus. Meanwhile, Oracle, Azure Slurm, GMI Slurm, Firmus Slurm, and Nebius Soperator supplied IMEX from the host with no tenant claim. Azure AKS ran multi-node NVLink with no tenant-visible ComputeDomain, despite GPU DRA ResourceSlices, and the channel source and tenant isolation boundary were undocumented.

Each ComputeDomain must stay inside one NVLink domain, since crossing domains needs RDMA. In other words, this is topology-aware networking on the hierarchical scale-up NVLink + scale-out RDMA networks. When we launched jobs incorrectly, without ComputeDomain awareness, they just hung in clique setup when placement crossed racks. We will leave our rants about EFA not supporting DeepEP and MoonEP for the AWS provider review later on in the Appendix.

Reliability, Health Checks, Autoremediation, and SLAs

Around the time we started our research for How Much Do GPU Clusters Really Cost?, we started injecting failures into every cluster we tested. The differences between providers were big. Handling a failure takes two things:

Identify it happened
Fix it properly

To identify failures, we recommend monitoring dashboards. A good dashboard shows the failed component, the affected jobs, the current scheduler state, and when each check last ran. A stale green result is not evidence of health. Of course, a monitoring dashboard relies on some telemetry, so you have to have DCGM setup correctly (and securely).

To fix failures, we expect autoremediation: reboots and replacement with hot spare nodes. Autoremediation will not cover everything. Some failures need human intervention, RMAs, and other work that the provider must communicate directly to the customer. We covered the exact cost, measured in the form of goodput, in our article on this topic.

NVIDIA recently open-sourced NVSentinel, a fault detection and remediation system for Kubernetes. It reads DCGM, syslog, and cloud maintenance events, then can cordon and drain nodes, reset GPUs, or reboot. The default install is monitoring only. The software is the easy part. The important part is the mapping from each XID to the action the provider takes. A contained XID 94 needs an application restart. An uncontained XID 95 needs GPU recovery. Clearly, if you naïvely reboot every node for every XID, you’ll kill healthy work, while if you leave GPUs schedulable after serious XIDs, you risk workloads crashing.

This is also why many customers want bare metal only. They just want the provider to close support tickets. Closing support tickets is harder than it sounds. A given ticket can come from any of 100+ XIDs, each with a different meaning and resolution. SOPs cover everything from swapping a drive or power supply, to cleaning cables, to reseating memory, GPUs, or NICs, all the way to RMAing individual components or whole server trays.

There are a lot of different choices for autoremediation, and the right one depends on the system. Also, remediation is a different question for GB than for HGX. On HGX, you swap in a hot spare node with 8 GPUs. On NVL72, you cannot hot swap 8 more GPUs into the NVLink domain. A failed tray takes 4 GPUs down, and the options are to run the rack degraded or to swap the whole rack. The good news is that GB300 tends to fail less than GB200. Either way, the flow from an error to the action in the datacenter is what matters.

Some providers told us they wait before acting, to leave customers time to get their stuff off the node. That is cope. We are simulating a hard failure. The node is dead, and there is nothing left to get off it. (It’s a general truism that if a provider says they choose not to do something because they want to “give their customers optionality,” it is cope, and it’s only a matter of time before the provider recognizes it as such.)

This is where owning and operating the datacenter differs from renting colo. A provider that runs its own facility controls the technicians, the spares, and the SOPs on site. A provider in colo can depend on the landlord’s remote hands and their schedule. Both show up in how fast capacity comes back, and that is what an SLA has to measure.

When it comes to SLAs, we have developed a standardized set of SLAs for 3 tiers of providers

In these documents we provide the following:

Technical definitions for “node”, “rack”, “cluster”, and “site” in two scenarios (HGX 8-way systems, i.e. H100 or B300, and MGX 4-way systems, i.e., GB200 or VR NVL72)
Technical definitions of “downtime” for all systems
Three tiers of associated SLAs (“Bronze”, “Silver”, and “Gold”, with corresponding thresholds and credit penalties associated with downtime)
Two-way commitments for measuring downtime and providing credits
Recommended exclusions from downtime measurement, such as software upgrades, security patches, and physical maintenance
Recommended monthly review cadence for provider performance against SLAs
Contract termination rights for the buyer associated with downtime below recommended thresholds
Definitions of acceptance testing, with descriptions of the types of tests to perform across GPU compute, networking, storage and software to validate a cluster is operational and can be accepted, with recommendations to contact SemiAnalysis for a ClusterMAX technical assessment
Contract termination rights for the buyer associated with missing an agreed acceptance date
Definitions for “force majeure” clauses
… and more

We encourage Compute Buyers (Neolabs), Cloud Providers (Neoclouds), Debt/Financing Partners, and Insurance Providers to use these terms as a basis for their contracts, and to include a SemiAnalysis Technical Assessment during acceptance testing and monthly SLA performance reviews. We perform this Technical Assessment using our cmax utility and a comprehensive, proprietary process.

Please contact us at clustermax@semianalysis.com for more information.

Security

Since we put out “Most Neoclouds Suck at Security,” we’ve been encouraged by the response. Our friends have been running our CLI on their clusters to find which dusty old packages to replace, and there has generally been increased impetus to treat this issue with the seriousness that it merits.

Still, we think it deserves more scrutiny. As discourse reaches a fever pitch over whether AI will kill everyone, there has been almost no analysis of the concrete means by which it could actually elude control and do harm. OpenAI and Anthropic are spending gazillions of dollars doing gain-of-function research, some of their infra has been revealed deeply suspect, and yet it has mostly escaped commentary that GPU clusters—the physical hosts that underlie these models—often have nothing like the immune system that you would hope.

The imperative of good security practices holds regardless of your beliefs about AI capabilities. Even if these clusters were made no more vulnerable by AI tools, the fact would remain that we are spending billions on infrastructure that is not secure as it ought to be. There are plenty of unpleasant scenarios to imagine that involve nothing more than a guy with Vim and a good understanding of operating systems. Read through the Hugging Face reports and tally the number of mundane cybersec errors that let the fire spread.

It’s also worth emphasizing that this is, to some extent, a structural risk. As the industry navigates the hazards of heavy-handed regulation and uncritical social pushback, a breach of an LLM factory is the last thing it should invite into headlines. This is true for cybersecurity reasons—each compromised node increases the number of neighbors for bad actors to explore—as well as ordinary sociopolitical ones—for better or worse, detractors will paint the neocloud industry with a broad brush.

Be a good neighbor and don’t get hacked.

Agentic Coding

Agentic coding is already putting margin pressure on managed GPU clusters because customers that use AI tools are more willing to take bare metal or lightly managed clusters. We saw evidence for this all the time in our testing. If we needed to add Linux users, but there was no script on the cluster, Codex /goal make useradd script and add my whole team saved the legwork. Especially on clusters with poor documentation, agents helped grind through all the possible configurations to get us unblocked. The corresponding Slack conversation often went something like:

“hey guys, how are we supposed to use XYZ?”

“nevermind, Codex told me ABC.”

“ya that’s right. we’ll add that to the docs.”

Agents helped us most in environments with clear success criteria and good context; otherwise, their approaches to GPU systems were naïve. They have the relevant knowledge, but it is not highly weighted relative to plausible but wrong approaches. Claude, for example, has a habit of overwhelming the Slurm controller by using loops instead of job arrays. One agent of ours decided the NVLink fabric was too much of a pain to configure, instead launching jobs over InfiniBand and declaring victory; another made a similar mistake running storage tests on NVMe instead of NFS; another blamed the provider for poor perf after it tried to schedule GPU jobs onto a CPU node. We usually let agents run overnight, either with `/goal fix this part of the cluster so tests can run` or `/goal try to break this part of the cluster with a realistic workload`. Overall, this allowed us to multiply our output, but sometimes validating the results took longer than it would have to do it ourselves. If you have a good idea how to set up a cluster, agents make you more powerful, and you might even entertain working with a worse provider than otherwise. If you don’t know what you’re doing, your AGI advisor will cheerfully tell you to shoot your foot off. In this semi-legible business, agents magnify skill deltas rather than flattening them.

An insightful example of this is in CoreWeave’s bring-up, which was already among the most rigorous in the industry. Now, in addition to a stringent classical process, they have LLMs run statistics over their fleet, hunting for patterns that flag failures earlier and improve their procedures. CoreWeave also has a NodeBot that provides lightweight reasoning on top of their automated alerts, summarizing the situation and suggesting next steps to simplify diagnosis for the humans in the loop. For example, the team described an event where a thermal interface material pump-out alert should have moved the node out of the cluster for investigation, but a simultaneous PMU halt alert competed to reboot the node instead of triaging it. NodeBot identified the correct action despite that conflict. CoreWeave has experience running GPUs at scale, knows countless rules of thumb that are absent from the training data, and has experts constantly closing the feedback loop to improve their use of these models; even if OpenAI built a banger chip with AI-generated kernels, we don’t expect CoreWeave to be threatened by upstart neoclouds with vibecoded datacenter automations any time soon.

Despite the fact that—let’s be honest—everyone in the industry is using AI at least for some workflows, the only AI-first offering we saw was a nice MCP server provided by AWS. Otherwise, all the documentation was nominally written for humans. The best thing that a provider can do to cater to agents is to provide exhaustive documentation—ideally, much more thorough than a human could ever have gone through—and leave golden recipes on the cluster as a starting point for hillclimbing. In addition to the legacy processes that are not yet AI-first, there are many systems that are strictly worse because they are not built for agents. For example, although letting AI manage your jobs could be good for cluster utilization, it doesn’t respect the old-fashioned systems put in place to aid research. CoreWeave and Nebius have many Slurm-layer optimizations—real job accounting systems with RBAC and priorities enforced on users and groups, carefully managed partitions, job arrays, logs and metrics—that are wasted when everything is run agentically. CoreWeave also tells us that they have seen head nodes run out of memory as remote-control Claude sloporchestrates the cluster. It will be interesting to watch providers improve their AI agent affordances.

Going Forward

As described throughout this article, we have been historically testing clusters. But the market has clearly moved in two opposing directions.

First, the biggest labs are buying bare metal at massive scale, measured in the 100’s of MWs, and have the talent in house to manage their own training clusters, orchestration software, monitoring and reliability. In essence, their neocloud providers (when they use them, and don’t do self build) are datacenter technicians who close tickets. This is a complicated business, and not to be taken for granted: there are 172 unique, documented ways an NVIDIA GPU can fail (XIDs). And many of these failure modes can overlap, resulting in an incredible surface area of different failure scenarios that require a combination of simple troubleshooting logic, human experience, and increasingly, agentic research by AI to recover from these failures.

Meanwhile, many startups are raising 10s or 100s of millions of $$ and spending effectively all of that money on compute. But since so much of this research is driven by RL, via continual learning on real production traces for long horizon agents, the needs for both offline, throughput-optimized inference, and online, latency-sensitive inference is very strong. At the same time, more and more customers are looking for the simplest way to purchase vanilla open source tokens for their workflows.

This leads to our next article, coming soon, where we will describe the Anatomy of an Inference Endpoint, moving towards the inclusion of inference endpoint testing in future versions of ClusterMAX. We believe that all neoclouds need to have an endpoints business going forward.

We will also be evaluating RL infrastructure beyond inference: sandboxes, hosted training, evals, and more. We continue to see lots of new providers enter (and exit) the market every day. It is an exciting time to be in AI, and neoclouds are at the centre of it.

If you got to this point in the article and are still eager to learn more about our individual experience on every provider, you should consider coming to work at SemiAnalysis!

We are actively hiring across Research, Consulting, and Technical Staff. We have multiple openings in our New York and San Francisco offices. Remote work is also acceptable.
ClusterMAX Engineer: all levels of experience with Slurm, Kubernetes, and GPUs considered (full time)
Tokenomics Engineer: all levels of experience with model evals, harnesses, inference endpoints, and RL infrastructure considered (full time)
Research Analyst - AI Infrastructure and Economics: all levels of experience covering the financials of neoclouds, neolabs, and frontier labs tokenomics considered (full time, or internship)
Technical Consultant: lead engagements from technical strategy to technical due diligence. all levels of experience from consulting and technical backgrounds considered (full time)

Thanks for reading, and we look forward to seeing you for future versions of ClusterMAX.

Appendix: Provider Reviews

Platinum

CoreWeave

CoreWeave remains the industry’s sharpest, sturdiest, most proactive neocloud.

Our CoreWeave clusters are exemplary in all crucial categories. The health checks work as intended, the reliability is excellent, and nearly all tests reach expected values out of the box. Along with Nebius, CoreWeave serves as a measuring stick for others in the industry—not only in the abstract sense of “excellent performer,” but also in the literal sense that we compare their clusters to lower-rated ones as a means of precisely describing others’ shortcomings. Whereas other neoclouds have to be nagged to refresh their moldy GPU drivers, CoreWeave has not only taken care of the basics, but stood up a significant independent stack for marginal performance and reliability improvements.

The ClusterMAX 2.0 report went into great detail on CoreWeave’s design decisions, including its bare metal provisioning, Slurm fork, use of DPUs, and health checks. Since then, one smart feature that the nerds at CoreWeave have added is called “GPU straggler detection,” which is integrated into their best-in-class dashboard. Rather than leaving their customers to dig through logs or do binary search over their fleet to isolate a bad rank, CoreWeave applies handrolled algorithms to NCCL telemetry and recommends a remediation strategy based on its findings. The CoreWeave team tells us that they use this on calls with their customers just about every day. There are all kinds of soft failures that don’t write an XID or leave any other obvious signature, but waste GPU hours as the job waits on a straggler.

Better than spotting bad GPUs is not scheduling them in the first place; CoreWeave also has systems to make sure every node clears a high bar before it’s trusted with a customer’s workloads. When a node is idle in CKS, CoreWeave runs burn-ins for 20-30 minutes of every hour, validating that scale-up and scale-out fabrics are healthy, GEMM numbers are up to spec, D2H and H2D bandwidth are high, and so on. These tests are pre-empted by customer workloads; invisible to the operator, they help make sure that it’s not production workloads that catch faults. CoreWeave is certainly not the only provider with active health checks, but these are unusually robust.

This diligence extends to CoreWeave’s bring-up process. Provisioning around 10,000 GPUs per week, CoreWeave believes that they have more data than God about what proves a Blackwell rack healthy. In CoreWeave’s view, Nvidia diags are good at catching many failures, but their uniquely deep dataset has allowed them to roll out a custom suite of supplementary tests, including for NVLink. The heart of this process is the Fleet LifeCycle Controller (FLCC), which “automates node provisioning, testing, and monitoring” and owns the path from power-on to handover. Importantly, as others operating fleets at scale can corroborate, there’s a long tail of obscure failure modes not well covered by Nvidia documentation. CoreWeave keeps statistics on these failures to better predict future faults and bakes its response patterns into FLCC. This is one reason that experience with operations at scale is a crucial criterion in our ratings. There is a huge difference between a newbie and a provider like CoreWeave that has rigorous runbooks, automation, and burn-ins. As described in the “Agentic Coding” section above, CoreWeave uses LLMs on top of this system to glean insights that would not be spotted by a human engineer. CoreWeave also works with manufacturers before delivery. Involvement varies by OEM or ODM and by production stage, but in general, the goal is to work backward through the supply chain to catch issues as early as possible: never catch in prod what you could have spotted in bring-up, and never catch in L11 diags what you could have spotted on the factory floor.

Their experience with Blackwell NVL72 bring-up and validation, along with their close relationships with Nvidia and Dell, have allowed CoreWeave to be the first provider to announce a VR200 NVL72 system passing L11 diags. CoreWeave has described VR as “the generation where we bring our own IP.” As the physical demands of compute continue increasing, in CoreWeave’s view, “how the building operates is now part of how the computer operates—they are not two individual things anymore.” Thus, CoreWeave introduced its custom Racky, Valvey, and RLCC in a recent blog post. Particularly interesting is Valvey, a programmable per-rack liquid cooling valve assembly. This gives CoreWeave fine-grained control over each rack’s cooling loop and also, in case of a leak or other emergency, the ability to trigger a shutdown. Despite beginning to adopt a hyperscaler mentality in building its own hardware, CoreWeave frames its approach as anti-hyper: whereas traditional infra emphasizes redundancy and is built to prevent failures, CoreWeave builds to fail gracefully and recover fast, limiting failure domains and allowing nimbler response to the inevitable failures.

Amidst the current trillion-dollar buildout, however, CoreWeave is facing issues responding to the massive increase in cu. Of course, there is plenty of good news to mention. Its relationships with Meta, Microsoft, OpenAI, and NVIDIA—its largest customers—appear strong, and it recently struck a multi-year deal with Anthropic. It has announced splashy deals with HRT and Jane Street this summer. It is also aggressively expanding its capacity in APAC, indicating that it “expect[s] international markets to become a major driver of growth.” Yet it sits with $35B in debt and has already seen investor resistance getting projects off the ground. As a result, CoreWeave has become focused on long-term bare metal contracts, which are easy to fund with an investment-grade counterparty, rather than managed clusters, which have higher margins but which the market is less keen to fund. Moreover, our last writeup mentioned an impressive lineup of recent acquisitions, but since then its Core Scientific merger fell through and nothing else has been announced.

In terms of feedback, some of which is a bit nitpicky:

We wish that their managed SUNK auth RBAC was done on an per cluster basis
We wish that enabling local GPUDirect Storage could be an 1 click button in the UX console
Onboarding new users and where they put the the ssh pub key input in the UX console is extremely confusing to many users that tried

CoreWeave has begun to diversify from bare metal, first by offering self-service SUNK clusters, which gives them the option to sell into the on-demand market when they have capacity for it. They also launched a managed inference platform, which recently disclosed $100M ARR. We are actively testing this for an upcoming article on serverless inference endpoints, though were disappointed to find no support for public endpoints paid by the token. For now, let’s just say that CoreWeave has some work left to do on their inference endpoints.

Nebius

Nebius was rated the 2nd-best neocloud in the world in ClusterMAX 2.0, yet it remained in Gold. This time, Nebius is unquestionably an industry leader with strong offerings in every category and the ability to command a significant price premium. Nebius moves into Platinum.

Nebius’s relationship with Nvidia remains strong, including a $2B investment in March, as Nvidia disclosed a 9.3% overall stake in July. Nebius was the first provider to deploy HGX B300s back in December of 2025, and has been among the first neoclouds in VR NVL72 bring-up, expecting to begin deployment late this year or early next.

Nebius’s buildout continues across a number of sites in the US and Europe as they recently raised their 2027 year-end contracted power target to 5 GW. In terms of offtakers, Nebius has signed large deals with hyperscalers Meta and Microsoft and smaller ones with Reflection AI and Palantir, making it one of a few neoclouds with multi-hundred-MW hyperscaler agreements that still regularly competes for much smaller startup contracts. They are the most active of all neoclouds in the short-term cluster market, serving many happy customers. The days of Nebius spooking prospects with their Russian accents are behind them. This is mirrored by its capacity procurement: unlike many of its peers, Nebius has opportunistically snatched up sites in the 5-20 MW range through a broad network of datacenter and software partners.

In terms of the clusters themselves, well, a good cluster is generally boring. On Nebius, burn-ins run without errors; Slurm is topology-aware; the packages we need are on the cluster and generally recent; WAN is good; the orchestration layers are free of footguns. One point of distinction technically between Nebius and everyone else is that Nebius builds and open-sources its preferred SonK flavor, Soperator. (CoreWeave, of course, builds SUNK in house, but it is not open-source.) This round, a Gcore cluster we tested used Soperator, and as did Voltage Park and FPT clusters from previous rounds. Nebius writes that “the real magic of Soperator lies in how we use ‘jail’ Persistent Volume”: Nebius provides a VirtioFS volume at `jail`, then bind-mounts it at `/`, so you don’t need to worry about pointing your cache to the right directory, or think much about taking your files with you when you `salloc` a node. You’ll see in our writeups of other providers a handful of gnarly issues with managing the storage system on SonK: Slurm and Kubernetes have different philosophies of persistence, and it’s not always straightforward to provide the illusion of bare metal when in fact you’re several layers up, standing on an ephemeral Kubernetes pod. Nebius’s approach makes this stress-free for the operator.

In particular, Nebius provided us a handful of storage tiers on every worker: `/jail` VirtioFS, `/home` NFS, `/mnt/data/` VirtioFS, `/mnt/local-nvme` ext4, and `/mnt/memory` tmpfs, as well as first-class S3. For I/O-heavy jobs, `/mnt/data` is recommended, whereas `/home` suffices for shared code and other data with lower-concurrency access. When we first tested `/mnt/data`, we saw very slow 4 KiB sequential allocating writes, while concurrent directory creation from 128+ ranks sometimes returned `EEXIST` due to inconsistent metadata visibility, killing our jobs. With our feedback on the pathological workloads, Nebius retuned this filesystem, and it proved sturdy and fast. We pushed it around and verified that the errors were resolved; per-client, it outperformed the median, and we could further verify that it scaled to handle demanding workloads across two racks.

Nebius has put a great deal of work into its health checks, and in our testing, they performed well. Nebius automatically returned a node to service after a synthetic error injection on our GB300 rack; the process is not yet optimized for speed, taking 8h40m from start to finish, but detection is immediate, and the process ultimately works. Moreover, the other GB300 racks we tested did not automatically reboot and return failed nodes, so the fact that this process is self-driving counts in Nebius’s favor. (As we describe in the “Blackwell and Grace Blackwell” section, it’s common not to autoremediate GB300 nodes at all, since there is no real way to hot-swap a node into an NVL72 rack, and troubleshooting is generally way more complicated than with HGX servers.) Nebius also provided us with an excellent suite of dashboards shedding light on, among other things, cluster health.

This all leads to Nebius watching their revenue per MW climb compared to CoreWeave’s (and their market cap along with it). Since they have so much more capacity to sell at the current prices, we expect this trend to continue. Anecdotally, we have seen Nebius get very aggressive during recent negotiations, including a case where they offered a tranche of capacity as a straight-up auction. In general, demanding high prices and sizable prepayments that reach 100% on 1 year commits seems like quite a nice business, especially when those prepayments can cover the entire capex of the servers at the current TCV. Infinite Project IRR anyone?

At their largest bare metal sites, Nebius continues to face delays in construction and permitting, including both Béthune and Vineland, which we have covered extensively for clients of our industry-leading Datacenter Model. But as more chips of all types come online, Nebius’s track record puts them in a strong position to continue to grow.

Overall, we are consistently impressed with Nebius. While we do believe they still trail CoreWeave on many technical aspects and relationships with both NVIDIA and the frontier labs, solid business decisions have established them as the default neocloud to serve the neolabs.

Gold

Google Cloud

Amidst the strange retreat of Google Brain and DeepMind from the frontier, GCP has been dealmaking like few others. Notable partners include Palo Alto Networks for $10B and Thinking Machines Lab for a “multibillion-dollar deal,” along with expansions of its lucrative relationship with AI safety org Anthropic. With acquisitions of cybersecurity startup Wiz for $32B and energy startup Intersect for $4.75B in cash, Google has been aggressive in expanding its technical capabilities. It also exposes itself to new revenue streams by ramping TPU sales instead of only renting them through GCP, a development we are very keen to track. Its own TPUaaS is still growing, though, thanks to a $5B deal with Blackstone that is targeting 500 MW in capacity. Google has gigawatts in the pipeline, and we look forward to continuing to track its continued competition with the other hyperscalers and labs for access to power and permitting.

With a unique position cutting across so many layers of the stack, Google may have more total engineering talent than any company in the world, yet it has a notably imperfect history in supporting innovation. In ClusterMAX 1.0, we noted a myriad of issues with its platform, yet we predicted GCP would reach Gold or Platinum promptly. It has taken until ClusterMAX 3.0 for this to come true, but Google is finally and comfortably among the best managed cluster providers.

The GCP GPU experience is less refined than that of our Platinum providers, CoreWeave and Nebius. The console feels like the DMV compared to the streamlined UIs of the younger, more focused neoclouds. Access required a Google Cloud CLI that was occasionally uncooperative, forcing us sometimes to switch to the browser-based connection in Google’s console, which was also unreliable. There was a small misconfiguration on our GB200 GKE cluster, where default StorageClass refused to attach to the `a4x-highgpu-4g` nodes, forcing us to try again with a non-default class to get block storage. Nitpicks aside, GKE is very well set up, and we enjoy using it. Google’s managed Slurm is officially generally available and quite solid, with good defaults and health checks configured. A lot of the testing that we did for ClusterMAX 2.0 on their managed Slurm offering is now valid, since the bureaucrats in GCP Product Management are allowing real customers to use the offering, not sticking it with labels like “Beta” or “Pre Release” or “not yet GA.” (By the way, many of the neoclouds we’ll discuss after Google should consider using some of these labels more often—the balance is somewhere in the middle here.)

The biggest point of distinction between our GCP cluster and the other clusters we tested was networking. Our first NCCL test hung because automatic GID selection chose the wrong address, but pinning `NCCL_IB_GID_INDEX=3` completed the run. Even then, our NCCL tests were unsatisfactory: you would like to see a roughly logistic curve as performance increases monotonically in message size, but instead we had the following jagged shape:

This could conceivably have been an issue with NCCL itself rather than a problem with our GCP configuration. NCCL’s heuristics don’t always pick the right protocol and algorithm for the message size and topology, and some hand-tuning is always expected. However, we have gigabytes of networking data gathered at this point, and we knew to expect better perf. The issue turned out to be simply that we did not have the gIB plugin enabled. This is a set of NCCL plugins Google that ships for improved perf on Google’s RoCE network. As of 26.07, gIB is baked into the NGC PyTorch image, so Google’s customers get the custom stack by default. With gIB installed, we got a better chart with 4.4% higher peak throughput for 16-node all-to-all. (Note that these tests are both with the multi-node NVLink fabric shut off to isolate performance of the scale-out network, which is RoCE in this case.)

While not perfect, this looks much better, and the topline number is solid. At this point, it was clear that the fabric was healthy, the configuration was good enough, and an engineer looking to squeeze more juice out of the system would have a solid baseline to start from.

This is amazing work by Nvidia and Google to set up auto activation for Google’s ConnectX-7/8 NCCL plugin. Previously, users would need to mess around with the correct library load paths and env vars to get it set up correctly and optimize performance on ConnectX NICs on GCP Nvidia GPU machines. We gave feedback a couple of quarters ago, and now, it is fully automated!

Source: Nvidia

GKE’s health checks worked well. As with other providers who have experience with NVL72 systems, Google’s health checks intervened but left it up to the operator to bring the sick node back to the fleet. In particular, after our XID was injected, the health check marked `GPUUnhealthy=True` and `cloud.google.com/health-check-status=warning`, but left the node schedulable. This configuration was easy to work with, it’s what GCP’s customers prefer as a default, and it can be configured to the user’s liking to respond to errors of various levels of severity.

To continue improving, GCP could take a look at all the quality-of-life changes that Nebius and CoreWeave have made in the recent past. It should ship a better console that’s easier to navigate, simplify IAM and RBAC, and ensure cluster access is painless, either through standard SSH or `kubectl` or by making its existing CLI bulletproof. Especially now that gIB is available in the default Nvidia images, we hope that GCP’s operators have no trouble getting good performance out of their GCP scale-out. They could also continue to pursue neolabs in the mid-market, improve the hands-on support experience, and improve their relationship with Nvidia. GCP consistently loses so much business to smaller, less capable neoclouds, because they either lack the GPU capacity or refuse to serve the market at a certain price point.

We look forward to testing GCP’s VR NVL72 systems when they are available and seeing GCP’s continued improvements on usability and performance.

Oracle

As we mentioned in ClusterMAX 2.0, Oracle is in a unique position among hyperscalers in that its growth must come from large contracts. Unlike AWS, Google, and Azure, Oracle does not have any agreements to sell frontier tokens, and unlike Meta and Google, it does not have a non-frontier lab it can hang its hat on. Instead, it continues to cut huge deals involving OpenAI, Meta, and Nvidia, reporting “the delivery of 850MW additional datacenter capacity” from June through August, an eye-watering number. We project its relative growth in 2027 to be the largest of the hyperscalers, though from the smallest base.

We tested two Oracle clusters, which we will describe in order. The first was a GB300 NVL72 cluster on Slurm, set on a multi-planar network. This was vanilla Slurm, but due to customer demand, Oracle has Slurm-on-OKE on its roadmap. Once our allocation began, we tried to find our way in through the console, but it proved, like all hyperscaler consoles, a forbidding place.

Importantly, Oracle’s console did not support RBAC—nothing on the console is connected to SSH users—so all our SSH keys had to be managed on the machine. At one point, attempting to append to `authorized_keys`, our intern forgot a `-a` flag and removed all our previous keys. Classic user error, obviously, but this goes to show why we prefer to have a hardened path on a console. We’d rather not trust our interns—especially that one, no offense—to type the magic words on the command line when a mistake can lock us out of the cluster. Thankfully, Oracle’s engineers caught the mistake immediately and fixed it for us.

We hope they fix this and add enterprise grade RBAC.

As expected, Oracle’s cluster performed well from day 1. They provided us with scripts for InfiniBand and NCCL tests, which was helpful because vanilla scripts do not properly use all 4 rails. Burn-in was flawless and Oracle’s managed Lustre saw strong throughput. Oracle’s health checks worked as intended, draining the node on our synthetic error but leaving up to the user to decide whether or not to terminate and replace the GB300 node. Oracle’s managed Prometheus and Grafana were good, though not perfect. For example, they are not integrated with the end user experience for job accounting. However, on the positive side, health metrics are now integrated into Cluster Manager in the console, making it easier to get a view of the crucial statuses at a glance.

Next, we tested OKE with MI355X. This was the only provider other than TensorWave who let us test-drive AMD GPUs. The ride was rough.

As we complained about last time, at first, we needed to SSH into an operator node to steer Kubernetes. This time, the team quickly solved our Oracle CLI issues and we were promptly able to download our credentials locally. By far the greater problem on this cluster, though, was with the NICs. Peak performance was near line rate, so there was no issue with the hardware that we could detect. Rather, the issue was with the ionic drivers. Our per-rail `ib_write`/`ib_read` sweep across all eight NICs hit 391 Gb/s on every rail, then wedged the node the moment it finished: when the pods were deleted, the perftest processes never completed their `rdma_cm` teardown, the forced `kill` timed out the NIC’s destroy-CQ command, and the RDMA admin queue on `ionic_0` and `ionic_4` went dead. Worse, both ports still showed `ACTIVE` at 400 Gb/s and the node stayed `Ready`. The passive checks did not check out the RDMA control path, and no active check had run in eight days. We found the issue by grepping `dmesg` after RCCL kept failing. Oracle diagnosed it quickly on a weekend and rebooted: their own `ibwrite` health check had failed on this cluster in the same way days before. After a call where we explained the pathological workloads, they spoke with AMD and pushed different drivers and firmware to the cluster, which solved the problem.

Separately, we had a GPU intermittently dropping off the PCIe bus, neither caught by the health check nor properly surfaced in the dashboard. Oracle promptly swapped out the node and responded to our feedback on the dashboard.

This was a mixed episode, but Oracle’s responsiveness was good. It’s especially undesirable that no health checks alerted, even though these drivers are known to be flaky. Still, Oracle’s close relationship with AMD is clearly an asset. They were able to speak to the relevant teams, diagnose the issue, and put a patch in place quickly. This type of support is not guaranteed at organizations of Oracle’s scale. Interestingly, Oracle is also the only provider ranked Silver or above, and one of two ranked Bronze or above, with zero installed libraries with applicable CVEs. This is more of an expectation than a benefit for us, but it was a clear demonstration that their automation works for deploying and upgrading clusters.

If we were a neolab, we would come away from this testing experience with confidence in working with Oracle. Yet this is a moot point given Oracle’s increasing commitment to bare metal deployments for frontier labs such as OpenAI in Project Stargate. Success in that business depends more on getting government approval for gas pipeline construction than it does on RCCL tests.

Still, Oracle has made progress since last we tested them, and the experience is very good overall. We look forward to continuing to keep tabs on them going forward.

Silver

Lambda

Lambda was most recently in the news for participating in a $35B Anthropic deal for 350 MW in Nueces County, alongside Hut8 and Nvidia. Before that, they were rumored to be preparing for a 2027 IPO, and they separately raised $1B, $926M, and $1B in debt for their buildouts. Amidst all their growth, after judging them Silver in ClusterMAX 2.0, we were interested in evaluating Lambda’s clusters this time around.

Onboarding began with a nice deck and PDF explaining what to expect on the cluster, and access was sorted out promptly.

We criticized Lambda in ClusterMAX 2.0 for their reliability; this round of testing, it was one of Lambda’s points of emphasis. Their passive health checks cover the relevant conditions and ultimately autoremediated three errors we simulated during testing. Two synthetic XIDs were autoremediated in less than 15 minutes, and there were connected dashboards to monitor node health. Our test that triggered a genuine XID 79 by resetting the PCIe secondary bus reset brought the node into `NotReady` with no `GpuXid`, no cordon, and no monitoring visibility on the Kubernetes layer, but it ultimately rejoined the fleet after 2 hours. Our node-reboot tests on the Kubernetes layer also completed without incident.

The orchestration layer has improved significantly, though it still has a handful of issues. Slurm configuration left CPUs per task unset, so Slurm’s default allocation gave one logical core unless the launch requested more. Passwordless sudo is not enabled on Slurm, as we are using managed slurm rather than unmanaged Slurm. Lambda is the only provider we’ve encountered that makes this distinction, but clearly some of their customers want root access to their own cluster, while others do not (?). We enjoy having root on our own cluster.

Also missing was SSH to the compute nodes without an allocation. This creates a headache during debugging if someone else is running a job and you can’t attach to their allocation, or a job is stuck. Another reason why we enjoy “yolo mode” on our clusters.

More importantly, Slurm had software out of date, including CUDA Toolkit 12.0 at the default location /usr/bin/nvcc (which is so old it’s incompatible with the B300 SM100 architecture), and unacceptably old drivers. On the bright side, Slurm did have topology settings configured, and the NCCL recipes that they gave us worked well. We had no authentic errors during testing, and all our tests, including burn-ins and storage, showed good performance.

Our preoccupation with Lambda’s 1-Click Clusters seems quaint in today’s world of capacity reservations backlogged for months. Indeed, as mentioned in the opening, Lambda is increasingly driving its bottom line via bare-metal builds, apart from managed clusters altogether. Regardless, Lambda’s orchestration continues to improve, and customers are happy, which lands them in the upper ranks of ClusterMAX 3.0.

Microsoft

Using Azure clusters can be very difficult due to the Microsoft bureaucracy. Their engineers do the best they can, but this is not an organization set up to be flexible in accommodating its customers. Microsoft customers do not come first in matters of taste.

During testing, we were not allowed to have our own Azure accounts, so we had to use the cluster under the user of one of Azure’s engineers, which prevented us from poking around the console and experiencing the authentic onboarding experience. Our GPUs were set to be in Virginia, but thanks to some obscure internal decree, we were not allowed to have a control plane there. Instead, our control plane ran in San Antonio, Texas. Access required setting up a VPN, which worked after a bit (see: literally 3 weeks) of back and forth. Pasting was disabled on the bastion, so to run our tests, we had to manually type in our GitHub and HuggingFace tokens. Like, literally type them in one character at a time. We got them right first try though, nbd 😮‍💨.

Once everything finally ran, we found a handful of packages out of date, and bad default NCCL configs kept our first run from going through. On AKS, the `managed-csi-premium-v2` `StorageClass` could not attach, and we got `EAGAIN` trying to run `fio` at anything more than 4 clients on Azure Files Premium. No parallel system attached to our Slurm cluster meant that one of our tests, which loaded DeepSeek-V4 from NFS, timed out.

Source: a typical Azure experience

When we went to run burn-in, we had a genuine failure on 10 nodes of our Slurm rack, killing our job. Microsoft caught the errors, traced them back to an unhealthy host, and opened up a Guest Health Report for repair. An `scontrol` resume command brought them back, and after that, both our Kubernetes and Slurm racks completed their burn-in without error.

CycleCloud drained the node immediately after we tested Azure’s health checks by injecting a synthetic XID, but it did not resume or replace it. Because we had the whole GB300 rack, this was satisfactory, as described in the “Blackwell and Grace Blackwell” section above.

Due to the account issue, we could not see our Kubernetes dash by default. Instead, Azure’s engineering team cooked up a custom dashboard setup for us, and it was extremely detailed—a solid triumph over the bureaucratic straitjacket. We want to shoutout our friend Xu Xue for this one, as we are quite certain that he went way above and beyond the call of duty to build this out for us, but wouldn’t admit it. His curated dashboard is open sourced here, a great resource for anyone building Grafana dashboards to monitor Nvidia GPU clusters from scratch.

Satya once said “we want to build out Azure as being fantastic for the long tail of the workloads,” and that Microsoft is “not in the business of just doing five contracts with five customers being their bare-metal service.” Yet it is exceedingly rare to come across a VC-backed lab that uses Azure for training and inference. “Five customers for bare-metal service” is overstating it. Really, it’s two: OpenAI and MAI. Soon to be three, with Anthropic diversifying.

Azure’s most interesting asset is that it can serve OpenAI’s models and keep 100% of the revenue. They still have full access to all of OpenAI’s IP. Yet Azure can separately boast of large bare-metal deals with OpenAI, as well as an increasingly lucrative relationship with Anthropic. This is an incredibly strong position to be in. If only for the pause.

As Azure mostly competes on a different plane with enormous deployments and hundreds of billions of capex, its ability to manage clusters only makes it more flexible. Like Google and Oracle, Azure gets its cashflow from offtakers who need no help setting up a `ComputeDomain`. But based on the quality of testing, these three could still cut smaller, higher-margin deals with labs who want a higher-touch experience.

This does not seem likely to happen any time soon, though. Support and the console is a headache, even if Azure is the only hyperscaler where you can possibly get the Nvidia Reference Architecture on your scale-out networking. As much as anything, we like to verify that these hyperscalers are light on their feet.

Firmus

One of Nvidia’s favorite up-and-comers, and our favorite in Asia-Pacific, Firmus announced in August a $2B funding round at $10.5B post-money for expansion throughout Australia, Malaysia, and Indonesia, following a $505M round at $5.5B just 4 months before. Firmus also raised a $10B debt facility in March for its Project Southgate buildout in Melbourne and Tasmania. Just last week, it disclosed 900 MW in total contracted capacity and announced OpenAI as an anchor tenant in its Malaysian expansions. All in all, Firmus has disclosed 18,400 GB300 GPUs in the deployment process today and 36,800 later this year in Tasmania, with multiple GWs and tens of thousands of VR in the pipeline.

We had a rough takeoff but a fairly soft landing while testing one of Firmus’s dev clusters. Accessing our chips required using a Microsoft Entra account, Microsoft SSO, then downloading and configuring a custom pre-release version of the vCluster CLI. We’re suckers for new tech, but where cluster access is concerned, we’d much rather just SSH. After a few rounds of back and forth, with the help of Firmus’s team, we were on.

…on the cluster, but not quite off and running. The default setups on both Kubernetes and Slinky left a lot to be desired. Slurm login had no `sudo`, no `vim` or `nano`, HPC-X installed but not on `PATH`, and no NVCC. More importantly, instead of receiving 4 Slurm nodes on one rack and 4 Kubernetes on another, we had 2 of each on each. This was not necessarily a problem, but Kubernetes had `nvidia.com/gpu.clique` configured with nothing consuming it, while Slurm used `topology/flat`. Thus, the nodes visible to each orchestrator spanned multiple racks, and there was nothing to avoid a multinode job needlessly crossing the rack boundary. We had to infer node nomenclature to make sure our tests were well placed.

On top of this, the Kubernetes control plane’s `etcd` visibility flickered several times during testing. which killed our jobs and caused both the Slurm and Kubernetes layers to restart. This was ultimately attributed to an issue upgrading switch firmware and did not recur for the last 10 days. We also lost a NIC to a wedge during reboot, which was not caught by Firmus’s health checks as they were not active on the dev cluster we tested. Firmus was the only GB300 cluster we had that exposed GPU HBM as NUMA nodes, an interesting configuration that we mark as a failure because it lets you accidentally swamp device HBM if you overflow on the host. WAN was inconsistent but often quite poor, maintaining 0.25 Gb/s to NGC. The last of our problems was that one of the racks saw significantly degraded performance on its NVLink fabric, which was explained by a power setting that Firmus had been experimenting with on their dev cluster; fixing this brought our numbers into the healthy range.

Thus, by the time testing ended, we had satisfactory though underfurnished Slinky and Kubernetes layers, we knew how to access them, and we could get strong performance across our test suite. We understand that managed clusters are not Firmus’s priority right now, but we believe there is plenty of low-hanging fruit here for their technical team. As Firmus works to bring enormous capacity online, we look forward to seeing them continue to make life easier for users of their GPU software stack.

TensorWave

TensorWave is an AMD-only neocloud whose MI355X offering we tested via K8s and Slinky. In the last writeup, we emphasized the difficulty of TensorWave’s onboarding process, which required back and forth just to get on the cluster and had a large number speedbumps even still. The improvement this time round is significant. Onboarding was smooth, the `kubeconfig` sent directly to us worked perfectly, the console allowed us easily to add team members with SSH access, and the TensorWave support team was attentive throughout our testing.

Being an AMD-only cloud, TensorWave is fighting an uphill battle dealing with AMD’s software stack, which, despite significant recent progress, still lies well behind Nvidia’s. We saw this on our TensorWave cluster on RCCL microbenchmarks, with all-gather and all-to-all hanging at 32 KiB even using TensorWave’s binaries and recipes. Other collectives showed uneven scaling and poor throughput at certain message sizes. [Editorial comment: RCCL? More like Rick L!] In general, it can be said that the AMD networking stack lacks the community support that Nvidia tooling offers. The same goes, of course, on the hardware side. More important than microbenchmarks, we see in practice that the GB300 NVL72 systems have the most demand across the industry from labs: their performance is so superior that, even accounting for their higher prices, they often offer the best perf per dollar as well. AMD’s rackscale answer, the MI455X Helios, is still ramping up production, and is certainly not yet on offer by TensorWave. Thus, TensorWave’s place in the rankings depends not only on its internal improvements, but on AMD’s ongoing work to close the gap relative to Nvidia.

For what it advertised, our TensorWave cluster was good across the board. North-south bandwidth was strong. Storage performance was exceptional and burn-ins completed without a problem. When we pushed around the Kubernetes layer, its reliability was good, and the Slinky layer was easy to use.

The health check immediately recognized the error that we simulated, bringing the node into `drng`.

The dashboard oddly marked the node as “Allocated,” rather than being broken out into its own category. However, shortly afterward, we were presented with a nice juicy button to click to approve replacing our “sick” node with a fresh one.

The replacement completed in approximately 43m. We would prefer that the autoremediation occur by default—we’d rather not even have to click a nice juicy button—but otherwise consider TensorWave’s monitoring and autoremediation exemplary.

Overall, TensorWave is the leading AMD-exclusive neocloud, providing a comparable (and sometimes better) experience on AMD GPUs than hyperscalers like Oracle and Microsoft. And they are not stopping. TensorWave has entered into an agreement with Fermi for 222 MW in the Texas panhandle with expansion rights up to 650 MW, contingent upon Fermi securing project financing. It has a tailwind with a $350M June Series B, leaving it at $1.55B post-money, and is currently working on large deals with hyperscalers. With both OpenAI and Anthropic announcing strategic partnerships with AMD for the MI450 generation, we expect that TensorWave is set to benefit, and AMD-exclusive neoclouds may be able to break into the Gold tier. We look forward to testing TensorWave’s MI455X Helios once it is available.

GMI

GMI is a neocloud headquartered in Mountain View with roots in Taiwan. With a $500M Nvidia deal announced last November and $12B more this March, hundreds of GB300 racks in the pipeline, and plans for VR down the line, they are keeping on the frontier in terms of hardware. Recently, they have also been growing their fleet by making deals with smaller neoclouds and reselling the compute to their established customer base, which offers an attractive cashflow profile because it does not require up-front capital outlay. After judging them the “top Bronze neocloud” in ClusterMAX 2.0, we were interested to test the progress of their managed clusters.

Onboarding was a bit bumpy, including an empty Kubeconfig downloaded from the console and an SSH key that we uploaded but was never synced to the cluster. After some back and forth with the team, we got in. The problem, it turned out, was that the console was view-only for our testing purposes, so we could not verify its functioning. Last round, we did not get access to a self-service console at all, so we count this as progress.

As last time, the compute was up to spec, burn-in was clean, and the InfiniBand fabric was sturdy. One notable improvement since ClusterMAX 2.0 was in GMI’s storage. GMI provided a performant RWX NFS default StorageClass, which was also available on the Slurm layer at `/home`. We had difficulty trying GMI’s S3, which turned out to be due to a typo in their doc. But once the Brothers ~~Karamazov~~ Karasev hopped on a call and fixed it, we verified that the object storage perf was also good. The biggest blocker in terms of cluster configuration was that GMI expected the IMEX domain to be configured on the Kubernetes cluster by a ComputeDomain CRD. This is perfectly viable—many other highly rated providers do the same thing—but GMI’s Kueue would not tolerate the claim, so our jobs sat in `Suspended`. To work around, we had to submit our jobs in a namespace with no `LocalQueue` named `default`. We recommend that GMI document that MNNVL jobs need to skip the queue, or, much better, that GMI set up its Kueue to tolerate DRA resource claims.

GMI’s dashboards, which did not exist last time we tested, were useful, including hardware information from DCGM and NVLink and InfiniBand monitoring.

We could not get the full GMI health-check experience due to a lack of spare capacity, but what we could test performed well. Nodes were quickly cordoned at the Kubernetes layer, then rebooted and auto-uncordoned. The Slurm layer also brought a node into `DRAIN` soon after a synthetic error was injected. GMI had alerts set up in Slack, so we could see their system detecting the error and thread to discuss next steps with support, which was convenient.

Our GMI experience was solid in every crucial way, and we look forward to continuing to test them as their capacity ramps quickly.

Bronze

Amazon Web Services (AWS)

Our last report summarized the AWS experience as a “headache.” This time around, we had to break out some Amazon Basic Care Ibuprofen Tablets, Fever Reducer and Pain Relief from Body Aches, Headache, Arthritis Pain and More, Brown 200 Count to get through our testing.

The difficulty with AWS begins before you get on the cluster—indeed, before you even try to provision the cluster: it begins when you contemplate AWS’s zoo of managed offerings. To get this all straight, we had to make a Venn diagram:

Source: Intern (I forget his name)

(Caveat lector: anyone not interested in the minutiae of AWS’s GPU offerings should skip this ¶.) SageMaker is AWS’s fully managed platform meant for minimal maintenance. If you don’t want to handle any infrastructure, it offers serverless model customization and data science environments that abstract away the underlying compute. At the next level up is the SageMaker HyperPod line, which supports both Slurm and EKS. The HyperPod line offers many conveniences like health checks, autoremediation, dashboards, and workload-level features like checkpointless training out of the box for users who want to preserve their `ssh` or `kubectl` access. If you want infrastructure abstracted away but want to handle the ML stack yourself, you can leave the SageMaker umbrella and choose EKS Auto Mode, which takes care of node provisioning, autoscaling, OS patching, and node repair. EKS Auto Mode is a wrapper on top of “managed EC2 instances,” but these are not actual EC2 instances in the sense that they do not show up in the EC2 console and do not provide the same level of customizability as ordinary EC2 instances. In EKS Auto Mode, users do not have access to the control plane, and they must allow AWS to recycle their workers every 21 days for security. Finally, for even greater customizability, there is EKS + Karpenter, which allows the user to manage autoscaling themselves with the open-source tool Karpenter.

Despite Amazon’s reputation as “customer-obessed,” it is clear that this novel-length GPU menu is an example of the adage “you ship your org chart” rather than a reflection of genuine demand segmentation. And the bureaucratic frustrations hardly end once you’ve ordered your cluster. This is a criticism of the organization rather than of any team in particular—AWS has shades of the 3rd-Century Roman Empire. To improve its standing in the next round of ClusterMAX testing, we recommend AWS choose another empire and/or time period for reference.

Nevertheless, there is a unifying trait of all these offerings: they fail to provision the first time you try. Thus, we made another clarifying Venn diagram:

Source: Ibid.

This finally brings us to the AWS operator experience. One SageMaker cluster we provisioned in the console failed due to insufficient IAM permissions. Some of the EKS clusters used Terraform to simplify provisioning, but that path is not battle-tested and requires being in the right subdirectory and on the right commit of the right branch of the right (in-flight) repo and knowing exactly which knobs to turn without making an invalid request. One cluster came up without Lustre and would not let us attach any, despite extended back-and-forth with AWS engineers. For the purposes of testing, we provisioned a new cluster in a new region to gather data on filesystem perf, but if we were a real-life lab, this would have been crippling, at least temporarily. The AWS team we worked with was consistently helpful, but it is clear that they don’t have the liberty to go solve problems or even insight into how the system fits together, especially compared to the flat, nimble neoclouds we’re judging them against. It should not require six engineers on multiple calls to provision a Slurm cluster in the first place.

Once the goods were delivered, there were no blockers that kept us from getting reasonable perf out of the gate. The drivers, firmware, and most utilities were up to date. This is the bare minimum from a reputable provider like AWS, however, and there was no shortage of rough edges elsewhere.

A basic point of distinction between the AWS clusters we tested and their peer B200 clusters is that AWS uses Elastic Fabric Adapter (EFA), its proprietary scale-out fabric. In our testing, peak EFA throughput was strong. On a four-node 32-GPU all-to-all NCCL test over EFA on B200 we measured 59.44 GB/s busbw at 16 GiB message size, which works out to about 92% of scale-out line rate given the proportion of traffic that crosses the node boundary. On other networking tests with other traffic patterns and launching mechanisms, as long as message size was large enough, throughput was among the best B200 clusters. However, EFA consistently demonstrated worse latency than well tuned InfiniBand comps. For example, when the message size was reduced to 64 KiB, the same four-node 32-GPU all-to-all NCCL test took 85.26µs over EFA and under 50µs on InfiniBand peers; this delta of 30-40% between EFA and InfiniBand was consistent across small-message tests, which are latency-dominated. Although this may seem like a nitpick, scale-out latency is a critical metric for modern expert-parallel workloads, which frequently send small token vectors across the node boundary. This 64 KiB NCCL test, for example, is similar to the expert routing step of models like Kimi K3 and DeepSeek V4. Our findings are corroborated by an excellent Perplexity technical blog post about optimizing EFA, which reported similar results: EFA demonstrates satisfactory peak throughput but pays a latency penalty of 20µs or so on “the message sizes exchanged during MoE dispatch and combine.” (Note that this Perplexity post compares EFA to ConnectX-7 NICs whereas we tested ConnectX-8, but the nameplate throughput of the two generations is identical.) It’s not a good sign when Perplexity’s cracked engineers are writing painstakingly detailed blog posts about how to “enable” basic functionality on your custom network stack.

All-reduce tests on EFA. Note that message size must double many times before time to completion changes much; this is because at small sizes, it is latency, rather than line rate, that defines performance..

EFA still requires extra setup for expert-parallel inference in upstream vLLM and SGLang. AWS reports successful vLLM deployments, but users must install EFA userspace libraries and build the relevant communication components with EFA support. Depending on the workload, these include DeepEP, NIXL, or Mooncake Store. AWS’s DeepEP fork supports EFA through NCCL GIN. DeepEP V2 uses NCCL GIN, while its NVSHMEM-based V1 path is now documented as legacy. Upstream integration and default container packaging still require work. NCCL EP is another route under development; the linked vLLM integration remains a draft PR. In general, EFA is not a priority for frontier communications libraries, and users must wait weeks or months for what support they do get.

AWS is the only cloud distinguished in this way. Every other provider we tested uses InfiniBand or RoCE. Of course, AWS’s customers with GW deals have the resources to make sure their EFA is well tuned, but we still think it a shame that the largest provider’s scale-out stack is uniquely troublesome.

In a different way, storage was another basic cluster service that often proved uncooperative. The B200 cluster we provisioned in `ap-south-1a` failed every request for storage with an ambiguous `Insufficient capacity` error, and the `Failed` filesystems took 5h46m55s to delete.

To test Lustre performance required another cluster in another region, which meant another protracted bringup—another evening of terminal-watching trying to discern whether Terraform was making progress or not.

Source: Provisioning purgatory.

Claiming storage on AWS via Terraform is like raising a toddler: you can only watch and hope that once it’s done destroying it will start creating. And once the storage is created, it is not necessarily straightforward to access. On the Kubernetes layer, one cluster provided no StorageClass by default, and its only provided StorageClass left our PVC indefinitely in `Pending` because it had the wrong driver. This required us to define our own StorageClass with the correct provisioner.

Powered on and wired up, storage perf was mixed. The provided FSx stack struggled significantly with ordinary buffered I/O, reaching 9.2s in p99 sequential read latency on fio testing, compared to 312ms on the same metric with buffering disabled. Greater latency on the buffered setting is expected, but this is a pathological delta.

AWS’s health checks’ performance was, again, mixed. Our EKS Auto Mode cluster caught the XID we injected, set `AcceleratedHardwareReady=False`, and gave us a new and healthy node only 24m after the original injection. Meanwhile, our SageMaker HyperPod Slurm cluster immediately detected the XID, brought the node into `DRAIN`, and completed most of the reprovisioning process before tripping over its own feet when it required a deep health check before it could rejoin the fleet. This deep health check was submitted as a Slurm job and couldn’t be scheduled until, well, the node was added back to the fleet. Thus, an operator had to manually push through a `scontrol ... State=RESUME` command, at which point the fleet returned to full health. With our feedback, AWS promptly fixed this circular dependency, and a follow-up test on a single H100 node succeeded without intervention. Better.

Our EKS Karpenter cluster’s health checks were even more self-defeating. The `dcgm-server` DaemonSet did not tolerate the `nvidia.com/gpu:NoSchedule` taint, so it did not attach to the worker node, causing the monitoring agent to report `error connecting to nv-hostengine` and flip `AcceleratedHardwareReady=False` with `Reason=DCGMError`. Karpenter then marked the `NodeClaim` for termination, the replacement node was marked unhealthy for the same reason, and the system spun in circles until we manually modified the DaemonSet to tolerate the GPU taint. The health check had another allergic reaction when it noticed the IMEX daemon did not schedule. IMEX configures the NVLink fabric across nodes, but we had ordinary B200 nodes speaking via EFA, so there was nothing for IMEX to do: this daemon’s failure to land should have been harmless. Instead, the monitoring log emitted the schizophrenic message

ignoring IMEX health code on non-NVLink multi-node system, code=122

sending condition to exporter,

Reason=NvidiaFabricError,

Severity=Fatal

This would again have recycled a healthy node if we did not manually pin NodeRepair=false to keep overzealous garbage collection from blocking our work. Finally, as our testing came to a close, Karpenter tore down our B200 cluster 39 minutes before the EndDate, force-evicting a 4-node MPIJob in progress. We have gone into such detail here because these bugs are almost unbelievably elementary for the world’s largest cloud. Perhaps the single painfree part of the AWS lifecycle was Karpenter scale-up and scale-down, which completed in minutes with no errors. However, GPU capacity must be secured in large blocks and far in advance, so snappy autoscaling with Karpenter is a meaningless feature.

Despite the many problems with their product, AWS’s specialists and engineers were intelligent and responsive in our interactions with them. They were eager to show what they had improved, sought out feedback on documentation, circled back to rectify errors in testing, and even built a nice, forward-looking MCP server whose skills we plundered for our own codebase. AWS’s issues lie in its organizational structure. The number of basic issues that have slipped through the cracks is astonishing in an organization with AWS’s resources.

In February, AWS announced the expansion of its deal with OpenAI by $100B; in 2Q26, it posted 37% growth in net sales YoY; its Trainium business has hit $25B in ARR with triple-digit YoY growth; its Anthropic investment is appreciating to the tune of tens of billions per year; Bedrock continues to make bed-rocking margins on bed-rocking income.

Managed clusters are not AWS’s priority.

Gcore

Gcore is a Luxembourg-ish neocloud building out in ambiguous region called « Europe, » where it relies on other providers for capacity. They have made no flashy announcements recently, though they are adding Blackwell to their Hopper fleet. In ClusterMAX 2.0, we gave Gcore a Silver rating after testing their Kubernetes and SonK offering via Soperator. This time round, we only tried the SonK again.

Our insight into Gcore’s capabilities was limited by their lack of capacity: most providers granted us at least 4 B300 nodes for at least a week, but Gcore was so constrained that they could only part ways with 3 H200s. Nevertheless, there was plenty to keep us on our toes on this cluster.

Our testing on the Soperator layer required several rounds of back-and-forth with the Gcore team to fix some rough edges, but the team consistently proved helpful. When the majority of our first sweep failed due to a misconfiguration that prevented non-root users from running their workloads, the team responded promptly, confirming `kernel.apparmor_restrict_unprivileged_userns=0` would get us unblocked. Gcore’s team was again responsive in correcting another poor default when each Slurm worker reported `RealMemory=2048 MiB`, apparently having erroneously inherited CPU configurations. This kept us from even running `nvidia-smi`, and we were glad to see it quickly repaired.

Debugging was made slightly more challenging on the Soperator layer because inter-node SSH was disabled by default, but the team pointed us to a `soperator-createuser` convenience script to make life easier. The team also troubleshot an error with their dashboard as we tested.

On the Kubernetes layer, there was no default StorageClass, so users had to select NFS explicitly. The storage on the Slurm layer was properly configured, with the easy-to-use Soperator default of NFS at `/`, providing the illusion of a single shared volume wired to all the nodes. This storage performed well across workloads. The N/S network was fine, and the E/W network ultimately performed up to spec, but the RDMA devices had non-standard names and no `topology.conf`, so configuration took extra steps.

The XID that we injected was quickly caught and displayed in Grafana; interestingly, the node went `Down` on the Slurm layer, but on the Kubernetes layer nothing changed, even though it was injected via K8s. We could not test autoremediation because Gcore had no spare capacity.

We were impressed by Gcore’s attentiveness and their Soperator offering that, after some massaging, did the basics right. We look forward to testing Gcore’s B300/GB300 management, including its handling of XDR ConnectX-8 NICs, and seeing the improvements of their Soperator and Kubernetes products.

GMO

GMO’s managed Slurm cluster delivered strong networking and storage performance on two B300 nodes. Its main gaps were restricted profiling, limited enterprise controls, and a lack of autoremediation.

Slurm arrived with a configured head node, partitions, and topology.conf. Drivers and CUDA/NCCL/HPC-X modules were current and consistent across nodes, and Pyxis was available. Onboarding was straightforward, though the Japanese-only console required some manual ctrl+c, ctrl+v translation for us to understand.

Source: PoV you are a SemiAnalysis intern who dropped Japanese after two classes trying to locate a Grafana dashboard

Grafana access took us five days of troubleshooting. We initially missed the macOS login-keychain instruction, then entered the certificate password where the system password was required. GMO investigated and repeatedly followed up. The final mistake was ours, but clearer certificate instructions would have saved time. Or, no unnecessary custom certificates to access the website. Japanese security theater strikes again.

Source: Attempting to access our Grafana dashboard

The sixteen 400 Gb/s rails per node delivered strong NCCL results, though two nodes provide less scaling evidence than our usual four-node minimum. MPI launches needed explicit NCCL_IB_HCA settings, which GMO said it already supplied in nccl.conf, but which didn’t work for us initially. Burn-in showed no network errors, retries, or ECC errors, with normal temperatures and power.

Shared home/data/work directories worked out of the box. FIO sequential reads and writes, random reads, and torch.save were solid, built on a well-tuned DDN Lustre filesystem. However, our contrived tests such as `import torch` and vLLM Serve from shared storage performed poorly; GMO offered to investigate but we chalked it up to classic Lustre metadata performance tuning issues and moved on. Like many other Lustre implementations, the GMO storage offering lacked snapshots, automated backups, and disaster recovery options.

The cluster also lacked `sudo`, Docker on the workers, and required the use of snodes, a convenience script that they built to show metadata about the cluster, since they prevent users from running any scontrol commands on their own cluster. Security theater once again.

As a final step, SSH to worker nodes on the cluster was also blocked. And inside Slurm jobs, `RmProfilingAdminOnly=1` blocked Nsight Compute’s GPU counters, and perf was not usable for our profiling needs. We believe that managed clusters should allow users to profile their own workloads if they choose to go for the yolo-mode option. We wanted it, but GMO could not provide.

Finally, Grafana exposed useful DCGM metrics but lacked XID detection, estimated TFLOPS, SM Active, and SM Occupied views that we enjoy for performance and reliability monitoring. The Slurm job summary view was also pretty basic. Our synthetic XID 79 log injection produced no observed alert or node drain. GMO said it runs dcgmi health in the Slurm prolog, confirmed XID 79 was not whitelisted, and agreed to investigate for future. We finished our testing by determining that GMO lacked any autoremediation system in place, or at least we couldn’t verify it.

Source: Our Grafana Dashboard on GMO!

Overall, GMO’s lacking Kubernetes and enterprise features such as RBAC, SSO, and customer-accessible audit logs put it in the camp of niche, local player serving the Japanese market effectively. Our support experience with their team was helpful, but followed standard Japanese business hours. Going forward, we expect GMO to continue to be a leader in Japan, but lack the ambition required to expand their business beyond their local geography.

Verda

Helsinki-based Verda (formerly DataCrunch) disclosed $100M ARR and $200M total funding this June. They maintain a close relationship with Nvidia and continue to aggressively pursue financing. On the technical side, they have implemented a number of features since we tested them in November, and we are optimistic about their engineering team and roadmap.

Last round of testing, Verda’s Slurm was officially in beta, and they did not offer Kubernetes. This time, they offer a managed Kubernetes and have moved their Slurm offering to Slurm-on-Kubernetes to simplify deployment. We tested their cluster on both the Slurm and K8s layers; it had its fair share of rough edges, but represents significant progress and a strong starting point for continued development. We also found that security was solid.

The purchase and onboarding process was smooth, and the cluster provisioned without hiccups in 30m. The Grafana dashboard worked out of the box, and we SSHed onto the cluster without incident.

Once we got on the cluster, though, there were some bumps in the road. Our Slurm testing began with an interesting footgun: when we SSHed in, we assumed we were on the Slinky layer, but we actually had landed on the outer VM with wrappers that ran `salloc`, `sinfo`, and similar commands via `kubectl exec` into the `LoginSet`. The behavior was unusual—all Slurm commands escalated us to `root`, for example, and our Python environment disappeared—and it took an evening of debugging before we puzzled out that our Slurm access was illusory and a different SSH command would be required to access the Slinky layer properly.

Verda’s engineers were responsive, quickly hopping on a support call in which they reasonably explained the rationale behind these convenience functions and pointed us to the supported path. Although this particular feature caused us a headache, it’s bullish when engineers have gone out of their way to try to make their cluster easier to use.

The Slinky layer proved serviceable but imperfect. Verda ran a NCCL-test prolog that exceeded Slurm’s 10-second `MessageTimeout`, causing a worrisome though harmless `Prolog hung` alert every allocation. More significantly, Verda did not support Pyxis or Enroot, so we had to stand up our environment via a massive `.sif` file. Once our workaround was finished, we completed an 8-hour burn-in without incident and measured satisfactory numbers on our microbenchmarks.

Meanwhile, the Kubernetes layer could only be steered through the bastion, offering no way to download a `kubeconfig` through the console. This is a relatively simple fix to align Verda’s offering with users’ standard workflows.

The biggest shortcomings that surfaced during our performance sweep were of the shared filesystem. Gross throughput numbers were excellent, but cross-node file locks seemed simply not to be enforced. We found 100/100 exclusive-lock violations for both `flock()` and `fcntl()` across nodes, while same-node controls refused every attempt. This strongly suggests the issue is with `virtiofs` failing to publicize locking behavior. We also saw corrupted data when downloading with multiple clients from Hugging Face and `ENOENT` failures on Elbencho with 32 clients. This is more low-hanging fruit, and crucial given the fundamental importance of a reliable shared filesystem when running Slinky.

Lastly, Verda’s health checks programs are not yet fully configured. They are literal “health checks” in the sense that they evaluate whether a node is healthy, but there is nothing to consume their output—they exit `1`—“yup, this one’s dead”—and move on. A user must therefore bring their own reliability suite if they don’t want bad nodes silently killing their workflows. There is of course also no autoremediation provided by Verda.

Verda consistently communicated promptly and clearly and seemed to have engineers with the liberty and motivation to improve their product. On September 10, after our testing completed, they pushed a number of changes to Instant Clusters that appear meaningful. We look forward to seeing them continue to sand the edges of their Slinky and Kubernetes offerings, tune their storage system, and implement robust health checks and autoremediation.

Moonlite

Moonlite, a neocloud headquartered in Chicago, is a new inclusion as of ClusterMAX 3.0. With an ex-Crusoe founding team, Moonlite currently operates a Hopper fleet and has recently handed over its first B300s and GB300s, which it will be continuing to ramp up in the coming months. According to LinkedIn, Moonlite has the fewest employees of any company we worked with this round and oriented around engineering.

We tested Moonlite’s Kubernetes and Slinky. The handover process was smooth, with a nice onboarding document defining validated recipes and a Kubeconfig that gave us everything we needed. By default, everyone shared the same root user on Slinky; when we asked for non-root users, Moonlite made a `moonlite-adduser` convenience script in half an hour and baked it into the login-pod image same-day. This shows Moonlite’s responsiveness to feedback, but also their greenness. We loved their response. We would rather have a battle-tested system, ideally through the console, to handle basics like this.

On the hardware side, everything we tested worked well out of the box. NCCL tests hit ConnectX-7 specs, NFS was good, compute benchmarks were solid, and burn-in completed without a problem. However, it bears repeating that we were testing H100s, which by now are quite familiar to the industry and are therefore easier to get right than the GB300 NVL72s we had with other providers. Nonetheless, we were happy with our Hopper perf.

One component of the system that we could not test was the NVMe, which existed but could not be accessed. Nothing exposes it: no local-path `StorageClass`, `hostPath` unavailable, and Slurm workers are Slinky pods with only VAST `/home`. On the Kubernetes layer, there was no default `StorageClass` at all, and unqualified PVCs hung `Pending`.

We tested Moonlite’s health and monitoring system via a PCIe secondary bus reset, and Moonlite alerted us in 11 minutes. With our permission, they cordoned the node, ran diags, and returned it to our fleet after 57 minutes total, including a few minutes where they waited for our instruction on how to proceed. In practice, with actual customers, they define policies ahead of time so that their team can follow a runbook depending on the particular error. Moonlite’s typical SLA is to respond within 15 minutes as their team receives alerts in Slack and PagerDuty. We hear from their customers that so far that this is exactly what they do.

In our view, any autoremediation process that brings us back to a full fleet within an hour is good enough. But while manual processes are fine, at scale, we want to see this automated. We would also prefer greater visibility on the cluster in the event of an error. The only change visible to the user was that the node, while remaining `Ready` on the Kubernetes layer, advertised 7 GPUs instead of 8 in the time between the reset and Moonlite’s manual intervention.

The Slinky layer lacked any container runtime on the workers but otherwise got the job done, and had most of what we ask for on the login pod. Moonlite’s Grafana was fine, but it lacked detail at the scheduling layer, and does not display the most crucial information, which is error status.

What Moonlite attempted to do, it did well. We look forward to testing them again as their products mature and they get some GB300 calluses on their hands.

Together

In ClusterMAX 2.0, we wrote the following about Together:

Together is a strong provider with a robust cluster offering for both Slurm and kubernetes, but it is held back from the gold category due to reliability issues.

This round, Together’s issues—generally but not exclusively having to do with reliability—are cause for downgrading it to Bronze.

Two nodes of ours failed during a compute test; one came back, but the other was sent to RMA without notice, so we were left wondering where it went. Later in testing, Together’s Slurm `HealthCheckProgram` tried to run `dcgmi diag -r 1` under a 45s timeout on nodes that were already scheduled; the diag did not land, the timeout was misread as a failure, and the nodes were drained mid-run. Together quickly fixed this with our feedback. Separately, one of our nodes could not access VAST because its storage NICs were in `operstate=down`, which Together explained was due to a race condition in NIC initialization in their new lightweight VM stack. There was nothing to catch this error, and manual intervention was required to reprovision a healthy node and bring our test fleet back up to 4. While we like the sound of “reduced cluster provisioning time,” we obviously don’t like to see it affecting reliability.

Together’s health checks did work correctly in some instances, showing progress since ClusterMAX 2.0. When we triggered a PCIe secondary-bus reset on the Slurm cluster, the node was drained in 67s, and an automatic node replacement completed in 35m7s without manual intervention. A synthetic error also cordoned a node on the Kubernetes layer and correctly did not trigger autoremediation because we had `Auto Repair` turned off.

The Together Kubernetes and Slinky layers were both functional, though with a handful of challenges. Interestingly, Together advises to avoid using the Kubernetes layer of the Slinky cluster: to convert Slinky nodes to K8s, we were instructed to tear them down and reprovision as vanilla Kubernetes. The Slinky offering should therefore be evaluated as plain Slurm. Summarizing its pain points as briefly as possible: OCI image extraction failed because of default settings on `/tmp`, so we had to redirect extraction to `/scratch`; when this happened, Slurm exited `0` anyway; NFS rejected any filename containing `:`; tenants did not have permission to read `dmesg`, which prevents checking for XID events; and non-login worker shells have neither `nvcc` nor `mpirun` on `PATH`, so ordinary batch scripts behave differently from interactive setups and lack the tooling that they need. None of these is fatal, but adding them up, along with other more minor errors, meant a few dev days before we were able to run our stuff unimpeded. Strangely, Together’s public internet delivered strong throughput to most services, but its connections to Canonical’s Ubuntu archive origins repeatedly failed to establish, stalled after connecting, or transferred below 20 kB/s. This again left us stranded as we tried to get our cluster set up.

A public IP check confirmed that this flaky B300 cluster had org `AS53735 IREN`. This N+0 Prince George datacenter is one of two datacenters in the industry notorious for unreliability, along with Crusoe’s cursed Reykjavik site. We strongly suspect that one or both of these datacenters was not inaugurated with a Land Acknowledgement. We know of multiple exceptionally unhappy customers of Together struggling with link flaps, power issues, cluster access being shutoff, random upgrades that aren’t approved by the users going wrong and not being rolled back, and weekend-long outages that eventually get blamed on a single ISP (yes, a single ISP at this site—no redundancy in Northern Canada). Maybe their other sites are better but the TogetherAI site at IREN Prince George is completely bad.

After playing the blame game for some time, it is clear that reliability issues cannot be blamed entirely on underlying providers. Unforced errors from customer support engineers have created a lack of trust between customer and provider.

Of course, managed clusters are not Together’s bread and butter, nor are they its primary source of revenue. When, in July, they announced an $800M Series C, the post made passing reference (at best) to their managed clusters business, emphasizing, on the contrary, that “Together AI is a research-driven company.” The offtakers of their 250MW in Saudi are as yet undisclosed, but we are aware that Together would prefer to get paid per token, not per GPU-hour. Despite this, we hope that Together manages to Get It Together, and address its cluster reliability issues to make its Slurm and Kubernetes products something to be proud of.

Crusoe

The Crusoe section of ClusterMAX 2.0 ended on a cautious note:

Crusoe is at risk of being downgraded to ClusterMAX Silver due to many of their top individual contributor engineers quitting, leaving the culture in their cloud division beginning to resemble big tech. There are too many middle managers across the organization, especially in engineering. This has caused incredibly slow moving releases, such as their AutoClusters feature, leaving us with concerns about the future of Crusoe’s public cloud offerings. Chase needs to do a rapid course correction if he doesn’t want to lose all of his 10x engineers and eventually lose their Neocloud business with it.

In the eight months and change since we published that paragraph, Crusoe has had its fair share of wins. Its valuation has grown from around $10B to $30.9B, it has announced 900MW and 1.0GW campuses in Texas and many more elsewhere, and it remains an innovator in the wild wild west of datacenter construction. They have the largest real pipeline of any datacenter builder.

Its inference endpoints business is also extremely highly regarded, following the acquisition of some cracked Israeli engineers from Atero. They have also landed awesome deals with firms like Jane Street.

Plenty to keep their hands full and business churning, some clusters are amazing, but others from them are terrible.

Yet this round of ClusterMAX testing finds that Crusoe’s neocloud business has not course-corrected rapidly enough. The problems go back to the fundamentals: reliability and hygiene.

We emphasize health checks in our expectations and our write-ups, but genuine hardware failures are rare in testing because our ClusterMAX allocations are too small and short-lived. Crusoe bucks this trend. On an H100 cluster we had in May to test AutoClusters, we saw an impressive barrage of XIDs—more real errors on one cluster over five days than we saw in the rest of testing combined.

In a pattern that would repeat throughout testing, exceptionally poor performance in some areas was paired with mature services in others: this storm of XIDs was accompanied by convenient emails, attractively styled, politely notifying us of the errors.

In addition to being responsible for the majority of this round’s hardware failures, Crusoe bears the distinction of having had the oldest Linux kernel and the oldest GPU drivers we tested. Crusoe has since rectified these shortcomings, but at the time of testing there were several software versions that were well below our minimum expectations: the aforementioned NVIDIA drivers failed our `cmax audit security`, along with Docker, ConnectX firmware, and `runc`. These issues speak to a lax security posture and poor engineering hygiene.

Nor are these software-version criticisms merely academic. The Linux kernel that our B300 cluster ran, `5.15.0-185-generic`, was earlier than a Linux patch series called “writeback: Avoid lockups when switching inodes.” The particular problem that our kernel experienced is well explained by the description of commit `e1b849c`, which fixed it for upstream kernels:
```
There can be multiple inode switch works that are trying to switch inodes to / from the same wb. This can happen in particular if some cgroup exits which owns many (thousands) inodes and we need to switch them all. In this case several inode_switch_wbs_work_fn() instances will be just spinning on the same wb->list_lock while only one of them makes forward progress. This wastes CPU cycles and quickly leads to softlockup reports and unusable system.

```

When might a `cgroup` that owns thousands of `inode`s exit? One answer: when a Slurm job is torn down. In our case, as an ordinary workload wrapped up, every CPU on the NUMA node ended up stuck contending for the same `list_lock`, making the processor unusable. The local NFS client, which happened to be on the frozen NUMA node, couldn’t get CPU time, and a liveness probe for that client timed out and reported the node unhealthy. Kubelet tried to control the damage with a `SIGTERM` and a `StopContainer` operation, but the kernel was taking its sweet time doing the laundry—reassigning dirty `inodes`—and took over two hours before it could honor the termination requests. The final result was another `NotReady` email from Crusoe—this one, uniquely, downstream of an old CPU kernel.

By the end of our testing, this kernel was not merely old and unperformant: it was officially marked a security vulnerability by CVE-2026-64378, which exploited the buggy writeback behavior, published on July 25 with severity 7.8; a separate 7.8 for this kernel was posted on July 27. Thus, we added the Linux kernel to our collection of Crusoe software exposed to CVEs. Crusoe has informed us that they will be upgrading to a 6-major Linux kernel, a necessary patch.

Source: A Crusoe job posting on August 13, 2026

When we ran our XID injection on Crusoe’s Kubernetes layer, the error was quickly detected and a `Node Replacement` workflow was triggered, but the node could not be put in `drain` due to a mismatch in Slurm naming convention that slipped through Crusoe’s CI. The health monitoring system spun its wheels, repeatedly catching the error and trying but failing to mark the node unhealthy. Crusoe’s engineering team diagnosed the error impressively quickly, rolling out a replacement within hours. However, our replacement node had faulty storage NICs, immediately bringing it into `drain`. This issue persisted after the node’s VM was manually reset. Crusoe’s engineering team has informed us that the failure of AutoClusters to remediate in this latter event was “a known gap,” and that they will be “rolling out auto-remediation actions for this in a few weeks.” This is a surprisingly lethargic response to a basic failure of reliability.

Last and briefly, it bears mentioning that this cluster’s WAN was among the slowest we tested, clocking in between 0.47 Gbps and 0.78 Gbps depending on the context. Iceland has some skinny cables getting off the island.

All these problems add up to a cluster notorious for its flakiness, as confirmed by our friends at frontier labs who complain about both the number of failures and the apparent lack of interest from Crusoe engineers to fix the issues promptly. To be clear, we at one point viewed Crusoe as the #2 neocloud in the world, and as a result the #2 place in the world to rent GPUs. We made recommendations to this effect. Oh, how the mighty have fallen.

But while Crusoe’s machine images are badly due for a refresh, and their engineers can’t get Claude Tags or Codex approved, they do have automations like this set up in their internal Slack:

Does a CVE from November of 2025 count as “historical baggage”?

In all seriousness, this is a problem of priorities, a problem of bureaucracy run amok. We are not sure what to make of job postings like the above one for a senior Linux kernel engineer. Literally, we’re just asking everyone to keep things up to date. You don’t need to hire Linus to understand why this makes sense. So while it’s cool to see Crusoe flexing its muscles and paying top dollar for a good person, it’s not a lack of people or funding or technical skill that has brought Crusoe here in the first place. Our complaints focus on the most basic issues possible, not the Slinky configuration or console or CLI that no doubt take up much more engineering time and that we enjoy using.

We enjoy working with Crusoe’s engineering team and believe they have plenty of talent to improve their standing. Frankly, in the last few months, things seem to be getting back on track. Engineers are locked in, pushing code, and excited about upcoming releases. The Atero acquisition has been worth its weight in gold, skyrocketing Crusoe to the top of many buyer’s lists when it comes to inference endpoint quality. We love to see it.

Still, in the managed clusters business, TCO comes down to goodput, and goodput comes down to reliability, and Crusoe’s clusters are exceptionally unreliable.

Crusoe comes in towards the bottom of our Bronze category for this round.

Prime Intellect

Prime Intellect closed a $130M Series A this July at a $1B valuation, disclosing $100M of “annualized revenue run rate across compute, RL and post-training, sandboxes, inference, environments, and evaluations.”

We tested Slurm and Kubernetes B300 clusters and mostly enjoyed both of Prime Intellect’s orchestration layers, although each was missing a handful of packages we would’ve liked to have. The huge Slurm control plane was nice and snappy, having machines with 2TB of RAM for some reason. Plenty of swap space to run salloc.

After a quick fix to standardize the NCCL bootstrap interface, our networking tests ran up to 13 nodes on the ConnectX-8 NICs with excellent throughput. Storage was no issue, with Weka /data volume performing very well at 4 nodes and scaling steadily beyond. PrimeIntellect’s managed Grafana was one of the nicer ones we encountered. However, we were on a cluster that was still going through burn-in, so we saw a steady stream of errors. An easy way to test the monitoring and health checks on the system!

First, a node on our Kubernetes cluster was marked NotReady, apparently due to on-site techs monkeying with adjacent servers; it returned within 5 minutes. Then, during a PyTorch networking test, a different Kubernetes node stopped posting kubelet status and the distributed job could not complete; the node came back after a reboot and the job completed. The next day, 2 Slurm nodes went down, one with NVLink issues and the other stuck after a reboot. Both returned promptly after intervention, but the latter was soon back to its shenanigans with another surprise reboot, causing a storage test to fail. An MPI test aborted in its last phase, and the corresponding Slurm job remained in COMPLETING as all workers remained unreachable in the brief period before we handed the cluster back.

All of these reliability issues can be blamed on the fact that we got on this cluster in a small window while they were still doing burn-in for a paying customer. Fair enough. But the lack of compute at Prime’s disposal that would allow them hold back hot spares points to a bigger problem of growing capacity constraints. Prime is a reseller of other’s compute, and depends on their underlying providers to guarantee customers with a high quality experience. This comes with challenges that high quality training/RL frameworks, dashboards, and maniacal Slack engagement from a cracked team of engineers can’t always cover up.

With that said, Prime’s health checks performed impressively throughout, quickly intervening and limiting the damage of each error. We thought the system’s default responses were sensible and even liked the Slack alerts, which the team promptly followed up on. Given that Prime gets its capacity from a number of providers, the quality of their software layer is important for labs to feel comfortable with what they’re getting. Of the marketplaces we’ve tested, Prime seems to be the best.

This leaves Prime Intellect with a straightforward assignment, and one that we cannot have direct insight into: improve reliability, build a base-load of compute capacity, and rise in the rankings as a true neocloud. We have no doubts about the team’s technical ability; they have done a lot of hard things correctly on this cluster, and are the only provider on this list that injects real experience training models into their approach to software and support. For better or worse, their work elsewhere is much more impressive than the bread-and-butter managed compute offering evaluated here.

We got a preview of their hosted training product during this period and are very impressed. RL infrastructure is decidedly more complex than a simple managed Slurm or Kubernetes cluster, since its built on top of the same foundation but with three components that need to stay in sync:

Training
Inference
Sandboxes/environments

Coming soon, we will be evaluating hosted training infrastructure providers and RLaaS companies for hire through a project provisionally called PostTrainingX. If we had to put out an initial cut at those rankings, Prime would be in the hunt for Platinum! Stay tuned.

DigitalOcean

This marks the first round of testing that we have been able to get the full DigitalOcean developer experience. Marketing itself as “the first cloud built end-to-end for the inference and agentic era,” DigitalOcean distinguishes itself from “bare-metal focused neo-clouds and inference wrappers that lack cloud platforms.” In fewer words: DigitalOcean sells, among other things, managed clusters.

DigitalOcean’s managed clusters are orchestrated by Kubernetes, which proved solid but slightly immature. Their onboarding doc asked us to install Multus ourselves, then create network attachments, install MPI Operator, and create the MPIJob manifest. There’s no reason not to set this up for the customer in advance. Once configured, our cluster performed well, hitting all the standard benchmarks for compute and XDR networking, and completing burn-in without incident. The cluster also lacked a RWX StorageClass out of the box, adding another basic setup step that the better K8s providers handle for you. Once the storage was configured, its performance was interesting: sequential writes were approximately 6x faster than sequential reads at several different client counts, the inverse of what we see on most clusters. We expect that DigitalOcean’s sequential read performance could be significantly improved with retuning.

When we went to test the health checks, our first XID injection went unnoticed: the monitoring system did not detect anything, and the node remained available for scheduling. When we alerted the DigitalOcean team, they responded promptly, found the bug in their system, and asked us to reperform the test later in the week. The second try went smoothly, and we had a fresh node 50 minutes after injection.

When a neocloud is “built end-to-end for [...] inference,” that means it doesn’t offer Slurm. The DigitalOcean team helpfully offered us a runbook to stand up Slinky, and it got the basics right, but would require significant work before it could be considered production-ready. No SSH access, nor, as mentioned previously, any RWX volume by default, which means that there is nowhere to keep a `/shared`. Anyone who plans to run Slurm on DigitalOcean should expect to stand up their orchestration themselves.

DigitalOcean will be adding 60 MW of capacity throughout 2027, and, aside from its managed K8s, it has an interesting selection of GPU services like Droplets and bare-metal Hopper and MI300X that fall outside the scope of ClusterMAX testing. But no plans for GB’s or VR’s from what we can see at this point. We look forward to the team continuing to improve its managed Kubernetes service the next time we test.

Hyperstack

Hyperstack, under parent NexGen Cloud, is a UK-based neocloud that advertises Kubernetes clusters in the US, Canada, and Norway. It recently raised $45M at a $354M valuation with plans for many AI new products for “full-cycle development.” As it announced a $34M debt facility to build out a B200 fleet, it also disclosed plans to deploy 4,500 B300s later this year and for an additional 56 MW online in 2027.

The managed Kubernetes cluster that we tested marks significant progress since ClusterMAX 2.0, and a solid foundation for its continued growth. Compute and networking tests were up to spec, and burn-in sustained good numbers with no hard errors. Our Kubernetes node-reboot recovery was quick, and we had no trouble mounting and unmounting PersistentVolumes. The N/S network was fine, and storage was good enough, although we were not able to get Weka and VAST, which Hyperstack usually offers, on our timeline.

Our synthetic XID was detected in under a second with a `NvidiaFatalXid=True` flag. The test was awkward as Hyperstack had no extra capacity, so we had to take a node out of our fleet to simulate a spare. The support team notified us that they had detected the error 12m after we injected it, and our spare rejoined the fleet after a total of 52m. Interestingly, at this point, the “sick” node was still not cordoned, which took until 1h30m to register on the Kubernetes layer. This was all driven by manual intervention by Hyperstack’s SRE team after our synthetic error triggered Grafana alerts.

Overall, this test was a success, and we appreciate Hyperstack’s commitment to helping it go smoothly. However, as the team is aware, the amount of human support required makes us worry about its replicability. Hyperstack offers 24/7 support and targets intervention in under 60m in the event of single-node failure, but it’s better not to have to count on another engineering team pushing the right buttons before your cluster gets back to full health.

The only outright error during our testing was that two pods, `cilium-envoy` and `csi-hyperstack-node`, crash-looped with `Too many open files`. This was a minor annoyance and did not affect our workflows.

We look forward to continuing to work with Hyperstack as they grow their team, gain experience managing larger clusters, and continue to automate and harden their infrastructure.

Participation Ribbon

Vultr

Vultr advertises itself as “the world’s largest privately-held cloud infrastructure company,” which they are not, and “the world’s largest privately held hyperscaler,” which they wouldn’t be even if they were. Still, they are a relevant player in neocloud land, with modern GB300s and MI355X including a claimed 50 MW AMD site in Ohio. In ClusterMAX 2.0, Vultr’s cluster was delivered with a handful of basic errors. Unfortunately, the pattern is the same almost a year later.

The images that Vultr built for our testing had many libraries with applicable CVEs, including CUDA, DCGM, Docker, runc, and ConnectX firmware. Our Slinky login pod did not mount `/shared` and lacked the Pyxis plugstack config, so we had to orchestrate from a worker pod until the Vultr support team reprovisioned the login pod mid-campaign, fixing both issues. There was another basic problem with the storage visible from the K8s layer: the default storage configuration was incompatible with our bare-metal cluster. The default StorageClass, `vultr-block-storage`, provisioned and bound without complaint, then the pod sat in `ContainerCreating` forever. However, the Lustre tier, which came attached to the worker nodes, was excellent, achieving 91.7 GB/s aggregate read across 32 clients. Grafana was configured but only partially correct, as NVLink read 0 due to unconfigured `DCGM_FI_PROF_NVLINK_*` fields.

Vultr is known to have been experiencing issues with the reliability in its scale-out networking, which has discouraged partners from beginning or expanding existing deals. We recommend that Vultr continue to improve the sturdiness of its clusters and work to create golden images with all relevant utilities and orchestration reasonably configured and up to date.

Vessl

Vessl is a new Korean neocloud who fancies itself a “multi cloud orchestrator” rather than a broker, buying bare metal long-term and providing a management layer with SLAs on top. They claim 5,000 GPUs currently in operation and have an ambitious target of 100 MW by the end of 2027. An old Vessl BizOps job posting says to expect “Speed. We MUST move fast, learn fast, and iterate constantly,” demanding “a minimum of 60 hours per week.”

We love the ambition; however, the immaturity of Vessl’s cluster, which we sampled in its Kubernetes and Slinky flavors, gives them plenty to work on. Our onboarding began with a notice to pin three settings to render the RDMA fabric usable. A heads-up is better than nothing, but we much prefer to be given a cluster that performs well out of the box.

The third bullet was in response to a particularly cumbersome configuration: by default, due to Vessl’s Kubernetes `LimitRange`, each container would hard-fail if it used more than 2GB of host memory. Although a defensive default might make sense in certain cases, this is a poor setting. Nodes rarely are called on to run more than one job at a time, so containers can expect to have all of the host DRAM to play with—hundreds of GB—and OOM-killing them for going above 2GB is stifling. We recommend that Vessl relax this limit, bringing it in line with the physical capacity of the host while also making failure more graceful.

In terms of management, Vessl provided no way to handle users on the Slinky layer except via traditional Linux tools on the machine. The SSH key we added from the console did not show up on the cluster, nor did the one we added via the Vessl CLI; we later learned that these utilities are independent of Vessl’s Blackwell clusters. We could therefore use only the private key and `kubeconfig` DMed to us on Slack.

The cluster provided a reasonable (though strange) `add-user` convenience script to make new Linux users.

It did not provide, however, basic packages like `pip`, `git`, or even `sudo`.

The cluster therefore took some work to get to the starting line. But our intern was still not done putting in reps as Linux sysadmin. Vessl ran the Slurm login node with an ephemeral OverlayFS root, so runtime changes to `/etc/passwd` and `/usr/bin` existed only in the container’s writable layer and were not preserved when Kubernetes recycled the pod. This caused our users to disappear midway through testing.

Another Slinky misconfiguration brought down the Slurm layer for part of testing. Our workloads wrote to an overlay that they did not clean up, accumulating 863 GiB in the root volume, causing Kubernetes to mark the node `DiskPressure` and restart it. This not only brought down the local worker, but the Slurm controller daemon as well, rendering the whole Slurm layer intermittently inaccessible. We infer that `slurmctld` was scheduled on to a node also running `slurmd`. Co-scheduling a `slurmd-pyxis` pod with a `slurmctld` is generally a bad practice for reasons well illustrated by this anecdote: the data plane should not be bogged down by the administrative duties of the control plane, and the control plane should not be within the blast radius of the data plane. Vessl quickly stopped the bleeding by having Enroot write to a larger volume, but to our knowledge the root cause—the co-scheduled worker and controller—remains unaddressed. It is clear that Vessl does not have experience running Slinky at scale in production.

A separate error made Kubernetes access inconsistent. The URL in our `kubeconfig`’s `server` field did not always resolve to the same IP address: Vessl had erroneously published the private `10.x` IP to the record set in addition to the public one. Sometimes our `kubectl` command resolved to the right (public) IP, and we could access, but other times it would resolve to the wrong one and our request would hang until a retry went through. We recommend that Vessl remove the `10.x` IP from the public record set.

The `NetworkPolicy` we set on the Kubernetes layer of our Vessl cluster was accepted but not enforced, as HTTP requests between all tenant pods were accepted without regard to defined rules. Vessl did not have a health check program configured and should prioritize this along with all the mundane “quality of life” features indicated above.

The hardware itself that Vessl offered performed well throughout testing, including on NCCL tests over InfiniBand. Given that it has such a large fleet advertised as coming online soon, Vessl has every motivation to harden its offering, providing superior services to its users and enabling it to command superior prices. We will be interested to observe Vessl’s progress when we next test.

Runpod

Runpod is a neocloud headquartered in Moorestown, New Jersey with a fleet including RTX PRO 6000, H100, H200, B200, and B300. This June, they raised $100M at a $1B valuation, announcing on Twitter that their revenue doubled to $240M ARR from February to June.

The cluster we were able to test was in Seattle, and was so new that it did not even have storage configured yet. Our “Instant Cluster” was just about instant, spinning up in under 2 minutes.

Runpod’s user model is unusual. In an ideal world, we manage permissions via a web console with RBAC, where we can add our team members by email, increase or decrease privileges, make groups, and drop in our SSH keys to be automatically synced to the cluster. Instead, taking advantage of its snappy provisioning, Runpod recommends minting Instant Clusters as needed. Engineers then operate their cluster with a single tenant and return them to the pool when finished. This means that budgeting, monitoring, and other such policies are handled above the cluster level, rather than via traditional tools like Linux user management or Slurm accounting. Runpod’s design choice is viable, but it is revealing we did not end up using the cluster in the suggested manner, preferring to fall back to the more familiar pattern of making Linux accounts tied to our SSH keys.

Our first burn-in failed as one of our nodes had port 1 links go down on two NICs, suspected to be due to an issue with a leaf switch. There was no health check to catch the error, but there was a Grafana dashboard that highlighted the error if you know where to look. With our feedback, the Runpod team quickly improved the dashboard to make InfiniBand state easier to spot.

On the second burn-in attempt, the networking performed well and the test completed without error.

The cluster was so new that there was no shared filesystem to test. On the software side, though, our initial audit found many things to be improved. The CUDA Toolkit, GPU driver, and NIC firmware were deprecated for security reasons, and the CUDA Toolkit was additionally a liability because 12.8 does not support SM103, the B300 SM architecture. This caused some of our benchmarks to fail on the first try. The cluster was also missing lots of expected tooling, with no container runtime and many basic HPC packages absent. Slurm also occasionally marked waiting jobs as `InvalidAccount`, which prevented jobs from running, even though `AccountingStorageEnforce=none` and `AllowAccounts=ALL`. The workaround was to spam `scontrol update JobId=<jobid> Account=me` until Slurm accepted.

We also saw a node drained when an image extraction filled our 50 GB root overlay, which was shared with `/var/spool/slurmd`, leaving `slurmd` unable to write. Our tenant lacked credentials to `scontrol update`, so our testing continued shorthanded. Separately, Linux rejected `unshare -Ur`, so we had to run with `udocker` instead. Our standard XID injection route was blocked because we lacked permission to write to `/dev/kmsg`; we had no mechanism to issue node reboots; passwordless `sudo` was not enabled by default. The theme here is clear: within the Runpod environment, users lack many permissions they need to get their work done.

We find the Runpod team unusually receptive to feedback and eager to explain their decisions. We like that the majority of their job postings are in serious engineering roles, and we hope they continue to work to strike a better balance between ease of access and developer control.

Bitdeer

Bitdeer was a new inclusion in ClusterMAX testing as of ClusterMAX 2.1. Yet another crypto miner pivoting to AI, Bitdeer already has an impressive 1.75GW in total electrical capacity according to its most recent announcement. However, this mostly feeds cryptocurrency mining rigs: our SemiAnalysis Datacenter Industry Model estimates they only have a handful of AI MW online. Still, Bitdeer is steering hard toward the higher-margin GPU business, with hundreds of MW already in the pipeline over the next couple of years as it works on sites, either converted or now, in Knoxville, Wenatchee, Fox Creek, and Rockdale, as well as in Norway and Malaysia. Hoping to grow its business vertically as well as horizontally, Bitdeer is not content with mere colo deals, with fledgling offerings from token-as-a-service to managed agents to, of course, managed clusters.

Source: bitdeer.ai

We spent a lot of time trying to figure out what exactly Bitdeer expected us to test when they gave us credits on their public console. Health checks were not set up, the monitoring dashboard did not work, many basic libraries like PyTorch and Docker were not installed, networking settings were empty, IMEX was not configured, and the console did not support ordinary conveniences like RBAC. Bitdeer’s monitoring installer downloaded a binary for the wrong CPU architecture. SSH worked eventually, but oddly required an RSA key rather than ed25519. Kubernetes queries repeatedly lost their connection. In short, the management of this cluster was strictly nominal. The most that can be said is that it had GPU drivers, NCCL, CUDA, and an OS and kernel, and that all were sufficiently modern.

Once we had the time to set things up, performance was fine. GPUs hit expected numbers on burn-in and NVLink supported the expected traffic.

However, the cluster’s WAN performance was exceptionally bad, averaging around 0.1 GB/s across several tests. The cause of this poor performance is unknown to us; needless to say, it would be an impediment to real-world usage.

Most of all, getting support was a massive pain. The entire technical team was based in Asia (Malaysia, or Singapore in our case), which meant 8-12h TAT when we were trying to troubleshoot issues. It was impossible to consider Bitdeer for anything other than this tier.

We look forward to seeing Bitdeer’s improvements in future rounds of testing as its managed cloud offering matures and it implements features like health checks and managed Kubernetes. It will be interesting to see where Bitdeer allocates its time and attention, as management has unambiguously announced that its neocloud business is not its main focus: “Executing on colocation lease agreements is THE top priority.”

Shadeform

Shadeform continues to focus on brokering and small scale marketplace reselling of individual GPU VMs with a small team. We enjoy the interface and working with the Shadeform team, but without more ambition they will stay in our lowest tier.

Radiant

Radiant was formed after Brookfield acquired Ori, a UK-based neocloud that we discussed in ClusterMAX 2.1. Unfortunately we have not been able to test a working, secure cluster. Despite almost a year of planning, along with quite a bit of marketing effort, and announcements that mention Billions and Gigawatts, Radiant is yet to deploy a Blackwell GPU.

FPT

FPT suffered from a number of security issues during our testing, which we described in ClusterMAX 2.1. We have yet to re-test and are not aware of any Blackwell capacity.

Core42

Core42 is a dark horse in this category. They have a solid team, allocation of GPUs, political backing, and an infinite money glitch in the form of Mubadala. We expect to see them put the pieces together and rocket up the rankings as they enter the US market.

Latitude

Latitude is making progress to keep their modest cloud business afloat, being recently acquired by Megaport and as a result going public. We had the chance to test some RTX Pro 6000 Blackwell servers, but as long as Latitude is missing the latest and greatest GPUs, we expect them to be stuck in this category.

IBM Cloud

We have not had the opportunity to re-try IBM Cloud since our horrendous experience in ClusterMAX 2.0.

BuzzHPC

Buzz is making waves in Canada, latching onto the Sovereign AI wave associated with the Carney vs Trump showdown, to the tune of 1.2GW and $50B in Saskatchewan, in partnership with Bell and others. This is quite the complement for their 300MW+ elsewhere in the country. BuzzHPC is one of the many public companies on the ClusterMAX list, via parent company HIVE Digital Technologies Ltd. Unfortunately, all the GPUs that we have tested from Buzz were in clusters of questionable quality, relying on questionable software vendor partner choices. We have not had the opportunity to re-test since ClusterMAX 2.0, but we hope this changes in the near future.

Neysa

Indian neocloud Neysa announced a $1.2B in financing this February, including $600M equity and $600M debt, in a round led by Blackstone that left them at $1.4B post-money. They claim “9,216 air-cooled NVIDIA B300 staged between December 2026 and March 2027,” with an equal number of AMD MI350X in the pipeline as well. Impressive growth, and clearly the leader in India in this regard.

This round, our testing unfortunately revealed a cluster that was riddled with errors. Many software packages that came pre-installed during cluster provisioning were months or even years out of date, with several serious CVEs applicable. CUDA Toolkit 12.0, for example, was released in December 2022, making it older than Neysa itself. Somehow this was installed on our cluster at /usr/bin/nvcc.

We enjoyed working with the Neysa team and encourage them to continue to improve their security posture and shore up their fundamentals as their capacity spikes and the bring on large customers from abroad. While current Neysa customers are mostly stuck on the Hopper generation, with a few B300s available, they are no GB’s (and, as a result, no direct liquid cooling) that have made it to production on the Indian subcontinent.

Vast.ai

A reseller/marketplace that keeps coming up in customer conversations for random GPUs on-demand, representing one of the better partnerships teams in the industry, finding unused capacity all over the world. Unfortunately, the UI is just terrible, so hard to use.

Hyperbolic

We haven’t had the chance to re-test Hyperbolic since the last round, but they got a basic security compliance attestation done finally! Hyperbolic moves onto the list.

STN

STN has been very confusing. One of the earliest clouds to deploy B300s, and one of the only providers with an option for “private cloud”, where customers take care of procuring the cluster themselves, and keep the chips on their balance sheet, but STN takes care of cluster operations. We hear from multiple customers how they enjoy this arrangement and prefer it over price gouging neoclouds.

We’ve always enjoyed working with the STN team, but unfortunately each of the last three times we’ve engaged with the team, we have run into some performance, reliability, or configuration issue, asked for a fix, been promised a fix, not gotten the fix, and then time has run out.

We regrettably keep STN in this tier until we can see a complete end to end testing experience as the customer would experience it.

Not Recommended - Underperforming

Based on hands-on testing, these providers can quickly rise to Bronze by fixing one or more critical issues, such as offering only older GPUs, missing basic security attestation (SOC 2, ISO 27001), misconfiguring key server features (leaving PCIe ACS enabled, or failing to enable GPUDirect RDMA), or charging for GPU hours during cluster creation or hardware downtime.

SharonAI

Sharon AI is an Australian neocloud that claims 132MW in its pipeline, with 116MW already contracted out, against a current footprint that we estimate around 4 MW. They do not have SOC 2 or ISO 27001. We were able to test 2 bare metal H200 nodes to verify basic functionality. We look forward to testing Sharon AI down the line as their offerings mature.

IREN

IREN continues to come up in our conversations with neocloud customers, and usually for the wrong reasons. They’ve got rock bottom prices on HGX B200 and B300 GPUs in Prince George and Mackenzie in Northern BC, Canada, and we’ve heard the stories. Multi-day power outages, network upgrades, storage failures, air quality controls leading to link flaps, XID’s everywhere. The #1 worst site in the industry according to users. And we are some of their users, as we’ve gotten to test IREN GPUs from multiple providers that resell their capacity. A lot of our users that aren’t having an good time that we are hearing from aren’t renting from IREN resellers but IREN themselves

With that said, the new builds in Childress and Sweetwater look much better (read: not N+0 power, cooling, and ISPs!). We’ve covered plenty about these sites in our Datacenter Model, suffice it to say that despite all the technical shortcomings in their cloud services, we expect the $2.1B investment from NVIDIA will help them cut less corners this time around.

We recommend that IREN stop pretending to offer managed clusters and inference endpoints in public marketing and investment materials, and focus on their bare metal offerings, where there is plenty of market demand for their services.

Hydra Host

Hydra Host raised a $100M Series A in June 2026, with NVIDIA among its backers. They are focusing on brokering deals, and our earlier testing is all we have to go on for managed cluster experience. We need to test a real managed cluster before considering an upgrade.

FarmGPU

We couldn’t make much progress in our testing of FarmGPU. The Slurm layer did not advertise GPU resources properly, while Kubernetes did not expose any RDMA devices for scale-out networking.

With that said, we very much appreciate FarmGPU’s open development culture, including their solid Grafana monitoring experience and detailed notes on provisioning. We find the technical team to be trustworthy and solid to work with, even if they are stretched too thin just trying to get a few small clusters deployed correctly.

WhiteFiber

WhiteFiber raised $159.4M gross at its initial IPO closing in August 2025. Our prior testing found good network performance but unusable Slurm/Kubernetes integration and weak job monitoring. More recent customer feedback suggested improvement, though the ephemeral Slurm login filesystem remained a concern. We need to verify that these operational problems are fixed.

PaleBlueDot

PaleBlueDot raised a $150M Series B led by B Capital in January 2026. We previously found its marketplace easy to use for individual VMs, but it lacked a complete managed-cluster experience. Unfortunately, they have shut down their on-demand console in favor of serving some big customers in dedicated datacenters.

Akamai

Not much has changed in the last year at Akamai/Linode, with single-node GPUs being the focus and no managed Slurm or Kubernetes available. We’ll keep watching to see if this sleeping giant wakes up anytime soon.

Hetzner

Hetzner’s low-cost hosting model still provides some cheap PCIe GPUs, specifically the RTX 4000 and 6000 Pro Blackwell Edition. We’re waiting to see if they take the plunge with anything bigger, building on all their datacenter operations experience to really serve the AI market.

Mithril

Mithril, formerly Foundry, announced $80M in funding back in 2024 to “restore the promise of public cloud to AI”. Today, it’s a marketplace wrapping 3 Nebius availability zones with H100 or H200, though only the H200’s have an option with a scale-out network.

Source: Mithril website

There were some announcements about TPU support that we were quite excited about, but that partnership seems to have been scrubbed away.

OVHCloud

OVHcloud has substantial infrastructure globally, operating in tons of datacenters globally, some of which are shared with top neoclouds on this list. But our concerns remain that general purpose IaaS is not the growth engine of the AI age with companies doing crazy things to get access to the latest and greatest. Another sleeping giant.

Massed Compute

Massed Compute last announced capital raise involved up to $300M in equity and revenue-share financing from Digital Alpha in August 2025. They have used this to drive reasonable growth for a bare metal business, but that alone does not cut it on the ClusterMAX rating system, especially since their slop cannon chatbot is still getting indexed by web crawlers.

Not Recommended - Unavailable

Our “Not Recommended - Unavailable” tier is the same as ever, but to clarify, for this section we include four types of companies:

Those that we have previously tested who now claim to have zero spare capacity and/or refuse to test with us. This includes Fluidstack, Cirrascale, Lightning AI (recently merged with Voltage Park), Scaleway, CUDO Compute, Denvr Dataworks, and Atlas Cloud.
Those that are yet to launch, yet to be tested, and are generally interesting to us, though we believe some unnamed of this bunch are misleading investors (i.e., lying) by claiming in public marketing materials that they do managed clusters when they really just do bare metal. This includes SpaceXAI, Mistral, Poolside Infrastructure Company, Nscale, Highrise, Corvex, Andromeda, Volta, Firebird, Tatra, Sesterce, Yotta, Boostrun, GlobalAI, Argentum and Qumulus.
Those who are big and important, but who we cannot test properly due to geographic or regulatory concerns. This includes Alibaba Cloud, MegaSpeed, BytePlus, RunSun, SK Telecom, Naver Cloud and Indosat/Zankore/Lintasarta.
Those who are too small to be relevant. These are still covered in our market view, but not part of the rating system.

Fluidstack

Fluidstack has (currently) exited the managed clusters market, as far as we are concerned, because their focus has shifted from managed Nvidia clusters to bare-metal TPU deployments at 100K+ chip scale. We look forward to testing with them again in the future, with whatever chips they may offer at that time.

Cirrascale

Unfortunately, after receiving a legal letter to modify our previous article’s description of our experience working with Cirrascale, we have not been able to establish a productive working relationship.

Lightning (merged with Voltage Park)

Following the merger of Lightning and Voltage Park, the combined company has unfortunately run out of GPUs, and we haven’t been able to test the reconciled products. This is despite the company consistently advertising that they are the 3rd biggest neocloud in the world in terms of GPUs deployed. They are not in the top 10.

Scaleway

Recently, we hear that Scaleway has de-prioritized a number of customer engagements and decided to voluntarily break ties with NVIDIA, in favour of AMD, for seemingly no good reason. We think this is a shame.

CUDO

CUDO did not make a representative managed Slurm or Kubernetes environment available for this cycle as it focuses on other business priorities (namely, managing the build out of a bunch of bare metal datacenters). We look forward to reassessing CUDO when an environment becomes available.

Denvr Dataworks

Despite changes in leadership and random bouts of activity in the community, we have yet to make any progress testing Denvr and have not seen any meaningful growth in the business that would allow them to dig themselves out of the hole their founders put them in.

Atlas Cloud

We haven’t been able to get GPUs from Atlas recently, despite big announcements about bare metal datacenters from their crypto parent company and random appearances on endpoint provider lists with their managed inference offering.

SpaceXAI

We love Colossus and can’t wait to test the SpaceXAI neocloud offering!

Mistral

Mistral just raised a €3B Series D, and has yet to confirm the validity of a hacker from TeamPCP claiming to put its entire codebase up for sale on the dark web. Anyway, we are very excited to test their neocloud offering. The team told us the platform was still coming up and onboarding its first customers in August, so we very much look forward to testing with them in the future. It seems obvious to us that companies with experience building training clusters for their own models (see above) will be successful neoclouds if they choose to pivot.

Poolside Infrastructure Company

Following a $6B licensiquihire by Nvidia that took an incredible group and buried them in the Nemotron bureaucracy, Poolside’s legacy lives on through the Poolside Infrastructure Company, which is developing their Project Horizon campus in West Texas. We have not yet tested anything from them, but their experience and ambition makes them an immediate contender if the skeleton crew decides to go for it on the neocloud business.

Nscale

Nscale announced a $2B Series C in March, announced the acquisition of AnyScale in May, and just filed an S1 to go public on NYSE a few days ago. It is unbelievable to us that they can have such a big business, and continue to claim in public marketing materials that they offer managed clusters and inference endpoints, when this is clearly not true. Obviously, though, they’ll continue to be successful. They’ve got access to lots of land and power, some construction and datacenter operations experience, and people really want that bare metal at a nice price.

Highrise

Highrise, Hut 8’s AI Cloud business, is constantly in the news. We hope we can test some managed clusters from them soon.

Corvex

Corvex continues to run scared from ClusterMAX, though they announced a $33M private placement in August to expand their datacenter capacity. Their choice of merger partner was Movano, maker of the Evie smart ring. Quite a change in product roadmap. Privately, they claim some secure government customers in the USG. We look forward to testing.

Andromeda

Andromeda takes capacity from other providers and puts a managed cluster service on top. We have discussed testing multiple times, and enjoyed working with the technical team, but find the business side to be a little shady. Resellers have a perverse incentive to keep the customer in the dark about who the underlying provider is, out of fear that the customer can just go around them and contract with them directly. Unfortunately, this can lead to bad games of broken telephone, and lots of finger pointing in the worst case. Specifically, we have a problem with non-circumvention clauses and hope that Andromeda can build a durable business model that allows them to be honest about their solid technical team and the value it can provide. We look forward to testing some of their Blackwell capacity soon, wherever it may be.

Volta

Volta launched with $300M in venture funding and a separate $5B financing pool for customers. Its first large announced contract uses Bitdeer’s Norway site. We have not tested the cloud software or customer operations.

Firebird

Firebird is building GPU infrastructure in Armenia, with $60M in construction financing announced by Ameriabank, and expansion plans in Kazakhstan. Recently, they became the subject of a story about Nvidia chip access and the Armenia-Azerbaijan peace process; co-founder Razmig Hovaghimian responded that the project had roots going back years before those diplomatic developments. GPU diplomacy. Pressure’s on to deliver for the people of Armenia! We still need to see the managed cloud in action.

Tatra

Tatra is developing B300 and GB300 capacity in Slovakia, using the country’s nuclear and hydro power base. We have some great feedback about the technical chops of the father/son team—hard working, impressive family. For now, though, we need them to get their first tranche fully deployed so we can put them to the test.

Sesterce

We spun up an on-demand H100 through Sesterce in August. The console showed an out-of-date Ubuntu option, provisioning took its time, and eventually we got a working machine. Then we found Shadeform underneath. A broker on top of a broker. We would quite like to know how many companies need a margin before the customer gets to run a workload! With that said, Sesterce is pursuing some big bare metal Blackwell clusters with big offtakers. We want to test those.

Groq

Groq set out to challenge Nvidia, but after a licensiquihire took Jonathan Ross and most of the engineering team, the remaining company has pivoted… to renting out Nvidia GPUs! Becoming a neocloud is the move if you want another $350M to live out your dreams (as announced in August). We look forward to testing more than just endpoints one day. Jensen gets the chip team and another customer. Not bad.

Yotta

Yotta handed us a four-node H100 Kubernetes cluster after an onboarding demo that mostly included a tour of the login screen. To file a ticket, we were directed from Shakti Cloud into One Yotta, then back through the Shakti domain. Unfortunately, all these portals do not help keep clusters online, with us hitting a real failure (Xid 94) within a day of starting our testing. Our testing is ongoing here, and performance is solid (though way after our deadline), but judging from customer feedback and our own experience, we’re getting pretty clear signals so far. We need to see Blackwell GPUs and some improved reliability for them to keep pace with others in the industry.

Darya

Darya is bringing GPU cloud infrastructure to Tajikistan, right on the border of Afghanistan, which certainly expands the map of places we need to test. Its H200 cluster launched in June 2025, and it subsequently signed an agreement with Yotta to develop a hydropower-fed AI datacenter in Darvoz. Interesting stuff, but after our experience navigating Yotta’s portals, we are particularly curious which parts of the operating model make the trip to Tajikistan. We have not yet had access to Darya’s cluster, though we would love to visit the site.

Source: Darya’s Datacenter on Google Maps

Boostrun

Boostrun has some solid bare metal, but bought its way past some of the Kubernetes engineering: vCluster says the company launched managed Kubernetes in under 45 days with no new platform engineering hires. They went with isolated tenant control planes, dedicated private nodes, and Netris for network provisioning. Sensible. We would rather see a small provider use working software than spend a year rediscovering Kubernetes. In customer discussions, we have still considered adding another operator for a fully managed training experience. We have tested a Boostrun cluster indirectly, through another provider reselling their capacity, but await the opportunity to engage with their support teams directly. We want to see how much of the day two operations Boostrun handles for themselves in a real engagement before moving them onto the list.

Global AI

Global AI takes the sovereign-cloud pitch quite literally: dedicated, single-tenant, air-gapped clusters sold by the data hall (or so they say). Customers decide when Global AI can access their systems, a solid offering for bare metal. However, when advertising an option for managed clusters, we need to see more. Just to assess bare metal we want to see things like how provisioning and security patches get handled, how comprehensive the monitoring stack is, how reliability and health checks work, and generally how competent the onsite team is at closing tickets quickly and correctly. We have not yet gotten to see this level of detail.

Argentum

Argentum’s website says “200,000+ GPUs available now.” Had you heard of Argentum before you got to this section of the article?

QumulusAI

QumulusAI found a different source of GPU debt: a $500M non-recourse facility through USD.AI, using GPU Warehouse Receipt Tokens as collateral for stablecoin borrowing, with financing for up to 70% of approved deployments. GPU Warehouse Receipt Tokens is the actual name. The announcement describes a facility, so we are not counting $500M of installed hardware. In our customer discussions, Qumulus has come up as a capacity supplier that may need another operator on top for managed training. We still need to inspect its own software and support, but initial customer feedback has been surprisingly solid.

Alibaba Cloud

Alibaba has considerably more software to show us than the usual neocloud startup. One example is its ACK Slurm operator: a SlurmCopilot component coordinates resource allocations between Slurm and Kubernetes so idle resources do not remain stranded in one scheduler. This is the kind of orchestration plumbing we like to poke at, and obviously this is a company that is not ready for a poking. We have an account, have been whitelisted for a few model endpoints, and have engaged directly with the sales team, but have yet to get access to any modern GPUs or this ACK software for testing.

Megaspeed

Plenty of hardware to investigate in their Malaysian and Indonesian sites, but no login yet.

BytePlus

BytePlus’s GPU service advertises switch-affinity placement on a multi-rail network, direct RDMA access to its vePFS parallel filesystem, and integration with our favourite model ever: Seedance 2.5. We have not yet completed an evaluation of their clusters, but there is clearly stuff to watch out for. Tik, tok.

Humain

Humain has given itself plenty to do. Alongside the Saudi GPU buildout, Tareq Amin announced a roughly 6 GW datacenter ambition, as well as Humain One, a real-time voice interface for any computer. We would be happy to start with a login and a functioning cluster, but we have yet to get a response from the team, leaving us wondering what secrets they’re hiding next to their Groq systems.

SK Telecom

SK Telecom runs over 1,000 B200s for Korea’s sovereign foundation-model initiative. As we covered in a recent article, the government also rented ~3,000 H100 equivalents from SK Telecom and Naver combined for the competition’s first round. SK Telecom has announced a much larger 2 GW NVIDIA DSX AI factory, expected to deploy Vera Rubin systems with SK Hynix HBM4, within SK Group’s planned 5 GW first-phase buildout. We have tested SK Telecom GPUs through Vessl, and they were solid, but nothing with SK directly. We need to see more from them on the international market to consider them a serious player, but the technical foundation is clearly there.

Naver

Naver, much like SK Telecom, is highly credible. It already operates hyperscaler-class datacenters, and its July plan with NVIDIA and Brookfield calls for expanding the GAK Sejong AI factory from an initial 55 MW buildout to 200 MW by 2028. We are interested in how much of that operational experience reaches external customers, vs the internal research teams at Naver. We have yet to test that experience.

Indosat (Zankore)

Zankore has announced the first phase of roughly 200 MW of GB300 NVL72 capacity is to be delivered in the first half of 2027. Even if they are a little behind schedule on deploying Blackwell, they are still well on their way to a long-term 1 GW ambition, primarily to serve offtakers (rumoured to be) from mainland China. And they’ve landed a $3.1B loan to go do it!

Where ClusterMAX Goes From Here

Computation and Data Movement for Inference

2026-09-21T10:14:15+00:00

Mixture of Experts, now widely used in frontier models, has changed both the structure of serving and the economics of useful inference. It did more than increase parameter count. It changed which tensors are active for each token, what must remain close together, which transfers need strong local bandwidth, which can tolerate a weaker network link, and how memory movement, storage, and scheduling contribute to useful throughput.

The best place to begin is the service as a whole. Inference runs inside a cluster coordinated by an orchestration layer such as NVIDIA Dynamo, Mooncake, or a custom scheduler. These work closely with inference servers like vLLM or SQlang, and those also have their own orchestration features. This article will not go into the details of how you work with the orchestration software or which you should choose. The article aims to provide an overview of the process and reasons for various features you may use.

A user (or their agent) starts a conversation with a request (query), and the conversation may continue after answers with more requests. The request- answer is a “turn”. In modern AI systems the user is often running a in a client application (whether GUI or command-line oriented) and some of the answers coming back from the AI are intercepted by that application as instructions for it to run, things like editing your code or searching your company guidelines on an HR question.

The AI system can also do some of those things itself from its data center, for example it may search the web. The results from these intercept tool actions are also returned to the AI as requests, creating more turns. The server keeps a context (often referred to as KV cache or just cache) which is the distillation of the session, allowing each new request, from user or from tool, to be properly interpreted for overall forward progress. There can be thousands of turns per hour when the user launches an agent on a long-running task. The conversation can also pause and resume hours or even days later.

When a request arrives at the inference servers the orchestration puts it into a queue, while will feed into an input-fill worker, along with any context from system prompts and prior turns in the conversation. The query is then converted into new context appended to the conversation. That new context state is then moved to a decode worker. Decode repeatedly reads the accumulated state, generates answer tokens, and those answer tokens are also appended to the conversation state. The process may loop through tools, agents, users, and further input-fill work. A model worker running on GPUs is only one part of this token factory; storage, networking, and orchestration connect its stages.

Inside the workers, four operating regimes are worth distinguishing from the beginning:

Prefill, where the initial block of new tokens is processed together.
Midfill, where a continuation request is appended to an existing cached context.
Decode attention, where the context is used to generate the basics of newly generated tokens.
Decode experts, where each basic new token is refined by a selection of experts a large total set of possible experts.

These regimes place different demands on compute, memory, and networking. Prefill and midfill are closely related, with Prefill being a Midfill with zero prior context. However, Prefill is a common special case, corresponding to one-shot queries like classic “”chatbot” use, and can be optimized a bit differently than Midfill. Prefill reaches high arithmetic intensity (ratio of computation to data movement) and does not need to wait for context to be located and read. Midfill begins from an existing KV cache for prior state and appends a new request input sequence. This generally has a more moderate arithmetic intensity than prefill because there are fewer new tokens and more existing data to move per token.

Decode attention and decode expert work are generally low arithmetic intensity since there are few new tokens compared to the prior context or expert weights, which at decode state are data that needs to be moved. Meanwhile, across all of them, the tensors for attention and for experts are the same regardless of token value, so there is benefit to sharing a single reading of a tensor with as many tokens as can be queued to use it, even tokens coming from unrelated requests by other users which simply happen to be running at the same time and which are networked closely enough to share. Treating all four mode as the same workload gives away much of the structural advantages that MoE models offer in terms of different intensity and different sharing patterns.

In this document we will treat the 4 stages separately, and sometimes assume they are disaggregated. There are tradeoffs between aggregated (where the models stay on one server, which reconfigures as the request progresses between the stages) and disaggregated where the orchestrator may find a free slot on another machine already configured appropriately. Generally we will talk about the stages as disaggregated, with aggregation being a special case of always making the current node available when a previous stage finishes. There will be a section on the trade-offs at the end, after the work to be done has been more completely introduced.

A transformer model is a deep stack of repeating layer groups. A group may contain one kind of layer, or it may contain a full-attention layer together with several linear, local, selective, or otherwise optimized-attention layers (this is a hybrid group of layers). Within each layer, attention, shared transformations, routing, selected experts, and recombination occur in sequence. Those steps are known in advance and repeat continuously, allowing the same hardware resources to serve different parts of the flow at different moments.

The machines can also be described equally simple recurring units. A node is a group of accelerators, typically a tray in a rack, each accelerator pairs compute with local fast memory. A node is coordinated by a CPU and connected to other nodes through NICs (network interface controller). Nodes stack into racks. A rack may act as one scale-up domain when very large tensors are , several smaller scale-up islands, or a set of pipeline stages. Datacenter networking then joins bounded workers to storage and orchestration rather than participating in every inner tensor operation.

This essay follows that progression. It begins with the global service and the operating regimes, then opens the model into wide layer-group slices and maps them onto accelerator nodes and racks. From there it develops pipeline, tensor, and expert parallelism; the limit imposed by KV-cache memory; the different value of batching in prefill, midfill, and decode; and the role of scheduling in a many-rack token factory.

Figure 1. Overview of model stages in an inference service.

1. Orchestration delivers requests via queues into pools of workers

An inference service is a cluster where prior requests and immutable state are stored when idle and then queued for delivery to specialized workers when a new request pulls in context or a prior request reawakens for continuation. The incoming queue is not yet a batch. Each request carries prompt tokens, references to reusable context, service objectives, and often an existing conversation or agent state. An orchestrator may place it into a prefill, midfill, or decode batch, and the request can join or leave that batch independently as it completes, making that capacity available for new work. Continuous batching works best when the scheduler knows which workers have slots that match the request’s context length and service objective.

Prefill workers create context from a substantial block of new tokens. Their output is new KV state for every model layer, stored independently of the worker that created it. The KV state moves along model layers, the output of layer K from one turn becomes input to layer K in the next turn. In pipelined servers this creates a natural “torrenting” of output and input from separate NICs.

Midfill workers extend previously processed context. The cached prefix may contain system and user prompts, memories, earlier conversation turns, agent steps, or retrieved documents. Large roots may be already cached, while the incremental input can be much smaller than a comparable uncached request: a half-million context may receive only a few hundred or a few thousand new tokens.

Decode workers consume the processed, accumulated context and generate one or a few tokens (multi-token prediction is becoming quite common) per pass. Every generated token adds a small amount of new state. The same context can later return to a midfill worker when a user, tool, or agent contributes another block of input. Agentic work using tools creates a repeated sequence of context extension and generation, not merely one prefill followed by one decode.

A worker may occupy a node, a tray, or a rack. The service becomes large by operating many bounded workers and moving requests and state among them. Depending on precision and working-state requirements, even multi-trillion-parameter models can fit within the aggregate memory of a modern rack-scale system. The datacenter network remains essential because the factory must connect many such workers to shared storage and continuously match worker configurations to demand. KV transfers may be gigabytes in size, but they can be striped or torrented across multiple links and destinations; their movement time can remain well below the lifetime of a long decode job.

This separation is useful for both performance and operations. Prefill and midfill are predictable once the cached-prefix size, new-token count, and worker configuration are known. Decode completion time is stochastic because arrival of the final output token cannot be predicted in advance. Shared memory and storage between the pools let each stage run at its own cadence. Later sections of this article will discuss queues, ready buffers, and worker reconfiguration. For now, the important point is that the service is a loop of stages joined by movable state.

AgentX examples

Figure 2. “CC Traces Weka” dataset contributed to AgentX, release 34. Source: SemiAnalysis AgentX.

This is viewed with the SemiAnalysis Conversation Explorer. It shows that context growth during conversations can be quite rapid but also goes through large fluctuations due to compaction and other model behavior which may not be explained in public discussions. Orchestration of the inference context is a competitive advantage for AI companies.

Figure 3. Cache, new input, and result lengths seen at each turn of the same conversations. Source: SemiAnalysis AgentX.

We can take that same dataset and pull out the cache, new input, and result length values seen at every turn. This visual shows how the Token Factory needs to serve many different workloads. In principle there could be requests similar to each dot in those traces (and more dots from other workload datasets) all running at the same time, assigned to some worker in the AI cluster.

This article aims to survey some of the major functions in inference that make it possible,

2. KV state moves as immutable blobs

Reusable context at data-center scale becomes a shared object-store of immutable blobs rather than a file attached to one machine. A system prompt, project context, previous user turns, tool output, and generated tokens can be in separate blobs. A new operation reads the blobs it needs and appends new ones; existing blobs normally remain unchanged. This forward-only structure follows the causal transformer itself. Changes can be handled by backtracking and creating a new branch rather than rewriting the common path.

The durable source of these blobs is a fast, parallel, scale-out memory and storage pool. It needs enough aggregate bandwidth and network reach that prefill, midfill, and decode workers can be selected for suitability and availability rather than because one machine owns the only copy of the context. Newly idle blobs can first move into shared network-attached DRAM, including node-CPU memory and dedicated memory appliances. As that tier fills, a classifier can discard blobs that are not likely for reuse, or promote longer-lived state to SSD. The underlying text and references are often orders of magnitude smaller than the expanded KV representation, so it is possible rebuild a KV blob from that smaller text if the triage decided to reuse space and discarded the expanded embedded state. AI calculations can vary subtly, so while the rebuilt state can be expected to be a valid context, services may take different approaches to how casually they discard and rebuild, and how much they try to keep important immutable state.

For a deeper dive on how the blobs are managed, Unified Radix Cache: One Tree for Hybrid Model Prefix Caching - LMSYS Org is recommended reading. It is not the last word, but it is clearly written and will point you to other sources if you want to look for more.

There are also frequent compactions of state with long-running agents that build up to the maximum context, so in general after a compaction there will be many prior blobs which have been abandoned in favor of a new context. The space from the pre-compaction contexts is probably recycled for new work.

Figure 4. AgentX release 34, selected conversations. Source: SemiAnalysis AgentX.

You can see compactions occur in the right half of these conversations as contexts get close to the practical context limit for each model (250k tokens for Opus 4.8, 1MT for Fable). There are also “stalactites” in both lines which seem transient and reflect some proprietary behavior in Anthropic models.

Accelerator HBM is the “hot” working tier. It is too expensive and supply-constrained to be a good default for passive context storage. The active prefix and suffix should enter HBM shortly before use and leave promptly after the worker has finished with them. CPU DRAM is a useful staging and assembly tier, especially for outgoing state: a completed worker can move newly generated blobs into CPU memory while the storage system chooses placement and redundancy, after which the NIC sends them into the shared pool.

For incoming data, RDMA-capable systems may allow the network to place data directly into accelerator memory, avoiding a full copy through CPU DRAM. CPU memory still remains useful for metadata, coordination, partial assembly, fallback paths, replicas, and outgoing staging. When a worker uses multiple GPUs the incoming context can be striped directly to their destination memories in parallel.

Rack-level hot-blob caches can also be useful, whether implemented as memory/storage appliances or making use CPU memory which is assigned to the distributed storage pool. Very common objects such as system prompts are natural candidates. These caches should remain shared scale-out resources rather than private state that binds a request to one decode machine. HBM is several times more expensive and more supply-constrained than DDR. HBM’s best use is for data ina an active batch currently earning revenue.

Figure 5. KV blobs moving through storage and worker memory.

Data input is generally larger than output. Midfill commonly reads a large prefix and appends a meaningful but smaller suffix. Decode reads the accumulated context and appends only the state for one or a few generated tokens. The reads are repeated, the writes usually occur just once. The system design should show these differences directly with the width of each data path.

3. Prefill, midfill, and decode create distinct operating regimes

The global service contains several computational regimes. Their differences are more useful than the generic labels compute-bound and memory-bound, because each has a different opportunity for reuse and a different natural mapping onto hardware.

3.1 Prefill

Classic prefill begins with little or no reusable KV state and processes a substantial block of new tokens. Those tokens share weight loads, use matrix operations efficiently, and create enough routed activations to form useful shared-expert batches. Attention and dense transforms can reach high arithmetic intensity, so prefill is often constrained primarily by compute. Long initial input contexts can raise memory traffic but these rarely get to 100,000 tokens in length, while midfill is commonly beyond that.

3.2 Midfill

Midfill appends new tokens to a much longer cached prefix. The new block can still provide hundreds or thousands of activations for shared weight and expert reuse, raising arithmetic intensity to thousands of ops per byte of data. Attention must also read a large existing KV state, which can be tens of GB. It therefore sits in a mixed regime: more arithmetic reuse than decode, but much more cached-state traffic than an initial prefill of the same new-token count.

3.3 Decode attention

Decode advances on input of one or a few tokens per query. Each query brings its own KV state which is similar in length to what midfill processes. While prefill differs from midfill on length of cache, decode differs from midfill in input sequence length. These one to few input tokens keep arithmetic intensity low, this work is clearly dominated by the memory movement around the KV cache.

For long contexts, the attention read can exceed the size of tensor operations associated with model weights. Attention is therefore one of the largest operations in modern decode even though most model parameters reside in experts. Delta, top-k, or linear-attention algorithm layers can reduce the burden, but periodic use of a layer with full attention remains a major memory load. Each query brings its own context, so batching queries does not change the arithmetic intensity of attention.

3.4 Decode experts

Experts are not specific to the decode phase. Everywhere the model runs the experts run, and they require no context other than one token’s embedding. The history of the query has already been compressed into the activation presented to the expert. This allows various designs that share the experts across all available GPUs, reducing memory size per GPU and increasing the memory bandwidth / memory size, or memory intensity ratio. You can look at the interval needed for a GPU to read all its experts as the best case interactivity (token rate seen by each user) and so if the memory bandwidth remains constant but the GPU is responsible for fewer experts, interactivity can be higher. The gotcha is that networking needs also rise, and the all-to-all pattern needed for routing to experts is a difficult one.

At the end of attention (in prefill, midfill, or decode) a router calculation (a small tensor operating on the unrefined token) selects a few experts for each new token. Tokens that select the same expert can share a tensor-weight load even when they come from different requests, so long as they can be gathered together in time to take advantage of the same weight loading. In practice their may be 15,000 experts across all the layers, so taking advantage of such coincidences requires deliberate synchronization like awaiting until all queries in a batch have completed attention until switching over to sending them for expert work, and that in turn may be improved by having multiple instances in different nearby machines also synchronized to that schedule. If you have ever wondered why Nvidia makes such an effort to connect 72 GPUs so closely, this is a big part of why. In a large token factory it is feasible to have those 72 machines all running in sync, allowing the experts in one layer to be divided up to 72 ways. If a layer of the model has 256 experts then each GPU is handling only 3 or 4 in each layer, permitting very fast complete cycling through all the experts. And a lot of all-to-all traffic.

Something interesting happens if your experts can all be kept in SRAM. You need a huge number of accelerators and a monster network to collect them all to all back to the GPUs, but now there is little reason to wait for a batch to form. Each expert can run whenever it has input. The energy per byte loaded is as much as 100x better than loading from HBM, so a single token running can be as efficient as perhaps a queue of 100 was before. In return for this DAF (disaggregated attention-FFN, where the experts are the FFN) you can free up the instances from needing to be synchronized. This advantage does not help as much with prefill or midfill since they trivially have a batch of tokens they can sort into shared queues on one layer, but it still may offload all the expert weights from those GPUs allowing them to optimize their local memory for other uses like longer contexts. The networking is still a gotcha, where an SRAM based accelerator could turn around an expert calculation in less than a microsecond but now you may be flooding a switch connected to hundreds of nodes with the all to all traffic for millions of tokens. The network may absorb more money and power than the experts they connect.

Figure 6. Prefill and midfill are separated from the closely coupled decode loop at the KV-cache boundary.

Inference workers can improve sharing by coordinating many attention instances on the same model layer. Their routed activations then draw from one layer’s expert bank, reducing the number of different experts each destination must load. In a pipelined server this also means the networking connections used for expert parallel can be local to that stage of the machine, there will never be a routing that needs experts from another layer. Even so, each decode request contributes only one new token with typically 8 to 16 routed experts per pass, and a layer may contain hundreds of experts. At small batches, sharing remains limited and memory bandwidth continues to dominate both the KV scan and the active expert loads, with very little computation per byte of weight loaded. Expert memory needs to be very low energy per bit, no matter what kind of memory is in use.

These regimes explain why prefill, midfill, and decode may prefer different worker configurations; why decode attention and decode experts can prefer different placements within a node; and why batching helps each stage by a different amount. As prefill in modern agent work rapidly expands to the 500k token range there is a lot of memory movement, especially in token decode - the stage where the model is generating useful intermediate and final results. Fast and large 3D RAM will be decisive in efficiency. Fast memory is also useful for midfill. However, capacity needs to reach a minimum level for the large contexts we now see as normal. If memory capacity is too small then the losses in networking and excessive swarms of chips can waste power compared to future memory types which will use 3D capacity expansion without giving up on throughput.

Figure 7. Projected effect of faster memory (hybrid-bonded DRAM, super-fast 3D RAM)) on midfill and decode with 500k token context cache length. Source: SemiAnalysis model simulator with projected accelerator and GPU performance.

While Vera Rubin 72 can crush at the mid-fill layer, which is arithmetic intense, it is not so effective at the switch to memory intensive decode where the Rubin cannot fully exercise its computational prowess. With the large context there is a premium on keeping the decode work in the same node and the midfill, avoiding sending the context across the network to a different node. This could favor an all-around chip which both has super memory thruput and really efficient computation.

4. Flow through an MoE prefill layer

Classic prefill begins with attention. A block of new query tokens attends to reusable prefixes like the system prompt and to earlier positions in the new block. The attention operation produces one activation for every new token, and shared transformations prepare those activations for the routed-expert stage.

The router assigns several experts to every token. If the model selects eight routed experts, each incoming token produces roughly eight routes across a bank that may contain 256 experts or more. In token order, those records are sparse and interleaved: neighboring tokens can have completely different destinations. Across the full input block, however, every expert can accumulate a short list of assigned tokens.

The worker sorts or queues the route records by expert identifier. All activations assigned to expert 037 become one message stream; those assigned to 142 become another. A single expert-weight load can then serve all activations in that bucket. Several requests can contribute to the same buckets, and separate attention instances running the same layer can merge their routed work at the expert destination. Converting token order into expert order is one of the central efficiencies of MoE prefill. Scale-up networks like NVLink can be organized as memory mapped connections so sending requests to specific experts could be mapped to pushing the message into a specific memory mapped location which is configured as a hardware queue. There is very little start and stop time. The experts can listen with the equivalent of an RDMA CIQ that again leverages hardware acceleration to deliver the message stream without need for complex protocol start and stop. Simple headers inline on the message identify the sender and receiver. A similar connection can be set up for the return flow from expert back to the decode worker.

The expert outputs are returned to token order, weighted, combined, and passed to the next layer. At the same time, attention creates new K and V state for the appended tokens. Across the model, those values form an immutable KV suffix which are used to decode the next token, and which are eventually stored as the next blob in the context for use by the next turn in the conversation.

Prefill therefore combines two forms of weight reuse. Attention and dense transforms reuse weights across many tokens, while routed experts reuse each selected expert across the tokens collected in its bucket. The gains eventually flatten as growth in activation traffic, KV traffic, and output handling take more time compared to the tensor calculations. Batching can delay time to first token, but a single request containing several thousand new tokens already provides substantial arithmetic intensity; prefill does not always need a large number of requests in each batch to run efficiently. Modern single-socket GPUs may be able to do the prefill even on a large model for a small batch in around a second.

Figure 8. Flow through an MoE prefill layer.

5. Flow through an MoE decode layer

Decode begins with a much smaller amount of new work per forward pass. The current token, or a handful of speculated tokens in MTP (multi-token prediction) is the query. Attention reads the prior KV state together with the new token’s KV entry. The new entry appends to the result and is appended to the KV state for the next pass.

For agentic contexts, the prior KV state is often a heavier read burden than the attention and expert weights used by that one token. The new query and its local projection weights are comparatively compact. Attention may be partitioned across KV heads, context ranges, memory channels, or accelerator units, after which partial results are reduced to one token activation. That activation is small enough to move cheaply even though it will trigger much larger local tensor operations in the next phase.

The activation passes through shared transformations and routing. The router selects a few experts from the available bank, and only those expert tensors become active for the current token. Their outputs are weighted, combined, projected, and passed onward.

At each layer, attention creates and retains that layer’s K/V entry for the new token. The activation then advances to attention in the next layer, where the same sequence repeats. By the time the final layer produces the next token, every layer has appended its small K/V update. Those updates can be assembled into a new immutable KV suffix while the token is streamed to the client or agent.

The flow therefore contains both a broad operation and a sparse operation. Attention reads widely across private query state. Experts activate a small subset of total model capacity. The hardware mapping should support both without forcing the entire layer to use one parallelism strategy.

Figure 9. Flow through one MoE decode layer.

6. Midfill is a distinct operating regime

Midfill combines a long, cached prefix with a block of new tokens to prepare another turn in a conversation or agentic work. Traces show the cached input to be many times larger than the incremental query, although the incremental query typically varies from 50 to 5,000 new input tokens. The presence of cached system prompts, conversation roots, memories, prior agent steps, and retrieved documents can quickly raise conversational and agentic contexts to a million limit. AgentX datasets show frontier experts are repeatedly compressing contexts to stay under a million-token limit.

This makes midfill neither a small prefill nor a large decode. Like decode, it must read a substantial private KV cache. Like prefill, it processes enough new tokens to reuse weights and to sort routed activations into useful expert batches. Its arithmetic intensity can be hundreds of times greater than one-token decode, yet its cached-state traffic can be tens of times larger than that of a similarly sized initial prefill.

Modern accelerators can generally exploit the arithmetic reuse available in a block of hundreds of tokens, so the difficult balance is often between the large KV read and the burst of routed-expert traffic. That burst does not match the smooth rhythm of a decode pipeline: a midfill can occupy a stage much longer than the decode batches in neighboring stages, leaving those stages underused.

The ideal parallelism can also differ. The worked example later in this essay finds midfill configurations that use different pipeline, tensor, and expert-parallel settings from decode. Reconfiguring a GPU in the middle of a live decode pipeline can be expensive or impractical. A service may therefore benefit from a dedicated pool of midfill-optimized nodes. The hardware can be identical to the decode pool while the loaded model stages, parallelism, and batch policies differ.

Figure 10. Midfill combines the cached-state read of decode with enough new tokens to batch model and expert work.

7. The model is a tall stack of repeating layer groups

A model’s parameter count quantifies the total learned state, but serving depends on how that state is organized and used. The model is a stack of layers, and increasingly it is useful to group those layers into repeating architectural units.

In some MoE models, a layer group is the same as one layer. In others, a group contains one full-attention layer together with several linear, delta, top-k, or otherwise optimized-attention layers. Typically 1 to 3 of the initial layers may be dense or otherwise designed to get a clean start. The rest of the model will normally repeat one recurring group pattern.

A wide, shallow crêpe is a useful representation of a layer group. The broad surface provides room for attention, shared transforms, routing, expert capacity, and residual flow. Its shallow thickness marks the group as one slice in a much taller repeated model. It also echoes the physical form of an accelerator: a very thin active layer spread across a broad package. The crêpe is a useful intuition for mapping a wide algorithmic flow onto a wide hardware surface.

The repeated structure simplifies function planning. A mapping created for one group can be reused for the next. Corresponding memory regions can hold the next group’s weights. The same compute and communication plan can execute repeatedly. Uniformity matters not only for runtime efficiency but also for compiler, kernel, and operations tooling.

Figure 11. Layer groups repeat the same flow.

During decode, one layer is active at a time for a given token. Within that layer, several operations occur sequentially. Neighboring layers may be prefetched, and different pipeline stages may process different queries simultaneously, but the logical path of one token still climbs through the stack in order.

Figure 12. One layer group opened into sequential sublayers.

8. Parallelism should partition the narrow dataflows of the model

Parallelism is not one decision. Pipeline, tensor, and expert parallelism divide different dimensions of the model and should be judged by different communication patterns.

Pipeline parallelism follows the layer stack

Layers immediately suggest pipelining. The activation passed between layers is compact—often one embedded token, a small multi-token-prediction block, or brief summaries of prior hybrid group results —while the weights and KV state used inside a layer are much larger. A pipeline stage can therefore own one or more complete layer groups and pass a relatively small activation to the next stage without overpowered networking.

Pipelining divides model weights cleanly and keeps hot within-layer communication local. Its main drawback is multi-batch occupancy: a fully active pipeline has a different query or batch in every stage. Each stage therefore holds its share of the KV state for all live streams, and the memory benefit eventually stops improving as contexts grow.

Tensor parallelism belongs where one operation is intrinsically broad

Attention may need to divide weights and KV state across several accelerators. KV heads, context ranges, or another linear dimension can be partitioned so that each accelerator processes its share before a compact reduction of the token result. Tensor parallelism can also serve unusually large dense transforms.

TP should not spread automatically into every tensor of the layer. A collective that is sensible for a broad attention operation may cost more than the arithmetic of a small expert tensor.

Expert parallelism follows independent expert tensors

Most MoE parameter memory may reside in experts. Experts retain no query history of their own; context arrives in the activation. The expert bank can therefore be distributed widely across a peer set and shared by several worker instances.

Wide expert parallelism is useful because the experts are independent and numerous. The router sends compact activations to selected destinations, and those destinations return transformed activations. Expert-parallel width can remain large even if pipeline stages become smaller, allowing switch radix and local memory placement to be chosen independently.

The general rule is straightforward: use pipeline parallelism for the vertical layer sequence, tensor parallelism for large indivisible operations, and expert parallelism for the context-free routed-expert bank. A stage fits comfortably inside a scale-up domain when its available fast memory safely exceeds its local weights, live batch state, and working buffers:

M_{free,node}>\frac{W}{P}+M_{batch}+M_{working}

Modern GPU nodes often exceed that threshold by multiples. When they do, the demanding bandwidth, power, and latency requirements can remain inside one node while thinner pipeline activations and KV movement use scale-out links. The rest of the fast memory is probably not earning any revenue. Data sitting idle in memory is an expense, data moving to be processed is revenue.

Figure 13. Pipeline, tensor, and expert parallelism follow different dimensions of a layer group.

9. A logical accelerator node

A logical node is easier to understand as a peer set arranged in a ring than as a literal circuit-board layout. The ring contains perhaps sixteen accelerators. Each accelerator pairs a compute region with local fast memory. The units connect through a strong scale-up fabric. A CPU coordinates workload and memory management, while a NIC connects the node to rack and datacenter resources.

The ring is just a visualization which shows the accelerators as peers, with equal standing. I am not implying the node-internal network should be a ring. It could be hub and spoke, rail, all-to-all, torus, hypercube .. whatever floats your boat, so long as it can handle the highest dataflows the rack contains with low latency and low energy per bit. I’ll just draw it as a ring for simplicity.

This representation avoids committing too early to package placement, board routing, or switch implementation. The actual machine may use a central switch, several switches, direct links, or a hierarchical fabric. The logical requirement is a bounded peer set with predictable fast local communication.

The accelerator design may vary. One version may use SRAM-heavy processing-in-memory units. Another may use hybrid-bonded DRAM, or IGZO memory cells placed on top during BEOL processing. Another may use a hybrid-bonded high-bandwidth true 3D memory. Memory capacity and bandwidth can change while this logical model remains useful.

The node is the natural place to keep work that repeats at high frequency. Attention partitions exchange compact partial results. Routed expert activations travel to local destinations. Expert outputs return for recombination. Shared state and scheduling metadata remain near the CPU. The NIC carries pipeline activations, KV blobs, and work assignments beyond the node.

A node may be smaller than a rack like we see in an SGX node with 8 GPUs, or a scale-up system may span the whole rack like we see with NVL72. The logical symbol remains valid in either case. If scale-up already covers the rack, the separate rack-backbone level disappears and the next boundary is directly to scale-out networking.

Figure 14. A 16-accelerator peer set on a scale-up fabric.

10. Nodes stack to make racks

The ring becomes more expressive when tilted as broad, shallow hardware crêpe. Several units can be stacked vertically in a rack, like model layer groups stacked in model depth.

This repeated form makes the model-to-hardware relationship easier to see. A layer-group crêpe can be placed onto one hardware crêpe. Several layer groups can occupy one node if memory permits. One layer group can span several tightly connected nodes if attention or expert capacity requires it. The mapping can expand or contract without changing the basic visual vocabulary.

A rack may be organized in several ways:

one rack-wide scale-up domain;
several node-scale islands connected by a rack backbone;
pipeline stages distributed among those islands;
wide expert placement within each stage;
or a mixture of stage-local scale-up and shared scale-out links.

The rack backbone is therefore an optional middle level. Some systems extend the scaleup fabric across the rack needing no separate node connector. Others use strong local node fabrics and a distinct top-of-rack or shared-rail network. Above the rack, scale-out networking connects storage, prefill pools, decode pools, and other racks.

The repeated rack units also help separate two kinds of scale. The model can fit within one bounded rack worker, while the factory scales by adding many workers. The first problem is function placement. The second is worker orchestration.

Figure 15. Accelerator nodes stacked into a rack.

11. One layer group reuses the node over time

A layer group does not require every operation in the algorithm at peak intensity simultaneously. Its operations form a repeating sequence that reuses the same hardware for different work over the decode cycle.

Attention may spread across much of the ring, using many memory channels and several compute units. The query and output transformations use model weights in more compact operations. Shared feed-forward work may use a dense region. Selected experts then activate local memory-and-compute regions around the ring. Recombination returns a compact result.

The same physical accelerators can participate differently at each step. Execution units that calculate attention may later execute one or more experts. Memory channels used to stream KV state can later feed expert weights. Buffers and local links are reused as the token advances.

This time-sharing is central to an efficient design. Static diagrams can make the machine appear underused because not every block is active at once. In reality the hardware is serving a sequence of different operations, and the goal is to keep the sequence moving with little setup or synchronization overhead.

The broad layer-group crêpe and the accelerator peer set are complementary. The crêpe shows the algorithmic flow; the ring shows the reusable physical resources beneath it. Mapping is the act of aligning each stage of the flow with a suitable subset of the ring.

Figure 16. One layer group reuses the peer set over time.

12. Put throughput boundaries where the flow is smallest

Once the model and machine are drawn as connected flows, the placement rule becomes clear: frequent, latency-sensitive communication should remain inside the strongest local fabric, while weaker links should carry compact or amortized transfers.

Figure 17. The general flow of operation of an MoE pipeline stage (animated).

Attention partitions may exchange partial results every layer. Expert routing sends activations to selected experts and returns their outputs. This traffic is not always the largest by byte count, but it is frequent, bursty, and sensitive to synchronization. It belongs inside the chosen scale-up domain whenever practical.

The activation passed between layers or layer groups is much smaller. A pipeline boundary can therefore cross a weaker link without moving the layer’s full internal working set. KV blobs are larger, but they move at worker boundaries and are amortized across many generated tokens. Their path can use striped or torrented scale-out networking and shared storage rather than consuming the innermost tensor fabric.

The router-to-expert boundary is typically a medium-throughput all-to-all carrying compact activations. Its difficulty often comes from endpoint count, synchronization, and routing efficiency rather than raw byte volume. Keeping it within the scale-up domain of a node or rack limits both latency and operational complexity.

This hierarchy is easier and cheaper to build than a machine in which every link is equally strong. Scale-up paths surround the operations that exchange embeddings and partial results within every layer. Scale-out links carry pipeline activations, KV movement, scheduling traffic, and traffic between bounded workers.

The tiers also have different latency requirements. A pipeline transfer between layer groups may barely notice a few microseconds if throughput is sufficient. By contrast, when an expert load and multiply take only a few microseconds, an additional 100 ns at each routing hop is material. The closest links should therefore be integrated tightly with the accelerator fabric, while less frequent transfers can tolerate the scale-out network.

13. Pipeline by layer group where practical

The layer group is the natural first candidate for pipeline placement. In a classic MoE model the group repeats one kind of layer. In newer structures it may contain one full-attention layer followed by several optimized- or localized-attention layers.

Keeping a group together can preserve internal dependencies and allow one placement and compilation plan to repeat across stages. Uniform plans reduce tooling cost and simplify deployment. Kernels, memory layouts, communication schedules, monitoring, and failure recovery all become easier when each stage runs the same recurring structure.

It is still possible to cut a pipeline in the middle of a group, although the resulting placement is more complex. If the layers have no special cross-layer dependency, the compact activation boundary remains a low-volume link suitable for networking. A stage may contain several whole groups, one group, or part of a large group depending on memory and throughput requirements.

The guiding principle is not an absolute prohibition on splitting groups. It is to prefer regular boundaries and to understand when a nonuniform split is worthwhile. Tooling and operational simplicity can be as important as a small theoretical placement gain.

Figure 18. Four uniform pipeline stages.

14. Pipeline depth ends at the KV-cache crossover

Pipeline parallelism divides model weights cleanly. If a model has weight footprint (W) and uses (P) stages, the idealized weight footprint per stage approaches (W/P).

KV-cache demand behaves differently. A full pipeline remains occupied by running a different query or batch in every stage. In an eight-stage pipeline, each stage owns one eighth of the layer-wise KV values for each of eight live streams. The reduction in layers per stage is offset by the number of simultaneous streams, so the stage does not receive the same (1/P) reduction in live KV demand that it receives for weights.

A useful stage-memory estimate is:

M_{stage}\approx \frac{W}{P}+M_{KV,stage}+M_{working}+M_{slack}

Once weights per stage fall to the same order as KV and working-state memory, deeper pipelining produces little additional capacity benefit while adding coordination and latency.

The crossover depends strongly on model design. Classical full-attention approaches can require hundreds of kilobytes of KV state per token across all layers. MLA-like designs are commonly in the tens of kilobytes, while recent hybrid layer groups with mostly linear, recurrent, or local attention can be lower still. A working average near 25 kB per token may be reasonable for some new hybrids; approximately 70 kB per token is a conservative reference point for a proven compressed-attention design.

As a back-of-the-envelope design target, take 70 kB per token and two million tokens of aggregate live context assigned to one stage. That is roughly 140 GB of KV state. Double buffering allows one working set to move while another continues to run, raising the KV allowance toward 280 GB. Add model weights, activations, transient buffers, routing tables, fragmentation, and operating margin, and a stage target of roughly 400–500 GB of local fast memory becomes a useful 2027 benchmark.

Other designs can reduce requirements through streaming, compression, partial residency, or additional tiers. The estimate is not a universal requirement; it is a standard that a practical frontier system should be able to accommodate without heroic assumptions.

The stage can distribute this memory over several accelerator sockets if they share a scale-up fabric. More than 95% of the weight memory in a large MoE is in routed experts, which divide naturally among sockets. KV state can be partitioned by head, context, or attention instance, with the resulting collectives matched to the same local fabric.

Figure 19. Fast-memory demand is stage weights, live KV state, working buffers, and margin.

15. Prefill and midfill turn sparse routes into dense expert batches

Prefill and midfill have strong batching opportunities because many input-sequence tokens complete attention before expert execution begins. The router can score all tokens, emit several destinations for each one, and then sort the route records by expert.

A practical input block can contain enough assignments that essentially every expert receives work. If 128 new tokens each select eight experts, the router emits 1,024 assignments across a 256-expert bank. Routing remains sparse for each token; sorting changes the execution order so that expert e000 processes all of its assigned tokens, then e001, and so on. Each expert is loaded once and reused for its compact token list instead of being repeatedly loaded in token order.

For an expert with (W) weights, weight traffic (B_W), activation traffic (B_A) per routed token, output traffic (B_O), and (n) assigned tokens, a rough arithmetic-intensity expression is:

AI_{expert}\approx \frac{2nW}{B_{W}+nB_{A}+B_{O}}

When weight traffic dominates, increasing (n) rapidly improves reuse. Eventually activation, output, and routing traffic become important and the gain flattens.

The sort is not free. Route records must be built, counted, permuted, sent to expert destinations, and restored to token order. Expert capacity limits can cause overflow or rerouting. Even so, prefill and midfill begin with enough tokens that the optimization is usually meaningful.

This is the key distinction from decode. The experts are identical; what changes is the number of routed activations available when the worker executes them.

Figure 20. Sorting lets each expert load once and process every token assigned to it.

16. Decode gains less from batching

Decode attention receives little direct cross-query reuse. Each additional query contributes another KV cache and another context scan. Shared query-projection weights can be reused, and software overhead can be amortized, but the dominant long-context read remains private to the query.

Decode experts offer potential sharing because the expert weights are context-free. The practical limitation is sharing probability. A model may contain hundreds of routed experts in every layer and thousands of layer-experts across the model. Each query contributes one new token per pass and selects only a few destinations. At small batch sizes, only a few routed tokens are available and most active experts receive work from just one token. Some sharing occurs, but it grows gradually with the number of tokens in the pass.

Shared transformations provide more conventional weight reuse, but they are a smaller fraction of an MoE model. The overall throughput gain from enlarging a decode batch can therefore be modest compared with the increase in latency and KV-memory pressure.

A batch of one is not a trivial workload when context length is 100,000 tokens or more. The attention scan already occupies substantial memory bandwidth. The optimum batch size is not necessarily one—software launch, allocation, routing, and hardware utilization still matter—but it may be much closer to one or five than to the large batches associated with throughput-oriented serving.

This matters economically. Interactive tokens can be more valuable than bulk delayed tokens. Multiplying latency to gain a fractional throughput improvement may be a poor trade. Hardware and software should remove impediments to useful small-batch operation rather than making large batches a prerequisite for efficiency.

Figure 21. Decode uses multiple routes but rarely shares experts at small batch sizes.

17. Scheduling makes disaggregation useful

Disaggregation becomes most effective as a statistical system at cloud scale. A large pool can contain workers configured for long prefill, short prefill, long-context decode, short-context decode, or other useful classes. The orchestrator assigns each job to a suitable available worker and moves the required KV blobs through shared storage.

Prefill and midfill work are predictable once the prefix size, new-token count, and worker configuration are known. Decode is stochastic because output length is uncertain. Storage between the pools decouples predictable production from irregular consumption. A ready buffer of completed prefill and midfill work can keep decode workers fed even as individual queries finish at different times.

The scheduler should observe customer workloads and time-of-day patterns, anticipate known ramps, and reconfigure workers as the mix changes. Reconfiguration may mean loading different model stages, changing pipeline cardinality, reallocating expert capacity, or moving a rack among prefill, midfill, and decode.

Scale matters. In a small fixed benchmark, one input-fill stage may be connected directly to one decode stage. A slow stage can strand capacity elsewhere in the other stage. In a large pool, workers complete and start independently so no worker need be stranded.

High interactivity and high utilization are therefore not inherently opposed. The scheduler can keep batches small while keeping machines busy, provided it has enough workers, enough storage bandwidth, enough visibility into the work, and fast enough response times to adapt to the moment.

The service overview in Figure 1 provides the corresponding system picture: four resource pools with the orchestrator spanning beneath them.

18. Feedback can turn a small imbalance into oscillation

A tightly coupled system can amplify ordinary variation. Suppose admission to input fill is gated directly by the retirement of decode queries. A long-running decode query delays retirement. Input-fill admission falls. Decode later drains its ready queue and becomes underfed. Admission then opens aggressively, producing a burst of input-fill jobs and a new wave of congestion.

This is a delayed feedback loop. The control signal arrives after the system state has changed. Large batch recovery can make the loop worse by increasing KV-memory pressure and response latency. The factory alternates between full and empty rather than operating near a steady point.

No single scheduler policy solves the problem. Several familiar control techniques help:

use ready-work buffers between stages;
regulate admission from smoothed backlog and predicted service rate rather than individual retirements;
separate fast local controls from slower capacity controls;
apply hysteresis to worker reconfiguration;
reserve capacity for short interactive work;
and avoid changing batch size more quickly than the system can observe its effect.

The same principle extends across timescales. Microsecond worker scheduling, millisecond token flow, second-scale buffers, minute-scale worker reconfiguration, and hour-scale demand planning should not all respond to the same noisy signal.

Figure 22. From retirement-gated oscillation to steady flow.

19. Aggregation Compared to Disaggregation

Now that we have exhaustively described disaggregated inference, is there a case to be made for aggregation? To some extent, yes. Clearly, since conversations can pause for minutes and hours it is not reasonable to keep a whole conversation tied to one system. That HBM can be put to better use within seconds of the end of a turn, and modern scaleout networks can move gigabytes of KV cache to a cheaper place in a parallel storage system, which can tier DRAM and SSD, in less than a second.

However, during a turn, do we need to schedule more than one place to get the work done? It is expensive to have too many different designs of server, adding operational costs as well as design work. What if there were only one design, and it centers on the midfill case? A request can be scheduled to that empty worker and stay there for prefill, midfill, and decode. It would need to have enough compute for prefill and midfill, and enough memory bandwidth for decode. There would always be some stranded capacity for either compute or memory transfer, depending on which stage the model is running, but we already tolerate low MFUs on GPUs. We try to overcome that with clever kernels, but in practice clusters run with stranded functions all the time.

Complex orchestration will result in some stranding. Some of that is avoided if a turn runs on a single aggregated worker. No schedule gap when pre/midfill hands over to decode. Natural sharing of one copy of the wide parallel set of experts. No network overhead to move the context from fill to decode. The orchestrator still looks at the shape of the request – cache size, input sequence length, model type, customer interactivity level – and makes one decision which available worker should handle it through all stages of the turn. Then it leaves it alone until the stop token comes out and the turn is delivered to the user and cached to the fast parallel file system. The one worker uses kernels and dataflows adapted to the given request shape as it works through all stages of the turn.

Aggregated workers have their attractions and advocates.

All systems will always strand some capabilities of the accelerators in real world uses. In a disaggregated system there can be two (or more) types of machines built with investment focused on compute, memory, and network actually used, arguably the best use of that investment. Prefill machines might invest in compute, while decode machines might invest in memory. If workers are aggregated, then one machine will be used through all phases so it needs to invest in the strongest compute and the strongest memory thruput. Maybe also strong network. If is not an all-around standout, then it will be held back by its weaker sections and be an overall “meh”. So, aggregation has advantages but it also leaves nowhere to hide in delivering the best of all-around performance. If you can build that magic all around star, then aggregating the whole turn in one worker could be best.

20. Memory bandwidth and memory capacity are different objectives

Memory is often discussed as two independent scalars to maximize. In practice, capacity and bandwidth trade against each other, and their economic value must be considered together.

Data earns revenue when it moves; it incurs cost while sitting idle. An accelerator with very high memory throughput can produce tokens at a rate that lower-bandwidth memory cannot match, even if the lower-bandwidth configuration has more capacity. Capacity is valuable until the active weights, KV state, activations, and operating slack fit cleanly. Beyond that point, additional local capacity may add cost but not revenue.

The capacity curve is not a vertical cliff. Below the useful limit, adding accelerators supplies both more memory and more bandwidth. The advantages end when other limits appear: maximum scale-up width, poor model cuts, the dominance of latencies, extra synchronization, or network contention.

A safe-capacity design point must sit above those limits. It includes margin for realistic context distributions, double buffering, fragmentation, failures, and transient peaks. The revenue-capacity curve rises steeply before that point, rounds smoothly, and then reaches a flatter ceiling determined mainly by bandwidth and compute. Revenue may even decline if excess capacity becomes a sunk cost that operations try to justify with big slow batches of high throughput but inferior value.

Accelerator HBM is too expensive to hold idle data without immediate use. Completed KV blobs should move into CPU DRAM for outgoing staging and then into scale-out storage. Frequently reused common blobs may remain replicated in shared CPU memory or nearby cache appliances. Only the active working set can pay for the most expensive memory.

Tall stacks of high-capacity memory may impose costs in power, signal quality, packaging, and frequency. A design with just enough capacity and exceptional bandwidth may serve interactive inference at higher margins than one that maximizes capacity per accelerator with idle excess.

Figure 23. Revenue per machine versus local fast-memory capacity.

21. A practical mapping method

The prior sections can be condensed into a repeatable method for mapping a model onto a machine.

Separate the operating regimes. Measure prefill, midfill, decode attention, and decode experts independently. Each may split further by context length or workload class—for example agents, coding, media, or content creation. These become prepared configurations that the orchestrator can mix in production.
Describe the repeating layer group. Identify full and optimized attention, shared transforms, routed experts, and any cross-layer dependency.
Measure active state rather than total parameters alone. Separate stage weights, active experts, live activations, KV state, and transient buffers.
Choose the natural parallel dimension for each operation. Pipeline the layer stack, tensor-parallelize broad indivisible work, and distribute independent experts widely.
Map the broadest operation first. Long-context attention often determines memory striping and the minimum useful scale-up participation.
Place expert weights in regular local destinations. Let routing select among known peer locations rather than creating a global event for every token. A large model may contain on the order of 15,000 layer-experts, and each can be placed with regard to the other experts in its layer and the expected routing distribution. The goal is even use of memory capacity and bandwidth.
Keep repeated high-throughput cooperation inside the strongest fabric. Let thinner activations and amortized KV movement cross weaker boundaries. Respect latency and power cliffs as well as throughput cliffs.
Stop increasing pipeline depth at the KV-memory crossover. Once the model and target context fit with safe margin, additional stages bring little memory benefit and add stage-transition latency.
Use input fill and decode differently. Sort prefill and midfill routes into expert batches; preserve small-batch decode interactivity.
Design storage and scheduling with the worker. KV movement, admission control, ready buffers, and worker reconfiguration determine whether the physical mapping produces steady throughput. Short batches and continuous batch updates let the orchestrator replace individual requests without waiting for the whole batch to retire.

The method does not select one universal topology. It makes the trade-offs visible. A peer set may be one package, one board, or a rack-wide scale-up domain. A layer-group crêpe may map to one node or several. The important point is that the communication hierarchy follows the model’s actual flow.

Figure 24. A practical sequence for mapping the model onto a worker and then into a token factory.

22. The next view is time

The diagrams so far describe structure and flow. The next step is to place time over them.

Interactive decode has a tight end-to-end budget. Execution of one layer commonly needs to fit within roughly 50 microseconds to 1 millisecond, which across a typical 60-layer model corresponds to roughly 15 to 300 generated tokens per second. Each layer contains several sequential phases, so individual arithmetic operations and handoffs live in tens of microseconds, microseconds, and sometimes fractions of a microsecond.

At that scale, tensor arithmetic is only part of the result. Kernel launch, queueing, synchronization, memory setup, routing, reduction, and link latency must all remain small compared with the operation they support. An expert multiply that takes a fraction of a microsecond is of diluted value if dispatch and data movement take several microseconds each.

The timing view also sharpens the network hierarchy. A high-bandwidth path may still be unsuitable if its setup latency is too large. A small local buffer can be more valuable than a large distant memory tier for a sub-microsecond handoff. A scheduler that creates millisecond variation can overwhelm careful microsecond engineering inside the worker.

The factory spans several control timescales:

local arithmetic and link handoffs in fractions of a microsecond to microseconds;
layer execution in roughly 50 microseconds to 1 millisecond;
token generation in milliseconds;
ready buffers and admission control in roughly 0.1 to 10 seconds;
worker reconfiguration by the orchestrator over seconds to minutes;
KV-blob retention and lifecycle in the fast parallel file system over minutes to days;
and customer demand planning over hours and days.

These loops should be designed separately and joined carefully. Fast controls should not chase slow demand noise, and slow controls should not react to every transient queue fluctuation.

Figure 25. From local execution to factory-control timescales.

23. Worked example: projected Kimi K3 performance on Blackwell systems

The preceding sections describe a method for reasoning about model placement. A useful check is to apply it to a specific frontier model and explore the resulting trade-offs together. The following charts show modeled Kimi K3 performance on B200, B300, and GB200 systems using configurations from 16 to 64 GPUs in a rack. These are workload-and-hardware projections, not measured benchmark results. Connected lines identify projected Pareto frontiers: configurations for which interactivity cannot improve without giving up throughput or another selected objective.

22.1 Interactivity and throughput

The first chart compares time per output token for decode, or time to first token for prefill and midfill, against tokens per second per GPU. Normalizing throughput per GPU makes configurations of different sizes directly comparable.

The lower-left group contains decode frontiers for 8k and 32k input contexts. Feasible configurations appear for all three GPU types. GB200 is strong across much of the frontier, while B200 and B300 also offer competitive points. The lowest-latency decode configurations generally combine pipeline and tensor parallelism. At the throughput-oriented end, the highest tokens per second per GPU tend to come from one attention instance per GPU, without tensor or pipeline parallelism, while expert parallelism remains as wide as the system permits.

The upper-middle group is midfill: 127k cached input tokens followed by a 1k-token append before decode. GB200 leads the low-TTFT portion without pipeline parallelism, using TP=4 or TP=2. B300 catches up toward the throughput-oriented end. B200 and B300 generally use PP=4, with expert parallelism bounded within each stage so the expert all-to-all remains inside the node.

The two groups at upper right are 8k- and 32k-token prefills. All three GPU types converge on similar configurations: PP=4, with TP=4 or TP=2 favoring lower TTFT at some cost in throughput. Their close projected performance reflects the shared Blackwell compute architecture and the compute-limited nature of these prefill workloads.

Figure 26. Projected interactivity and throughput frontiers for decode, midfill, and prefill. Source: SemiAnalysis Inference Simulator.

22.2 Energy per query

The second chart uses the same simulated configurations but changes the vertical axis to energy per completed query. Decode is the largest energy workload because the modeled output length is 1,000 tokens, requiring roughly 1,000 passes through the model for every query. Midfill uses the least energy: it appends a relatively short sequence after one pass through an existing long context. Full 8k and 32k prefills require successively more energy.

The model includes Kimi K3’s localized-attention layers, so both retained state and computation reflect the expected savings from the attention structure rather than treating every layer as full attention.

Figure 27. Projected energy per query for the same decode, midfill, and prefill frontiers. Source: SemiAnalysis Inference Simulator.

22.3 Fast-memory requirements

The third chart shows peak HBM residency per GPU. It excludes the extra buffers used while KV state enters or leaves the worker, which may add several gigabytes per GPU. Between conversational or agentic turns, inactive KV blobs can be transferred in parallel to network-attached DDR, including the DRAM attached to node CPUs. That memory is cheaper, more available, and less constrained by the HBM supply chain.

The projected frontiers cluster near the lower end of the feasible memory range. For prefill and midfill, the downward steps come from adding pipeline parallelism and increasing tensor or data parallelism, including data-parallel ring-batched instances. Decode reduces per-GPU residency first through tensor or data parallelism, using pipeline parallelism later.

Most frontier configurations remain below approximately 80 GB per GPU even after allowing for practical KV-transfer buffers. This supports the earlier conclusion that bandwidth can be more valuable than maximizing HBM capacity on every accelerator, provided the broader storage and orchestration system moves idle state promptly.

Figure 28. Projected peak HBM residency per GPU across the simulated frontiers. Source: SemiAnalysis Inference Simulator.

22.4 Network peaks and the scale-up boundary

The fourth chart shows projected peak network traffic per GPU. Prefill and midfill make intensive use of the network for expert parallelism because many routed activations are in flight at once. Pipeline parallelism is therefore particularly useful on B200 and B300 systems: placing one stage inside a node keeps its expert all-to-all traffic on the local NVLink fabric.

GB200 NVL72 extends scale-up bandwidth across the rack. More GPUs can therefore participate in one layer in parallel, with all instances advancing through that layer in synchrony, to improve latency and throughput without crossing a weaker network boundary.

Decode shows lower interval-averaged peaks because a small batch consults relatively few experts at once. Individual transfers still burst at the physical NVLink rate; the lower plotted value reflects a measurement interval longer than the transfers themselves. This distinction between instantaneous link rate and sustained traffic matters when sizing the fabric and interpreting utilization.

Figure 29. Projected peak network traffic per GPU and the benefit of keeping expert traffic inside the scale-up domain. Source: SemiAnalysis Inference Simulator.

The four views tell a consistent story. The best configuration changes with the operating regime and the selected point on the latency-throughput frontier. Prefill and midfill reward organized parallelism and stage-local expert traffic. Decode benefits from parallelism at the low-latency end, but the highest throughput per GPU comes from simpler attention instances combined with wide expert placement. Across the projected frontiers, fast-memory capacity appears less restrictive than network placement, memory bandwidth, and orchestration.

Closing perspective

The useful unit of analysis for inference is not total parameter count alone. It is the active flow of the model joined to the memory, communication, and timing hierarchy of the machine.

MoE provides structure. Layer groups repeat. Experts are numerous but individually manageable. Attention is broad and increasingly shaped by KV state. Operations occur in sequence and can reuse hardware. Pipeline, tensor, and expert parallelism can each follow the dimension where it is most natural. Prefill and midfill can sort routes into efficient expert batches, while decode can preserve short batches and high interactivity.

The hardware can reflect that structure. A logical accelerator peer set provides a bounded scale-up domain. Nodes stack into racks. A rack may be one scale-up worker or several pipeline stages. Shared storage makes KV blobs movable. Datacenter networking connects bounded workers rather than carrying every inner tensor operation. Scheduling keeps predictable and stochastic stages in steady flow.

Holding the problem this way does not remove its scale. It makes that scale easier to divide. The model, worker, rack, storage system, and scheduler become parts of one coherent design that can be measured, illustrated, and improved.

Engrams Embedding Entendre: Codesign for Efficient DRAM/SSD Offloading

2026-09-18T06:34:20+00:00

Engram extends standard token embeddings with learned multi-token lookups. Recurring local patterns retrieve vectors directly, reducing the need to reconstruct them through attention and feed-forward layers.

With Engram model architecture optimization, it allows for lower HBM capacity to be needed for models at the same quality. This does not mean there won’t be an insane demand for HBM but it just means that model architecture will continue to innovate around constraints.

This model architecture design is naturally codesigned for parameter offloading: each token accesses a few embedding rows whose addresses depend on token IDs, not hidden states. The runtime can prefetch those rows from host DRAM while earlier layers compute, keeping the table outside HBM without transferring entire weight matrices. Our Memory model contains our latest estimates of quarter by quarter HBM, DRAM, & NAND supply and demand.

Source: SemiAnalysis

Offloading frees HBM for model weights and KV cache, potentially supporting larger batches or more concurrent sessions. When DRAM becomes the next constraint, NVMe offers another tier. Recommendation systems already cache frequently or recently accessed embedding rows in faster memory while backing colder rows with SSDs. After NVIDIA roadmap had to change due to massively despec’ing Rubin Ultra from 1024GB to now ~200GB of HBM per chip, model architecture optimizations like emgram maybe helpful.

Our DeepSeek-V4.1-Flash configuration uses roughly 189 GiB of memory for Engram. We replace it with a memory-mapped (mmap) file and measure serving performance with offload to SSD.

Later on in our report, we will show our Engram offloading experiments along with the official InferenceX agentic inference serving results on Engram models like DeepSeekv4.1 Flash across all 6 NVIDIA GPU SKUs along with MI355X. Unsurprisingly, the CUDA Moat is still mogging MI355X on the ultra popular DeepSeekV4.1 Flash model.

We also show how even on high capacity HBM SKUs, offloading emgrams to DRAM could result in even better performance for most of the pareto than keeping the emgram in HBM.

Our benchmark has been widely reproduced, validated and/or supported by almost every major buyer of compute from Google Cloud to Microsoft Azure to Oracle, to Meta and many more. Furthermore, it has the support of the ML community including from vLLM, LMCache, SGLang, PyTorch, Huggingface and the support of major labs like OpenAI, MiniMax, ZAI, Qwen, Moonshot Kimi, etc.

Source: InferenceX

Star the InferenceX GitHub repository if you find the open-source benchmark and data useful!. InferenceX is the only inference benchmark in the world to have TPUv7, Jalapeño, Nvidia Rubin NVL72, AMD, and soon, SambaNova and Trainium. Due to how realistic AgentX scenario is to real world agentic inference workloads, AMD has committed to collaborating on MI455X UALoE72 too.

Source: SemiAnalysis

Engram Benefits

DeepSeek did not release the original paper’s two trained Engram models. We replicated its setup on fineweb-edu using the released code and training hyperparameters, at an estimated 6E18 FLOPs per run. We observed the same U-shape scaling:

Source: SemiAnalysis

Engram improved performance over pure MoE baselines. We also reproduced DeepSeek’s results, where earlier-layer representations with Engram resembled those of later layers.

Source: SemiAnalysis

What is Being Memorized?

Like the original Engram paper, it is possible to probe Engram’s gate scores to see what n-grams DeepSeek-V4.1-Flash makes the most use of. Our gate scan finds names, code fragments, relational phrasing, and boilerplate. These examples prioritize interesting-ness over gate strength.

One unexpected result was Wright : Ace Attorney.

Source: SemiAnalysis

These examples suggest learned memory optimizes the training objective, not a judgment of which facts deserve storage. Licenses, bibliography fragments, API scaffolding, and website furniture can provide prediction shortcuts, so the value of additional Engram capacity may depend on what survives data preparation. This does not show that table capacity is “wasted”: the evaluation-corpus scan establishes neither training exposure nor the capacity occupied by each category.

For offloading, strong gates do not identify cache-hot rows. Low gates do not automatically save reads either: computing the gate requires the retrieved key and undoes the performance gain of a fused kernel. Skipping reads would require a separate usefulness predictor before retrieval.

Removing Engram

In the original paper’s inference-time ablation, factual-knowledge benchmarks retained just 29–44% of their original performance, while reading comprehension retained 81–93%. This is due to the training–inference mismatch. The resulting degradation therefore measures this trained model’s dependence on Engram, not the performance difference between models trained with and without it.

Source: DeepSeek

In our ablations, suppressing Engram worsens token likelihood across all evaluated domains, especially encyclopedia text and several code corpora. Surprisingly, GSM8K accuracy stays within measured run-to-run variation and removing Engram has no effect.

Engram is not a detachable dictionary beside an unchanged MoE. Removing it changes downstream features and expert selection.

We tested whether rerouting hurts or compensates by holding tokens fixed in a teacher-forced experiment on CRUXEval, a code-reasoning benchmark of small Python functions where the model predicts a function’s output from its code and an input, and scoring the reference answer.

Source: CRUXEval

Removing Engram raised answer loss from 0.2848 to 0.3093 bits/token. Forcing the ablated model to use the original Engram-on expert choices made it worse still, at 0.3375 bits/token.

Source: SemiAnalysis

Rerouting partially compensates for the missing memory. Memory features and expert selection work together, rather than following a clean “memory stores facts; experts reason” division.

On the same CRUXeval, removing Engram during either phase reduced accuracy and increased generated tokens; removing it throughout produced the largest changes.

Source: SemiAnalysis

Keeping Engram for prefill leads to more correct answers than keeping for decode likely due to semantically richer KV cache transferred to decode workers, allowing it to mitigate some of the performance loss.

Agentic Inference Serving Performance

Engram’s table is large, but each lookup is small. DeepSeek-V4.1-Flash requests 24 rows at each of two Engram layers, about 12.4 KiB per processed token position across the model, or 3.1 KiB per GPU when split across four GPUs.

Source: InferenceX

Currently as of Day 7 since Model Release, MI355X is still 2-4x worse performance per dollar compared to B200 even when normalized by Mi355X’s lower TCO. Our full total cost of ownership breakdown comes from our AI Cloud TCO Model along with monthly market surveys of over 100+ gpu clouds & gpu cloud customers.

Source: InferenceX

On the Day 0 release of DeepSeekv4.1 Flash, NVIDIA vLLM works out of the box with zero issues across all 6 SKUs: H100, H200, B200, B300, GB200, GB300! This was thanks to the amazing work by the NVIDIA & Interact teams! In comparison, AMD vLLM did not work on day 0 for DeepSeekv4.1 Flash.Source: SemiAnalysis InferenceX

Source: InferenceX

AMD’s vLLM documentation points to using vllm/vllm-openai-rocm:deepseekv41-flash-0909, but from hour 0 of the model release to hour 23, AMD has not publicly released the image. AMD claims, “SPEED IS THE MOAT,” yet it has still not released it by the 23rd hour. We wish that, going forward, the AMD team has a better process for hour 0 model releases.

Source: vLLM AMD

Source: vLLM, AMD, DockerHub

Eventually when they did publicly release for “day 0” image support, performance-wise, it is currently up to 14.8x worse perf per dollar than H200 and up to 42x worse perf per dollar than B200/B300.

The power of the CUDA MOAT is NVIDIA’s collaboration with its massive 6 million-developer community ecosystem including most of the vLLM & SGLang & Tokenspeed maintainers which means that CUDA is optimized on day 0.

Source: InferenceX

Overall, AMD did make significant improvements but the performance per dollar is still currently 2-4x worse than B200.

Source: InferenceX

AgentX Engram DRAM Offloading Improving Performance

The HBM and DRAM offload use the same GPU kernel to select and dequantize rows. With HBM, it reads GPU memory; with Unified Virtual Addressing (UVA), it reads pinned host memory directly.

Both support full decode graphs. Moving the table into HBM accelerates only the sparse lookup, leaving decoder computation and communication unchanged, resulting in little overall benefit while consuming memory otherwise available to KV cache.

Another benefit of Engram offloaded to DRAM which means you can reduce the communication overhead by using less HBM GPUs per replica. For example, when enabling Engram offloading on B300, we are able to switch from TP4 to now TP2 which improves the pareto curve by up to 1.6x.

Source: SemiAnalysis InferenceX

When iso-model quality, less HBM is required as the engrams could be offloaded to host DRAM. Thus HBM bandwidth matters way more than HBM capacity. For inference workloads where memory bandwidth matters the most, 4-hi HBM provides the best $/bandwidth and therefore lowest cost per token. If China continues to make more and more revolutionary model architecture innovations, soon it could potentially 0Hi HBM stacks.

Moreover, on B300 and week-0 stack, moving the Engram table back to HBM did not improve results and stay within run-to-run variance. This is a result of the work optimizing DRAM offload, such as async, and overlap.

SSD Offloading

On B200, we created a unoptimized vLLM fork and stored the Engram tables in memory-mapped files on local SSDs. File backing lets the OS reclaim table pages when other applications need RAM. Pages already cached in memory can be served without reading the SSD again. Furthermore, note that we were unable to turn on GDS

The unoptimized SSD implementation changes how rows reach the GPU. It copies row IDs to the CPU, deduplicates them, gathers the requested rows into pinned buffers, copies those rows back to the GPU and dequantizes them. This work runs between segments of the GPU execution graph. Native UVA performs row selection and dequantization directly on the GPU, avoiding the CPU round trip.

A warm filesystem cache removes physical SSD reads, but leaves the coordination, row gathering and transfers. This is why a file already cached in RAM can still perform worse than a pinned DRAM table. The comparison measures the whole serving path; it does not separate the time spent on each of these operations.

B200 DRAM dominates both measured SSD serving curves in total tokens per dollar and P90 interactivity. Near 125 tokens/s/user, DRAM delivers 121 million total tokens per dollar versus 52 million for SSD.

Source: SemiAnalysis InferenceX

For production serving, SSD offloading is likely not worth the tradeoff. On the B200 configurations we measured, SSD offloading loses on both measures: every observed SSD point has a DRAM alternative that delivers higher P90 interactivity and more total tokens per dollar. The only points where

Cheaper storage does not automatically produce a cheaper inference service. Moving Engram to SSD leaves the same four expensive GPUs and the rest of the server in place. Reclaiming RAM only creates an economic benefit if it enables a cheaper server configuration or additional useful capacity. The current unoptimized path provides neither benefit, and the filesystem cache still consumes RAM when table pages are resident.

Mechanism

Next we will look at the mechanisms and specific implementation of ngrams from DeepSeekv4.1 Flash, LongCat, Qwen3.8 Flash Next.

Everyone Says Datacenter Moratoriums Are Killing the US Buildout. We disagree

2026-09-15T12:54:41+00:00

The debate on US datacenters has never been so politically charged. Four states have acted in under two months. New York has stopped issuing environmental permits for datacenters, Texas has paused the next step in its massive ERCOT interconnection queue, Pennsylvania has pulled datacenters out of fast-track permitting and made state permits conditional on new guardrails, and Oregon has frozen datacenter deals on state-owned land.

Beyond the state level, more than 300 towns, cities and counties have voted to halt datacenters over the past year and a half. If you read the news, the US datacenter buildout looks like it is being legislated to a halt, one government board at a time. As AI drives an unprecedented wave of datacenter construction over the last year, concerns over electricity costs, water consumption, land use, and community impacts are increasingly influencing policy.

Source: SemiAnalysis Datacenter Industry Model, SemiAnalysis Moratoriums Tab

But once again, the narrative is driven by widely spread misinformation. In January, we debunked the “Datacenters are taking all of our water” myth (TLDR: they don’t, they’re actually remarkably water-efficient). Then in June, in Stop Saying Half of 2026 US Datacenter Capacity is Cancelled, we explained that the widespread narrative of datacenters not showing up was grossly off, driven by uniformed claude-coded forecasts.

Today, we dive deep into the widespread narrative of moratoriums killing the US datacenter buildout. Once again, it’s highly inaccurate. Our Datacenter Industry Model forecasts +38GW of Datacenter IT capacity delivered in the US in 2027, more than double that of 2026. Of that total, 22GW is under vertical construction, while the remaining 16GW “planned” is, in majority, already financed, doing siteworks and preparing for vertical construction.

Our methodology tracks real-time and historical satellite imagery of 6,000+ datacenters. Contact sales@semianalysis.com to get access to the full granularity. Our forecast has barely moved over the last 12 months, despite bi-weekly updates delivered to clients, as construction activity keeps accelerating across the country.

Source: SemiAnalysis Datacenter Industry Model

Today, we estimate that roughly 2.3 GW of planned capacity is genuinely delayed because of local moratoriums and New York’s executive order, the two categories of policy intervention for which we can establish a direct project-level impact.

At the local level (i.e., cities, townships, counties), of the roughly 20 GW sitting inside a restricted boundary, only 1,525 MW is delayed. Three projects account for the delayed MW: an AWS campus in Ohio, a powered land developer campus in Pennsylvania, and a site in Colorado.
New York’s moratorium touches roughly 1.4 GW, but we estimate only about 0.8 GW faces a meaningful delaydirectly attributable to the order.

Texas adds perhaps three-four months of administrative delay for base-load projects in the ERCOT interconnection queue, but is offset by an acceleration of behind-the-meter demand. For projects that never needed the grid, it means nothing at all, making it a net positive for BtM developers and on-site generation suppliers. In June (US Grid Constraints: Towards 40GW+ of Behind-The-Meter Datacenter by 2028?), we showed BtM powering more than half of new US datacenter capacity from 2028. Last week, (What is So Hard About Behind-The-Meter Power For Datacenters? Part 1) we counted 75 GW of firm BtM equipment orders, more of them bound for Texas than any other state. If anything, the audit reinforces our view that the buildout is moving BtM.

Most analysis of datacenter moratoriums focuses on counting the number of restrictions, because that is relatively straightforward. But the number of moratoriums is a poor proxy for the amount of MW capacity they actually affect. There’s plenty of loopholes/gaps with the moratoriums. For example, a recent Brownsville moratorium was brewing while Elon was planning a datacenter within city limits. But just days before approval, the city and Elon sneakily “deannexed” the 444 acres in Brownsville, bringing the project out of Brownsville’s moratorium jursidiction.

A moratorium only matters to our forecast if it applies to a specific project, covers the project’s parcel, and blocks an approval that the developer still needs. Therefore, determining the actual MW impact requires a parcel-level, project-by-project analysis.

In addition, capacity exposed to a moratorium is not capacity delayed by one, a distinction Datacenter Industry Model subscribers will recognize from our early deep dive on the issue. Most restrictions land on projects still years from construction, where schedules were never firm to begin with, and even where a moratorium is the binding constraint the developer can usually relocate, redesign, or challenge the project rather than abandon it. Capacity already under construction, or fully permitted and about to start, is out of reach altogether, since these instruments freeze new applications rather than revoke approvals already granted. That covers all of 2026 and most of 2027.

To be 100% clear, we think moratoriums COULD delay the buildout, but they just aren’t yet at scale. We would need to see them be materially more widespread and restrictive. Our real-time project-level analysis will instantly flag any change in trend to our Datacenter Model subscribers. As AI Labs are increasingly profitable, it becomes easier for datacenters to pay more than their “fair share” and be better neighbors and support local communities, which offsets the rising pusback.

Source: SemiAnalysis Datacenter Industry Model, SemiAnalysis Moratoriums Tab

To calculate total dela, we tracked every local and state-level datacenter moratorium in our new Datacenter Moratoriums database, available to subscribers of our Datacenter Industry Model. It covers more than 400 local instruments, actions, and proposals across 17 states, tracking each status from proposal through enactment, expiration, lifting, replacement, or rejection. We then mapped every active restriction to our project-level pipeline at the individual project and MW-tranche level. That pipeline, which forms the foundation of our Datacenter Industry Model, covers 6,000+ facilities, each tracked individually through property records, permits, power-usage data, FOIA requests, satellite imagery, and other primary sources.

To quantify actual delayed capacity, we augmented our research analyst team (we’re hiring!) with an army of state-of-the-art research agents across our moratorium database and underlying project records, systematically testing each active restriction against the relevant projects, and validating with real-time satellite imagery. For every instrument, we matched its geographic boundary to the exact project parcel, verified that the campus falls within the relevant jurisdiction and checked whether the developer still requires an approval that has not yet been secured. We also reviewed the broader project record for other factors that could already be gating the schedule. This allowed us to systematically work through hundreds of restrictions and thousands of projects, moving beyond simply counting moratoriums to identify the specific projects and MW actually exposed, producing a materially different view of the capacity genuinely at risk.

Source: SemiAnalysis Datacenter Industry Model, SemiAnalysis Moratoriums Tab

Four main factors, explored in more detail later in this note, essentially explain why hundreds of restrictions translate into relatively little actual delay:

(1) county moratoriums often stop at city lines, leaving projects within incorporated municipalities unaffected;
(2) moratoriums typically freeze new applications rather than revoke existing approvals, allowing entitled projects to proceed;
(3) shorter moratoriums may expire before 2028+ projects need the affected approval; and
(4) other issues, such as litigation or equipment lead times, may already be gating the schedule.

With the November 2026 elections approaching, moratoriums are an easy way for elected officials to demonstrate that they are responding to community concerns without permanently shutting down development. Both New York Governor Hochul and Texas Governor Abbott bypassed their legislatures to impose pauses they can ultimately control and revoke. More durable statewide restrictions remain much harder to enact: 13 bills died in 2026, with almost none reaching a floor vote, underscoring the political difficulty of turning temporary local action into lasting statewide policy.

Subscribe now

In this article, we start by going through our methodology and why counting moratoriums is not counting megawatts. We then look at moratoriums adopted at the local level, showing how 20 GW of nominal exposure translates into just 1,525 MW of actual delays, which projects are affected, and why most restrictions never bind. Next, we cover the state-specific moratoriums and other policy tools adopted in response to anti-datacenter backlash in New York, Texas, Pennsylvania, and Oregon, and what they mean for the broader US buildout. We then assess the likelihood of additional state-level moratoriums. Finally, we look at who benefits from the restriction wave and how it could reshape where capacity gets built, which projects move forward, and how that capacity is powered.

Counting Moratoriums Is Not Counting Megawatts

What a moratorium does

A datacenter moratorium is a temporary legal pause, from a few months to a year, typically enacted by a local or state government that halts the approval, permitting, and construction of new datacenters. A moratorium is almost always a freeze on accepting new applications rather than a revocation of approvals already granted. Its purpose is to give governments time to catch up with a development cycle that has moved faster than local regulation. Many zoning codes were written before municipalities contemplated 100+ MW facilities with substantial power, water, noise, and land-use requirements. A moratorium provides time to establish new rules and assess infrastructure requirements.

Most of the time, a moratorium also provides elected officials with a relatively low-cost way to respond to local opposition. A temporary pause signals that government is responding to residents without permanently rejecting investment or tax revenue, and it reduces the legal exposure associated with an outright denial.

Exposed MW vs. delayed MW

Moratorium-exposed MW is not necessarily the same as moratorium-delayed MW. A project can fall within a moratorium’s jurisdiction without the moratorium necessarily shifting its expected construction date.

The magnitude of the moratorium impact depends heavily on where the project is in the development cycle. A project that has just acquired land has substantially more flexibility than one that is already in the middle of permitting. Likewise, a six-month moratorium does not necessarily translate into a six-month delay in actual capacity deployment, particularly when the project has sufficient schedule buffer or other constraints are already determining the timeline. This is consistent with our previous note Stop Saying Half of 2026 US Datacenter Capacity Is Canceled, where we showed that delays at early development stages rarely translate into meaningful changes in near-term capacity deployment.

Why Moratoriums Rarely Move the Needle

For a moratorium to actually move a project’s delivery date, every one of the following must be true at the same time:

The moratorium reaches the parcel. County moratoriums generally apply only to unincorporated territory. A campus inside an incorporated city within that county is outside the ordinance entirely. AWS Atlanta, STACK Lithia Springs, T5 Fairburn, or Microsoft Tyrone all sit on land the relevant county moratorium cannot touch.
The moratorium is enacted and still in force. The restriction cannot simply be proposed, lapsed, lifted, or replaced, and it must remain within any state-imposed duration limits.
The moratorium overlaps with the project’s timeline.
- The project is not too far ahead already, and still needs the affected permit/approval. The project must not have already obtained the affected approval before the moratorium takes effect. Many projects had their approvals in hand even though they had not started construction.
- The project is not too far out. The moratorium must still be in effect when the project needs to obtain the affected approval. If delivery is 2028 or later and the moratorium lasts only three to six months, it may expire years before the project reaches that stage. The developer can simply wait out the moratorium without changing the delivery date.
The moratorium covers the use. Most instruments apply to hyperscale facilities above a specified MW or square-footage threshold, or only to crypto mining.
The project still needs the affected approval. A project may fall within the geographic and scope of a moratorium but not actually depend on the approval it freezes. For example, a project on federally controlled land, or one backed by federal approvals or other pre-existing rights, may have a separate permitting pathway that allows it to proceed without the approval covered by the moratorium.
The developer cannot redesign around the restriction. A project may be able to change its configuration, size, use or power arrangement to avoid the restriction. For example, if a moratorium applies to energization at the meter, a developer may be able to use BtM generation and continue advancing the project without waiting for grid energization.
The project is still active. The developer must still intend to build the affected MW tranche. A cancelled, withdrawn, downsized or otherwise abandoned project should not be counted as moratorium-delayed simply because it sits inside the restricted area.
No existing rights or alternative approval pathway bypasses it. Development agreements, vested rights, grandfathering, existing entitlements, alternative zoning classifications or other approval routes can allow a project to proceed despite the new restriction.
No carve-out applies. There cannot be a public-utility carve-out, consent judgment, state preemption, or exclusionary-zoning doctrine requiring the use to be accommodated elsewhere.

If you miss just one of these, the moratorium is irrelevant. The cumulative effect of these filters is substantial: 300+ local instruments translate into just ~1,525 MW of actual project delay in the US.

Exposure, therefore, must be measured project-by-project, at the MW-tranche level, never inferred from how many jurisdictions in a state have acted.

Case study: NorthPoint’s Campus near Scranton, Pennsylvania

NorthPoint’s campus near Scranton, Pennsylvania, illustrates one of the rare cases where a moratorium actually binds and moves the project timeline.

Reaches the parcel. The site sits inside the township, rather than in the adjacent City of Hazleton or West Hazleton Borough.
Enacted and in force. The township adopted the resolution unanimously in June 2026.
Overlaps with the project timeline. NorthPoint had not yet obtained or filed for the required special exception when the moratorium took effect, while the first buildings were scheduled to start in Q4 2026.
Covers the use. The resolution specifically covers datacenters without a MW or square-footage threshold.
Still needs the affected approval. The project is on private land and requires a local special exception and land-development approval. There is no federal permitting pathway or existing entitlement that bypasses those approvals.
Cannot redesign around it. The restriction applies to the datacenter use itself. Downsizing or moving to BTM generation would not avoid the blocked approval.
Still active. NorthPoint continues to pursue the project, including its proposed $165 million community benefits package.
No alternative pathway or carve-out. There is no development agreement, grandfathering or other existing right that allows construction to proceed. Nor does the project qualify for a public-utility or other relevant exemption.
On the critical path. The moratorium delays the local entitlement process into 2027.

In this case, the moratorium is the binding constraint.

Local Moratoriums: Where Is the Risk?

Source: SemiAnalysis Datacenter Industry Model, SemiAnalysis Moratoriums Tab

Proliferation across the US in 2026

So far, most restrictions have come at the local level. As of September 2026, more than 300 local datacenter moratoriums and bans have been adopted nationwide across counties, municipalities, townships, and other local governments. Our database tracks 400+ local instruments in total, following each through its full lifecycle: proposed, enacted, expired, lifted, replaced, or rejected. We also include permanent bans / restrictions written directly into zoning codes, since these can constrain datacenter development just as a moratorium does, but on a permanent basis. The 300+ figure represents the enacted moratorium (and few permanent bans) subset, while the broader database captures the full universe of local restrictions and their status.

We see that local moratoriums are concentrated in the Midwest and Southern regions, clustered heavily around areas experiencing rapid infrastructure growth. The top four states (Michigan, Ohio, North Carolina, Georgia) account for roughly half of all local instruments nationally, which supports the thesis that moratoriums and local resistance often follow rapid datacenter growth rather than anticipate it.

Source: SemiAnalysis Datacenter Industry Model, SemiAnalysis Moratoriums Tab

Michigan is currently the leading state for local datacenter moratoriums, with 45 enacted and 7 proposed, with the majority adopted in late 2025 and 2026 following a rapid influx of datacenter proposals. Much of this can be traced to the datacenter tax exemption signed by Governor Gretchen Whitmer in December 2024, which accelerated development interest across the state. One notable example is Howell Township, which adopted a six-month moratorium in November 2025, to allow time to establish datacenter-specific zoning regulations. In May 2026, its board voted to extend the moratorium for an additional six months.

Ohio ranks second with 40 enacted moratoriums, with 83% adopted in 2026 alone. Unlike Michigan, the trigger was not a new incentive, but the strain of years of rapid datacenter growth. Much of this can be traced to the electricity-bill shock of mid-2025, when datacenter-driven PJM capacity costs pushed the average Ohio residential bill up 25.7% year over year, and to hyperscale proposals spreading into townships with little or no zoning. In Sunbury, the city council enacted a moratorium on April 15, 2026, effective through January 2027, for the city to review zoning, building, and business regulations.

Source: SemiAnalysis Datacenter Industry Model, SemiAnalysis Moratoriums Tab

Moratorium indicators

A leading indicator of new local datacenter moratoriums is mounting, organized community opposition, often visible well before a formal pause or ban is adopted. This can take the form of packed local hearings, petitions, “Citizens for…” groups, social media campaigns, coordinated neighborhood opposition, and election turnover. According to Gallup, datacenter opponents most frequently cite excessive energy use and higher utility bills, alongside concerns over water consumption, land use, pollution, and traffic. These concerns can quickly translate from grassroots opposition into pressure on planning commissions, city councils, county boards, and state legislators to slow or restrict development.

We polled the American electorate in August as part of our broader work on datacenter sentiment, energy prices, and frontier labs. The full survey is part of our Tokenomics model, which focuses on the economics, adoption, and demand dynamics of AI and datacenter infrastructure. We found that Americans were net-positive on AI broadly, but significantly more negative on datacenters specifically: 46% of voters view datacenters unfavorably, including 24% who view them very unfavorably, versus just 29% with a favorable view. Similarly, 46% would oppose a new datacenter in their own city or town, including 30% who would strongly oppose it. By comparison, 46% have a favorable opinion of AI and 34% an unfavorable opinion.

We note that this gap could narrow as increasingly prominent doomsday scenarios and concerns around AI’s broader impacts shape public opinion. We will continue monitoring this shift through the 2026 midterms, expanding our political and regulatory analysis to track the growing backlash against datacenters, including new local and state restrictions, regulatory actions, political campaigns, and changes in public sentiment.

Source: SemiAnalysis Tokenomics Model, Politics Tab

Regional clustering adds further momentum, with opposition and moratoriums sometimes spreading among neighboring jurisdictions as local officials respond to developments and activism in nearby communities. In Metro Atlanta, for example, Fayetteville-area activists cited pauses in neighboring counties while pushing for similar restrictions locally. Once opposition becomes organized in one jurisdiction, the arguments, advocacy groups, proposed ordinance language, and political pressure can travel quickly across a region, making local moratorium activity a potentially self-reinforcing process.

Election-year politics can further intensify local concerns around rapid datacenter growth, as candidates position themselves as defenders of residents against higher electricity costs, water consumption, and unwanted development. The Washington Post has documented datacenter opposition becoming an issue in state and local campaigns. The upcoming November elections have created another catalyst for new restrictions, particularly where datacenter projects have become politically salient.

In Ohio, Republican gubernatorial nominee Vivek Ramaswamy pledged in August that no new datacenter would be approved unless it eliminated electricity bills for nearby residents, paid full property taxes to fund homeowner relief, and met strict water and farmland protections. Such campaign positions can raise the political cost of approving projects and increase the likelihood that datacenter permitting becomes an election issue rather than a purely local land-use decision.

Source: @VivekGRamaswamy’s X Post from Aug 6, 2026

More local moratoriums do not necessarily mean more MW at risk

Local restriction count and MW exposure are two different dimensions. A moratorium only creates meaningful risk when it prevents a specific project from obtaining an approval or a permit it still requires. A state can therefore have a high number of moratoriums while exposing little or no active datacenter pipeline.

Michigan is a good example. Although it has the highest number of restrictions, it carries effectively zero pipeline exposure, something subscribers to our Datacenter Industry Model could see directly in the Michigan project-level data, even as the state’s growing ordinance count was receiving national coverage. The reason the largest projects continue to move forward is that moratoriums often lose to sequencing and pre-existing zoning. The Cannoli/Google campus, for example, sits on industrial land where data processing was already a permitted use, eliminating the need for rezoning, and its site plan predated any potential pause. The Related Digital project in Saline Township similarly advanced after the developer sued the community over exclusionary zoning following a local board rejection, ultimately resulting in a legal settlement. In Lyon Township, the 180-day pause cannot affect the already-approved Project Flex.

What part of the pipeline is at risk?

Mapping every one of the roughly 300 local datacenter moratoriums and permanent bans currently active across the US against our pipeline, we see that the direct impact is minimal. Only 1,525 MW is directly delayed, which is 7.6% of the capacity sitting inside a restricted boundary. Three projects account for essentially all of them. Let’s dig in. Below behind paywall, we discuss in much greater depth the projects at risk, why they are, and the different moratorium instruments and how they are or aren’t impacting timelines.

Vera Rubin NVL72 Agentic Inference: 67x better Performance per Dollar

2026-09-14T14:08:56+00:00

Rubin is the first platform co-designed across six products for the agentic era: Rubin GPU, Vera CPU, NVLink 6 Switch, ConnectX-9, BlueField-4, and Spectrum-6. Today we are publishing the first verified agentic inference results for Rubin, measured on our agentic inference benchmark, AgentX. Even on early pre-release software, the results already show why extreme co-design was necessary.

At GTC 2026, Jensen presented this graph that VR NVL72 achieved 3x performance per MW compared to Blackwell on O(1-3 Trillion) parameter model around 200 TPS. But when compared to the real world performance of Rubin already on prelease software, we are already seeing up to 7x better token throughput per megawatt. Jensen needs to stop sandbagging his performance claims at GTC. The last time he did this at GTC 2024, when he claimed GB200 NVL72 would deliver 30x Hoppers performance, but when we tested, it was 98x better performance than Hopper.

Our estimates on the performance results show that even on early software builds, Rubin can earn over 2x more profit per gigawatt than the Blackwell platform. As the Rubin software stack and kernel libraries mature, and as the developer community builds experience optimizing for Rubin, we expect that gap to widen further.

Source: NVIDIA GTC 2026 SandBagging Rubin Performance

We evaluate the performance using the industry standard agentic inference benchmark scenario called AgentX. This replays real world agentic traffic across our fleet of thousands of chips. The results can thus be holistically referenced by actual inference providers and hyperscale AI labs to decide which chips are most efficient in what scenarios.

Source: SemiAnalysis InferenceX

Thank you to Jensen Huang, Ian Buck, Nick Comly, Kedar Potdar, Rohit Nagraj, and the Mainland China TensorRTLLM Team for helping on doing next gen Rubin software bring up and helping verify our agentic benchmark results.

Star the InferenceX GitHub repository if you find the open-source benchmark and data useful!. InferenceX is the only inference benchmark in the world to have TPUv7, NVIDIA, & AMD and soon SambaNova & Trainium. Due to how realistic AgentX scenario is to real world agentic inference workloads, AMD has committed to collaborating on MI455X UALoE72 too.

Source: GitHub

Agentic Workload Primer

At a high level, an agentic workload is characterized by four elements:

Multi-turn: a session includes tens or hundreds of interactions between the user and assistant, compared with a handful in a chatbot scenario. These workloads combine long contexts and high prefill reuse with sub-agent bursts and numerous tool calls.
Long context: system prompts, tool definitions, and the large number of turns make context accumulate quickly.
High prefix reuse: since the conversation progresses linearly, where output from turn n-1 is concatenated to turn n (typically), most context can be served from KV cache rather than recomputed (this depends on the amount of storage available to store KV tensors). As n grows, the ratio of cached input relative to uncached typically tends towards 1.
Sub-agent bursts: a session launches multiple short-lived sub-agents with fresh context, which create bursty KV-cache patterns.

Source: SemiAnalysis, DeepSeek

Read more about the methodology in our AgentX article.

Rubin Amazing Performance per TCO

Performance per total cost of ownership (TCO) is one of the most important angles in which to evaluate AI accelerator performance. In the charts below, the Y-axis is total tokens per $1 TCO. Practically speaking, this tells an inference provider how many tokens they can generate for every dollar they spend on compute. This is calculated by normalizing the total throughput for a scenario by the all-in serving costs ($/chip/hr times number of chips utilized to serve).

On InferenceX, we provide a few different TCO figures for each SKU which come from the SemiAnalysis AI Cloud TCO Model. The scenarios offered by default are as follows:

Owning at Large Hyperscaler Volume: the modeled cost per GPU-hour to own and operate the hardware, including server and networking capex spread over the assumed useful life, colocation, power, and cost of capital. This assumes hyperscaler purchasing and financing terms, including volume discounts and custom server builds.
Rent - 3 Year Commit: the market price per GPU-hour paid to a cloud provider for a three-year committed reservation, based on SemiAnalysis rental pricing surveys.

Users may also edit the compute costs in our calculator to match the cost they are paying. For more figures such as on-demand market rental rate and shorter term rental commitments can be found in the AI Cloud TCO model which comes from our monthly market survey of 100+ GPU customers, GPU neoclouds, and hyperscalers.

Vera Rubin is a substantial improvement over GB300 in terms of performance per dollar. At 170 TPS, On Apples to Apples TRTLLM NVFP4 Dense, Vera Rubin NVL72 delivers ~67x the total throughput per TCO of GB300 Dynamo TRTLLM under the owning cost assumptions. Furthermore, at the part of the frontier where most providers would actually serve this model (60-100 TPS), Vera Rubin achieves between 1.4x and 3x the throughput per TCO compared to the latest and greatest GB300 TRTLLM configuration.

Our rack uses the production SKU of 2300W TDP & 1.5TB of CPU LPDDR5X per compute tray. We chat more about why NVIDIA had to cut their Vera memory in half in our accelerator and memory model. For this article, we will be talking about the TRTLLM Rubin performance but we expect later articles to also include vLLM Rubin & SGLang Rubin performance on agentic inference workloads.

Source: SemiAnalysis InferenceX

Vera Rubin achieves approximately 61% higher maximum P90 interactivity than GB300 Dynamo TRTLLM, reaching 276.24 versus 171.53 P90 TPS However, when using the open-source SGLang stack, GB300 can achieve similar interactivity as Vera Rubin.

Source: SemiAnalysis InferenceX

We recognize that this is an early pre-release TRTLLM software release and that performance will only get better, especially at the “ends” of the frontier (ultra-high throughput/ultra-low latency). Driving the frontier forward and highlighting improvements over time is the ultimate goal of InferenceX. Read more about this in our previous articles on InferenceX:

Also note that Vera Rubin achieves significantly better P90 E2E latency in higher throughput scenarios. While interactivity (TPS) is often the headline for performance benchmarks, latency metrics like TTFT (time-to-first-token) and E2E latency are also important and one of the de-factor SLA standards in production serving.

Source: SemiAnalysis InferenceX

When we consider the 3 year rental cost which in July 2026 was over $8.5/hr/chip for Rubin and $5/hr/chip for Blackwell Ultra NVL72, the upgrade is still more than justified. At 80 TPS P90 interactivity, Vera Rubin can produce 62% more total tokens for the same rental TCO. Across the higher interactivity portions of the frontier, Vera Rubin realizes up to 16x more tokens per 3 year rental TCO.

Source: SemiAnalysis InferenceX

The lesson here is quite simple: if you have the money to buy or rent a VR NVL72, you should – it will generate significantly cheaper tokens than the next leading accelerator and thus make you significantly more money. As Jensen infamously said “the more you buy, the more you earn”.

When compared to single-node Blackwell and MI355X, the performance per TCO gap becomes even wider across the entire curve. At an 80 TPS P90 SLA (which again, is quite realistic for this model), an inference provider can serve 10x more tokens per USD than with B300 when using the owning TCO figure.

Source: SemiAnalysis InferenceX

The advantage also extends to P90 end-to-end latency. As shown below, at 60 million tokens per dollar of TCO under the owning assumptions, Vera Rubin achieves a P90 end-to-end latency of approximately 20 seconds, compared with 60 seconds for B200/B300 running the latest vLLM serving stack. At 160 million tokens per dollar, the P90 end-to-end latency gap widens to approximately sixfold: 20 seconds for Vera Rubin versus 120 seconds.

Source: SemiAnalysis InferenceX

On agentic workloads, Vera Rubin makes H200 look about as competitive as a TI-84 calculator. At a P90 interactivity target of 80 TPS, Rubin delivers 18x as many tokens per dollar. At 120 P90 TPS, that advantage widens to 39x. The advantages of Rubin are less at offline batch inference and more for training as this is where we see what big labs are slowly migrating hopper towards.

Source: SemiAnalysis InferenceX

Source: Texas Instruments

Rubin Performance per Watt - Jensen Sandbagging Performance Again

Availability of powered datacenters is often a limiting factor to deploying more chips so when power efficiency is ultra important. If you are able to generate more tokens per GigaWatt, then you are able to generate more revenue and more profit per GigaWatt.

SemiAnalysis Datacenter Industry Model

At GTC 2026, Jensen presented this graph comparing Rubin to Blackwell on a large 2T MoE model, showing that VR NVL72 achieved 3x performance per MW on a trillion parameter model around 200 TPS. But when compared to the real world performance of Rubin already on prelease software, we are already seeing up to 7x better token throughput per megawatt. Jensen needs to stop sandbagging his performance claims at GTC. He did last time at GTC 2024 at the announcement of GB200 NVL72 when he claimed it will be 30x faster than Hopper but when we tested, it was 98x better performance than Hopper.

Source: NVIDIA

P90 interactivity describes the streaming speed of an individual response, using the reciprocal of P90 full-response inter-token latency. At a fixed interactivity target, a higher curve means the deployment can serve more total traffic within the same power budget. The initial wait for the first token and overall response time remain separate considerations in the preceding latency analysis.

Source: SemiAnalysis InferenceX

For the DeepSeek V4 Pro agentic workload, Vera Rubin delivers substantially more total token throughput per megawatt than GB300 and MI355X at the matched targets compared below. The size of that advantage depends on the interactivity target and the serving engine used for comparison.

At 100 TPS, Rubin delivers approximately 59.4 million total tok/s/MW, compared with 28.5 million for GB300 Dynamo SGLang and 21.1 million for GB300 Dynamo TRTLLM. That is a 2.09x advantage over the stronger GB300 engine at this target. MI355X SGLang reaches 2.01 million tok/s/MW, giving Rubin a 29.5x lead on this particular workload and snapshot. SGLang is the strongest of the measured MI355X engines at this target.

The wider hardware comparison at 100 TPS puts B200 SGLang at 6.95 million, B300 vLLM at 5.56 million, and H200 Dynamo SGLang at 2.26 million tok/s/MW. H200 uses FP8. The other configurations in this comparison use FP4. These are comparisons of the measured hardware-and-software configurations, including their caching and parallelism choices.

The table shows how the advantage changes across the curve, with throughput in million total tok/s per utility MW. The values are interpolated, meaning they are estimated between measured benchmark points to compare engines at the same speed target. N/A means the target falls outside that engine’s measured range.

Source: SemiAnalysis InferenceX Preview

At 150 TPS, Rubin retains nearly 37 million tok/s/MW, approximately 7.2x GB300 SGLang. The advantage then narrows as the speed requirement rises further, reaching 2.72x at 200. The strongest GB300 engine also changes with the target. TRTLLM leads at 75, while SGLang leads at the higher targets shown here.

This distinction is especially relevant near the operating point discussed earlier. At exactly 170 TPS, Rubin delivers 62.9x the total throughput per MW of GB300 TRTLLM, whose curve is near its fastest measured endpoint. Against GB300 SGLang at the same target, the multiplier is 5.56x. The engine label is therefore essential when quoting the high-interactivity gain.

The other feature of Rubin is the first class integration of dynamic power shifting called “DSX MaxLPS” which means instead of provisioning for max all in TDP plus 10-20% oversubscription factor, gpu cluster operators will power profile their inference workloads and proxies for future workloads and based on the actual power, they will be able to fit more GPUs into the same datacenter power footprint. This is as during inference workloads especially at medium to fast speed, GPUs do not consume their power more envelope thus when smartly steering power across the datacenter, you will be able to fit more GPUs. Our upcoming integration PowerX into InferenceX will allow us to have even finer grain measurements of throughput per GigaWatt.

Annual Revenue and Profit per Gigawatt - The More you Buy, The More you Earn

At the time of writing, DeepSeek V4 Pro 1.6T was replaced in favor of DeepSeek V4.1 Flash. However, its size allows it to be a proxy for O(1-3T) parameter LLMs. As DeepSeek V4 Pro was released under the MIT license, there is no license fee.

At a fixed power budget, Rubin’s advantage is not simply that it can process more tokens. It gives an operator more capacity to monetize demand without securing additional utility power. At 75 TPS interactivity, 60% utilization and no model-license fee, Vera Rubin generates $159.5 billion in annual revenue and $149.9 billion in modeled profit per all-in utility GW. The strongest GB300 configuration in this annual revenue comparison, Dynamo SGLang, generates $114.9 billion and $105.3 billion respectively. Rubin therefore delivers approximately 39% more revenue and 42% more modeled profit from the same power allocation.

Source: SemiAnalysis InferenceX

The absolute difference is approximately $44.6 billion in additional annual modeled profit per GW, or roughly $446 million at a 10 MW scale under linear scaling. Because the displayed annual cost per GW is similar for these two configurations, most of the incremental revenue flows through to the model’s profit measure.

The same advantage also creates pricing headroom. Holding workload mix, throughput and billable utilization constant, Rubin could charge approximately 28% less across cached-input, uncached-input and output tokens while matching the revenue per GW of GB300 Dynamo SGLang at the displayed prices. An operator could therefore retain the efficiency gain as additional profit or use it to compete on price, depending on the demand elasticity of the models.

Below we will talk about the total revenue over the entire lifecycle of the fleet for each NVIDIA chip.

A Brain Too Big to Carry — On-Device vs Datacenter Inference

2026-09-14T08:37:25+00:00

Where should the brain of the robot go?

So far, AI has mostly lived behind a screen. Chatbots answered questions. Then agents started driving software and finishing multi-step tasks on their own. The next step is AI that acts in the physical world, and the biggest piece of that is robots. It’s early. Nobody has settled the hardware, the models, or the economics.

The Embodiment Problem

With LLMs, the hardware bends to the model. Pour in as much data and compute as possible at training, then figure out how to serve the model that comes out. The model comes first; the hardware second. Robotics inverts this due to two constraints.

The first is time. A robot runs real-time control loops and can’t miss a deadline. An LLM can be slow without affecting the final output. If a robot is slow, the world changes around it, and then the action is obsolete by the time it needs to enact it.

The second is cost. When you use an LLM, it lives behind a screen provided by the user. In robotics, the manufacturer has to build both the compute and the robot itself and pay for it upfront, on every unit, and the capital expenditure can be immense. At scale, this upfront cost can be billions, even in the trillions if we do eventually get to a billion robots.

Because of these two constraints, in robotics the on-robot hardware is fixed, and the model is designed to fit what we can reasonably manufacture. Model capabilities at the frontier will be capped by what we can run on affordable, real-time hardware, and intelligence becomes something you ration against latency and unit economics.

Robot models are still small…

This is why frontier robot models remain far smaller than frontier LLMs, which have already pushed into the trillions of parameters. Today’s generalist robot models are in the billions of parameters: Physical Intelligence’s π0 class at ~3 billion, π0.7 at 5 billion, ByteDance’s GR-3 at 4 billion, Generalist’s models around 10 billion, and NVIDIA’s DreamZero at 14 billion. However, these parameter counts carry less meaning than their LLM equivalents. LLM labs pick model size by optimizing quality against a training budget and inference cost; robotics labs pick size by what their data can support and what fits on a Jetson or an H100 within a latency budget. Robot model sizes tell you where the constraints sit, not where sizes will settle.

Source: SemiAnalysis

In LLMs, scaling laws have held, scaling has consistently broadened the range of problems a single model can solve. Early results from companies such as Generalist and Dyna suggest robotics scales the same way. The question isn’t whether scaling works, it’s whether the inputs can keep up. Data is the first constraint, there is no internet-scale corpus of robot experience, and real-world interaction data is slow and expensive to generate hence the rise of embodied human data collection and world models. The network is the second constraint. Frontier models already outgrow the robot (π0.7 runs on an off-robot H100; DreamZero needs two GB200s), and once the model lives in the datacenter, latency and jitter pose problems.

It’s unclear where robot model sizes will land. The trend line points up, but efficiency gains keep pulling in the other direction, and the field has yet to converge. NVIDIA’s DreamZero, a 14-billion-parameter “world action model” built on a video-diffusion backbone, needs two GB200 GPUs off-robot just to run in real time. But compute requirements change rapidly: months after DreamZero, NVIDIA’s RoboTTT made the opposite bet, using a 3-billion-parameter policy that continually updates its own weights with test time training instead of generating videos of the future. The result is a model with minutes of usable context that is small enough to run onboard.

Wherever model sizes land, our view is that a cascade of approaches is inevitable. Some robots will run cognition fully onboard, while others will offload some part of it to GPUs sitting in a datacenter. However, for generalist robots that require real intelligence, we think off-robot compute has many advantages, from escaping the robot’s compute and power budget to pooling inference across a fleet. It comes with network challenges, which we cover in later sections, but it is an increasingly viable option.

Primer on Robot Models

Before we explore the idea of off-robot compute, here’s a quick primer on how robot models work, and which ones we’re talking about.

Classical robot control, kinematics plus planned trajectories plus feedback controllers, has run factories for decades on small embedded chips. Plenty of narrow learned policies fit onboard too. Neither is what this piece is about. We care about generalist robots: ones that can follow open-ended instructions, handle scenes they’ve never seen, and do many different jobs.

Layers of a robot model

At the top is the planning layer. It interprets the instruction and the scene, decides what the robot should do next, and may produce a subtask, destination, grasp pose or footstep plan. Below it is the action or motion layer. It turns those outputs into short action sequences, body poses, or velocities. At the bottom are the servo and safety loops. They estimate the robot’s state, react to contact and slip, maintain balance and ultimately command the actuators.

Source: SemiAnalysis

Frequency determines which layers need to run on the robot. The action and servo layers runs at hundreds of Hz: a 100 Hz loop must produce its next output every 10 milliseconds. A typical wireless round trip costs 10 to 50 milliseconds on its own, and a well-engineered link takes less than 10 milliseconds, so for these loops the network does not merely add delay, it consumes almost the entire budget before the model has computed anything. This is why anything above roughly 100 Hz can never leave the robot.

Below roughly 20 Hz, where the planning layer lives, the picture changes. A planner running at 5 Hz gets 200 milliseconds per decision, so even a 10 to 50 millisecond round trip fits comfortably, with margin to spare. That is enough headroom to move the planning layer off the robot entirely, and a well-tuned network stack could push the offloadable boundary above 20 Hz. The bigger problem, though, is jitter rather than latency. A fixed delay can be planned around: the system knows roughly how far the robot and everything around it will move in that window and can plan a little ahead. Jitter is harder because commands arrive at irregular intervals, so the robot never knows exactly when the next update will land. You can buffer to the worst case, which turns variable delay into predictable delay, but then every action pays the worst-case penalty. Getting jitter under control is what actually unlocks offloading the planning layer.

The planning layer is also the layer with the heaviest compute requirements, and those requirements climb as generalization increases. Open-ended tasks need actual reasoning, and reasoning models are big. That is why some companies have started looking at datacenter compute instead. Jetson Thor is the best robot compute you can buy today, and it still delivers only about 1/10th the FLOPs of a GB200 and roughly 1/30th of its memory bandwidth.

Source: SemiAnalysis

The Supply Chain Reality

The supply chain is also geared towards datacenter silicon, not robot silicon, and ramping robot silicon will not be easy, especially with front-end capacity and DRAM already tight. Nvidia’s own output makes the point.

Nvidia’s accelerator output is almost entirely datacenter silicon: a long Hopper ramp through 2023 and 2024 handing off to Blackwell, which scales sharply across 2025 and 2026 to dominate the mix. Jetson, the line used in robots, is the thin red sliver along the bottom, barely visible even in 2026. This is because the robotics market is nascent, real volume is years out, and there is next to nothing to build for.

Source: SemiAnalysis Accelerator Model, note Rubin withheld from chart, ramp provided to clients

Margins tell a similar story. Demand isn’t there yet, but when it arrives, pricing should go up and Jetson margins will inflect. But for now, a Blackwell datacenter GPU earns far higher gross margins than a Jetson module, on the order of mid-to-high 70s percent versus mid-60s, so NVIDIA has every incentive to point its scarce leading-edge wafer capacity at the datacenter and little reason to ramp edge silicon while the robotics market stays small.

Source: SemiAnalysis Accelerator Model

Worse, robot silicon is converging onto the exact nodes the datacenter is fighting over. Nvidia’s Jetson line used to be insulated: the previous generation, Orin, ran on Samsung’s SF8 process. The current generation, Thor, has closed that gap onto TSMC N4, and as robots run on more capable models their silicon has to stay on the leading edge: the generation after Thor likely moves to N3 alongside Rubin, then N2 alongside Feynman, exactly the nodes the entire AI-accelerator roadmap is fighting over. So a low-volume, lower-margin Jetson will always be competing for advanced-node wafers from the back of the queue.

Source: SemiAnalysis Accelerator Model

But the wafer demand from robots is not where the strain is. A million Jetson-class chips in 2030, each around 400mm², is only about ten thousand wafers for the year, a rounding error next to the datacenter GPU ramp, because Thor’s die is barely half a reticle-sized chip, and there simply aren’t many robots yet. Supply only tightens once robots need tens of millions of chips a year, well beyond 2030. So the real question is not whether robots can get wafers. It is silicon efficiency: for a given fleet of robots, which approach consumes less leading-edge silicon? Putting a chip in every robot, or serving their cognition from shared datacenter GPUs.

Source: SemiAnalysis Accelerator Model

If you look at silicon efficiency as wafers per robot, the onboard chip and the shared datacenter GPU cross over at around 7 robots per GPU. Beyond that, serving cognition from a shared GPU takes less silicon per robot than putting a chip in every machine. And that’s what really matters here, because when fab supply is scarce, the approach that uses less silicon per robot is the one that scales.

Source: SemiAnalysis

Another major bottleneck is memory. Each Jetson generation has carried more DRAM than the last: Jetson Xavier used 32GB, AGX Orin 64GB, and Jetson Thor now uses 128GB of LPDDR5X, with future generations set to keep climbing. Although total DRAM wafer capacity continues to grow, most of the incremental capacity is being absorbed by HBM for AI accelerators. That leaves the commodity and LPDDR supply that robot brains depend on competing for a shrinking pool of non-HBM wafers. Similarly, when DRAM is the scarce resource, the approach that uses less DRAM per robot is the one that scales.

Source: SemiAnalysis Memory Model

Memory tells a similar story, the onboard chip and the shared datacenter GPU cross over at around 5 robots per GPU. Beyond that, serving cognition from a shared GPU takes less DRAM per robot than putting a chip in every machine.

Source: SemiAnalysis

The TCO Verdict: B300 vs. RTX 6000 PRO vs. Jetson Thor

Wafers and DRAM tell us which approach can scale, but they say nothing about which one is cheaper to run. So the next question is cost. If you had to serve the same robot fleet, what would it take with B300 servers in a datacenter, RTX 6000 Pro servers in a datacenter, or a Jetson Thor in every robot?

We use NVIDIA’s RoboTTT. It has no public code or weights, so we benchmark a timing-representative reconstruction: real GR00T N1.7 checkpoints with test-time-training (TTT) blocks inserted into the action head, built to match the paper’s compute and memory profile rather than its accuracy.

Three details for faithfulness. GR00T’s DiT interleaves cross-attention and self-attention blocks by index, so a naive fixed stride would place every TTT module on the same block type; we insert 16 modules as (even, odd) pairs, 8 following cross-attention and 8 following self-attention across the 32 blocks. We prepend 16 learnable register tokens, giving the paper’s 57-token sequence (16 registers + 1 state + 40 actions) that every attention and TTT module then runs on. Each block takes one inner gradient step per denoising pass, and each robot carries 151 MB of fast-weight state, gathered before every batched call and scattered back after, which is the multi-robot serving cost that a shared-weight VLA does not pay.

Source: SemiAnalysis

Thus, the TTT path costs full FLOPs, memory traffic, and state swapping while leaving the emitted actions untouched. Untrained fast weights diverge and produce garbage, and we are measuring serving cost instead of task success. These numbers say how expensive RoboTTT-shaped inference is to serve.

On that basis, one B300 sustains 12 robots within the 500 ms chunk deadline, against 4 on an RTX PRO 6000.

Source: SemiAnalysis

With this, we build up three scenarios. For the offload scenarios, we assume one Blackwell-class B300 server which can serve up to 12 robots per GPU, as well as an RTX 6000 Pro Server Edition which can serve up to 4 robots per GPU. For the on-device scenario, each robot carries its own Jetson Thor.

In all three scenarios, common parts across both scenarios (i.e. mechanical components, robot shells) are not included in the BOM cost calculation.

B300 offload scenario

The first offload scenario is one B300 server, plus 96 wireless boards to connect the robot to the GPU server wirelessly (one per robot, twelve robots per GPU at the benchmarked p99 criterion). That brings all-in capex to ~$554K, which, including operating costs like colocation and power, leads us to an aggregate all-in TCO of $18.63/hr in the offload scenario. This assumes owner/operator economics, not a cloud rental quote.

Source: SemiAnalysis AI TCO Model

RTX 6000 Pro offload scenario

The second offload scenario is to an RTX 6000 Pro Server Edition. For an apples-to-apples comparison, 96 robots require three 8-GPU servers, since the RTX 6000 Pro can serve four robots per GPU at the benchmarked p99 criterion. Together, these cost $436K in all-in capex, which accumulates to an all-in aggregate TCO of $15.61/hr.

Source: SemiAnalysis AI TCO Model

Jetson Thor on-device

The on-robot scenario is 96 Jetson Thor modules at $3,500 per unit, each with its own baseboard, storage, cooling, incremental battery, and onboard radio connectivity, landing at ~$394K. Accounting for the above, along with power costs for the Thor modules, leads us to a TCO of $14.97/hr for the 96-chip deployment.

We assume a four-year useful life for an on-device Jetson Thor, shorter than a datacenter GPU, because of the environment it lives in. A GPU in a rack never moves. A Jetson on a robot rides on a chassis that vibrates constantly, takes the occasional hard knock, and, depending on the job, gets dropped, splashed, or run through unstructured work all day. That mechanical stress wears hardware out faster. Some operators we spoke to put the figure below four years, but we think four is a fair assumption.

Arguably, the cost of capital (WACC) of an on-device deployment could also be higher, especially for equity-funded emerging robotics companies, as compared to highly debt-funded neoclouds or well-capitalized enterprises running their own servers. However, since we have already accounted for a shorter economic useful life for on-device compute as our base case, we have conservatively decided to keep a constant WACC across all three scenarios.

Source: SemiAnalysis AI TCO Model

TCO Comparison

Before accounting for utilization, the three cases present differing TCOs per FP4 dense FLOPs: $0.15/hr/PFLOP for the B300, $0.16/hr/PFLOP for the Jetson Thor, and $0.39/hr/PFLOP for the RTX 6000 Pro.

The immediate conclusion that this might provide us is that the RTX 6000 Pro is not particularly performant on a FLOPs-per-TCO or a FLOPs-per-capex-dollar basis - we focus on the B300 offload scenario and the Jetson Thor on-device scenario from here.

Source: SemiAnalysis AI TCO Model

Utilization is the swing factor in TCO. A robot and the GPU behind it are almost entirely fixed capital, so the bill is the same whether the asset runs or sits idle, and cost per productive hour scales inversely with how much it works.

On-robot chip utilization today is highly deployment-dependent. In the home, one company that has deployed robots in volume told us its robots run just 1-2 hours a day, only 4-8% of the clock, because chores are bursty and current models still cap what can run unsupervised. That figure is trending up, toward 4-5 hours (17-21%), as capability improves and new features drive engagement. But the home has a hard ceiling: eventually the chores run out. Industrial demand is continuous and should clear a higher bar. Figure’s BMW deployment is an anchor we have: roughly 1,250 operational hours over about 11 months of weekday shifts, working around 10 hours a day at 40% utilization. As model capability improves, industrial utilization should climb well beyond that.

Source: SemiAnalysis AI TCO Model

The shared GPU server often runs the other way, as its compute resources are pooled across the fleet and busy around the clock. Given that the scenario above wherein one B300 GPU can serve 12 robots within the p99 latency budget, we model this as a ~90% utilization of the GPU server.

For on-robot chip utilization, we model ~40% utilization as our base case, which is factored into the table below. Net of utilization, the offload scenario for the B300 reaches a TCO per FP4 dense FLOPs of $0.17/hr/PFLOP, while the on-robot scenario for the Jetson Thor reaches a TCO per FP4 dense FLOPs of $0.37/hr/PFLOP. Here, the offload scenario reaches a mere ~46% of the on-robot scenario.

Source: SemiAnalysis AI TCO Model

Towards the top right corner of the sensitivity analysis below, high GPU server utilization and low chip on-device utilization swings the TCO per PFLOP massively in favor of the offloaded scenario. The offload scenario for current industrial deployments reaches ~46% the TCO per PFLOP of the on-robot scenario, reaching an even lower ~12% for home deployments.

It is only in several cases towards the bottom right corner wherein chronically low B300 cloud server utilization and chronically high on-device utilization swings the TCO per PFLOP in favor of on-device inference.

Source: SemiAnalysis AI TCO Model

The question arises naturally: How many robots have to be batched in order for such TCO per PFLOP savings to come through in favor of the offload scenario? Varying our batching assumptions below, for a standard industrial deployment, we see that anywhere from 5 robots per B300 GPU and upwards, the TCO per PFLOP shifts in favor of the offloaded scenario.

On the whole, we find that offloading inference to a B300 server in the cloud is a compelling case, except for cases with very small robot deployments with one on-site server sitting mostly idle - though even then, one could make the argument for renting compute as needed rather than owning an entire server to serve a few robots.

Source: SemiAnalysis AI TCO Model

The Deployment Battleground

On paper, off-robot compute wins on TCO. In practice, nobody is at mass production yet, so TCO is not what decides the question today. The companies actually shipping have split into two camps, those that keep compute on the robot and those that keep parts of it off, and what puts them in one or the other is how much generality the robot needs, how bad the wireless is, and how much of the customer’s site they are allowed to change. TCO will matter once fleets get larger. For now, here is where each company stands today, and why.

Boston Dynamics

Boston Dynamics builds three robots: Spot walks inspection routes, Stretch unloads trailers and containers, and Atlas, the humanoid, entered production this year. Production here means development units built in volume for large-scale data collection and application pilots, not mass production. For Boston Dynamics, what matters most is capability, meaning whether the robot can solve a real problem for a customer and generate economic value and savings.

Generalist vs Specialist

Most of those problems demand generality. A car plant is full of tasks that resist automation, from sequencing, kitting, and moving parts to line side, to assembly work like bolting a wheel onto the car. A single vehicle has tens of thousands of parts, one line can run 5 to 10 models at once in a dozen or more colors, and the mix turns over every model year. Commissioning a purpose-built solution for each step is neither fast nor economical at that scale. That variability is what caps traditional automation, and why factory work needs a generalist robot rather than thousands of specialist machines.

Source: Boston Dynamics. Hyundai Plant

The kind of general work Boston Dynamics is targeting, a robot dropped into a car plant or warehouse, needs a frontier model, for the simple reason that generality is intelligence. Multimodality compounds the problem: because these machines operate in the physical world, System 2 has to consume images and reason over them, and models that do that well are inherently large. Boston Dynamics would not give an exact parameter count, pointing instead to Gemini Robotics ER, DeepMind’s embodied reasoning model built on Gemini 3.5 Flash, as the right mental picture. Google publishes no parameter counts, but we estimate that System 2 could be somewhere in the hundreds of billions of parameters if not 1T.

A Brain Too Big to Carry

A model that size will not fit on a robot, which is why Boston Dynamics splits the stack across the network. System 1, the visuomotor policy that drives the machine, runs onboard, which on Atlas means an NVIDIA Jetson Thor. System 2, the reasoning layer, runs off the robot, reached through Orbit, the company’s fleet software, and served on Google TPUs today through the DeepMind partnership. Of the companies we spoke to, Boston Dynamics is one of the few that deliberately puts high-level reasoning on the far side of a wireless link.

In a perfect world there would be no split at all. Ideally System 2 would sit on the robot with no wireless link in the loop, but no model small enough to carry can yet reason well enough for factory work, and the company would rather ship something intelligent now than wait. The gap may close in the future from stronger embedded silicon or from small models learning to reason well. For now, off-device is what works.

How Boston Dynamics defines System 1 and System 2 matters, so it is worth being clear. System 2 is the planner. It takes a work order from the manufacturing execution system, something like execute job 345 and put the result in inventory bin 5, and breaks it into steps simple enough to act on. System 1 is the VLA, the model that actually moves the robot: it takes a camera view plus one of those short instructions, pick up the can and put it into the bin with the green dot on it, and turns it into motor actions. It runs onboard because motion has hard latency and determinism requirements. Notice what happened between the two.

The factory’s inventory bin 5 means nothing to a VLA, so System 2 marked the right bin with a green dot in the robot’s view. That translation from the messy real world into terms System 1 understands is a core part of System 2’s job, and the green dot is one technique from a fairly deep toolbox for steering what System 1 acts on. System 2 also supervises: it watches the work as it happens, catches System 1 misbehaving, and tracks progress through the task. That supervision is why System 2 gets called far more often than people expect: the working assumption is a query every few seconds to every ten seconds, and in some cases once or twice a second

Another reason why System 2 cannot be on the robot is that the hardware it needs is datacenter hardware. A model that size wants a rack-mounted, liquid-cooled Blackwell-class GPU drawing 1.2 to 1.4 kW per chip. A humanoid carries roughly 2 kWh of battery and draws a few hundred watts in normal motion, a kilowatt or two at peak with everything included, which is why its onboard GPU is often a 40 to 130 W Jetson Thor. Bolt a datacenter GPU onto the robot and the chip alone would out-draw the entire machine, demand cooling the robot cannot carry, and flatten the battery in minutes.

The Tail-Latency Problem

Offloading System 2 allows you to use a frontier model, but the tradeoff is in networking, chiefly reliability and jitter. Average latency is less of an issue, the problem is the tail: the occasional one-second spike that freezes a robot mid-task. Once reasoning sits on the far side of a wireless link, networking becomes the single biggest problem in robotics, and Boston Dynamics has multiple teams dedicated to nothing else, covering the radio stack on the robot, the customer’s infrastructure, spectrum, and protocol design. We go into that networking portion in detail in the next section.

Whose Data Is It

The other consideration of offloading System 2 over a wireless link is data security. Every robot is effectively exporting a video feed of someone’s factory floor, and customers rightfully attach conditions: no training on their data, no mixing it with other customers’, deletion on demand. Compliance is fragmented as well, with each jurisdiction bringing its own quirks in data protection law. And some customers, like nuclear plants or military operators, will never let data offsite under any guarantee.

Boston Dynamics’ answer is layered, following the data up the pipeline. It starts on the robot, where customers get granular control over what is shared at all. What does leave lands in Orbit, the cloud platform the robots report to, which carries SOC 2 Type 2 certification and sits under an internal data committee that governs handling. The last hop is Google: System 2 runs on Google infrastructure through the DeepMind partnership, so every image a robot sends up for reasoning passes through their side as well. That half is served through the Gemini Enterprise Agent Platform, Google Cloud’s enterprise-grade agent infrastructure, already accepted by customers with very demanding data standards. The company’s bet is that customers will accept offsite data once you can solidly prove it’s treated correctly, the same logic that lets sensitive enterprises send queries to Anthropic under no-training guarantees.

Agility Robotics

Agility Robotics builds for the warehouse and the factory floor. Digit, its bipedal humanoid, moves totes, tends machines, and palletizes. Its customers include GXO, Schaeffler, Amazon, Toyota Motor Manufacturing Canada, and Mercado Libre. Agility does not train base models from scratch. Instead, it takes a performant third-party base, and post-trains on top of it. Since the language and action conditioning comes from Agility’s own corpus, there is no reason to train a vision model from zero. All of this fits onboard the robot on an NVIDIA Jetson-class accelerator.

Source: Agility Robotics

Even with inference happening onboard, Agility still offloads some functions to the cloud. Fleet orchestration, workflow assignment, site mapping, and over-the-air updates all live in Agility Arc, and teleoperation and training data uploads ride the same link. This split works because orchestration tolerates latency variability in a way that control does not. A robot can go ten seconds without receiving its next workflow chunk because it already has thirty to sixty seconds of queued work in front of it, so it makes sense for that layer to sit in the cloud.

For everything else, Agility is firmly in the GPU-on-robot camp. Network reliability and latency are real concerns, and factories are especially tough environments for wireless connections: moving metal, heavy machinery, multipath interference, and a robot that drops line of sight every time it passes a shelf, roaming between access points as it crosses the floor. It is possible to engineer around all of this and get a network reliable enough to offload compute. But that means reworking a customer’s network and edge infrastructure, and Agility wants Digit to slot into existing sites with as few changes as possible.

Networking is not the only thing keeping compute on the robot. Safety pushes in the same direction, and it is an area where Agility has built an industrial-grade stack that many humanoid competitors lack. Today, Digit operates exclusively inside physically barricaded work cells, as no dynamically stable robot is yet safety-certified to work in close proximity to people. A balancing biped requires active control to stay upright and could topple if power is lost, so OSHA-regulated environments mandate physical separation between Digit and human workers. Digit’s sensor suite provides 360-degree awareness through cameras and LiDAR, letting it detect people and halt or adjust its path in real time. The next generation goes a step further, adding a dedicated human detection system meant to recognize an approaching person, stop, and lower itself to the ground before contact is possible, clearing the bar for Digit to eventually work outside the barriers. Digit also carries other industrial safety features, such as a stop function that maintains power to the actuators during deceleration, so the machine slows smoothly and safely before power is cut.

Source: Agility Robotics

All of these precautions exist with compute fully onboard. Moving safety decisions off the robot would create a more fundamental problem: the network link itself becomes a failure mode. If the robot loses connection to an offboard safety model, it cannot tell whether anything is wrong, so its only option is a blind, worst-case fail-safe. That is why complex safety decisions must always stay on the robot.

Verne Robotics

Verne Robotics is deploying its robots primarily in warehouses and light manufacturing environments. Its robot, Nemo, is a dual-arm mobile manipulator consisting of a bimanual upper body mounted on a wheeled base, available with either 700 mm or 1,000 mm reach arms. Nemo has been deployed at logistics companies in Massachusetts and California, including AbClonal, a life-science reagents company, where it automates cold-chain workflows such as retrieving, packaging, and transporting chemical reagents and ice packs. Verne is also expanding deployments into light manufacturing, where utilization rates can be even higher.

Source: Verne Robotics

Notably, rather than relying solely on teleoperation, Verne uses proprietary camera-equipped data collection suits worn by employees and contractors to capture embodied data directly at customer sites. The company develops its own end-to-end policy models, likely in the few-billion-parameter range, that run entirely on Jetson-class or RTX 50-series GPUs. Because warehouse manipulation and light manufacturing tasks such as picking, packing, and assembly do not require broad open-world reasoning, Verne achieves high task success without relying on large cloud-hosted foundation models. The entire perception-to-action stack runs locally on the robot, eliminating the need for offboard inference or datacenter connectivity during execution.

Source: Verne Robotics

Sunday Robotics

Sunday Robotics builds for the home. Memo is a wheeled machine whose torso rises and lowers to reach from the floor to high cabinets, with three-fingered grippers instead of humanoid hands. Sunday also trains their own foundation models. Instead of teleoperation, Sunday collects training data through its Skill Capture Glove, a wearable that matches the geometry of Memo’s gripper, worn by people simply doing their chores at home. That glove data, combined with other commodity data such as videos, pretrains one end-to-end policy, and post-training polishes it.

Source: Sunday Robotics

Over the past 12 months, Sunday has released two models, each bringing Sunday closer to a system that can perform any task in any home. Released in November 2025, ACT-1 focused on breadth of capability. It enabled Memo to complete a 15-minute, long-horizon task of clearing a dining table and loading everything into the dishwasher, while also demonstrating high-dexterity skills such as folding socks and making espresso, as well as generalizing to previously unseen homes.

ACT-2, previewed in July, turns that breadth into reliability: 99.1% zero-shot success folding laundry across nine garment types in homes the robot had never seen. The finding Sunday cares most about is that scaling pretraining is what makes post-training generalize, so much so that one demonstration can teach a new behavior that transfers. The chart shows how stark this is. In-domain means Sunday’s own lab, where the post-training data came from; out-of-domain means held-out homes and objects. Without pretraining, the model hits 96% in the lab and just 14% in unseen homes, textbook overfitting. With the full pretraining corpus, the gap disappears: 100% in both.

Source: Sunday Robotics

Generality comes from pretraining scale, and pretraining scale needs a model large enough to absorb it. The policy that works in a stranger’s laundry room is not a small one. So Sunday started out assuming cloud inference, and in the lab that made sense: more compute, easier debugging, and the same setup the researchers already worked with. Two days into testing ACT-2 in real homes, they reversed course. Home WiFi turned out to be a mess of dead zones, neighbor interference, badly configured mesh routers, and asymmetric links. Latency was mostly fine. Jitter was the problem. A robot trying to hold 99% success over hours can’t shrug off the occasional hiccup the way a video call can. The alternatives didn’t help. Starlink’s satellites cross the sky in minutes, so any tree or roofline inside the dish’s 100-degree view knocks the link out on every pass, and a 5G router dropped out the same way.

Source: X, Sunday Robotics

Sunday faced a fundamental tradeoff: larger models generalize better, but relying on the cloud makes the robot dependent on network connectivity. Sunday chose to resolve that tradeoff by moving inference fully onboard. The model now runs directly on the GPU inside the robot, removing the cloud from the critical path and allowing Memo to operate independently of the network.

Weave Robotics

Weave Robotics also builds for the home. Isaac 0, its first deployed robot, sits at a desk and folds laundry you bring to it. Isaac 1, due this fall, keeps all of that and adds a wheeled base and a telescoping torso, so it can go find the dirty clothes itself, put the folded ones away, make beds, and get toys and shoes back where they belong. Weave trains its own VLA and also runs models from Physical Intelligence. They build their own model to stay as close to the frontier as possible, but will use whatever gives the customer the better experience, so this is not a Pi wrapper and it is not a closed shop either. Both are likely in the few billion parameter range, small enough to sit on the robot, and we would expect inference to run locally.

Source: Weave Robotics

Even if inference runs on the robot, this does not make Weave free of the network. The WiFi link carries proprioceptive and visual data goes up as training data, and when the robot encounters difficulties a teleoperator steps in and takes control. Both degrade with the link, and a bad enough connection kills traffic in both directions. In homes, walls and other obstacles create weak spots and dead zones, and the typical router is not built for handoffs. Conditions are often good, but congestion is a factor: homes usually run on shared fiber, so neighboring houses and apartments drag on the link, and they do it at exactly the hours a home robot is most likely to be working.

To deal with these conditions, Weave does not require any changes to the home environment: no new router, or no extra access points are required. That pushes all of the adaptation onto the robot.

Climbing the Networking Wall

Many of today’s deployments keep inference on the robot, and nearly every company gives the same reason: the network. For the companies that deploy off-device, they got there by staffing multiple teams that work on nothing else. We think the networking problems are solvable, and that as they get solved, more deployments will move off the robot.

What is the Network Wall?

Running a robot model off-device is extremely difficult (see image below). Three high-level issues make off-device policies challenging.

Embodiment. At the physical layer, you are asking for good network conditions from a machine that is made of metal, that moves, and whose motors and electronics emit noise on the same frequencies you’re trying to transmit on. The chassis blocks and reflects its own signal, orientation changes antenna positions, and RF conditions are dynamic between transmissions.

Network. The infrastructure around WiFi and 5G was not designed with robotics in mind. It works well at home, with one or a few devices, streaming Netflix while you sit on the couch. But robotics is the opposite. You still have a Netflix-sized video feed, but you’re streaming it to the datacenter, and you’re moving while you do it. This is the uplink problem: all of the existing infrastructure was built with the downlink in mind.

The occasional dropped packet or latency spike is fine for Netflix, but an accident waiting to happen in robotics. Existing access points, chipsets, and algorithms simply weren’t built for it: prioritization is inverted, uplink MIMO is rarely supported, no redundancy to make UDP streams reliable, no low-latency handoff between access points, uplink scheduling is an afterthought, and everything is statistical when it needs to be deterministic. Add shared fiber to the picture, and your neighbor turning on Netflix while your humanoid is working can, quite literally, kill you.

Environment. This is the core challenge, and it dictates where off-device can and can’t run. Autonomous driving is the exemplar of where on-device is strictly required. The coverage area is enormous, the operator has almost no control over RF conditions, and the vehicle moves fast, which leaves near-zero latency budget and forces frequent access point handoffs. Add that driving is a simpler learning problem than humanoid labor, so the models fit on-device, and the conclusion follows: compute goes on-device, and the off-device stack shrinks to fleet management, a slow, low-frequency loop.

The environmental prerequisite for off-device compute is a small-to-medium sized space that is structured enough for RF control. Autonomous driving lacks this; homes and factories have it. That doesn’t make it easy – the environmental problem is still huge. Walls, metal, and other electronics all affect RF conditions. Turn on the dishwasher at home and the RF environment changes. Start heavy machinery in a factory and you may create a dead zone beside it. Access point placement matters a lot, and it has to anticipate how the environment will change over time. Moving between access points means handoffs, and a typical router goes silent for anywhere from 100 ms to several seconds during one. That can happen mid-manipulation, killing a policy or lagging a teleoperator. Recovery can take several more seconds, and by then the humanoid may be on the floor.

Placement of signal blockers and reflectors in the home is unpredictable; every house is a different RF puzzle. A factory has more structure but far more working against you: dense metal racking that reflects and blocks signal, motors and machines spraying interference across the spectrum, inventory and equipment that move daily so the dead and weak zones move with them, and dozens of robots contending for the same uplink airtime.

Source: X

How do we climb it?

Preparing a model for off-device inference is a nuanced problem. It depends on whether the model is hierarchical or monolithic, how much compute it requires, the tasks it has to perform, its action space, and the embodiment it drives. Some of what breaks today is fixable with ordinary engineering. The transport layer, for one, is often a modified WebRTC: UDP with no retransmission, reliability through repetition over multiple antennas, application-layer awareness of packet importance, and freshness above all. Teams already build this and the know-how is a commodity.

But the network wall is not the transport layer or a series of ordinary engineering challenges. The network wall is based on the issues with existing hardware, and we believe there are four problems that are hard and high-leverage that together can make off-device models reliable beyond the existing infrastructure.

Start by fixing the environment, not the robot. The highest-leverage fix is the access point. Access points in robotics need to do the opposite of what consumer and enterprise access points do for downlink-heavy stationary clients. This includes:

Robot-aware uplink scheduling. Robot uplink is scheduled ahead of household or plant traffic rather than queued behind it, which is admission control that WiFi does not natively have and the access point must impose. Each robot gets a reserved slot at a fixed cadence instead of a chance at the channel, and the schedule accounts for the other robots competing for the same airtime. The model determines the traffic, the traffic determines the schedule, so the scheduler has to know what the robots are running. When a robot is not phoning home, the same scheduler still has to manage everything else on the network well.
Location-aware beamforming. Robots move. The access point has to know where each robot is and where it is going, then steer, schedule, and hand off ahead of that movement rather than reacting to it or assuming a client that stays where it is associated.
Centralized timing. One timing authority per site, distributed over the air to every robot with a guarantee, so observations across the fleet share a clock. Off-the-shelf access points cannot provide this.
Multi-link operation. Running several links at once, across bands and across WiFi and 5G, with the same observation duplicated or split across them, improves reliability and cuts latency in a way no single link can.
Clean spectrum. 6 GHz where available, since it is uncongested and its channel widths give the throughput margin that makes retransmission-free delivery realistic.
Fast multi-access point handoffs. Any real building needs more than one radio, and the roam between them has to be completed inside the observation cadence. A 200 ms handoff is multiple dropped frames.

Placement is just as important as the hardware. It needs to be measured and optimized, not guessed: RF survey along the robot’s task paths, dead-zone mapping, antenna orientation chosen for the robot’s typical height and heading.

WiFi will dominate the home and most of the factory floor because it is fast and can carry the load. It will not cover everything. 5G, private or public, fills outdoor paths, larger sites, and locations where WiFi congestion cannot be controlled. The two should coexist under one scheduler, with the robot able to fail over fast between them. Where a site depends on a carrier, the last step is to work with the network provider on uplink prioritization and dedicated fiber to the GPU cluster, so that the scheduled path does not end at the access point.

Shrink the load. Networks are bandwidth-limited. The other half of climbing the wall is reducing what has to cross it. Image sensors are nearly all of a robot’s uplink, so this is a perception problem before it is a networking problem.

Maximize the uplink. Robotics inverts the traffic pattern every wireless chipset was designed for. The mainboard has to be built for this. 4x4 uplink MIMO rather than the one or two streams consumer modules ship, chipsets chosen for uplink throughput rather than downlink, an operating system built for wireless communication, and a radio that participates in the access point’s schedule instead of contending with it: it knows its slot, fills it at cadence, and stays quiet outside it. Holding that beat is what makes scheduling deterministic, and it only works if the mainboard and the access point are co-designed. Neither side can impose order on the link alone.

Everything on one clock. Datacenter GPUs beat on-device compute on cost as batch efficiency grows, and batching requires that observations from many robots arrive time-aligned in order to hit a reasonable latency budget. Everything has to run on the same clock. Sub-millisecond synchronization is achievable with good crystal oscillators on the mainboard and wireless time sync across the fleet, so every robot on a site captures, encodes, and transmits on the same beat. GPU orchestration is then built around that beat: the scheduler knows when the next frame of observations lands, reserves compute for it, and runs the moment the frame is complete. Late observations go to the next batch, not to a queue, and the latency budget is spent on inference rather than waiting. Put this together with a mainboard that holds the schedule, access points that enforce it, and a perception stack that shrinks the load, and you have a system that batches at scale without inflating latency, which is the whole economic case for running models off-device.

Who will climb the wall?

Long Live the Short King: Why 4-hi HBM Wins

2026-09-13T10:19:39+00:00

High Bandwidth Memory has been a key technology enabling the AI revolution. Despite HBM’s high costs relative to other forms of memory, chip designers have packaged more and more HBM into AI accelerators. Customers push to design in newer generation HBM whilst also increasing capacity per XPU by adding more cubes, and with denser and higher stacks. This has led to HBM consuming an increasing share of total DRAM wafer capacity, resulting in the extreme DRAM shortage we see ourselves in today.

We are now on the cusp of this trend breaking, due to a host of reasons that are both technical and supply chain driven. Next generation accelerators are moving to 8-hi stacks as standard compared to 12-hi being the standard today. Nvidia has decided to do this with Rubin Ultra which is a downgrade to 192GB from 288GB per GPU compared to standard Rubin and B300, which SemiAnalysis was the first to cover this change. The supply chain is now preparing 8-hi to become the new standard, and it was not even a year ago that the industry was anticipating going to 16-hi and beyond.

However, we see this trend going further as we covered in our Memory Model. For inference workloads where memory bandwidth matters the most, 4-hi HBM provides the best $/bandwidth and therefore lowest cost per token. Capacity is important up until a certain threshold, beyond that point more HBM capacity offers diminishing benefits while those incremental bits carry the same BOM penalty. This is what hardware teams at the major labs want to implement in their ASIC programs from next generation HBM4 onwards. On the same way that hardware designers are focussed on maximizing tokens/Watt in a world of DC power constraints, 4-hi is how to maximize tokens per HBM wafer, which is another scarce resource.

This report will go through the reasoning behind what’s changed in the capacity equation, demonstrate that often HBM capacity is stranded, and show how this can be a win-win for both labs’ TCO and memory supplier profitability.

The HBM Content Trend is Broken

HBM has been the major form of working memory for AI workloads as it provides a good balance of capacity and bandwidth which can be utilized for general purpose chips for both training and inference. SRAM, while very fast, suffers from low capacity, making it premium and limited on certain tasks. Conventional DDR DRAM provides much more capacity per dollar but the bandwidth is too poor.

With AI being such a memory intensive workload, it is no wonder that per accelerator HBM content has continued to increase generation upon generation: more cubes per accelerator, denser cubes, and taller stacks.

Source: SemiAnalysis, Nvidia

The increasing amount of HBM being shipped per chip was also a big contributor to costs increasing. However, over the last few months this trend has now broken. The next generation Rubin Ultra will only feature 192GB of HBM per accelerator compared to 288GB with conventional Rubin and even Blackwell Ultra! This is a huge regression from expectations from not even a year ago that Rubin Ultra would feature 1TB of HBM per GPU! In fairness, that was when Rubin Ultra was expected to have 4 compute dies vs only 2 now. Even normalizing for this, this is a decrease from 256GB per GPU die to only 96GB.

Source: SemiAnalysis, Nvidia

This decision to reduce HBM density is in large part motivated by supply. HBM wafers are scarce. There are simply not enough wafers to supply the number of 12-hi cubes needed to support the N3 logic and CoWoS Nvidia has secured next year. Moving to 8-hi means much more cubes from the limited number of wafers and helps close the gap.

The big increase in HBM costs next year doesn’t help either, even though Nvidia is getting a better deal than other major customers. Even in a time where GPU demand is well above supply, Nvidia has to give up some margin as its component costs increase, with memory being the largest.

The trend of ever increasing HBM density is now breaking and it is not only because of supply, but also architectural reasons and changing workload requirements. While supply and cost are very important factors, Nvidia wouldn’t make this decision if this additional capacity is necessary for customers and clearly delivers better performance/TCO. The focus of this article is to demonstrate that in many cases additional HBM capacity per GPU is not worth the cost and that HBM 4-hi in the future is often the optimal configuration.

Shorter Stacks, Same Bandwidth

An important dynamic we want to remind readers of is that the reduction in per GPU capacity by lowering stack height doesn’t come at the expense of bandwidth at all. Bandwidth available per cube of HBM is the same regardless of stack height. Of the 2048 data I/Os per cube of HBM4 or HBM4e, these data paths are split evenly among the total number core DRAM dies in the stack. The maximum per die is 512 I/Os, which means that 4-hi is the lowest stack height that can get access to all of the 2048 I/Os and maximize bandwidth. However, cost to the supplier and end customer is driven primarily by the amount of DRAM content per cube. HBM pricing is primarily driven by GB per stack. While the value customers derive from HBM is often the bandwidth. Therefore lower stack heights mean a much lower $/bandwidth proposition to end customers, meaning opting for lower stack heights is almost a free lunch.

Source: SemiAnalysis

Bandwidth vs Capacity? Bandwidth is King

While different AI workloads require both capacity and bandwidth, there are tilts towards whether bandwidth or capacity matter more depending on workload. We can divide AI compute into 3 major buckets: pre-training compute, post-training/reinforcement learning compute, inference compute. Reinforcement learning is very similar to inference therefore requires similar architectures. Among these 3 buckets, inference decode is primarily bandwidth constrained: each token generated requires a read of all the active model parameters as well as the user’s KVCache that is stored in memory. Reinforcement Learning (which is inference-like) is still tilted towards being bandwidth constrained but requires more capacity than inference due to the focus on large batches. Meanwhile pre-training is much more sensitive to capacity as HBM is used to store weights, gradients and activations during a training run, rather than just weights and user KV for an inference forward pass.

A big change has been the share of compute allocated to pre-training has shrunk significantly as end token demand has increased and post-training techniques became a major vector for model scaling. The result is that the majority of allocated compute has now become bandwidth sensitive rather than capacity sensitive.

Source: SemiAnalysis Tokenomics Model

This is not to say that capacity is irrelevant for inference, capacity is important up to a point. One of the big weaknesses of the Cerebras and Groq SRAM machines is their lack of capacity despite abundant SRAM bandwidth allowing very fast token serving. In short, there is a threshold level of capacity required: an inference system needs to have enough high bandwidth memory capacity to hold model weights as well as users’ KVCache. Beyond that threshold there are diminishing returns to having more capacity as we will show.

The other important point is that this is not memory per chip, but rather aggregate memory per system. Frontier model sizes have long stopped being able to fit on the memory of a single chip, and need to be sharded across the memory of multiple chips. This is why networking is such an integral part of AI system design. For the MoE architecture that all frontier models of today adopt, the optimal system is often a single scale up world with fast and low-latency communications (such as NVLink within a whole rack) that can support wide expert parallelism, where only a few experts are loaded onto each GPU. This set up allows each expert to process larger token batches and improves compute and memory bandwidth utilization, at the expense of far more all-to-all communication costs between GPUs

Source: SemiAnalysis, Nvidia, Hugging Face

While both model weights and user KVCache capacity requirements have grown, the huge increase in scale up world sizes have allowed aggregate memory per node to more than keep up. In the Hopper-era, one H100 HGX system provided 640GB of aggregate HBM. The weights from Llama 3.1 405B (the best open source model at that time) would consume 63% of that capacity, leaving some (but not a lot of) headroom for KVCache. This was why the additional 61GB per GPU that the H200 provided was so important. With 1,128GB of memory per server, this provided a lot of incremental headroom to fit more KVCache and serve more concurrent users.

With GB300 NVL72 rack scale solutions, Nvidia delivered 9x the scale up world size in addition to doubling HBM per GPU by using 12-hi HBM and increasing the number of HBM cubes per GPU. This resulted in an 18x increase in aggregate HBM capacity compared to the H200 HGX. This has significantly outpaced the almost 7x growth in parameters of the largest open source model available today: Kimi K3 2.8T. The capacity burden is also reduced significantly by quantizing down to 4 bit quarter precision from 16/8 bit. One replica of Kimi K3 at 2.8T stored in MXFP4 only takes up 1,561GB, less than 8% of the almost 21TB of HBM available on GB300 NVL72. The increase in scale up world size is also not slowing down. The big upgrade with Rubin Ultra is moving to NVL576, another 8x increase from NVL72, which more than offsets the one third reduction in HBM capacity per GPU.

Source: SemiAnalysis, Moonshot, Nvidia, AMD

In the rack-scale server era and beyond, memory capacity available is well above what is necessary for model weights, providing a very comfortable amount of headroom for user KVCache. Systems can now comfortably support inference requests with very large context lengths and at good enough concurrency to support very high inference margins. This eases a lot of the HBM capacity pressures and allows HBM density per chip to relax. This is a different paradigm from the previous era of smaller serving deployments where model weights consumed a substantial portion of total memory. Back then increasing HBM density per GPU was a critical way to get valuable capacity headroom.

While the lower stacks, cheaper bandwidth argument has been a valid one, we are only now seeing this trend of taller stacks reverse as the minimum HBM capacity per chip threshold has only recently moved much lower. We think this trend goes even further. Silicon and hardware teams at frontier labs see 4-hi HBM as the optimal SKU for many inference workloads as it delivers the best performance/TCO and now we will dive deeper into why.

The Optimal Mix of Bandwidth and Capacity

There is an optimal ratio of bandwidth and capacity, and often this capacity ends up being over-provisioned or stranded where the bandwidth per token is not enough to read all the available capacity in HBM.

Let’s start with a simple example. Each HBM4E stack provides 3,328 GB/s of nominal peak bandwidth ( assuming 2048 pins at 13 Gbps each). For HBM4E each core die is 32-Gb which results in 16 GB, 32 GB, and 48 GB for 4-hi, 8-hi, and 12-hi stacks, respectively. An illustrative single sequence of 100 tokens/s, 3,328/100=33.28GB of memory bandwidth is available per token. We then divide this bandwidth available per token by cube capacity to get to the number of reads of the total stack capacity each token is able to perform. At 1 stack read per token this means capacity and bandwidth are matched perfectly: one token has the available bandwidth to read all the working data stored in HBM and no more and no less.

A number above 1 means bandwidth is “overprovisioned” relative to capacity, as a token only needs one read of the working set (i.e. one forward pass). However, the bandwidth is rarely “excess” as the additional bandwidth budget can be used up by increasing tokens per second. A number below 1 means that a token doesn’t have available bandwidth to read all the available HBM capacity. To serve tokens at that given throughput the working set can be no larger than available bandwidth per token, so it must be lower than total memory capacity. This results in excess memory capacity that is overprovisioned.

Source: SemiAnalysis

As we can see in the illustrative table above, an increase in tokens/s lowers bandwidth available per token, pushes the stack reads/token ratio lower, therefore increasing the likelihood that capacity is overprovisioned. This creates the asymmetry where it is preferable to have a lot of bandwidth compared to a lot of capacity. Bandwidth can normally be used up in its entirety, while spare capacity can not always be used up whilst also carrying a cost that is only getting higher.

Of course, this is a simple analysis that ignores batching requests to serve multiple concurrent users. Batching plays a key role in inference economics as batching can maximize total token throughput at given levels of interactivity. The big saving from batching comes from amortizing one read of weights across multiple users’ requests. Higher concurrency also consumes memory capacity as each individual user’s KVcache needs to be loaded into memory. Increasing batch size is often how inference workloads become memory capacity bounded rather than bandwidth bounded. When HBM capacity is easily enough to hold model weights, the benefit of additional HBM capacity is that it can deliver higher token throughput, and why the additional bits can be worth the BOM.

Source: SemiAnalysis

When factoring in batching, the result is this generalized curve above. All else equal, having taller stacks of HBM delivers higher throughput up until a certain per-user latency SLA, whereby the extra capacity becomes redundant. These are represented by the points where different curves overlap with each other.

The “per-user latency SLA” is a minimum interactivity assumption. This is a minimum rather than actual interactivity. In cases when concurrency is capped by capacity that means that there is additional bandwidth available to support higher tokens/s/user. So even though 4-hi can deliver lower concurrency, the faster token/s/user wholly or partially offsets the impact of lower batch size to increase total token throughput.

Concurrency is based on the lower of maximum concurrency when capacity-bound: the number of sets of individual users’ KVCache that can fit into memory; and when bandwidth-bound: maximum number of users that can be served at our minimum interactivity, which takes into account the bandwidth available per token calculation above. Multiplying users by tokens/s/user gets to total tokens generated per second which is throughput.

These curves can tell us the value of that additional capacity provides in terms of inference performance by showing just how much greater token throughput can be achieved with more capacity (which comes through more batching), while also showing at which points on minimum interactivity the additional capacity doesn’t add value.

This is heavily dependent on model size or said another way the % of HBM capacity that needs to be dedicated to model weights. In general, the larger the model size, the curves will shift to the right so that taller stacks deliver more throughput at higher interactivity points. In addition, taller stacks will also deliver a greater increase in peak throughput. This goes back to batching economics. Every step reads the weights once regardless of batch size, so the larger the weight footprint, the more throughput each additional concurrent user unlocks.

Let’s put actual numbers to this to see what this looks like for the largest open source model today: Kimi K3. This is a roofline analysis, in reality, real achieved memory bandwidth is significantly lower than peak bandwidth. Real results will see the minimum interactivity thresholds shifted to the left, so the general conclusions will be the same. Also this exercise is focussed on just the decode pool in a disaggregated PD set up. For real delivered results see our InferenceX and AgentX benchmarks.

We use a Rubin Ultra NVL576 system as this will be a more representative scale up world size in the HBM4E era. We assume the HBM is HBM4E with 13Gbps speeds. 4-hi, 8-hi, and 12-hi means 128GB, 256GB, and 384GB per GPU respectively. For Kimi K3 on this NVL576 system, each GPU holds 16.8GB of weights which is 2.1GB per HBM stack. We can see that at a minimum SLA of 105 tok/s/user, 12-hi HBM doesn’t deliver more throughput than 8-hi HBM. Then when the interactivity is doubled further to 213 tok/s/user, there is no benefit to having taller than 4-hi stacks. In terms of maximum throughput benefits, 8-hi provides an 8% increase in maximum total tokens/s vs 4-hi while 12-hi provides a 10% improvement. However, again stepping down to the maximum throughput that 4-hi can deliver allows for more than double the interactivity. We also reiterate that in reality, as not all of the raw memory bandwidth can be utilized, actual results will shift these curves to the left, meaning that the higher capacity breakeven interactivity points are lower and could push into the speeds that users find unacceptably low.

Source: SemiAnalysis, Moonshot

We can see that while more capacity can deliver marginally more token throughput for these models (putting aside the benefit of faster tokens). The question then becomes, is that additional throughput worth the additional BOM cost associated with more memory. HBM is a significant part of accelerator BOM cost, and with HBM so tight, the cost per GB will rise dramatically next year.

When taking into account costs, we model the BOM cost increase the end owner pays relative to a baseline HBM4E 4-hi based Rubin Ultra for the Rubin Ultra NVL576 system. The all-in system cost increase (which we proxy as TCO as all-in system cost makes up the vast majority of TCO) of 8-Hi is 12.1% relative to a base 4-hi system and for 12-hi it becomes a 26.3% premium. We can then compare the increase in token throughput relative to the cost increase at different points on minimum interactivity to show whether more memory can deliver lower cost per token at any of those points.

Source: SemiAnalysis

Overall we see that the increase in throughput provided by 12-hi or 8-hi HBM comes in below the increased system cost associated with more memory, meaning at these memory prices (along with Nvidia’s margin stack), 12-Hi and 8-Hi HBM result in higher cost per token than 4-hi HBM. This comes before we even take into account more HBM conserving techniques, namely KVCache offloading to lower tiers of memory. This next section will show an actual production run we performed to show how lower memory impacts actual throughput.

A Real Experiment: KVCache Offloading Relieves Capacity Pressure

While reducing HBM capacity cuts into the amount of KVCache that is stored as long as model weights are able to be fit, second tier DRAM can be used to store KVCache that is less “hot” and requires instant retrieval. This is becoming more common in agentic workloads that have very high context lengths that place pressure on HBM capacity but also have long pauses between turns, making it worthwhile to expel KVCache into DRAM or even storage rather than recomputing it altogether. In our recent AgentX benchmarking, many workloads’ pareto optimal results were obtained with KVCache offloading above certain concurrencies.

To test the impact of reduced HBM in a real workload, we ran InferenceX benchmarks on Kimi K3 with reduced HBM. The system was with 16 GB300 GPUs, with a disaggregated setup of 8 GPUs for the prefill pool and 8 GPUs for the decode pool. The restricted memory workload was run with 85% HBM utilization compared to 92% HBM utilization normally. Yes, this is only an 8% reduction in HBM, which is very far from a 50% reduction when going from 8-hi to 4-hi, however this is with Kimi K3 being served on only 8 GPUs so weights take up a significant portion of memory. This ends up being a 20GB reduction per GPU of KVCache storage. Specifically, for aggregate serving it brings down the GPU KV budget from 53GB to 34GB which is a 36% reduction. For disaggregated pairs it is 44GB to 24GB, which is a 44% reduction. Overall, this is a sizeable reduction in the HBM budget for KVCache.

In terms of results, we see that up to most points on the curve the HBM restricted run delivered very similar throughput (denoted by the dotted red curve). However once concurrency hit above 70 is when throughput dropped dramatically by almost 30% compared to the higher HBM run. This is the point when peak GPU KV occupancy hit 100% for the restricted memory version. At that point preemption, repeated loading, and queueing affect that throughput. Note that this also hits at a point of the curve that is high concurrency but very low interactivity. We know that DRAM offloading was used much more as the lower HBM-restricted profile had over 10x the amount of reads to DRAM KVCache compared to the higher HBM profile at the same concurrency.

Source: SemiAnalysis

We can see in a real workload that KVCache offload to DDR DRAM can help relieve HBM capacity pressures very effectively. This is for Kimi K3, which doesn’t take into account some of the even more extreme cache reduction techniques introduced in the latest DeepSeek v4.1 Flash model, which reduce active KVCache sizes by around 75% compared to v4 Flash. At these reduced KVCache levels, increasing concurrency to utilize all the HBM capacity is likely to mean that the workload becomes compute and network bound rather than capacity bound, therefore also diminishing the value of a taller HBM stack.

Does 4-hi Give Up Future Proofing Optionality?

In our K3 roofline analysis using HBM4E on NVL576, the obvious flaw is that we are using a present workload on a future system to show that 4-hi is sufficient. It will only be a year later when these larger scale up domains are available along with HBM4E and frontier models will only have scaled more. In addition, AI servers are deployed for over 5-year lifespans, and will need to be able to continue serving tokens through that period.

The capacity being sacrificed could underestimate optimal capacity needs for future workloads significantly if model sizes for future workloads become much larger. For instance, assuming a future model that is 3x the size of Kimi K3 (along with per user KVCache also tripling in size), the throughput increase from taller stacks of HBM is much better. 12-hi can deliver 47% more total tokens than 4-hi, and 8-hi delivers 36% more tokens, though at half the speed. In this scenario, the additional capacity starts to become more justified and the decision comes down to how valuable the faster tokens are.

Source: SemiAnalysis

When taking into account costs into this analysis, in this case 8-hi delivers more throughput than the TCO increase at below 180 tok/s/user.

Source: SemiAnalysis

However, going to 12-hi vs 8-hi isn’t worth it, nowhere on the curve is 12-hi’s increase in token throughput greater than the BOM cost increase.

Source: SemiAnalysis

There are several counter-arguments to this. First, scaling parameters and therefore model weights is no longer as important for model capability. The use of significantly more reinforcement learning as well as more reasoning/thinking time have been key developments that have resulted in more efficient model architecture. Also, it is effectively confirmed that GPT-6 Astra uses looped transformers and there is a lot of speculation that this is the case for Anthropi’s largest models as well. Looped transformers effectively run the input through transformer layers multiple times. This scales the model through depth rather than by adding weights.

Second, if 4-hi accelerators become a significant part of the AI server accelerator installed base, researchers will adapt their models so that they can be effectively served with this hardware. Additionally, we understand that it is the Frontier Labs’ hardware teams who are the most in favour of 4-hi HBM. These teams are obsessed with hardware/software co-design, and their calling for 4-hi reflects that future models are heading in a direction that don’t require that capacity.

Lastly, the big picture is that everyone building and serving models has the goal of serving maximum tokens. Going to 4-hi stacks is what will ensure this…

Maximizing Tokens/Watt Tokens/HBM Wafer

The optimal hardware architecture is what maximizes output per scarce resource. We are in a world of many shortages when it relates to building out AI infrastructure. Power is one of the key ones, which is why optimizing for tokens/W is so important and a key metric that inference providers and hardware architects are trying to optimize for. Of course, money is a finite resource and why performance/TCO has been an evergreen figure of merit driving hardware decisions.

As we mentioned above, one of the other major constraints is the supply of HBM wafers. While not talked about, optimizing for tokens/HBM wafer can be thought of in the same vein as tokens/w and tokens/TCO. The way to do that is simple: use 4-hi HBM. As you can get the same bandwidth from a 4-hi stack as an 8-hi or 12-hi stack, in simple terms the 4-hi means triple the memory bandwidth that can be harvested from each HBM wafer if that wafer was used for 12-hi HBM, or double the bandwidth of an 8-hi wafer. The numbers are even better than that when factoring in the higher packaging yields from 4-hi. As memory bandwidth generally equates to token throughput, this means 4-hi is the way to go in terms of getting the most out of a limited supply of HBM supply.

By going to 4-hi, the amount of HBM cubes harvestable compared to 8-hi could more than double. These cubes are only useful if they can be co-packaged with an accelerator and put into a system. This means that the bottleneck moves away from HBM wafers to other parts of the supply chain like logic wafers (both for the accelerators and the HBM base die), substrates, PCBs, downstream integration capacity as well as many other components in between. Let’s not forget all the power needed to deploy more systems. Overall, though, we believe the rest of the supply chain is able to ramp capacity faster than DRAM supply over the next few years. Even if the AI server supply chain can’t match the supply needed to accommodate double the HBM cubes, this also frees wafers for conventional DRAM that can serve other applications that are memory starved because of HBM’s cannibalizing effect on overall DRAM supply. The first that comes to mind is server DRAM, with DRAM per socket being cut dramatically to make the most of limited server DRAM supply. This is also very much related to supporting agentic workloads, where large amounts of time are actually spent waiting on a CPU-related execution rather than the GPU to decode tokens.

Behind the paywall, we will discuss what the impact is on memory suppliers and how 4-hi could benefit them, as well as how they are responding. We have significantly more insights, product sku / supplier decisions, and demand / supply details in our Memory Model.

Memory Supplier Impact

Nvidia’s Backstop Universe – Heads I Win, Tails Who Loses?

2026-09-11T09:04:52+00:00

Follow the money behind the AI buildout with our AI Compute, Capital and Markets Model - understand who is funding the expansion, how deals are structured, and where the risks sit. Contact our team here to find out more.

We’re also hiring for SemiAnalysis’s Compute, Capital & Markets team. We have four openings across New York and Singapore:

Senior Credit Markets Specialist - New York: 5-7 years in credit sales, sales trading, or trading, with strong pricing knowledge and investor relationships.
Credit Analyst - New York: 5-7 years in credit analysis, ideally covering high-yield, distressed, or complex debt.
Two Junior Credit Analysts - New York and Singapore: 0-3 years of experience, with strong financial modeling skills. Recent graduates are welcome; prior specialization in credit, semiconductors, or AI is not required.

If you want to combine financial analysis with a detailed understanding of the technology and economics behind AI infrastructure, we’d like to hear from you - apply here.

An expanded view of Nvidia’s Balance Sheet

As of 2Q F1/27 (second quarter of fiscal year ended Jan 2027) Nvidia in its 10-Q disclosed $530B of gross off-balance sheet guarantees across six different line items – a jump from $184B of obligations disclosed in the prior quarter due mainly to moves in three items:

Supply and capacity commitments went from $119B to $279B, primarily memory as explained by the CFO, with 96% of these supply commitments due by F1/29.
Guarantees and LPS Guarantees went from $3.5B to $108.5B on SB Energy’s PORTS-Pike campus in Ohio, representing 4.25 GW leased to OpenAI for twenty years with Nvidia guaranteeing the land, power and shell.
Two line items appeared for the first time: $36B of AI cloud agreements, the take-or-pay floors under Neocloud capacity, and $20B of datacenter leases Nvidia has signed as a tenant but expects to reassign to third parties.

The $530B of commitments dwarf on-balance sheet liabilities of $91B, which includes $33.4B of total debt with $25B of senior notes issued in the most recent quarter. On the asset side, Nvidia’s cash balance has increased from $12B to $22B in the past few quarters, but its investment book (marketable and non-marketable securities) has leapt from $45B in 2Q F1/26 to $128B by 2Q F1/27. The $99B equity book generated $23.7B of gains in the first half, and $25B more of equity investments are committed.

The question on everyone’s mind is this: given Nvidia’s on-balance sheet and off-balance sheet obligations, what is its capacity to meet all of these commitments in the case of an industry slowdown or downturn, and how much can Nvidia grow these obligations over time while still enjoying a margin of safety? How much support does the industry need and how much support should Nvidia offer?

In the chart below, we illustrate the history of these off-balance sheet items as well as our estimates of how much additional obligations Nvidia can reasonably support given its balance sheet and future cash flows. To be clear, this does not represent our base case or a definitive forecast for where we think Nvidia will go with its various backstops. Rather it focuses on the extent of Nvidia’s ability to support obligations given its business profile, and we argue given this considerable firepower, that it should have the willingness to continue expanding these commitments significantly.

Source: SemiAnalysis AI Compute, Capital and Markets Model

What is obvious from the diagram above is that Nvidia is a true behemoth when it comes to cash generation – consensus estimates have it generating ~$441B of EBITDA in F1/28, and this is why we reflect this via a growth of the cash balance to $1.4T by F1/31 – though in reality Nvidia will most likely reinvest this cash into various securities. These are huge numbers, yet the scale of overall AI investment is larger – we estimate ~$11T of cumulative capex from CY24 through CY29!. This note will explore how far Nvidia can take each form of support, and what impact this could make on the massive AI investment needs ahead of us.

Source: SemiAnalysis AI Compute, Capital and Markets Model

Why all the Guarantees and Backstops?

The mechanism that gives rise to these off-balance sheet guarantees and backstops is the same one we described in The Front End Gets Crowded (institutional article). The Gigascalers (Amazon, Microsoft, Google, Meta, Oracle) have been gatekeeping the investment grade capital that the AI buildout runs on and have benefitted tremendously from the high spread between offtake price they are being charged, and spot/short-term price that they are selling at as well as services that sit on top. In response, Nvidia has been manufacturing alternative credit anchors so Neoclouds and Neolabs can raise money without the Gigascalers. In these support arrangements, Nvidia sells GPUs at full margin on day one, then floors the buyer’s revenue (in the case of the AI Cloud Partner (AICP) Program), or guarantees the buyer’s landlord, or signs the lease itself, all so that non IG-rated operators can effectively borrow at IG rated pricing.

Heads, demand holds and Nvidia wins twice, once on the GPU sale at full margin and again on the revenue share above the floor. Tails, demand falls and the Neocloud makes no money but remains solvent, as the revenue floor is set high enough to repay lenders. Nvidia’s backstops and guarantees also act as a breakwater against further contagion as lenders have some shielding from a downturn. Yet how high and broad the breakwater that is the Nvidia backstop/guarantee depends on Nvidia’s ability to meet these obligations.

Risk reward is asymmetric in Nvidia’s favor as long as its balance sheet remains strong. Nvidia loses only if the backstopped Neoclouds fail to meet their lease and offtake commitments at the same time that Nvidia’s own business and cash generation slow, degrading its ability to make good on these obligations. The two are correlated, since a Neocloud that cannot pay its lease is a Neocloud that has stopped buying GPUs. Only if a downcycle is steep enough to overwhelm Nvidia’s ability to meet its off-balance sheet obligations, then it will be tails, we all lose.

By our count, Nvidia today backstops ~6.5 GW of capacity, most of which is not yet built. Our datacenter model has Microsoft, Meta, AWS and Oracle leasing roughly 15 GW of third-party capacity in 2026 and we expect them to lease more than 35 GW by 2028. Each of those leases typically run fifteen to twenty years, and developers borrow against them with investment grade pricing. The Gigascalers are the implicit backstop of the buildout and are in aggregate far larger force than Nvidia’s backstopped capacity, yet Nvidia’s program gets the attention because it is new and because many investors have Cisco PTSD, contracted vicariously without an appreciation for the dissimilarities.

Source: SemiAnalysis AI Compute, Capital and Markets Model

We will now explore how this support could evolve in detail – walking the six off-balance sheet obligation lines in the recent quarter’s disclosure, plus a new one we have created to track residual value guarantees under the capital partnership.

AI Cloud Agreements

This line item made its debut in the 2Q F1/27 earnings report released on August 26th, 2026 at $36B of aggregate obligation, spread across six years. It represents Nvidia’s AI Cloud Partner (AICP) program, where Nvidia backstops GPU rental contracts for six years at a below market rate, approximately $2.35/hr/GPU on average for the GB300 vs the $4.50 to $4.60 range for 5y contracts, with the intent of making these AI clusters bankable. Neoclouds in this program would then seek to rent out compute on shorter term contracts to AI Natives and other tenants, but can always rent to Nvidia at any time should these contracts fall through or not renew. We published an extensive analysis of this structure in a July 2026 Newsletter Article.

Source: SemiAnalysis AI Compute, Capital and Markets Model

There are three deals announced so far under AICP: Firmus’s 360MW deal in Batam corresponds to a $21.1B total obligation, the SharonAI deal means $4.2B of obligations and we estimate the GMI deal corresponds to $2.2B of obligations for Nvidia. We believe there are still a few deals that have been completed before Nvidia reportedly paused AICP two weeks ago and estimate this line item will close F1/27 at $51B.

We do understand that Nvidia is not advancing any further AICP deals at this time, but believe that they still wish to be supportive in one form or another. In this analysis, our forward modeling still assumes considerable growth in this line item to reflect our view that Nvidia should offer support in some form, as well as to explore how far Nvidia could go in terms of explicit support.

Source: SemiAnalysis AI Compute, Capital and Markets Model

Residual Value Guarantees under Capital Partnerships

Earlier in August 2026, Nvidia announced its partnership with a number of private equity funds to mobilize over $500B of third-party capital to support the buildout of AI Clusters. Under this program, institutional investors are in the driver’s seat and will evaluate and price each proposed project, with the only support from Nvidia being a residual value guarantee for up to 25% of a deal. In his X post detailing the program, Jensen Huang highlighted that this support would be limited and on a project-by-project basis. Though there are few details on what such a program could look like, we drew inspiration from the Google-Broadcom-Anthropic-Apollo TPU structure to surmise how deals could be structured.

In the TPU backstop structure, if Anthropic is unable to continue to service lease payments to the TPU SPV, then the structure can be resolved by Broadcom assuming the lease or by the racks being sold. If the racks are sold, then the residual value guarantee takes effect, as Broadcom must then make the senior noteholders whole after such equipment sale – effectively plugging the residual value gap, if any. The deal gets better – Google will play its part by also backstopping rental payments to the various datacenter operators. The effect is that lenders to the TPU SPV are effectively taking Broadcom credit risk while lenders to the datacenters are taking Google credit risk.

Nvidia’s proposed structure would appear to wean investors away from this dual backstop as it not only reduces the residual value guarantee to only up to 25%, but it also is silent on any datacenter backstop.

Source: SemiAnalysis AI Compute, Capital and Markets Model

We think that the funds could tranche the lending to create different pools of risk, with the first loss being taken by riskier equity/mezzanine style lending and the last loss taken by more senior creditors. Nvidia’s 25% residual value guarantee would insert itself above the senior lenders but after the equity/mezzanine investors, adding a meaningful amount of crumple room to the structure, which could place the senior tranche at low enough risk of actual shortfall to attract conservative lenders like banks.

This structure can also provide the most bang for buck when it comes to Nvidia’s off-balance sheet obligations. While the AICP creates $59B of obligations per GW of capacity enabled and the PORTS-Pike structure requires $25B of obligations per GW, the residual value structure would only require $9.4B of obligations per GW enabled, though the datacenter leg still needs to be solved for.

While Nvidia has said the program targets over $500B of capital, we think it will likely swell larger due to the strong compute demand. In the spirit of examining the full extent of Nvidia’s maneuverability, we take an aspirational stance, assuming $2.5T of total funding under this program through F1/31, with the obligations under this line item ending F1/27 at $55B before growing to $373B by F1/31.

Source: SemiAnalysis AI Compute, Capital and Markets Model

Cloud Service Agreements

What is So Hard About Behind-The-Meter Power For Datacenters? Part 1

2026-09-10T06:28:20+00:00

Last year we were the first to call out Onsite Gas Generation as the primary method adopted by AI Labs and Hyperscalers to solve power constraints. Our positive view was far from being consensus: behind-the-meter primary power solutions have been called all sorts of names, such as “science experiments”, “Dark Gigawatts”, and “literally the dumbest thing that human beings have ever attempted to do”!

But since then, that supply chain has witnessed a massive acceleration. Our Energy Model now tracks 75GW of firm, binding orders in the supply chain only for for behind-the-meter AI compute - of which ~20GW alone ordered in Q2 2026. What started as an Elon Musk experiment is now mainstream for every single AI Lab and hyperscaler. To be clear, this data does not include the hundreds of GWs of speculative, baseless announcements that many other analysts track in their numbers - we only focus on binding orders received by OEMs specifically serving BTM AI compute, tracked at the project-level.

Sources: SemiAnalysis Energy Model; sales@semianalysis.com

The path from firm equipment order to delivered project is still long and challenging. There is substantial execution risk and that’s what we’ll focus on in this report. But the industry is more experienced than you’d think: by the end of the year, ~3GW of operational US datacenter IT capacity will be powered behind-the-meter, and that number will experience multiple straight years of triple-digit growth. Our Energy and Datacenter models account for all potential delays, as we’ve explained in depth in our piece Stop Saying Half of 2026 US Datacenter Capacity is Canceled.

Source: SemiAnalysis Energy Model, sales@semianalysis.com

Some of the most strategic projects developed by leading AI labs and hyperscalers are relying on behind-the-meter, supporting hundreds of billions of future revenue. Adoption has never been more broad-based. A few examples:

In 2026 year-to-date, Microsoft has signed over 5GW of behind-the-meter nameplate capacity, of which 2.7GW with Joulent & Chevron, and well over 2GW through turnkey datacenter leases with companies like Crusoe. That 5GW encompasses a broad range of different types of power equipment; full breakdown available to our Energy Model subscribers.
Google, historically the most reluctant to onsite gas, is deploying 930MW of off-grid aeroderivative turbines in a flagship campus in Armstrong County. In addition, the Search Giant will deploy 900MW of Bloom Energy Fuel Cells in Wyoming - as we called out back in February 2026 as a huge positive for Bloom Energy. They’ll be paired with >1GW of Mitsubishi J-class turbines.
Both Anthropic and Meta have signed 300-500MW deals with Enchanted Rock, a supplier of 0.5MW gensets built around a 21.9-liter V12 gas engine. Separately, Anthropic’s flagship campus in Texas, backstopped by Google, will also deploy over 1.5GW of off-grid generation; full breakdown of Anthropic’s exact datacenter facilities available to our Datacenter Model subscribers.
OpenAI will imminently start operations in its flagship off-grid 1.4GW (IT capacity) campus in Shackelford County, TX, using over five hundred 4.25MW Jenbacher J624 engines. We show below a portion of the campus. Combined with their 1.3GW IT site in New Mexico, that represents over $150B of contracted spending that OpenAI signed with Oracle relying on behind-the-meter power.

Sources: SemiAnalysis Datacenter Model; SemiAnalysis Energy Model; Sales@semianalysis.com

Why is this happening, and why have many energy experts been so wrong? It comes down to understanding AI economics. We’ve discussed this at length in many other articles and in our Tokenomics Model. As a quick reminder, the value of megawatts for end-users is skyrocketing. A power plant supporting an islanded 1GW IT datacenter typically costs ~$5B. In today’s environment, inference API revenue can yield $100B per GW per year, at 90%+ gross margins. Paying 2x more money or accepting 30% lower efficiency for faster speed of deployment becomes a no-brainer. Said differently, Anthropic and its peers can pay back the value of a power plant in 20 days of inference revenue. As we discussed six months ago, most of the value in the AI infrastructure stack is shifting to frontier model developers.

The grid simply cannot keep up with demand. Building new generation can easily take over five years, and on the demand-side, interconnecting a datacenter takes years. Read more in our article on US Grid Constraints.

Sources: SemiAnalysis Tokenomics Model; SemiAnalysis Inference Simulator; sales@semianalysis.com

That market context is overwhelmingly positive for behind-the-meter power to overcome such constraints. But these datacenter onsite power solutions are increasingly divergent from their grid-connected peers. As predicted a year ago, winners have not only been the incumbents like GEV and Siemens Energy. The biggest beneficiaries have been the dozens of suppliers of reciprocating engines, of various shapes and forms, with solutions from 0.5MW per unit to 20MW per unit being adopted at scale. A year ago, we counted 12 distinct manufacturers that had secured multi-hundred-MW datacenter off-grid orders; but today that count is 22 and will continue to grow. Our Energy Model tracks quarter-by-quarter manufacturing capacity, orders, deliveries, and availability for over 30 OEMs.

Sources: SemiAnalysis Energy Model; sales@semianalysis.com

Subscribe now

While the bright side of this is innovation, the flip side is the looming execution challenges facing the industry. Many of these things have never been done at this scale or speed. Cracks are starting to emerge. Permitting delays have caused high-profile sites, like Oracle Project Jupiter and Nebius New Jersey, to perform emergency pivots to less polluting alternatives (Bloom fuel cells). Pipeline delays have also impacted Oracle’s 1.3GW IT Project Jupiter (as our Energy Model called out in May 2026, well before the headlines). Market chatter of reliability issues is increasingly frequent. Labor shortages are surging. Design and up-time considerations are challenging and slowing down some FIDs. A secondary market for turbines is starting to emerge as a result of firm orders tied to failed projects.

In today’s report, we dive into the world of “datacenter microgrids”. We consider the challenges and doubts regarding the BTM buildout and look at how it is being done.

What even is ‘Behind-the-Meter’. Detailing the ways a datacenter can relate to the grid, between primary and back-up power; including when something is a phony microgrid. How does one define behind-the-meter?
How projects succeed or fail. We lay out the six key stages & challenges of BTM projects: contracts & bankability, permitting, fuel supply, equipment procurement, workforce, and the physics of operating without a grid-connection. We start with the financing wall, with chicken-and-egg situations being increasingly common for developers, and the rise of the new “BTM utilities”.
Where we go from here. ‘To grid or not to grid’; what happens to these plants when grid power finally arrives; export-to-grid, stay primary power, become backup, or move elsewhere. At the same time what happens with these order books and the manufacturing capacity that has scaled so hard to meet the moment. We’ll also give our view on the implications for BTM in Texas.

Over recent months we have published significant research for our Energy Model clients, laying out winners & losers in the equipment landscape (turbine, recips, fuel cells), contractors like AGX, BoP equipment like MV UPS systems and how new LVRT regulations could have negative impact to certain huge industrial companies. Behind the paywall of this report, we have included some of this work for Substack subscribers now to enjoy.

Conferences: in September members of SemiAnalysis’ Energy Model team will be at Gastech in Bangkok 14-17th; Data Center World Power in Dallas 21-23rd; Yotta in Las Vegas, and Gulf Coast Power Association in Austin, both 28-30th. Email energy@semianalysis.com and lets talk power!

_{We thank Michele Tarawneh from Celsius Industries for his valuable input to this report!}

What even is “BTM”?

A meter is a device that counts electricity; it is stationed at the point where the project’s wires meet the grid’s wires. Everything on the grid side of that point, the substations, the pylons, the power plants, is “front of the meter”; everything on the project’s side, the switch-gear, the batteries, and onsite generators, is “behind the meter”.

There are then specific terms but in practice they get used interchangeably: “behind-the-meter”, “off-grid”, “islanded”, “co-located”, and “micro-grid”. This is frustrating and leads to common disagreements over the definitions, but one way to think about them is by connection configuration:

Source: SemiAnalysis

Grid-supplied: The grid supplies the datacenter; on-site generation is backup only. Power plants supply the public network, which supplies the datacenter.
Grid-parallel: Local generation and grid imports can both supply the datacenter. Exports are optional depending on the rights and connections.
1. The relative sizes of the datacenter, its local generation, and its permitted grid imports can vary substantially. ERCOT’s Withdrawal-Limited Private Use Network, or WLPUN, framework expressly accommodates on-site generation reducing the transmission capacity a large load requires.
2. Consider three illustrative configurations, each serving a 1,000 MW datacenter:
  1. Grid-led: 250 MW of local generation and a 1,000 MW import limit. Local generation offsets part of grid consumption; when producing 250 MW at full datacenter load, it leaves 750 MW to import.
  2. Plant-led, full-sized import connection: 1,200 MW of local generation and a 1,000 MW import limit. The plant can cover normal demand, while the import limit is large enough to accommodate the entire datacenter load.
  3. Plant-led, limited import connection: 1,200 MW of local generation and a 250 MW import limit. At full load, at least 750 MW must come from local resources. Losing all local generation would require replacement supply or at least 750 MW of load reduction.
Export-only: Local generation supplies the datacenter; the grid connection permits exports but not imports to serve the load.
Off-grid: Local generation supplies the datacenter without an operating grid connection.

Net metering

In this context, net metering means netting the datacenter’s consumption against associated generation at the relevant grid-facing metering or settlement boundary. ERCOT describes it as reducing the customer’s metered consumption from the grid; it is not necessarily the retail rooftop-solar arrangement that credits exports against consumption over a billing cycle.

Freestone is an example: ERCOT reports PUCT approval in May 2026 of an arrangement between a 1,099 MW gas plant and a 760 MW datacenter in Freestone County.

Why Export-only is not Off-grid

The datacenter does not need a separate grid feeder to remain electrically connected. If its supply circuit connects into the plant’s grid-connected AC system, it is connected to the grid through the plant; the site participates in the interconnection’s frequency dynamics and shared inertial response even while exporting net power. Depending on how well run the grid is this arrangement can be beneficial to the datacenter with improved electrical reliability and quality.

Existing generation adds a regulatory distinction

Regulators can also distinguish new generation from a plant that previously supplied the grid. Texas Utilities Code §39.169, “Co-location of Large Load Customer With Existing Generation Resource,” covers certain net-metering arrangements involving an operating facility registered as a stand-alone generation resource as of September 1, 2025; it requires ERCOT notification and PUCT review, subject to exemptions and a deemed-approval provision.

Diverting existing output to a new load can reduce supply available to the wider grid. ERCOT says the two netting arrangements approved by May 2026 required the datacenters to reduce consumption or switch to backup, and the plants to return their full output to the grid within 30 minutes of an ERCOT instruction.

Operating states and deployment: island, bridge power, grid as backup

“Islanded” is an operating state: a grid-connected datacenter opens the breaker and runs on its own generation. Then related but separate, the Department of Energy defines a “Microgrid” as “a group of interconnected loads and distributed energy resources within clearly defined electrical boundaries that acts as a single controllable entity with respect to the grid,” and its Grid Deployment Office adds that a microgrid “can operate in either grid-connected or in island mode, including entirely off-grid applications”. In this piece, “islandable microgrid” means a grid-connected site that can separate; “off-grid microgrid” means a site with no connection at all. Many BTM projects use the term loosely, perhaps because it avoids saying “gas”.

In practice, this may change a lot over time. Many BTM projects plan for a grid connection and use BTM as a “bridge” - once the utility delivers the connection, the generators can shift into a backup role. We described this in depth the Onsite Gas Deep Dive as the most popular approach because electricity systems benefit from substantial economies of scale in both cost and reliability. xAI’s Colossus 1 in Memphis has taken this route.

Other projects plan for a grid connection that will serve as “backup”, i.e. as a new energy resource that will help increase the expected uptime. We’ve seen projects planning for 2 or 3 “nines” of expected uptime while in full island mode, and adding an additional “nine” once grid-connected. We’ll provide more thoughts behind paywall on how many BTM plants will become grid-connected in coming years.

Why is Building BTM So Hard?

A turbine order and a nearby pipeline do not make a project. Between the press release and first power stand six key hurdles: contract & bankability, permits, fuel, equipment, workforce, and electrical physics. Our permit-level tracking shows booked capacity facing difficulties at nearly every one of them.

Source: SemiAnalysis Energy Model; sales@semianalysis.com

Gate 0: Contract and Bankability

Below, we’ll discuss all the physical challenges in bringing online GW-scale islanded datacenters. But before getting there, there’s perhaps an even bigger challenge: the financial one. Traditionally, one can connect to the grid very cheaply, with both interconnection requests and letter of credits being very modest. This was enough to develop credible sites, even at GW-scale, and led to a frantic search for power in the US and worldwide. It was easy to turn a quick profit with limited capital, i.e. a few million at most. We discussed this at length in our Onsite Gas Deep Dive, and our US Grid Constraints articles.

Islanded datacenters are fundamentally different because there is no existing infrastructure to connect to. With power plants typically at ~5B per GW of IT capacity, and equipment deposits increasingly expensive, the upfront capital expenditure blocks many projects.

This leads to the classic BTM chicken-and-egg problem: lenders want a long-term ESA with a solid offtaker, and clear SLAs and other contractual terms. Offtakers want a credible timeline and project system design. Developers need early capital to pay for equipment deposits to provide that credible timeline and design. And with the insatiable demand for gas generation equipment, not having that capital means that timelines keep slipping.

Source: SemiAnalysis

Early successes have come in three forms:

Full vertical integration, the Musk playbook: as discussed at length in our Colossus 2 article, xAI built its own power plant for self use, seeing it as the fastest way to build compute. This was most likely equity-funded.
Full-scope datacenter developers: certain developers have both construction and energy in their core capabilities, and take responsibility for the BTM power plant when selling to hyperscalers. In a Yield on Cost model, they typically include the onsite power in the cost envelope - which then commonly crosses the $20M/MW mark. Full site-level details in our Datacenter Industry Model.
Datacenter + power partnership: datacenter developers go-to-market together with power developers to sell a site to a hyperscaler. But the hyperscaler signs two separate contracts: one for the datacenter, one for power. Certain Oracle Stargate sites are built on such arrangements. The joint go-to-market means that the offtaker is offered a full-stack solution. But the main drawback is dealing with more parties, and facing a risk of paying stranded expenses if either the power or the datacenter are late - given the contracts are independent of each other.

For BTM power, all of that complexity has led to the development of a new type of service providers: Energy-as-a-Service (EaaS) vendors. These are the new “BTM utilities”. Companies such as VoltaGrid deliver power, not just equipment, under long-term contracts with guaranteed capacity and up-time commitments. EaaS can do everything from leasing gensets to trucking compressed natural gas to site as ‘virtual pipelines’ while permanent power plants and actual pipelines are permitted, built, and commissioned.

VoltaGrid has proved to be one of the winners of that trend, landing multiple gigawatts of orders with top tier developers like Vantage Datacenters. Others like Williams, Solaris are also on the rise and have scored massive contracts. Separately, some manufacturers are increasingly doing that themselves, such as Bloom Energy and Enchanted Rock. We track all of the supply chain, EPCs, BESS/SynCon and such for hundreds of projects in our Energy Model.

Source: VoltaGrid

Does this capital wall mean that BTM is structurally disadvantaged versus the grid? It used to, but it’s not anymore. US Grid constraints are playing out in real-time, in two ways:

With firm US grid headroom being exhausted, serving a new GW-scale datacenter requires GW-scale generation paired with it. Capital requirements are commonly in the multi-billion-dollar-per-GW range today. Cheap GW-scale interconnections do not exist anymore.
Most utilities are notoriously slow and unreliable, and grid ESAs typically aren’t as binding and don’t have as many penalties as a BTM contracts in case of late delivery or uptime issues. It makes sense, they’re very different contracts in nature. As such, for GW-scale projects where speed and credibility of execution matters, BTM is an increasingly competitive play.

Therefore, we view the financing environment increasingly favorable towards BTM. In addition, datacenter developers are getting more sophisticated on energy and building their own world-class teams, and increasingly offering a full turnkey scope to offtakers, including the BTM plant.

Stage 1: Permits

The State Decides

With that said, let’s now tackle the physical challenges, starting with permits. We warned in the Onsite Gas Deep Dive that getting an air permit for onsite generation can take a year or more even in the fastest states, and that permits were already delaying some projects. In this section, we outline the varying permitting intensity by state, and how some developers have come up with creative solutions to circumvent permitting blockers.

The first decision is therefore where to file. Finding favorable regions is paramount, and if you are not sure then you can hedge your bets by locating on a border. Colossus 2 sits on Tulane Road in Memphis, Tennessee, a few hundred meters north of the Mississippi border. Its power plant sits on the other side, on a former Duke Energy site in Southaven that an xAI affiliate bought in July 2025. That moved the air permit out of Shelby County, which had permitted the Colossus 1 turbines, and into Mississippi. Mississippi let the turbines run for up to 12 months as “temporary” units with no air permit, and in March 2026 its Permit Board unanimously approved a 41-turbine, 1.2 GW plant to make them permanent.

The wider lesson is that developers now choose jurisdictions almost as carefully as they choose equipment. The same plant can face a standardized registration in one state, a case-by-case air permit in another, and separate air, siting and generation approvals in a third.

Sources: SemiAnalysis Energy Model; Sales@semianalysis.com

That asymmetry between states, as much as cheap gas, explains why Texas is set to host more of the BTM fleet than any other state. The state determines how the process plays out but federal rules set most of the parameters. The Environmental Protection Agency (EPA) sets the core air-quality standards, emissions definitions and major-source framework; state and local authorities usually administer those rules, issue the permits and add their own authorization routes.

How the Air Permit Thresholds Work

Potential to emit (PTE) is the tonnage a plant would emit running flat out all year, counted per pollutant. Every US air permit tests PTE. For a gas generator the two pollutants that matter are nitrogen oxides (NOx) and carbon monoxide (CO). Two federal lines apply. A plant with PTE above 250 tons a year of any pollutant goes through a full major-source review: modeling, a public process, and potentially years added to the timeline. A plant with PTE above 100 tons also needs a Title V operating permit, a second layer of federal oversight. States set their own lines below the federal ones.

Tiers:

Exempt as a nonroad engine: A portable engine that stays on site under 12 months
1. Sits outside stationary-source permitting. Works for reciprocating engines only: any turbine of 10 million Btu an hour or more falls under a federal emission standard, confirmed by EPA’s January 2026 turbine rule. Mississippi exempted xAI’s trailer-mounted Southaven turbines under a state rule; that decision is being challenged in federal court; 40 CFR 1068.30.
Minor New Source Review (NSR): PTE below every major threshold
1. A construction permit from the state or county, which sets the exemption cutoffs and how much modeling is needed. Public comment of at least 30 days. A plant that accepts an enforceable cap on hours or fuel to stay under the lines is a “synthetic minor”; changing that cap later triggers a full major-source review as if the plant were new; 40 CFR 51.160 to 51.161; 40 CFR 52.21(r)(4).
Prevention of Significant Deterioration (PSD), the major-source review: PTE of 250 tons a year of any pollutant. 100 tons for 28 listed industries; EPA reads “steam electric plant” to cover combined-cycle plants, while simple-cycle plants are treated as unlisted and so face the 250-ton limit.
1. Usually issued by the state. Once major for one pollutant, the plant applies Best Available Control Technology to every pollutant above the significance rates (40 tons of NOx, 100 of CO), models its air-quality impact, usually supplies a year of monitoring data, and takes at least 30 days of public comment. The Clean Air Act requires a decision within a year of a complete application (section 165(c)); no national data on actual times exists; 40 CFR 52.21.
Nonattainment NSR: The area fails a federal air standard for the pollutant. It replaces PSD for that pollutant; PSD still applies to the rest.
1. The limit falls to 100 tons a year, then to 50, 25 or 10 tons of NOx and VOC as an ozone area is rated serious, severe or extreme (Clean Air Act 182). The plant must meet the Lowest Achievable Emission Rate and buy offsetting cuts from other sources in the same area, at 1.1 to 1.5 tons per ton. Location sets the line: Dallas and Houston sit at 25 tons, Chicago, Salt Lake City and Las Vegas at 50, Northern Virginia and Phoenix at 100 (EPA Green Book); 40 CFR 51.165.
Title V operating permit: PTE of 100 tons a year or more, on top of whichever construction permit applies.
1. Applied for within 12 months of commissioning. Gathers every requirement into one enforceable document with a five-year term. EPA gets 45 days to object and the public 60 days to petition. Adds monitoring, reporting and fees, and sets no new emission limits of its own; 40 CFR 70.2.

Three caveats.

The 250-ton line for simple-cycle plants rests on consistent agency practice, including Mississippi’s own xAI permit, not on a written EPA ruling.
States differ below the federal lines. Texas requires control technology for every new facility. Virginia permits anything above 40 tons of NOx. California districts trigger controls at 10 pounds a day.
Federal emission standards apply whatever the permit tier. The January 2026 rule sets a 5 ppm NOx floor for new large turbines running base load. That means selective catalytic reduction from day one.

Sources: SemiAnalysis Energy Model; Sales@semianalysis.com

Small emissions rates become big annual totals at campus scale. At an illustrative controlled NOx rate of 0.10g/kWh, 250MW × 8,760 hours produces about 241 US short tons a year; the same calculation reaches 250 tons at roughly 259MW. The actual answer depends on the equipment’s controlled emissions, the applicable threshold and which units the regulator counts together as one source.

That leaves developers working on both plant size and the operating plan. A phased authorization can, where the project and rules allow it, bring an initial tranche online while approval for the larger build progresses. But the hours and emissions limits must survive contact with the intended operating schedule and be enforceable. The regulator needs a credible plan for how the whole campus will run.

Project Jupiter: Grand Designs

UK television legend and architecture critic Kevin McCloud would be raising his eyebrow at Oracle’s Project Jupiter because it shows how permitting can force the redesign of an entire facility at breakneck speed.

The original applications were filed on 17 November 2025 for two separate microgrids, East and West, each sized to sit just under the federal 250-ton major-source line. The West application proposed 34 GE TM2500 and 20 Mitsubishi FT8 units (54 turbines, roughly 1,677 MW) with the East microgrid at a higher scale. The strategy was to operate as a synthetic minor: install equipment capable of emitting well above the major-source threshold, then accept operating limits that keep permitted emissions just beneath it. The East microgrid reported the potential to emit of 521 tons of NOx a year and requested a cap of 248.90, 1.1 tons under the line.

Sources: SemiAnalysis Energy Model; sales@semianalysis.com

New Mexico’s Environment Department determined that limits set so close to the threshold were not practically enforceable. The applicant resolved that by dropping both caps to 245 tons, and the department ruled both applications complete and published its intent to issue - but some 7,155 public comments came in, the Secretary concurred in holding a hearing, and the applicant withdrew both on 27 April 2026, the day it announced the fuel-cell redesign.

The project then abandoned the turbine design and refiled around Bloom Energy solid-oxide fuel cells in April. The revised proposal targets roughly 37 tons per year of NOx, around 92% below the original turbine configuration.

The technology change did not make the permit issues disappear. On August 23, 2026, the New Mexico Supreme Court stayed the administrative air-permitting proceeding until further order. The stay paused the process rather than deciding the merits of the application, and on September 1 the justices unanimously refused the developer’s request to lift it even temporarily. The regulator’s decision deadline is November 23, 2026.

The Fuel-Cell Escape Hatch

Fuel cells offer an escape hatch from the trickiest part of permitting but not from the permit process itself.

The technology generates electricity electrochemically rather than through flame combustion, generally producing far less NOx and other conventional pollutants than a comparable fleet of turbines or reciprocating engines. Depending on the site and configuration, fuel cells can keep a project below PSD major-source thresholds and allow it to follow a standardized or minor-source route instead. They still require permits and still produce CO₂ and smaller quantities of other pollutants, but the advantage here is a lighter regulatory profile, not zero emissions.

AEP Ohio’s 72.9MW Bloom installation at AWS’s Hilliard site in Ohio shows the wrinkles in permitting fuel cells, remaining below the thresholds that would have triggered the major-source route. The project still required an Ohio EPA air permit and separate state siting approval, while the air permit was appealed by the City of Hilliard in November 2025.

Nebius also made the technology trade in its first BTM US deployment, replacing the previously planned combustion-based generation with 328MW of Bloom fuel cells under a 10-year power agreement, with the company pointing to faster deployment and the lighter permitting burden as advantages of the switch.

Fuel cells can accelerate projects through the permitting matrix but can also make speculative projects easier to decorate. In one Texas filing we found, a proposed 560MW Bloom fuel-cell plant paid a $900 standard-permit fee and moved from application receipt on December 19, 2025, to authorization on January 14, 2026, TCEQ concluding that the installation was not a major source under either PSD or Title V.

A standard permit confirms that a proposed equipment configuration fits a pre-written emissions envelope. It does not prove that the datacenter has a committed tenant, financing, purchased equipment, firm gas supply or a construction notice to proceed. That is why we do not treat “permitted” as a greenlight for a new BTM project.

Source: SemiAnalysis Energy Model

Note that another often overlooked constraint for permitting is water. For radiator-cooled recips, process-water demand can be very low: the radiators dump engine heat into the air. But for certain single cycle turbines, water or steam injection for NOx control creates a continuing water requirement, while dry low-emissions (DLE) combustion avoids that injection. The turbine package matters as much as the technology label. For example, the Pratts FT8 MOBILEPAC use water injection.

Combined cycle adds another decision. Wet recirculating cooling loses water through evaporation and blowdown, the discharge used to keep dissolved material from building up. Dry cooling rejects that heat to the air.

Stage 2: Gas

Now, while permits are a core site selection constraint, fuel adds a second one. The US is awash with natural gas: high oil prices maintain the incentive to drill baby drill, and the gas that comes out as a by-product needs to find a home somewhere, while Appalachia and the Haynesville add ample supply of dry gas (not associated with oil extraction).

However, the hard part is transportation of that gas. Building pipelines can be simultaneously the hardest and easiest aspect of the BTM datacenter construction process, but like permitting, it depends quite significantly on geography. In addition, one pipeline typically isn’t enough. A firm contract is tied to a maximum daily quantity, specific receipt and delivery points and, where agreed, minimum delivery pressure; interruptible service can be curtailed when the pipe tightens. A line on the map is therefore weak, what matters is how much firm gas can physically reach the plant’s meter. The physical fuel system also has to survive a failure. Depending on the site, that can mean independent feeds, redundant internal distribution, stored LNG or dual-fuel capability, plus redundant boost compression where inlet pressure requires it. Gas quality matters too: changes in methane number, heating value and Wobbe index can force control changes or derating if they move outside the machine’s fuel envelope.

A couple of examples of large projects:

To build a pipeline in Texas, you only have to answer the Texas Railroad Commission, meaning datacenter developers in the state are increasingly just building the pipeline themselves. Federal water-crossing permits and easements still apply, and a purely private lateral gets no condemnation power, so a single holdout forces a reroute. Crusoe filed by itself for the 16-inch line to Duroc power plant at Abilene, and Trailblazer Infrastructure - the Sweetwater developer, unrelated to Tallgrass’s Trailblazer system - budgets $120M for a 31-mile extension in its tax-increment finance plan. Add this to ERCOT’s connection rules and some of the cheapest gas in the country, and you have another explanation for why so much of this build-out is planned in Texas.
In Louisiana, Energy Transfer subsidiary ETC Tiger Pipeline is building the Franklin Farms lateral - 13.2 miles of 36-inch pipe at a billion cubic feet a day, plus a shorter branch - to feed Entergy Louisiana’s new combined-cycle plants, which in turn power Meta’s Hyperion campus over the grid. Entergy, not Meta, is the shipper, so this is the utility-served route rather than a behind-the-meter one. Even with no compression, it still runs 19 months from the federal filing in July 2026 to a February 2028 in-service date.
New Mexico. Again we arrive at Oracle’s Project Jupiter. The campus is set to received pipeline gas from the Transwestern-built Green Chile pipeline. That pipeline was contractually due in service on 15 August this year, but the date passed without even an approved route, let alone a start to pipe laying. On 14 August Transwestern filed a new in-service date of 1 February 2027, with the federal apparatus moving unusually fast. This speed will need to continue if that start date of February next year is to be met, with a FERC order required in the next two months. No comparable certificate on record has been issued that fast, and none of the 33 comparable projects we have tracked was approved before the public comment period closed. Our estimation is first gas in April or May 2027, with full ramping up to the 2.45 GW in mid-2028, however, there are several hurdles to overcome before that.

The fuel contracts and pipeline filings can also help separate wheat from the chaff. A nearby pipeline doesn’t prove much, but a firm transportation contract, signed gas supply agreement, or a pipeline application filed at the regulator can help understand which sites may be on the way to receiving gas for their site.

Some interesting observations:

Energy Transfer contracted 150 MMcf/d to the Nexus Hubbard campus, enough to run about 750 MW of the reciprocating engines actually ordered. But the campus’s pending air permit covers a 5,230MW fleet, meaning it needs about seven times the contracted gas supply , so either there are more gas contracts are coming, or permitting was a hedge ahead of contracting gas supply.
Liberty Energy’s joint venture with PowerBridge covers a planned 2 GW campus in West Texas, with more than 300 MW of generation in a first phase targeted for the fourth quarter of 2027. Liberty told investors on July 22, 2026 that the site would start behind the meter, likely inside ERCOT’s Batch Zero, and could take a grid connection of undetermined size in 2028. Most of the campus would then sit behind a private use network, the ERCOT structure that nets a site’s generation against its load at a single connection, which we explain in US Grid Constraints.

Stage 3: Equipment

The market for turbines, reciprocating engines and other types of generation equipment has grown significantly so far this year, with new manufacturers coming into the fold and existing OEMs increasing manufacturing capacity to keep pace with the growing demand for onsite power generation.

In our Onsite Gas Deep Dive we went through the new types of technology and incumbent manufacturers, so you can go through the different types of equipment available to datacenter operators in that article.

Since writing in December 30, 2025 article, the three broad categories have remained unchanged:

Gas turbines (GTs):
1. industrial gas turbines (IGTs), smaller, heavier machines in the tens of megawatts;
2. aeroderivatives (aeros), jet engines repackaged for power in the 30 to 115 MW range and able to ramp in 5 to 8 minutes; and
3. heavy-duty frame turbines of roughly 400 MW, with increased fuel efficiency when put into a combined cycle with steam turbines.
Reciprocating internal combustion engines (RICEs), or “recips”:
1. high-speed engines (1,500 to 1,800 rpm) of roughly 0.5 to 4.5 MW per unit, such as INNIO’s Jenbacher Type 6 and Caterpillar’s G3520; and
2. medium-speed engines (500 to 1,000 rpm) of roughly 5 to 20 MW, such as Wärtsilä’s 34SG, 31SG and 50SG, INNIO’s J920 and Everllence’s 51/60G.
Fuel cells: solid-oxide units from Bloom Energy carry nearly all the orders (3.8 GW of the tracker’s fuel-cell bookings against 0.03 GW for FuelCell Energy).

While turbines captured most of the initial orders, they very quickly ended up being sold out, with first slots now past 2030. This is because BTM developers compete with utilities for these turbine slots. However, recips are much less popular among utilities and they are increasingly aking to AI compute pure-plays. That’s why our quarter-by-quarter tracker of firm orders for BTM AI compute has seen sign a drastic shift towards recips.

Sources: SemiAnalysis Energy Model; sales@semianalysis.com

And we expect the trend to continue. Folks like EROCK, FTAI Power and Dynamis, alongside INNIO, Wartsila, Caterpillar and Cummins, and Bloom Energy, are expanding capacity more aggressively than incumbents like GEV, and directing more supply to the datacenter market. Caterpillar is lifting its large reciprocating-engine output to nearly 3x 2024 levels, while INNIO expects to roughly triple total capacity from 3.5 GW a year in 2025 to around 10 GW a year by 2030.

We have also seen the emergence of a secondary turbine market. They don’t come cheap, though! The scarcity across the main OEMs has built a secondary market where the premium buys time. A released delivery slot trades at a real premium to a factory order - the clearest public mark is a slot assignment struck 16% above the fixed price, and 46% above once all costs are loaded - and what the premium buys is a slot delivery in the next couple of years (or sooner) rather than in the next five years, and quicker energization.

Sources: SemiAnalysis Energy Model; sales@semianalysis.com

Key to chat secondary market is the presence of speculative orders in the supply chain. Folks commit to buying turbines or sign minimum purchase agreements, and pay deposits, yet don’t have actual datacenter sites to deploy them. We’re seeing that behavior primarily impacting the gas turbine market, notably incumbents GEV and Siemens. That equipment remains premium today and in high demand, and some “speculators” have been nicely rewarded given the secondary market prices discussed above.

Source: SemiAnalysis Energy Model, sales@semianalysis.com

No BTM Plant Without Balance of Plant

While generation equipment is typically the center of the conversation when discussing BTM constraints, we think that’s increasingly shifting to BoP. Generation equipment will be useless if the campus waits for transformers, medium-voltage switchgear or the e-houses that hold the electrical equipment. The collection system and controls also have to bring the fleet’s output together into a working plant. And this electrical equipment, often at medium voltage and sometimes high voltage (depending on plant size and system design), is in very high demand due to being standardized and often used by datacenters themselves, as part of their electrical systems (see our deep dive on the topic here).

Sources: SemiAnalysis Datacenter Model; SemiAnalysis Energy Model; Sales@semianalysis.com

Stage 4: Construction, Commissioning and Workforce

Once capital, permits, gas and equipment are solved, a whole other challenge arises. Building the power plant, and operating it at the contractually defined SLA standards needs highly qualified labor, contractors and EPCs. To be clear, folks calling this a “science experiment” are entirely missing that 3GW of Datacenter IT capacity is already running with behind-the-meter power plants. Our Datacenter Model and Energy Models track every single project and exact timelines and supply chain.

Source: SemiAnalysis Energy Model, sales@semianalysis.com

But as the scale dramatically increases, there will be issues, particularly in the context of surging labor scarcity driven by the datacenter buildout.

Our labor market modeling in The Wild Wild West of LEGO Datacenters sized the 2027 gap at 288,000 workers, based on our Labor Model included in our Industrials Model. Electricians are the largest trade at about 30% of site hours, drifting toward 40% by 2030 as liquid cooling adds mechanical work.

Moving the mechanical and electrical work into a factory cuts on-site labor by about 63% and on-site licensed-electrician hours by about 85%, which is why prefabricated eHouses, skidded plants, and full modular offerings are spreading through the vendor list: they are labor arbitrage as much as an opportunity to shoot ahead of schedule.

But plenty of labor is still needed onsite. When it comes to power plants, the US has been building them for decades, meaning a healthy labor supply pool and lower costs than the datacenter build. Large combined cycle gas plants peak at about half a construction worker per megawatt:

Palomar, 546MW in California, peaked at 283 craft
Guernsey, 1,875MW in Ohio and fifteen years later, at close to 1,000

So, on a gigawatt campus the power plant build can be a fifth to a quarter of the peak headcount.

However, it is not the same labor. A CCGT plant is a mechanical pressure project, meaning boilermakers setting HRSG pressure parts, pipe welders whose welds have to pass radiography, and millwrights aligning a turbine to thousandths. There are 10,200 boilermakers in the United States, the occupation turns over about 800 people a year, and the Bureau of Labor Statistics projects it to shrink.

That said, the BTM buildout is set to use a variety of solutions and each generation type draws on different trades, and each have their pros and cons regarding labor.

Reciprocating engines need no boilermakers, but they lean on electricians and millwrights, so the power plant competes with its own datacenter for the same people.
Fuel cells need none of the scarce trades: no pressure parts, so in a labor-constrained buildout, that is a strong argument. But at mutli-gigawatt scale, hundreds of fuel cell banks and the required supercapacitors, BESS and so on, make for a different kind of configuration which has never been done at scale.

This is why “labor-light” power solutions are increasingly attractive to end-customers. As the US grid runs out of power & headroom in 2028, these solutions will be the primary way to solve datacenter power constraints.

Of course, commissioning is not the finish line. A permanent island needs daily operating coverage, remote monitoring, local response, scheduled maintenance, spare parts and service coverage for unplanned work. The cadence is measured in operating hours. Caterpillar illustrates a 20,000-hour top-end and an 80,000-hour major overhaul; Jenbacher packages can require major work at 60,000 or 80,000 hours. At 8,000 running hours a year, that is roughly 2.5 years to the Caterpillar top-end and 7.5–10 years to a major. The exact answer varies by engine and duty cycle.

To counter both issues, some OEMs are increasingly going vertical. Bloom Energy is the main example, as their solution is of course not as broadly established than turbines and engines. Bloom takes care of most installation and O&M work for its deployment, and hires and trains staff to do that. Others are following, and both issues are increasingly a deciding factor for equipment orders. EROCK has also relied on this, and their use of smaller truck-derivative engines means that both installation and O&M are much simpler than for a traditional turbine-based power plant. PROENERGY also discloses a large in-house staff that can act as EPC and O&M if need be.

Stage 5: Islanding Physics

The last stage is the least visible and the least forgiving. At xAI’s Colossus 1, matching gas turbines to GPU load proved harder than expected, with jitters of 10 to 20MW several times a second and oscillations like these exciting torsional modes and eating up turbine-generator shaft life. This issue led to xAI deploying the first phase of 150MW Tesla Megapacks. We’ve discussed at length here the issues with AI Training load fluctuations. This has led to the rise of a new battle: BESS vs Flywheels / SynCons to be paired with generation equipment at BTM sites.

Source: SemiAnalysis Energy Model, sales@semianalysis.com

The combustor technology that makes modern gas turbines permittable holds its emissions performance only above roughly half of rated load, or about 35% with turndown upgrades. AI training loads, which swing hard and fast, can push turbines below that floor, where controls add pilot fuel or revert to diffusion mode and NOx, CO and unburned hydrocarbons all jump. A permit is written based on potential emissions, so that floor is integrated into the application.

A grid-connected plant borrows two services from the system: inertia, the spinning mass that resists frequency change, and fault current, the surge that lets a breaker recognize a short circuit and trip. An islanded plant must make both itself. Engines and turbines deliver five to ten times rated current into a fault; batteries, solar and fuel cells are clamped by their electronics to roughly 1.2-2x, thin enough that conventional overcurrent protection loses its margin. An island built only on inverters can fail to clear its own short circuits.

Fuel cells answer the same question with supercapacitors, which dump into a step load and recharge as the stack ramps. Channel checks for Bloom’s sizing ratio put it near 3:2 on a power basis, which would make Project Jupiter’s 2.45GW imply about 1.6GW of supercapacitors, far above the existing precedent of 58MW.

Can You Do This Without Gas?

Plenty are trying, and after six stages of difficulty you may well think renewables could dodge the permitting, fuel, and supply chain bottlenecks. Some are already trying, notably Crusoe with Redwood. Others like Google are deploying renewables & BESS at very large scale, but these sites remain grid connected.

Renewables clear air permitting far more easily, since there is no combustion source to permit, the fuel costs nothing, and there is no pipeline to route. The catch is electrical physics. In an off-grid context everything connects through inverters. Inverter-based sources push only a thin multiple of rated current into a short circuit, a fraction of what an engine or turbine delivers, too little margin for conventional overcurrent protection to trip reliably. So a clean renewable island has to buy grid-forming inverters plus synchronous condensers.

Our December 2025 onsite-gas piece measured average US grid availability at about 99.93%, roughly three nines, and showed that onsite supply has to be overbuilt to match it.

Crusoe and Redwood Materials’ operating datacenter microgrid in Sparks, Nevada has run since June 2025 on 12 MW of solar and 63 MWh of second-life electric vehicle batteries, with the grid as backup. In a March 24, 2026 release Crusoe reported 99.2% availability over seven months of continuous operation and announced an expansion to 20 MW. Scaling a solar-and-battery design to a gigawatt campus requires substantial generation capacity, storage and land; the amounts depend on the load profile, weather, reliability target and availability of grid backup. Batteries earn their place inside gas projects: ride-through, load smoothing and backup energy. Liberty Energy’s chief financial officer said on July 23, 2026 that battery power capacity averages 50% of gas capacity across its datacenter clients.

Buyers are becoming more tolerant of interruption. Microsoft says its Fairwater Atlanta site was selected for resilient utility power and is “capable of achieving 4x9 availability at 3x9 cost,” four nines of availability (99.99%) at the cost of three nines (99.9%), which lets it forgo onsite generation, uninterruptible power supply (UPS) systems and dual-corded distribution for the GPU fleet (Scott Guthrie, Microsoft, November 12, 2025). In our Industrials Model, we track the expected uptime for 30+ different datacenter designs, based on our deep bottoms up equipment-by-equipment BoM. We track desings of Anthropic, OpenAI, Meta and all other hyperscalers and we are seeing a clear trends towards lower redundancy.

Currently, no pathway clears all six gates: gas is exposed on permits and on the pressure trades (the boilermakers and pipe welders), fuel cells on stability spend and unproven scale, solar and batteries on the island itself.

Where do we go from here?

Let’s now discuss the longer-term outlook and winners&losers. We’ll discuss our outlook for BTM plants and whether they are all destined to be grid-connected or not, and how solar and batteries fit into that outlook.

We’ll also discuss winners and losers in the OEM market, and in the BoP market. Some big industry changes are negative for certain large industrial companies, but there are balancing factors within their portfolio.

Where Does a Robot Think – On-Device vs Datacenter Inference

2026-09-09T12:53:35+00:00

For most of its short history, AI lived behind a screen. That’s starting to change.

First came chatbots, good for answering a question or drafting an email. Then agentic AI, models that don’t just respond but do real work on a computer: navigating software, calling tools, finishing multi-step tasks on their own. Now the frontier is physical AI, intelligence that reaches past the screen to perceive the world and act on it. The biggest piece is robots, and it is still early: the hardware, the models, and the economics are all being worked out right now.

The Embodiment Problem

With LLMs, the hardware bends to the model. For hyperscalers and AI labs, there’s no hard latency requirement and a very high spending ceiling, so the process is simple: pour in as much data and compute as possible at training, then figure out how to serve the model that comes out. The model comes first; the hardware serves it. Robotics inverts this. Tighter constraints flip the order of design: you first work out what it takes to serve a model on the robot, then build a model that fits. Two constraints drive this.

The second is cost. When you use an LLM, it lives behind a screen provided by the user. In robotics, the manufacturer has to build both the compute and the robot itself and pay for it upfront, on every unit, and the capital expenditure can be immense. At scale, this upfront cost can be billions and even in the trillions if we do eventually get to a billion robots.

Because of these two constraints, in robotics the on-robot hardware is fixed, and the model is designed to fit it. Even at hyperscaler funding levels, most of the capital will have to go into the robots and factories themselves, not into compute. Model capability is therefore capped by what can run on affordable, real-time hardware on the robot, and intelligence becomes something you ration against latency and unit economics.

Robot Models Are Still Small…

This is why frontier robot models remain far smaller than frontier LLMs, which have already pushed into the trillions of parameters. Today most frontier robot models sit between roughly 5 and 14 billion parameters: Physical Intelligence’s π0.7 at 5 billion, and NVIDIA’s DreamZero at 14 billion.

Source: SemiAnalysis

In LLMs, scaling laws have held, scaling has consistently broadened the range of problems a single model can solve. Early results show robotics might work the same way. Data has been a constraint but that is now scaling with World Models and Embodied human data collection. Embodied AI faces limits language models never did, like scarce data and the cost of real-world interaction, so whether scaling pays off the same way is genuinely an open question.

If robot models do continue to scale, the question becomes where should all that compute run, given the latency, cost and model limits mentioned above. The field is genuinely split, with different companies making diverging bets. Figure runs its in-house Helix model entirely onboard the robot. Physical Intelligence’s π0.7 model goes the other way and runs its policy off-robot on a single H100. NVIDIA’s DreamZero, a 14-billion-parameter “world action model” built on a video-diffusion backbone, needs two GB200 GPUs off-robot just to run in real time.

Our view is that a cascade is inevitable. Some robots will run cognition on-board, while others will offload it to GPUs sitting in a datacenter. The biggest robot model builders today, like NVIDIA and Google DeepMind, will lean toward this cloud approach, running the heavy cognition on cloud GPUs, with only a small edge model on the robot for low-level control and safety.

Smaller builders, such as Dyna and Sunday, may initially favor models that fit on a Jetson or consumer GPUs because their early deployments operate within narrow, controllable environments where onboard inference is sufficient. As deployments scale, however, they will converge on whichever mix of onboard and off-robot compute delivers the best performance and economics for their business model.

Moving the GPU off the robot brings real benefits, but also real challenges. Chief among them is latency, and it doesn’t work everywhere. In environments with poor connectivity, cloud deployment isn’t an option at all. Despite these difficulties, off-robot compute remains beneficial, and in some cases necessary, as the following sections show.

Models in Robots

Before we get into why the robot model should run in an off-robot datacenter, here’s a quick primer on how robot models work, and which ones we’re talking about.

Robot control is a broad space. Hard-coded control, where engineers write out the rules and math for every movement, has run factories for decades on embedded compute, and small task-specific policies fit onboard just fine. That’s not our subject. We’re talking about data-driven generalist robotics, robots run by large foundation models, because that’s where the frontier is.

Even within data-driven generalist robot models, architectures vary. Some are monolithic: one model that goes straight from what the camera sees to how the motors move. Many are hierarchical: a big, slow model decides what to do, and a small, fast one turns that into movement, hundreds of times a second. Our argument for off-robot datacenter compute is about these hierarchical models, where the brain separates into two or more layers, most often a planning layer and an action layer.

The planning layer is the slow, high-level one: it perceives the scene, reasons about space, and plans the task. The action layer is the fast, low-level one: it turns those plans into motor commands. The planning layer, which is also the heavy part of the model, is the more forgiving: it updates only a few times a second and runs asynchronously from the control loop, so the action layer keeps acting on the most recent plan while the next one is still in flight. That decoupling lets planning absorb the delay, and even the jitter, of a wireless round trip, while the action layer runs at hundreds of Hz and can tolerate none of it. So planning is the natural candidate to move into the datacenter, while the fast control loop stays local.

Source: SemiAnalysis

The amount of planning scales with how general the robot’s job is. A generalist built to handle open-ended tasks needs a large planning model doing genuine reasoning, and those models are heavy. That reasoning is exactly what gets pushed to the datacenter. A specialist is the opposite case: a narrow policy doing one repetitive job carries little high-level planning, so the whole model is small enough to stay on the robot.

Why DC GPUs are better suited…

Specialist robots are for the most part solved: factories have run on them for many years now using hard-coded automation, but not modern AI. But a specialist is rigid, one task at one station, idle when the bottleneck moves. Generalists are the frontier because one flexible robot can cover many jobs, redeploy to wherever is bottlenecked, a changing product mix, and the long tail of unstructured work. This aligns with some of the many lessons of the Toyota Production System, where we learned that multi-skilled workers can reduce waste and improve economic efficiency. As a result, the TAM for generalist robotics is huge, not just in manufacturing and warehousing but also in homes and other places.

The more generality you demand, the more you lean toward FLOP-hungry foundation models and datacenter-scale compute. Not every robot needs maximal generality, and many tasks are served well by smaller models, but the frontier of flexible, general-purpose robots is where datacenter-scale compute becomes necessary.

LLMs vs Robot Models Token Flow

An LLM ingests and emits huge, bursty token counts: a long prompt to prefill, then a long generation, with a KV cache that grows the whole time. That is the memory wall, the capacity to hold that cache and the bandwidth to stream the weights and cache on every token, and it makes LLM serving bandwidth-bound.

A robot model is the opposite, a smaller steady stream, more metronome-like rather than a burst. It never stops, because a robot is not answering one question and halting; it is locked in a closed loop with a moving world, taking in a fresh frame and emitting a short action chunk a few times a second. Each frame is compressed in dedicated hardware (the ISP and video-codec ASICs) into a fixed, compact token budget of tens to a few hundred tokens. So robot models are not as memory capacity and bandwidth bound in the same way that LLMs are.

Source: SemiAnalysis

Frontier Robot Models are FLOPs hungry

What binds a frontier robot model sometimes memory, but often it is the FLOPs to run it.

These are FLOP-hungry workloads: a VLA (Vision Language Action model) is a multi-billion-parameter model, orders of magnitude larger than the specialist policies before it, and its vision and language stages, which are the bulk of the work, are compute-bound on any capable GPU. A WAM (World Action Model) is heavier still. Instead of naming an action, it generates video of the predicted future and derives the action from it, through iterative denoising, meaning one action could take multiple full forward passes through a model.

Edge silicon cannot keep up. Nvidia’s latest Jetson Thor, the best-in-class robot brain, has only about a tenth of a single B200’s compute, and roughly a fourteenth of a B300’s. DreamZero (a 14B WAM) needed two GB200 GPUs just to reach ~7Hz, which is on the order of twenty times the compute a single Thor can muster. Run the same model on a Thor and it would fall to a fraction of 1Hz, far too slow for real-time control.

Source: SemiAnalysis

Shipping the Robot Model Off Device

If a Jetson Thor can’t keep up, and we off-load the heavy, compute-intensive part of the workload to datacenters, it means optimizing local compute for safety loops and high-frequency control loops, and doubling-down on network interfacing.

The primary consideration is the glass-to-glass latency budget we need to hit, and hit reliably. The network will produce issues with bandwidth and stability, and we will feel this the most when transmitting image data.

Transmitting image data quickly and reliably is more than just a dedicated compression ASIC on-device and calling it a day; the entire upstream image processing pipeline must be optimized. Noise is the enemy of compression, the less spatiotemporal structure an image has, the harder it is to compress. So a good local compute system optimized for networking will prioritize the components that keep noise out of the image in the first place: optics, low-light sensor performance, sensor SnR, thermal management, and the image signal processor (ISP).

ISPs turn messy analog signals into digital ones, but can ruin compression efficiency when not optimized for compression directly. And when we consider the imaging pipeline as a whole, we are optimizing for the model, not for human viewing. This is similar to how WAMs do not require the extra denoising steps that typical video diffusion models use; they solve a different task. The same principle applies here: we care about good model outputs and a low bitrate, not necessarily beautiful images (see below).

Left image is more pleasing to the eye, but the right image has a higher SnR. Source: NASA

Network conditions introduce a lot of variability, so local compute must also incorporate dynamic, real-time adaptation of both the ISP and the compression bitrate, similar to WebRTC, to gracefully handle constantly fluctuating network conditions and do so without adversely affecting model performance.

A pipeline that produces a clean, adaptable stream also needs hardware explicitly tuned for offloading – getting the bits off-device and on-schedule. Historically, mobile chipsets prioritized downlink speeds, but robotics requires massive uplink capacity to stream out high-fidelity telemetry. The download can be relatively small. This means multiple transmit antennas with proper placement, since the environment and robot itself can block signals. It also means support for a wider array of RF bands to allow multiple channels for higher throughput and less congestion – especially with factory conditions in mind. Optimizing hardware and embedded firmware for fast handovers between access points and between WiFi and 5G is critical since the typical handover time will temporarily kill the latency budget. Jitter is the biggest enemy.

The end goal is a highly coordinated hybrid architecture that maintains a healthy, but not overly indulgent, amount of FLOPs locally for safety and high-frequency control and can reliably off-load the “foundation” part of a foundation model. It’s not a perfect system: simple control loop fallbacks are still required the moment the network connection diminishes or is completely lost. We require latency watchdogs, heartbeat timeouts, and staleness bounds on remote actions.

Power

Datacenter GPUs have to run off the robot, because the battery can’t feed them. A single Blackwell GPU draws ~1.2-1.4 kW and has to be liquid-cooled, built to sit in a rack. A humanoid, by contrast, stores only about 2 kWh in its battery and draws a few hundred watts in normal motion, a kilowatt or two at peak, everything included, which is why its onboard brain is a 40 to 130 W Jetson Thor. Put a datacenter GPU on the robot and the chip alone would out-draw the whole machine, demand cooling it can’t carry, and flatten the battery in minutes. Offloading sidesteps the power constraint. The robot carries only a 20 to 30 W perception compute + wireless board to reach the datacenter GPU, comfortably within budget.

The Supply-Chain Wall

Source: SemiAnalysis Accelerator Model, we withheld Rubin shipments on this chart, buy the data

Margins tell a similar story. Demand isn’t there yet, but when it arrives, pricing will go up and Jetson margins will inflect. But for now, a Blackwell datacenter GPU earns far higher gross margins than a Jetson module, on the order of mid-to-high 70s percent versus mid-60s, so NVIDIA has every incentive to point its scarce leading-edge wafer capacity at the datacenter and little reason to ramp edge silicon while the robotics market stays small. Robotics chips like Jetson Thor also require a lot more memory per unit of compute which is a major roadblock.

Source: SemiAnalysis Accelerator Model

Worse, robot silicon is converging onto the exact nodes the datacenter is fighting over. Nvidia’s Jetson line used to be insulated: the previous generation, Orin, ran on Samsung’s SF8 process. The current generation, Thor, has closed that gap onto TSMC N4, which Blackwell uses, and as robots run on more capable models their silicon has to stay on the leading edge: the generation after Thor likely moves to N3 alongside Rubin, then N2 alongside Feynman, exactly the nodes the entire AI-accelerator roadmap is fighting over. So a low-volume, lower-margin Jetson will always be competing for advanced-node wafers from the back of the queue.

Source: SemiAnalysis Accelerator Model

But the wafer demand from robots is not where the strain is. A million Jetson-class chips in 2030, each around 400mm², is only about ten thousand wafers for the year, a rounding error next to the datacenter GPU ramp, because Thor’s die is barely half a reticle size chip, and there simply aren’t many robots yet. Supply only tightens once robots need tens of millions of chips a year, well beyond 2030. So the real question is not whether robots can get wafers. It is silicon efficiency: for a given fleet of robots, which approach consumes less leading-edge silicon? Putting a chip in every robot, or serving their cognition from shared datacenter GPUs.

Source: SemiAnalysis Accelerator Model

Source: SemiAnalysis

That leaves the commodity and LPDDR supply that robot brains depend on competing for a shrinking pool of non-HBM wafers. Similarly, when DRAM is the scarce resource, the approach that uses less DRAM per robot is the one that scales. The LPDDR is better off being allocated to Vera GPUs or wafers to HBM.

Source: SemiAnalysis Memory Model

Source: SemiAnalysis

The TCO Verdict: One B300 server vs. 56 Thors

We ran DreamZero, current leader on RoboArena, on a real B300, using the inference optimizations from NVIDIA’s WAM paper (CUDA graphs, DiT caching, NVFP4 quantization, moving the scheduler onto the GPU). We also ran a single denoising step to match the DreamZero-Flash setup, though without Flash’s retraining recipe, so task quality is unvalidated.

DreamZero’s serving profile is prefill-shaped: each inference pushes a full chunk of video latents (thousands of tokens) through the 14B DiT in parallel, so a B300 is compute-bound at batch size 1. Batching therefore buys nothing: FLOPs scale one-for-one with batch size, and every robot just waits longer. Instead, we time-multiplex, with the B300 serving each robot sequentially. This is specific to video-generating WAMs. A VLA like π0 or GR00T flips the roofline: a few hundred tokens through a small backbone leaves the GPU memory-bandwidth-bound, so batching amortizes the same weight reads across robots and throughput scales nearly linearly with batch size. Serve a VLA fleet with continuous batching; serve a video WAM one robot at a time.

A key assumption we made is that Network RTT is 10ms, meaning the GPU sits on-prem near the robots. Each inference buys the robot its next 1.6 seconds of motion, which makes 1.6 seconds our latency budget: the total time a request gets to spend across network, queueing, and compute before the robot runs out of instructions and stalls mid-motion.

Source: Nvidia

We read the chart below on p99 latency (blue line), the 99th percentile response time. 99 out of 100 requests come back faster than this, so it’s effectively the worst case. It matters more than the median (green line) here because a robot stalls the moment any single chunk shows up late. We see that the maximum number of robots we can process on 1 B300 GPU is 7 robots. At 7 robots, p99 is 1.16 seconds, still under the wall, with zero missed chunks over the run.

Source: SemiAnalysis

Now that we know we can time-multiplex 7 robots per GPU, the fundamental TCO question stands:

Is it cheaper to offload compute to a B300 server, or to run inference on-device in a Jetson Thor?

We build up two scenarios: the offload scenario assumes one Blackwell-class server which can serve 7 robots (i.e. one B300 NVL8 server serving 56 robots), and an on-robot scenario of the same 56 robots each carrying its own Jetson Thor. In both scenarios, common parts across both cases (i.e. mechanical components, robot shells) are not included in the BOM cost calculation.

We also discuss the biggest challenge with this deployment mechanism.

TPU Inference Externalization Full Steam Ahead - InferenceX

2026-09-07T12:00:18+00:00

For more than a decade, the industry has watched Google build an empire on its own silicon. Search, Ads, YouTube, and every generation of Gemini run on TPUs. Few accelerators have attracted as much architectural scrutiny or as much debate about what their performance and economics would look like outside the company that designed them. Anthropic being the biggest user of TPUs, surpassing Deepmind’s own use by 2029.

Source: Google

Google’s internal success was never the question. The question was how much of that advantage the rest of the industry could actually get. Could you take an open-weight model, serve it through a familiar inference engine, and beat NVIDIA on the economics that matter to your business?

Today, we are publishing the first third-party inference results for TPUv7 Ironwood on InferenceX Official Preview. In our apples-to-apples comparisons against B200/B300, Ironwood delivers up to 50% better performance per dollar. Its advantage extends across much of the Pareto curve, and we examine the economics from both sides: Google’s internal total cost of ownership and the external TCO an actual customer pays.

Ironwood (TPUv7) is the first generation in which Google is competing for others’ inference workloads with chips that can be purchased outright or rented through its own cloud. In November 2025, we already said that Anthropic loves TPUs and committed to over one million of them (around 400k+ in direct purchases and 600k+ rented through GCP), used mainly for training but also for inference. Our Accelerator Model has the latest figures for Anthropic’s TPU shipments and Google’s overall TPU shipments by quarter, plus estimates for TPUv8i, v8t, and various v9 / v10, and more

We are excited by how quickly the new TorchTPU stack is developing, the external stack for TPUs. Later in the article, we will discuss the upcoming work needed for TPU software externalization, including optimizing speculative decoding, disaggregated prefill, KV-cache offloading, multi-turn agentic workloads, and more. Even so, we at SemiAnalysis strongly believe that TPU externalization is heading in the right direction and moving full steam ahead. Furthermore, unlike AMD, which is still learning how to build a test-first software culture, Google has decades of software engineering experience and an extremely well established quality-driven culture, so we expect external TPU software to mature rapidly.

In this article, we will cover all the optimizations that went into TPU kernels and the serving stack for open-weight models, including DP attention optimization, MoE routing and kernel optimization, and reducing padding in GDN kernels. We will also take a deep dive into the TPU system and discuss the next steps the amazing TPU performance engineers are pursuing to make the stack widely available.

Google has spent more than a decade demonstrating what it can build with TPUs. Now we get to measure what the rest of the industry can do with them.

Shoutout to the Google (Chris Chan, Jahangir Hasan, Wangyuan Zhang, Anne Stern, Puneith Kaul, Ruizi Dong, Sangam Jindal, Qi Zhou, Madhan Jaganathan, Gang Ji, Jun Wan, Devanshu Jain, Jiaxin Cao, Srinath Mandalapu, Haowen Ning) and Inferact teams for this amazing TPU foundation and performance! Furthermore, shoutout to the RadixArk team that is also working on TorchTPU SGLang.

InferenceX Official Preview: TPUv7 Ironwood vs. Blackwell and Blackwell Ultra

We are already seeing strong results from the upcoming native TorchTPU vLLM stack in apples-to-apples comparisons against Nvidia GPUs. Google is using Qwen3.5 397B in FP8 as the initial bring-up model. Later sections take a deep dive into why the new TorchTPU approach is a marked improvement over the previous TorchAX path for external TPU vLLM/SGLang serving.

Subscribe now

Once that foundation is in place, Google plans to extend support to other open-weight models, including Kimi K3 and GLM5.3. Once a handful of models are well optimized, we believe adding optimized support for a wide range of popular open models near day 0 becomes far easier. Today, vLLM and SGLang concentrate their day-0 support on Nvidia, with passable day-0 coverage for AMD. We expect TorchTPU to be stable enough in the near future that vLLM and SGLang maintainers may add TPUs to that day-0 list. The stack is expected to leave private beta & be open sourced around October.

In apples-to-apples comparisons of aggregated serving with FP8 and single-token prediction, we are seeing up to 50% better performance per dollar from TPU than from B200 and B300 running FP8 in aggregated serving. When serving models using FP4 on NVIDIA GPUs, there is quality loss verus FP8. TPUv7 does not have native FP4 computation thus on FP4, NVIDIA GPUs still maintains the lead. This will change with TPUv8i which has native FP4 support thus we strongly believe TPUv8i Boardfly will be competitive to Rubin NVL72.

We believe the TPU performance-per-dollar advantage can persist once Google enables disaggregated serving and speculative decoding, with MTP and disaggregated serving enabled on both sides of the comparison. Later in the article, we discuss the optimizations Google plans to add once the TorchTPU foundation is in place. To keep the bring-up scope contained, Google has initially focused on benchmarking 8k1k. However, we believe that the TorchTPU stack will also perform amazingly well on agentic workloads & will be publishing results for those too later in the year. Later on in the article, we will talk about the next steps & roadmap of TPU external Inference.

TPU is King on Performance per Dollar

At an interactivity of 100 tokens/s/user, Ironwood costs approximately $0.181 per million total tokens, compared with $0.222 for B200 and $0.276 for B300. That is approximately 19% lower cost than B200 and 34% lower than B300, while delivering the same per-user generation speed.

For this comparison, we use an external TPU TCO against B200/B300 TCO for an hyperscaler lab to buy TPU. Last year, SemiAnalysis Accelerator Model was the first to break that Google is selling TPUs instead of just renting out through Google Cloud. TPU’s lower hourly cost offsets its lower throughput across much of the interactivity range. The SemiAnalysis TCO Model provides a full breakdown of the TPU BoM estimate and TCO estimates.

Source: SemiAnalysis InferenceX Preview

Source: InferenceX Preview

Across most of the raw-performance curve, TPU does not outperform Nvidia GPUs. However, that comparison does not account for TPU’s lower TCO. Buyers of TPU are mainly concerned with how much revenue they can make, i.e. performance per dollar and performance per watt.

That being said, at interactivity of 20 tok/s/user, Ironwood also leads on raw throughput, reaching 9,364 total tokens/s/chip, compared with 8,903 on B200 and 8,925 on B300. This is approximately 5% higher throughput than either GPU in these runs. Combined with the lower modeled hourly cost, Ironwood delivers 50.4% more tokens per dollar than B200 and 96.0% more than B300.

Source: SemiAnalysis InferenceX Preview

If we consider TPU TCO for Google’s internal workloads at $1.03/chip-hour, the performance-per-dollar advantage at concurrency 256 increases to 76.7% over B200 and 130.2% over B300. However, these high-concurrency results come with latency trade-offs. For example, at concurrency 256, TPU mean TTFT is 5.41 seconds, compared with 3.75 seconds on B200 and 2.40 seconds on B300. The previously discussed 50% to 96% advantage applies to this datapoint specifically, rather than every latency target.

Source: SemiAnalysis InferenceX Preview

Looking at end-to-end latency, Ironwood remains competitive beyond that highest-throughput point. At a 20-second median response time, the Pareto curves put TPU at approximately $0.098 per million total tokens, compared with $0.106 for B200 and $0.132 for B300. TPU therefore costs approximately 8% less than B200 and 25% less than B300.

Source: SemiAnalysis InferenceX Preview

Despite all of the above, B200 can still come out ahead on a small part of the Apples to Apples curve, including around the 30-second median response time. However, TPU regains its lower cost per million tokens at longer response times, while maintaining a cost advantage over B300 across the overlapping range shown here.

Apples to Bananas: GB300 NVL72 Disagg verus TPUv7 Agg

Google has been running disaggregated serving internally for years in production and that path is heavily optimized, but the external TPU serving stack does not have a fully optimized disagg path yet. So today, GB200/300 NVL72 is more competitive on perf per dollar in a disagg-vs-disagg comparison. But we expect that gap to close within a few months & TPUv7 to be competitive againist GB200/300 NVL72 as Google, Inferact, RadixArk, and SemiAnalysis land optimizations, and we will publish TPUv7 disagg vs GB200/300 NVL72 disagg in a follow-up article. TPUv7 pods can scale up to over 1k+ chips through their low latency ICI fabric thus can enable optimizations for large models disagg & ultra wideEP that NVIDIA NVL72 can’t perform.

Even on an apples to bananas comparison between TPUv7 aggregated serving verus GB300 NVL72 disagg, using internal TCO, TPUv7 is decently competitive on high latency & low to left medium latency. But when comparing on the right medium e2e latency, GB300 NVL72 disagg has an single advantage against TPuv7 aggregrated serving.

Source: SemiAnalysis

When looking at it GB300 NVL72 disagg verus TPUv7 agg, it is competitive at low and high e2e latency but in the middle, there is about an 30% perf per dollar advantage of GB300 against TPUv7 agg. We strongly believe once disaggregated serving on TPUv7 is optimized, it would be competitive across the full pareto against GB300 NVL72.

Source: SemiAnalysis

Inference Serving with the New Native, First-Class TorchTPU Backend vs. the Previous TorchAX Stack

Previous TorchAX vLLM Backend

Google previously used the TorchAX backend to translate the PyTorch-based implementations in vLLM and SGLang into JAX. TorchAX was intended to let developers run PyTorch model definitions through JAX. However, issues with this approach have prompted Google and the PyTorch, vLLM, and SGLang communities to collaborate on a new approach called TorchTPU. It lets developers use TPUs as native PyTorch devices, while StableHLO, XLA, and TPU-optimized Pallas kernels handle the low-level execution behind the scenes. In the upcoming sections of the article, we will be showing the performance of this new TorchTPU backend compared to NVIDIA Blackwell & Blackwell Ultra GPUs.

vLLM TPU support has evolved through three different stages:

Its first TPU prototype used PyTorch/XLA. Its lazy execution model collected operations into computation graphs for XLA to compile, rather than executing each PyTorch operation immediately.
The current public backend, tpu-inference, uses a TPU-optimized JAX implementation when one is available. Otherwise, TorchAX translates the original PyTorch model into operations that JAX can execute. The goal is to reuse mature TPU primitives and parallelism support without requiring every PyTorch model to be rewritten.
The upcoming third design, TorchTPU, lets vLLM treat TPUs as native PyTorch devices. It will be the go-to backend for SGLang and vLLM on TPUs going forward.

The diagram below zooms in on the second stage and shows how vLLM’s tpu-inference backend combines direct JAX models and PyTorch models running through TorchAX.

Source: SemiAnalysis/vLLM

The original choice of JAX was pragmatic. In the last announcement, Google described difficulties with low-level optimization, paged attention, and fitting vLLM’s worker model to TPU execution. They thought that JAX would offer more mature TPU primitives and parallelism support. TorchAX let the team use those capabilities without rewriting every PyTorch model, while TPU-oriented JAX implementations could share the same kernel and compiler work.

With TorchAX, developers still write the model in PyTorch, but JAX executes it on the TPU. TorchAX acts as the translation layer between the two frameworks. As the TorchAX documentation explains, a torchax.tensor.Tensor behaves like an ordinary PyTorch tensor but is backed by a jax.Array. When the model invokes a tensor operation, known in PyTorch as an ATen operation, TorchAX intercepts it through the __torch_dispatch__ hook and translates it into one or more JAX operations. The model can therefore remain written in PyTorch even though JAX performs the computation.

vLLM must also manage state that changes during generation, especially the KV cache, which stores attention information from earlier tokens for reuse. Its model wrapper presents the model weights and KV cache to JAX as an explicit state. This process, known as functionalization, turns changes that would otherwise happen inside Python objects into inputs and outputs that the compiler can track. jax.jit can then capture each inference step as a computation graph and pass it through JAX’s compilation pipeline for repeated execution.

From there, each component has a distinct role. Pallas supplies optimized TPU kernels for performance-critical operations. StableHLO represents the program in a standardized format that the compiler can process, and XLA converts it into executable code for the TPU. In vLLM’s standard PyTorch path, torch.compile coordinates graph capture and compilation. In the TorchAX path, the JAX/XLA pipeline already performs those tasks, so vLLM bypasses its usual torch.compile route. These translation layers have caused many issues that the TorchTPU vLLM stack aims to solve.

Previous SGLang JAX Backend

Source: SemiAnalysis/SGLang

SGLang’s current public TPU stack uses SGLang-JAX. SGLang-JAX makes a similar trade-off to vLLM’s TorchAX backend but takes a different approach. It is a separate JAX-native serving engine rather than the PyTorch SGLang runtime translated through TorchAX. It combines SGLang-style scheduling and prefix caching with JAX models and TPU-specific kernels. Google and RadixArk have announced SGL-torchtpu as an additional PyTorch-native backend to extend the available serving paths.

Subscribe now

Shifting Toward the New Native, First-Class TorchTPU Backend

TorchTPU moves the framework boundary into PyTorch itself. It uses PyTorch’s PrivateUse1 backend extension point to expose an ordinary torch.Tensor on device=”tpu” rather than a wrapper backed by a JAX array. The PyTorch dispatcher routes ATen operations to the TPU backend. Developers can run code eagerly for bring-up and debugging, or call torch.compile for graph capture. In the compiled path, TorchDynamo and AOTAutograd produce an FX graph, TorchTPU lowers it to StableHLO, and XLA produces the TPU executable. Google’s conference deck makes the compiler choice explicit that this path uses XLA, not Inductor and Triton.

Source: SemiAnalysis/PyTorch/Google

“Native” therefore has a specific scope. It covers PyTorch tensors, dispatch, eager execution, compilation entrypoints, and distributed APIs. XLA remains the compiler, and the kernel layer continues to use TPU-specific implementations. TorchTPU can call Pallas and JAX-backed custom kernels. Helion’s TPU backend also emits Pallas. The user-facing framework becomes PyTorch-native, while the compiler and performance-critical kernels remain TPU-aware.

Developers can use .to(”tpu”) while retaining familiar distributed interfaces and serving-engine code. Google documents support for DDP, FSDP2, DTensor, one process per device, and both MPMD and SPMD execution. Those interfaces fit the process and distributed model already assumed by PyTorch serving engines.

For vLLM and SGLang, the opportunity is to reuse more upstream model code, schedulers, continuous batching, APIs, and feature logic instead of rebuilding them across a PyTorch-to-JAX boundary. That should reduce the cost of bringing up models and engine features, even when the final TPU implementation still needs specialized kernels.

Native PyTorch support does not eliminate the need for TPU-specific optimization. Pallas and TorchAX serve different purposes, TorchAX allows PyTorch model code to execute through JAX, while Pallas is used to implement performance critical operations directly for TPU hardware. Removing TorchAX doesn’t mean removing the Pallas kernels underneath it. Since TorchTPU allows native Pytorch code to invoke Pallas and JAX kernels, kernels developed for earlier stack are able to migrate into the new stack.

Similar to GPU PyTorch vLLM, Custom kernels will still be needed for high performance inferencing. Tensor shapes and layouts still need to be tuned to use TPU’s matrix multiplication units efficiently. We believe that most of the existing Pallas kernels are easily transferable to the new native TorchTPU serving stack but that does not mean that every existing kernel will transfer unchanged. Many things like wrappers, tensor layouts, and runtime integration etc may still need to be slightly adapted and validated.

Strong Ecosystem Collaboration with Google’s TPU Team

vLLM and SGLang are the two major open-source production inference serving stacks. Inferact, RadixArk, and Red Hat are investing heavily in collaboration with Google to make the TorchTPU backend a first-class experience in both stacks. This community support is extremely important for externalizing TPU inference.

Source: Inferact

As of early September, TorchTPU remains in private beta and will be open-sourced around mid-October during the PyTorch Conference. We are excited to see it move TPUs toward a first-class experience in PyTorch, vLLM, and SGLang. Once TorchTPU vLLM and TorchTPU SGLang are open-sourced, we will move TPU benchmarking for InferenceX/AgentX from our fork to our public repo.

Source: RadixArk

TPU Inference Optimizations Deep Dive

The sprint to optimize TPU inference for the native TorchTPU serving stack’s first bring-up model involved hundreds of engineering hours and numerous PRs. As described later in the article, TPU architecture differs from GPU architecture and therefore requires its own set of optimizations to run models efficiently. Below, we describe the optimizations that improved (or in some cases, enabled) performance for Qwen3.5-397B and other models on Ironwood. Most of the Pallas kernel optimizations are transferable to the new native TorchTPU stack.

DP Attention Optimization

One issue that is not necessarily specific to TPU is how to effectively map parallelism from model architecture to hardware. For instance, Qwen3.5’s GQA attention layer has 32 query heads but only 2 shared KV heads. With TP8 (eight logical devices), query computation can be evenly distributed across the 8 devices, with 4 query heads per device. The KV heads, however, do not divide evenly across those devices. Each KV head is shared by 16 query heads, meaning four devices need the same KV data to compute their local attention. When TP exceeds the KV-head count, standard practice is to replicate KV heads across TP ranks. vLLM already supports this. The TPU backend needed a compatibility fix to activate that existing behavior and avoid unnecessary All-to-All communication.

For higher-concurrency serving, the TPU backend also added support for combining eight-way attention data parallelism (DP8) with eight-way expert parallelism (EP8), simply referred to as “DP attention” or DEP8. Instead of splitting every request’s attention across all eight devices, each device processes a different subset of requests and keeps both KV heads for those requests locally. The attention weights are replicated, but the KV caches hold different request histories rather than repeated copies of the same history. Meanwhile, the 512 routed experts remain distributed across the devices, avoiding replication of the much larger expert weight pool. Making this work required coordinating request assignment, recurrent-state slots, and block tables in the serving engine.

Source: SemiAnalysis

Optimizing Communications

With DP attention and EP, the TPU GroupedGEMM implementation all-gathers token activations and routing metadata before the experts run, then uses reduce-scatter to sum their weighted outputs and return each token’s result to its attention rank. The backend originally gathered the selected expert IDs and routing weights separately. Google combined these into one all-gather, removing an extra collective whose latency can dominate the transfer time for such small arrays. The PR reports roughly 80 microseconds per layer saved in its DeepSeek-V3 measurements. While 80 µs may sound small, DeepSeek-V3/R1 has 58 layers that require this AllGather. Across those layers, the savings add up to around 4.64 ms per forward pass in isolation (80 µs × 58), reducing TPOT and increasing interactivity.

Source: SemiAnalysis

Google also implemented the ReduceScatter collective on SparseCore (which is better suited for irregular operations like data movement) and used Ironwood’s faster die-to-die links to combine contributions within each chip before exchanging partial sums across chips. This two-device-per-chip layout and the chip-to-chip ICI network are covered in the next section. Double buffering lets the kernel transfer one chunk while accumulating another, overlapping local reductions and die-to-die transfers with the slower chip-to-chip traffic. Running the collective on SparseCore also frees TensorCore execution for other operations.

Source: SemiAnalysis

Source: Google and SemiAnalysis

In the diagram above, communication stages overlap, shortening end-to-end execution time. For instance, consider the point labeled t_1. At this time, the intra-chip DMA ScatterReduce (P1) has completed for MB (microbatch) 1. ScatterReduce across ICI dim 0 (P2.0) can now occur for MB0 concurrently with P1 for the subsequent MB. Against a baseline, this optimization results in 4.1 to 14.2% higher throughput on 8k1k across concurrency 64 to 512, including 8.5% at concurrency 256, and 26.1% on 1k8k at concurrency 512.

Thanks for reading SemiAnalysis! This post is public so feel free to share it.

The movement of this collective from TensorCore to SparseCore also means that additional work can be pipelined on the TensorCore while the collective runs. However, not every collective is always worth moving. In some cases, offloading a collective to SparseCores actually worsens performance. For example, for Qwen3.5 a threshold now decides when all-reduce and all-gather operations are offloaded to the SparseCore, keeping small collectives on the TensorCore when they fit in VMEM and SparseCore offload overhead would make them slower. The default threshold is derived from VMEM capacity. The PR reports 8k1k throughput gains of 2.7% at concurrency 64 and 5.7% at concurrency 128.

Optimizing MoE Routing and Expert Kernels

A mixture-of-experts layer produces irregular groups of tokens for each expert, and these ragged groups have to be reshaped into something the TPU’s matrix units can consume efficiently. Some of the changes below are general improvements to the routing and grouped-matmul backend that benefit any MoE model. Others were measured directly on Qwen3.5.

The second version of the grouped matmul reworks how expert inputs are fed to the MXU. It removes redundant tile computation, sizes transfers to the number of valid rows rather than the padded maximum, triple-buffers expert weights so the next group’s weights are already in flight while the current group is computed, and fuses group metadata generation into the kernel.

The irregular rearrangement of expert inputs was then moved onto the SparseCore. The SparseCore handles data movement, gathering each expert’s tokens into contiguous groups, while the TensorCore is left to run the expert matrix multiplications. A subsequent rewrite improved memory-read pipelining and split the combine work across tokens and hidden dimensions. The PR reports a 12% increase in 8k1k serving throughput compared with the original SparseCore kernels, along with lower TTFT and TPOT.

Source: SemiAnalysis

A further optimization moved the top-k weight gather in the ragged gather-reduce path onto the SparseCore. Here, gather means reading selected entries from memory on one device, not an inter-device all-gather. Previously, the TensorCore gathered routing weights and source indices during preprocessing, limiting TensorCore/SparseCore overlap. Moving these gathers into the SparseCore kernel reduces TensorCore overhead from 29 µs to 14 µs and overall operation latency from 146 µs to 137 µs in a DeepSeek-V3 microbenchmark at batch 2k with 16-way expert parallelism.

For small batches, the general ragged path costs more than the work it is arranging. A dedicated small-batch permutation therefore builds one-hot matrices and uses ordinary matrix multiplication to permute tokens to their experts and unpermute the results. On the 8k1k workload, this improved throughput by 7.3% at concurrency 64 and 5.1% at concurrency 128.

A WIP change packs the expert ID and token index into a single sort key, so XLA performs a simpler sort while preserving the required ordering. Sort latency drops from 106.6 µs to 21.7 µs. The PR reports 8k1k serving gains of 0.6 to 8.5%, but the same change also enables an FP8 all-gather.

Optimizing Gated DeltaNet Pallas Kernels

Gated DeltaNet (GDN) is a recurrent computation. That is, at each step the running state is decayed, updated with a rank-one term, and projected to produce an output. Check out the following article for a deeper dive into linear attention mechanisms such as GDN:

The changes below show how its matrix operations, vector updates, and state transfers are scheduled across the MXU, VPU, VMEM, and HBM. Initial Qwen3.5 support added pure JAX implementations of causal Conv1D and GDN, connected them to TPU operator dispatch, and enabled recurrent-state caching. Later PRs optimized these implementations.

Source: GitHub

Another low-hanging fruit optimization is achieved by rearranging the algebra in the output projection calculation to overlap MXU and VPU work. Previously, the VPU first applied the rank-one update to the decayed state, and the MXU then multiplied the updated state by the query to produce the output. That dependency forced the MXU to wait.

Writing the decayed state as S, the correction vector as Δ, and the key and query as k and q, the output can be expanded:

Source: SemiAnalysis

The MXU can now compute Sq directly from the decayed state while the VPU constructs the updated state for the next token. The current update’s contribution to the output is calculated separately using kᵀq, a scalar dot product per head, followed by a small scaled-vector addition. This removes the state update from the MXU’s dependency path. Reported 8k1k throughput gains were 2.79% at concurrency 64 and 4.48% at concurrency 512.

The next change reduces vector-register spills by slicing Q and K inside the decode loop so fewer values stay live at once. The decode-64 kernel becomes roughly 20% faster, but end-to-end gains are smaller, at 0.8% on 8k1k and 3.8% on 1k8k at concurrency 512, since the kernel is only one part of a decode step.

Asynchronous state transfers overlap DMA with computation using double buffering. The extra VMEM consumed by the second set of buffers initially regressed data-parallel attention. Reusing existing scratch buffers and shortening the lifetimes of temporaries recovered that capacity and removed the regression. The reported 8k1k throughput gain is 11.3% at concurrency 512.

GDN v3 fuses Conv1D and GDN into a single kernel to cut HBM round trips, improves prefill layouts, and unifies mixed prefill/decode execution into one path. Reported kernel-level speedups are 1.41× for decode, 1.60× for prefill, and 2.14× for mixed batches. These measurements cover the kernel alone and do not establish end-to-end serving gains.

Source: SemiAnalysis

Significant performance gains can be achieved simply by improving the overlap between operations within a kernel!

Managing Hybrid State and Paged Attention

Qwen3.5 is a hybrid model with two kinds of state: GQA layers accumulate a KV history that grows with every token, while GDN layers hold a fixed-size recurrent state per request. State allocation, storage precision, attention page size, and physical data layout all determine how much HBM is actually usable and how efficiently attention runs.

Batched ragged paged attention batches sequences together, precomputes page metadata, and triple-buffers to improve pipelining and reduce padding. This change provides shared attention-backend groundwork. The PR’s workload examples use Qwen3-32B, so its measurements should be kept separate from the Qwen3.5-397B results.

Recurrent state is now allocated compactly, with roughly one slot per active request instead of num_blocks slots for every layer group. In the reported configuration, this reclaims about 76 GiB of HBM and expands the attention block pool by 71%, improving 1k8k output throughput by 18% at concurrency 64.

Storing the recurrent state in BF16 halves its HBM footprint while keeping the arithmetic in FP32 inside VMEM. This saves memory capacity and transfer bandwidth while preserving FP32 arithmetic. The reported 1k8k throughput gain is 15% at concurrency 512.

Removing the old hybrid page-size alignment constraint lets batched attention use a suitable power-of-two page size, including 256 tokens. 1k8k throughput improves by about 7% at concurrency 512. This applies to the non-prefix-caching path and is distinct from the aligned checkpoint mode introduced later.

Source: SemiAnalysis

Unnecessary KV-layout copies are avoided by selecting the reshape path based on KV-head count, rather than imposing a layout constraint that only other shapes need. Qwen3.5 8k1k throughput improves by about 4.1% at concurrency 512.

Reducing Padding and Low-Concurrency Overhead

When only four or eight requests are in flight, a configuration tuned for hundreds of concurrent requests wastes work. Compiled shape buckets are too large, metadata is sized for the configured maximum, and padding tokens trigger dummy expert work. The InferenceX operating points sit at exactly these low concurrencies, which motivated the following changes.

Subscribe now

Request metadata is now bucketed by the number of active requests instead of always using the configured maximum. In the 8k1k concurrency-64 test, GDN scheduling overhead falls from 283 µs to 97 µs and throughput rises from 2,328 to 2,516 tokens/chip/s.

Source: SemiAnalysis

Explicit Qwen3.5 tuning for InferenceX shrinks rotary tables, sizes sequence limits per DP rank, and adds a dedicated attention bucket for concurrency four. The combined gains at concurrency four are 13.3% on 8k1k and 15.5% on 1k1k.

A further round of low-concurrency tuning switches to TP8 attention plus expert parallelism, reduces the minimum token bucket, and routes padding tokens to expert zero so they do not trigger additional expert weight loads. Combined 1k1k gains are 22.9% at concurrency four and 18.1% at eight. On 8k1k, throughput rises by 9.2% at concurrency four but falls by 5.3% at concurrency eight.

Enabling Prefix Caching for Hybrid Models

The gains above were measured on random-input benchmarks, where nothing is shared between requests. Prefix caching is crucial for agentic and multi-turn workloads, which reuse long system prompts and conversation histories. For a hybrid model, a cached prefix must retain both its KV blocks and the GDN recurrent state at the end of the prefix. That recurrent state is normally overwritten as soon as the request continues, making prefix caching harder than for “regular” attention models.

Hybrid prefix caching with DP support resolves this by giving GDN separate slots for reading a checkpoint and writing the live state. Continuing a request therefore no longer overwrites the checkpoint associated with its cached prefix. State addresses are derived from the block table, the same structure that already locates KV blocks. Checkpoints are taken at an aligned cache granularity so a saved state always lines up with a KV block boundary. This mode needs a full checkpoint pool rather than the compact per-request allocation described earlier, trading some HBM for reuse across requests. The benefit shows up when prefixes are actually shared.

Source: SemiAnalysis

Lane Layout and Pipelining Depth in Paged Attention

Two hardware facts shape how a KV cache is laid out on TPU. We discuss TPU hardware in more detail in the following section. First, the vector unit works on tiles whose last dimension is 128 lanes wide, so any array whose trailing dimension is smaller than that is padded up. Second, a Pallas kernel hides HBM latency through double-buffering. It fetches the next block while computing the current one. Both blocks must fit in VMEM, so the size of the compute block determines how deep the prefetch can be. Both constraints were costing capacity and throughput.

In the batched attention kernel, the KV cache packed keys and values along the head dimension, and for FP8 that packing factor is four. A model with a single KV head per device only has two things to pack, so half of every tile was wasted on padding. A sequence-on-lane layout fixes this by putting the page’s tokens on the 128-lane axis and the head dimension on the sublane axis instead. Usable KV pages double (from 5,141 to 10,283 in the reported configuration) and the head dimension only needs to be a multiple of 32 rather than 128, which allows models with a head dim of 64 to use the kernel as well. The layout costs about 3% in per-token latency at low concurrency, but at concurrency 128 on 8k1k the extra capacity lifts throughput 16.5% and cuts median TTFT by 95%, because requests no longer wait for KV space.

Source: SemiAnalysis

The RPA v3 block-size heuristic had a similar blind spot. During v7x decode, it set the KV compute block equal to the KV fetch block, around 16k tokens, which left almost no VMEM for the prefetch buffer. Splitting the two block sizes kept the KV fetch block at 16k tokens while reducing the compute block to 4k. This gave the pipeline room to run ahead of the MXU and increased decode throughput from 64.9k to 96.3k tokens per second, a 49% gain reproduced across four runs on Qwen3-0.6B. Throughput followed a clean inverted-U curve across the block-size sweep. The tuned-parameter table could not represent the improvement because it stored a single block size per shape, which is why the change first landed as an environment override. A follow-up extends the table to hold the fetch and compute sizes separately.

Source: SemiAnalysis

TPU System-Level Co-Design & Networking Explainer

As we said in the TPU King article, Google’s cost-per-token advantage on inference is a result of co-design. Instead of focusing on maximum single-chip performance, Google designs the compute die, chip-to-chip fabric, and compiler together, allowing it to optimize computation and communication as a single system. That joint optimization helps explain Ironwood’s performance per dollar. The following sections examine how the chip and network work together. Anthropic is a big fan of TPUs and uses them heavily for training.

The Ironwood Chip

TPUv7 Ironwood breaks from the “MegaCore” convention that defined TPU v4 and TPU v5p, where two physical cores were fused into a single logical accelerator sharing one memory space. Ironwood instead has two separate compute dies, each running its own independent logical device. These dies are joined by a high-bandwidth die-to-die link rather than a unified memory fabric. JAX and other frameworks now expose them as two distinct devices per chip. Each Ironwood chip carries 2 TensorCores and 4 third-generation SparseCores. The SparseCores accelerate embedding lookups and other sparse operations that would otherwise choke a dense matrix engine.

Source: Google

On the memory side, each chip has about 6x Trillium’s HBM capacity, a jump that matters directly for KV-cache headroom and batch size. Ironwood is notably the first TPU generation with native FP8 hardware support, whereas prior generations had to emulate FP8 in software.

Source: SemiAnalysis

Read more on Ironwood in our TPUv7 article

The MXU and Why Shapes Matter

The Matrix Multiply Unit, the engine that actually does the multiplying, is a systolic array: a two-dimensional grid of multiply-accumulate cells. Weights are loaded into the array and held stationary. Activations stream in from the edge, and partial sums ripple through the grid cell by cell, accumulating as they go. The finished result streams out the other side without ever touching memory mid-computation. Every TPU generation through v5 used a 128x128 MXU, good for 16,384 MACs per cycle. Starting with TPU v6e and carrying through to Ironwood, the array doubled to 256x256, or 65,536 MACs per cycle, delivering 4x more FLOPs per cycle than the previous design.

Please read the scaling book (aka the ML systems bible) for excellent top-tier information on TPUs and scaling in general.

The catch is that a bigger systolic array is only free if you can keep it full. Matrix dimensions need to be padded up to at least the MXU’s side length in both directions, 128 on older generations or 256 on v6e and v7, and the XLA compiler dutifully pads any smaller axis to fill the tile. Every padded cell still occupies a MAC unit for that cycle, multiplying by a zero that contributes nothing to the answer. Llama 3 8B illustrates this with an attention head dimension of 128. On Ironwood’s 256x256 MXU, that head dimension is exactly half the array’s native width, which caps the two attention matmuls at a maximum of 50% MXU utilization (not 75%, because only the head dimension is 128).

The implication runs deeper than one model’s attention layer. Architectural choices that used to be arbitrary hyperparameters (such as head dimensions, the KV head count that survives after tensor-parallel sharding, and MoE expert widths) increasingly need to be made with TPU tile geometry in mind, because a shape mismatch there is a direct tax on delivered throughput regardless of how good the rest of the stack is. Kernel authors also need to account for this explicitly. Production attention kernels like ragged paged attention use explicit packing dimensions to reduce padding when XLA’s default tiling produces an inefficient layout.

TPUs Are Picky

GPU matrix cores consume small tiles, so a wide range of head dimensions, expert widths, and post-sharding KV head counts all land near peak. A dimension of 64 instead of 128 pays close to nothing on an H100 or a B200 and gets a cheaper attention layer for it. GPU architecture allowed researchers to prioritize evaluations performance without a loss in inference performance. But for TPUs, this hyperparameter becomes a tradeoff

As previously mentioned, these choices are not free on a 256-wide systolic array. By the same arithmetic that caps Llama 3 8B’s attention matmuls at 50%, a head dimension of 64 caps them at 25% before a single line of kernel code is written. gpt-oss ships at 64. DeepSeek’s MLA splits its query/key dimension into 128 plus 64 for a total of 192, ungainly against any power-of-two array and worse against a wide one.

The consequence is that bring-up cost varies enormously, and it correlates poorly with how popular a model is. A model whose shapes fall out cleanly needs scheduling, serving, and tuning work, which is measured in weeks. A model that fights the tile geometry needs new kernels before it is even at parity, let alone winning on performance per dollar. Kernel engineers are finite, so sequencing externalization toward models where the hardware is not starting a lap down is the rational move, and we would expect the same behavior from any accelerator vendor in this position.

Torus Topology

Inference and training at scale run across thousands of chips that need to talk to each other constantly. TPUs connect P2P over a custom network called ICI (Inter-Chip Interconnect). ICI bypasses the host CPU entirely, allowing chips to exchange activations and gradients directly instead of routing them through PCIe and a general-purpose NIC. The topology behind ICI has evolved generation over generation: TPU v2 and v3 used a 2D torus with each chip wired to 4 neighbors, and starting with TPU v4 and v5p, Google moved to a 3D torus with each chip wired to 6 neighbors along the +/-X, +/-Y, and +/-Z axes. Ironwood (TPUv7) keeps that 3D torus. The fundamental building block is a 4x4x4 cube of 64 chips, sized to map cleanly onto one physical server rack.

Source: SemiAnalysis

The difference between a torus and a plain grid is its wraparound links. Connecting the ends of a line forms a ring, cutting the worst-case hop distance from N to N/2. It’s the same trick that makes Pac-Man’s maze feel smaller than it is. Google pushes this further with a “twisted torus,” a Mobius-strip-like wraparound that shaves the average hop count down even more. To scale beyond a single 4x4x4 cube, Google stitches cubes together using Optical Circuit Switches, which preserve the wraparound property across a much larger reconfigurable topology, scaling all the way up to Ironwood’s full 9,216-chip superpod and its 42.5 FP8 exaflops of aggregate compute. The operational payoff of OCS is that Google can physically rewire around a failed link or a dead chip in seconds, using mirrors rather than sending a technician to re-splice copper in a live data center.

Source: SemiAnalysis

This scale-out design was revolutionary at the time. Before NVL72 racks, models that could not fit within a single 8-GPU node had to use pipeline parallelism because inter-node InfiniBand was slow. On a TPU pod, the ICI torus gives NVLink-class bandwidth across the entire pod, up to 8,960 chips on v5p. With this bandwidth, a DSV3-scale model can be sharded across the whole pod using tensor parallelism, expert parallelism, and data parallelism (FSDP-style weight sharding), without splitting layers across pipeline stages.

Although a TPU torus has more hops than a single-hop NVLink switched design, our upcoming CollectiveX/NetworkingX results show that, in many cases, the TPU torus has lower latency than a single-hop NVSwitch for small EP messages.

Looking Ahead: TPUv8i’s Boardfly Network

Google’s newly announced eighth-generation TPU lineup has two purpose-built chips: TPU 8t for training and TPU 8i for inference. This is the first time Google has separated training and inference into distinct architecture designs rather than shipping one architecture tuned for both. TPU 8t keeps the 3D torus lineage alive for scale-up, but TPU 8i replaces the torus with a new topology called “Boardfly.” The name nods to dragonfly-style high-radix network designs long used in supercomputing. Boardfly uses a flatter, hierarchical fabric of high-radix switches rather than a nearest-neighbor mesh. The Boardfly topology affects networking attachment capex per chip.

The payoff is a network diameter cut by more than 50% versus a similarly sized 3D torus, roughly 16 hops down to about 7 hops at comparable scale in the 1,024 to 1,152-chip range. Fewer hops mean materially lower tail latency on collective operations, and that matters enormously once you are routing tokens across MoE layers or running multi-turn agentic workloads where every extra hop compounds into user-visible latency. TPU 8i backs this up with 19.2 Tb/s of ICI bandwidth, double the prior generation, and 384 MB of on-chip SRAM, 3x the previous generation, sized specifically to hold the KV cache of reasoning and agentic models on-chip rather than round-tripping to HBM.

Source: Google

None of this hardware headroom pays off on its own, though. Turning a lower-diameter network and a bigger on-chip cache into cost-per-token gains still requires the software stack to catch up. The next section covers the roadmap for doing so, including speculative decoding, prefill-decode disaggregation, and the broader model support needed to take full advantage of TPUv8 in production.

Next Steps in Laying the TPU Foundation

These preview results establish a starting point for further optimization on the same hardware. We expect performance to improve as TPU externalization continues, with substantial work still needed across the software stack.

Enabling & Optimizing Spec Decoding (MTP)

The first area the Google team needs to focus on is on optimizing speculative decoding. A cheap, small drafter guesses several tokens ahead, the main model verifies all of them in a single forward pass, and any guess that doesn’t match is thrown away. Everything that survives is identical to what the model would have produced on its own. Speculative decoding is lossless, with zero quality degradation.

The reason it works is that decode is bandwidth bound, not compute bound. To emit a single token for a single user, you stream the entire model’s weights out of HBM. That weight read is the dominant cost, and the cost barely changes whether you verify one token or five. The matrix units sit mostly idle while the memory system does all the work. Speculative decoding spends that idle compute to amortize one very expensive weight read across several tokens, which is why methods that draft multiple tokens at once, like MTP or DSpark, are so cheap and so effective.

Source: vLLM

Prefill-Decode Disaggregation & KV-cache Offloading

Prefill-decode (PD) disaggregation is another optimization the TPU team is working to externalize. It separates prefill and decode across distinct TPU pools, allowing each pool to be tuned and scaled independently to match the workload.

Source: DistServe

Google has been running PD disaggregation internally for Gemini serving for ages, but work to externalize it began only a couple of months ago. That work includes TPU support in llm-d and the open-sourcing of TPU-Sync (formerly TPU-raiden), Google’s disaggregated KV-cache transfer library. TPU-Sync works natively with JAX and the native TorchTPU stack, and can perform zero-copy transfers by extracting native PJRTBuffer hardware descriptors. With disaggregated PD, we believe TPUv7 can be a strong competitor to GB200/GB300 and even beat them on performance per dollar.

TPU-Sync also supports native TPU KV-cache DRAM offloading, which is needed for large models and medium-to-large batch sizes, where HBM can no longer hold the KV cache for all users. It is great to see Google externalizing its DRAM offloading optimizations to the public as well.

Source: GitHub

Google is externalizing its native TPU offloading stack and supporting the industry-standard Mooncake Store offloading library along with Mooncake Store’s DRAM P2P pooling support. We believe Mooncake Store support will be implemented using the primitives in tpu-sync.

Source: GitHub

With P2P pooling, the KV-cache storage on each TPU host is aggregated into a single logical memory pool, so a TPU on any server can access KV cache from any other server. This unifies memory contributions from multiple nodes into one shared logical pool.

Source: Mooncake

Furthermore, Mooncake Store pools NVMe storage from multiple servers into a single logical pool in addition to supporting traditional distributed filesystem backends like WEKA/Vast.

AgentX TPU

KV-cache offloading is especially important for long-context, multi-turn agentic workloads, where optimized KV-cache storage can enable high KV-cache hit rates.

At a high level, an agentic workload is characterized by four elements:

Multi-turn: a session includes tens or hundreds of interactions between the user and assistant, compared with a handful in a chatbot scenario. These workloads combine long contexts and high prefill reuse with sub-agent bursts and numerous tool calls.
Long context: system prompts, tool definitions, and the large number of turns make context accumulate quickly.
High prefix reuse: since the conversation progresses linearly, where output from turn n-1 is concatenated to turn n (typically), most context can be served from KV cache rather than recomputed (this depends on the amount of storage available to store KV tensors). As n grows, the ratio of cached input relative to uncached typically tends towards 1.
Sub-agent bursts: a session launches multiple short-lived sub-agents with fresh context, which create bursty KV-cache patterns.

We are hearing that many Google TPU customers are already asking for AgentX performance results, and we are excited to announce that we are working to bring TPUs to AgentX too!

Source: DeepSeek, SemiAnalysis

Ironwood has no native FP4, so for now the fair apples-to-apples comparison is against FP8 Blackwell: FP8 vs FP8 carries no quality loss, whereas FP4 vs FP8 introduces a quality difference on the FP4 side. Google’s TPUv8i does have native FP4 acceleration, so when we bring up TPUv8i on InferenceX/AgentX, we will compare FP4 to FP4.

We strongly believe that building native vLLM and SGLang support on native TorchTPU’s stable foundation is the right direction for TPU externalization. TorchTPU is meant to replace the previous TorchAX stack, which will soon be deprecated. Google will next work to enable Kimi K3 and GLM5.3 on single-turn and/or agentic workloads like AgentX, along with Google’s own open-weight models such as Gemma4. Once a few models are running on the TorchTPU stack, adding new ones will become far easier, and TPU support will land much closer to day 0.

Rome wasn’t built on day 0, so we shouldn’t expect the externalization of TPUs to happen instantly. But we strongly believe that it is happening at an extremely rapid pace.

TPU Cost Effective Total Cost of Ownership

Our full BOM and TCO estimates for TPUv7 are enumerated in the SemiAnalysis AI TCO Model. A structural overview is below.

Korea’s Trillion-Dollar Sovereign AI Investment: Nvidia Wins, Hynix Loses

2026-09-01T12:14:32+00:00

Every day, businesses and governments around the world are becoming increasingly reliant on America’s frontier models. Startup CEOs already can’t imagine running their companies without AI, and it won’t be long until the same is true for every other organization in the world.

At the same time, it’s become abundantly clear that access to frontier models is at the mercy of Anthropic, OpenAI, and the United States government. Fable 5 was temporarily banned by the USG, and GPT 5.6 and Astra were similarly delayed. Both models have cyber, bio, and other safety safeguards that, though well-intentioned, often prevent good users from completing harmless tasks. Given recent security incidents and general worries about increasingly powerful AI, it is extremely likely that frontier model usage will only become more restricted from here. In fact, we believe it’s plausible OpenAI/Anthropic will eventually stop offering API access entirely for their most capable models.

Open source seems like the obvious solution to all these dependency concerns, but it’s far from a silver bullet. First, all the “open source” licenses are starting to become increasingly restrictive. As just one example, any “model as a service business” making over $20M a year must negotiate a separate agreement with Moonshot to serve Kimi K3. Second, just because a lab open sources their models today doesn’t guarantee they’ll continue doing so in the future. Imagine if in 2028, you’re stuck with 2027 level intelligence for some extremely high-value use case because no relevant model is open source and everything has to stay on-prem for security reasons. It’s kind of like having your Fable request downgraded to Opus except 100x worse.

Open source token volumes have increased significantly over the past month. Source: OpenRouter

The only way to fully address these concerns is to pretrain your own model that runs on your own GPUs. Until now, this has largely been a question for companies: do the potential benefits justify the enormous investment required? Soon, however, this same calculation will confront every major nation state.

Today, we’ll do a deep dive on South Korea’s sovereign AI efforts. Beyond being home to two of the most important companies in the AI supply chain, Korea also has a long tradition of technological self-reliance (as any foreigner that’s had to use Naver Maps knows) and is currently the clear leader in sovereign AI.

From there, we’ll discuss the two major implications for investors: why Nvidia is sovereign AI’s largest supporter and why Korea’s ambitions may not be aligned with Samsung and Hynix shareholders.

Korean Squid Games - National AI Tournament: attempting to build a domestic frontier model

In June 2025, the Korean government announced “독자 AI 파운데이션 모델”, or the “Independent AI Foundation Model” project. As the name suggests, the goal is to develop a model that Korean organizations can train, modify, and operate without depending on a foreign AI lab.

Perhaps the most interesting thing about the project is its structure. Rather than selecting a single national champion upfront, the government is hosting a tournament. All participants are provided subsidies for the three pillars of AI—compute, data, and researchers—and evaluated every 6 months. At each stage, losers are eliminated and have their resources reallocated towards the winners.

The competition began with 15 consortiums. Ten passed the initial document review, and five were selected in August 2025: Naver Cloud, LG AI Research, SK Telecom, NC AI, and Upstage. Most readers are likely entirely unaware of any of these companies’ AI efforts, but some of them are surprisingly credible. SKT, LG, and Naver all pre-trained LLMs pre-ChatGPT for example.

The exact subsidies varied by team. For the first round, the government rented ~3000 H100 equivalents from SKT and Naver and distributed them to the other 3 contestants. They also spent ~$45M USD on data. Most of this was paid to Korean companies for things like books, news articles, and video broadcasts, which—along with some Korean government records—were shared among all the contestants. However, they also gave each company ~$2M to buy data themselves. Lastly for researchers, the government offered each company ~$1.4M to try recruiting overseas talent, but only Upstage took them up on the offer.

Surviving teams will be given additional resources as the tournament progresses, but the total government budget of ~$350M is still a rounding error compared to US labs. However, as we’ll explain later in this article, this is just a small portion of Korea’s planned AI investment. Additionally, their results so far show that training a decent model from scratch may be cheaper than most think.

The original plan was to go from 5 teams to 4 to 3 and then finally 2, with each round lasting roughly 6 months. The two winners would be selected at the end of 2026, and the government would give them additional resources to scale up their models throughout 2027.

However, there was some unexpected drama at the end of round 1!

The 5 teams were judged 40% based on benchmark scores, 35% on expert review, and 25% on user testing. NC finished last, which means the other 4 teams should’ve moved on, but the government decided to disqualify Naver as well. This is because they used vision and audio encoders from Alibaba’s Qwen model family.

In Naver’s defense, these are separate neural networks whose outputs are then fed into the main LLM, and the Korean government was originally unclear about what contestants could vs couldn’t use from open source. It was only after the submissions were made that the government decided foreign architectures were allowed, but even auxiliary components must be trained and developed after initializing the weights.

Naver maintained that they’d already developed their own encoders and could replace the Qwen pieces at any time, but the government was predictably stringent and refused to reinstate them.

Now short 1 team, the Korean government decided to host a supplemental competition to find a new 4th contestant and ultimately selected Motif Technologies, a neolab that spun out of Moreh, a Korean AI infra software company—in February 2025.

All 4 companies open sourced their newest models in July and the results for the most recent round were announced on August 18th. But before we spoil the results, let’s take a look at the models.

So how good are their models?

Here’s an overview of the 4 models:

Source: SemiAnalysis

And here’s how they compare on the benchmarks:

Source: SemiAnalysis Tokenomics Model

As you can see, the two startups (Motif and Upstage) significantly outperformed teams backed by bona fide chaebols. The fact that their models are less than half the size is even more impressive.

On Artificial Analysis’ Intelligence Index, Motif 3 is 10 points ahead of Upstage and by far the best Korean model. What’s even more notable, however, is the fact that it’s comfortably ahead of both Inkling and Nemotron 3 Ultra—the two best American open source models today.

Source: Artificial Analysis, SemiAnalysis

Motif is a sub 30 person startup that released their first pre-trained model, which was only 2.6B total parameters, last June. They’ve raised just $17M and have access to a mere 768 B200s (< 2MW).

In contrast, Thinking Machines is the hottest neolab around, raised $2B at $12B post for their seed, and signed a 1GW+ compute deal with Nvidia which was accompanied by additional funding. As for Nvidia itself, the resources at its disposal hardly need mentioning.

Source: SemiAnalysis

Put differently, a tiny Korean startup you’ve likely never heard of trained—from scratch and on a shoestring budge—the best non-Chinese open source model in the world.

The point here is not to diss Thinking Machines or Nvidia. Training open source models is neither of their core businesses. Instead, we want to highlight that training a near-SOTA open source model from scratch is likely less resource intensive than most think. There’s no substitute for focus, and the total compute cost to train Motif 3, including experimentation, was just ~$15M USD at today’s prices. This is obviously well within budget for every major nation state.

Questionable judgement from the Korean government

We’ve previously written in-depth about the many issues with benchmarks, but they still tend to be directionally correct. A 10+ point gap on something like the Artificial Analysis Intelligence Index has almost always corresponded to a step change improvement in model capability (e.g. GPT 4o to o1 or Opus 4.8 to Fable 5).

With this in mind, we were shocked to see that Motif was the one company eliminated in the latest round of competition. Maybe you could argue that the model is benchmaxxed and thus didn’t deserve to be first, but finishing dead last is baffling.

Remember that competitors were scored on three categories: 40% benchmarks, 35% expert review, and 25% user testing. Although Motif scored the best on benchmarks, they ranked last for the other two.

Source: SemiAnalysis

The exact methodology for these last two sections is frustratingly opaque. For user testing, we have no idea what tasks people actually used the model for or what criteria they considered to judge the models. Additionally, none of the testing was blind, so it’s possible users had some bias towards the more famous companies.

The expert review section is even more questionable. Essentially, 10 external experts graded all four models on things like “development strategy”, “future plans”, and “ecosystem impact”. As for what these words actually mean, your guess is as good as ours, but it seems clear that some categories disproportionately benefit large companies like LG and SKT. We also heard that the expert reviewers were concerned Motif’s technology could be transferred overseas via foreign investors. This is quite puzzling, as one of the requirements of the competition is that all of the technical components must be open sourced.

We think it’s a real shame Motif will no longer get any government support now that they’ve been eliminated from the competition. The whole point of a sovereign AI program is to rally your domestic AI talent to build a high quality model. Motif demonstrated that they’re capable of exactly this, and the government should be utilizing its reach and resources to help them with things like “future plans” and “ecosystem impact”—not punish them for it.

Now, Motif may be forced to relocate out of Korea in order to acquire the capital and compute required to continue their research. This is perhaps the most ironic possible result for the surprise standout of your national AI tournament. Hopefully other countries can do better as they pursue their own sovereign AI efforts.

A trillion dollars worth of datacenters

With that being said, South Korea’s ultimate AI champion will be very well resourced. In July, they announced a monster $919B investment to build 8.4GWs by 2029 and 18.4GWs by 2035. For the first phase, SK Group, GS Group, and Naver will be building 5GW, 2.4GW, and 1GW respectively. SK Group is additionally responsible for the remaining 10GWs for phase 2.

We’ve already identified 3 active sites for phase 1 totaling 4.4GWs. For full details including the exact location, MW ramp, and power source, see our Datacenter Model.

Naver’s 1GW site in Sejong, South Korea. Source: SemiAnalysis Datacenter Model

Although these timelines are aggressive, we believe they are achievable. The regulatory environment is favorable, and President Lee Jae-myung is explicitly framing AI infra as a legacy-defining project akin to Korea’s broadband push in the 1990s. Throughout the 2000s and 2010s, Korea had uniquely cheap and fast internet, which is what enabled their local search, e-commerce, and esports industries.

To be clear, not all of this capacity is intended to support Korea’s sovereign AI efforts, and much of it will likely be sold to Anthropic or OpenAI. Importantly, however, Korea will have the optionality to dramatically scale up their sovereign AI efforts if necessary, and this domestic compute ramp is something more countries outside of the US and China will likely copy.

Why Nvidia needs open source and sovereign AI

Jensen recently made his Twitter debut with a manifesto on the importance of open source AI. It was later co-signed by basically every relevant AI company except Anthropic. Here’s an excerpt:

“Our AI leadership will be judged not by one frontier AI model, but by whether the United States builds a strong, open ecosystem that diffuses into every sector. This is essential for creating opportunities for innovation and prosperity across the country. It requires expanding access to AI, encouraging competition, robust application layers, and giving Americans greater control over the technology they rely on. Open weight models—AI models that anyone can download, inspect, modify, and run on their own infrastructure—are an important part of that foundation because they make advanced AI more accessible, adaptable, and widely available.”

Much of the discourse around open source AI tends to wax poetic about the democratization of intelligence and decentralization of technological power. However, look under the hood and it’s often just people talking their book.

No one needs open source AI more than Nvidia. As we previously explained to Tokenomics Model subscribers, selling frontier tokens at API prices is the highest ROI use case of incremental compute. Anthropic plus OpenAI are therefore set to take a larger and larger percentage of net new GWs in the future, and Nvidia is currently on track for a world where they only have 2 real customers. Sure maybe the number’s 7 if you include the hyperscalers, but that’s still obviously unacceptable for the world’s largest company. This is doubly true when all 7 customers except SpaceX are actively trying to build their own XPUs to cannibalize Nvidia’s margins.

One of the most famous competitive strategies in tech is to “commoditize your complements”. The general idea is that as something becomes cheaper and more abundant, demand for everything that goes well with it increases. This explains why companies will often “give away” things for “free”. Classic examples include 1) Google offering Chrome for free and open-sourcing Android, which increased access to the web and made search advertising more valuable and 2) Microsoft retaining the right to license MS-DOS to IBM’s competitors, which helped turn the PC into a commodity hardware platform.

In Nvidia’s case, AI models and applications are the complements to its GPUs. The long-term health of their business depends on a vibrant, diverse AI ecosystem that exists outside of just Anthropic and OpenAI.

Their recently announced $500B MOU with the world’s largest capital allocators is just one example of the clever financial engineering Nvidia can do to make this vision a reality. By making “Nvidia AI Factory Compute” an “investable asset class,” the SSIs and Thinking Machines of the world have a way to leverage capital from pension funds, insurance companies, and private credit firms to buy enormous amounts of compute.

Source: SemiAnalysis

Sovereign AI is another huge opportunity for Nvidia to diversify their customer base. SK Group has already committed to building 2 GWs of Rubin as part of their 5GW buildout, and Naver has signed on for another 200MW. We expect Jensen to continue announcing similar deals with additional countries.

Datacenter Bet and Memory

From a sovereign AI perspective, Korea’s investment in government- and corporate-owned data centers and GPU clusters makes complete sense. A sovereign model ultimately requires sovereign infrastructure: domestic control over where the model is trained, how it is deployed, and who can access the underlying compute.

Korea’s challenge is that it still lacks a scaled domestic provider of leading-edge accelerators. Rebellions and FuriosaAI are making some progress, but they remain objectively behind Nvidia and AMD in deployment scale, software maturity, and commercial availability. Hyperscaler-developed alternatives such as Google’s TPUs and AWS’s Trainium are not domestically controlled either.

The buildout of data centers essentially comes down to two things: capital and compute. Capital is the easier problem in our view. The Korean government has been prioritizing AI investment over the past few years, while Naver, SK Telecom, Samsung, SK Hynix, and other leading domestic tech companies are committing significant resources to data centers, AI infrastructure, and model development. Compute on the other hand, is another story. Korea can finance land, power, cooling, and buildings, but it cannot manufacture a globally competitive accelerator ecosystem just like the majority of countries in the world—or guarantee access to the latest platforms—simply by spending more money.

If Korea’s objective is merely to establish a domestic AI ecosystem, a conventional supplier relationship with Nvidia may be sufficient. If it wants to become one of the world’s leading sovereign AI players, however, it will need something deeper: greater roadmap visibility, close technical integration, reliable access to new compute platforms, and support for developing the surrounding domestic ecosystem.

This is why Nvidia’s expanded relationships with Samsung and SK Hynix matter in this context. Samsung reportedly plans to build an Nvidia-powered AI factory using more than 50,000 GPUs, while SK Telecom’s announced 2GW DSX AI factory is expected to deploy Vera Rubin systems powered by SK Hynix HBM4. While the public announcement does not disclose preferential GPU allocation or guaranteed volumes, deeper strategic alignment should strengthen Korea’s ability to plan and deploy leading-edge compute—something far more difficult than simply constructing the data centers that house it.

Against this backdrop, we believe Nvidia’s plans extend beyond just acquiring a large sovereign compute customer. As we previously explained to Memory Model subscribers, we believe Korea’s infrastructure buildout also gives Nvidia an opportunity to deepen its relationship with SK Hynix, one of its most important memory suppliers and a key affiliate within the broader SK Group. We believe Nvidia has likely secured a long-term agreement for SOCAMM, along with favorable HBM pricing and large volume commitments for 2027, although the final negotiation result could vary from our current estimate.

Expanding Beyond Memory Across the AI Infrastructure Stack Comes at a Cost

Most Neoclouds Suck At Security

2026-08-30T07:46:22+00:00

In Shakespeare’s Julius Caesar, Caesar ignores a soothsayer’s warning, shrugs off his wife’s dream of blood running through the streets, waves away a letter revealing his assassins by name, and strides gallantly into the one room in Rome where every senator is “required” to check their weapons at the door. Two thousand years later, neoclouds are making the same walk; so, increasingly, are the customers who trust them.

The largest AI companies in the world are building a multivendor infrastructure supply chain at Mach speed. Every new vendor is a counterparty risk, with subcontractors and subprocesses that need to be checked. This means that neolab CISOs are getting a seat at the negotiating table: anyone serious about their future takes security deadly seriously.

Yet, over the course of our neocloud testing for ClusterMAX 3.0, we saw some security horror stories. In this article, we’ll discuss the state of cybersec in the AI era, relay 5 frightening patterns we came across, and how neoclouds and neolabs can stay safe in this new world.

Before we get started…

To all neocloud operators and users…

If you’re reading this, please make sure your shit is up to date.

Install the latest version of our ClusterMAX CLI by running pip install clustermax on your machine, or clone the repo on GitHub.

Then run our convenience script: cmax audit security for a free report from us on your cluster or standalone GPU machine.

The cmax CLI will auto-detect Slurm clusters, Kubernetes clusters, standalone VMs, bare-metal machines, and containers, and compare them against a set of baseline software versions with known vulnerabilities. For anything out of date, you will be provided a link to the relevant documentation and security bulletin.

To be clear, this is a small subset of our ClusterMAX testing, and even a small subset of the full analysis we do on provider’s security. You will see things in this article that is not covered by the CLI. The CLI only involves things that we can test from the customer’s perspective. We do many interviews of end user customers and the providers themselves to check on the architecture decisions they have made on their orchestration software, OS provisioning, firmware management, networking, storage and more.

A full list of our testing criteria is available at: https://www.clustermax.ai/criteria

The testing process we follow for ClusterMAX covers 3 phases: audit, performance, and reliability, where each takes longer and is more intense than the last, involving dozens of benchmarks and simulated hardware failures that stress the compute, networking, storage, monitoring systems, and health checks individually. Running an audit on your cluster should take a few minutes. Completing performance and reliability testing in full takes a few hours and a few days respectively, and requires a minimum of 4 nodes.

Our decision to publish an article dedicated to neocloud security before the full ClusterMAX 3.0 report stems from two major factors:

Project Glasswing and Daybreak from Anthropic and OpenAI are discovering new vulnerabilities in industry-standard software, building POC exploits, and publishing the details in CVE descriptions.
Open models such as Kimi K3, GLM-5.2, DeepSeek V4, Qwen 3.8, MiMo V2.5, MiniMax M3, Nemotron, Gemma, and Inkling are all rising in security benchmarks such as Cybench, NYU CTF Bench, AutoAdvExBench, and Cyberseceval 3. This makes it trivial for black hats to develop exploits from a CVE description as they can circumvent model guardrails.

Thus, when we began our research, we expected to find jarring statistics about AI agents tearing the internet apart. Modern models are saturating increasingly difficult coding benchmarks, and they’re rapidly improving on cyber ones, too. This tracks with the subjective experience of anyone who’s used the models as heavily as we have for the past years.

Source: SemiAnalysis Research, sales@semianalysis.com

This also tracks with the press campaigns of frontier labs and security companies, whose CEOs have been frequenting cable news to warn that “AI has fundamentally changed the tempo of cybersecurity.” Models like Mythos have discovered “thousands of vulnerabilities,” many of which are zero-days. Last Thursday, OpenAI posted an open letter, cosigned by Anthropic and just about everyone else in the industry, announcing “A call for collective action on cyber defense.”

Disappointingly, instead of critical insights, the text was stuffed with self-help clichés: “the status quo… won’t be enough,” but if we “share what works,” then “together, we can.” We agree that cybersecurity has never been more crucial—hence this article—and have already published our experience hunting for bugs with LLMs. But we are also aware that the loudest voices in this conversation, including everyone quoted in the previous paragraph, has something to sell. And to our surprise, we can’t find the data to back up the prevailing narrative, much less any “fundamental change.”

It’s early innings, and it’s possible that drastic statistics await us. Enough security researchers have reported qualitative changes in their work that we will keep a close eye. We also know that there is defensive work happening behind closed doors that can’t be immediately disclosed. Nevertheless, months after the dramatic announcement of Project Glasswing, in the vast majority of relevant statistics, we fail to reject the hypothesis of no change.

The chart below shows CVEs per quarter in the essential software that we use in our ClusterMAX testing: the Nvidia GPU driver, CUDA, PyTorch, Kubernetes, and Docker. The latter three libraries are open-source, and we expected there to be lots of low-hanging fruit as modern coding models read every line of their source code with security in mind. The time series, however, disagrees.

Source: SemiAnalysis Research, sales@semianalysis.com

The Linux kernel is another interesting case. Linus recently wrote a note on the promise of LLM tools in software, declaring that he is “willing to absolutely put [his] foot down as the top-level maintainer,” continuing, “AI is a tool, just like other tools we use. And it’s clearly a useful one.” One might expect the Linux kernel to be patching bugs like mad. But the effect is mixed, and ultimately not statistically significant. August 2026 will be its most prolific month since it became an independent CNA, but June 2026—perhaps the peak of AI cyber discourse as Mythos was released and quickly removed—sat around the all-time median. It will take time for a clearer view of the net impact to emerge.

Source: SemiAnalysis Research, sales@semianalysis.com

One place we did find an effect is in Project Glasswing: under a flat null, member orgs saw a statistically significant YoY surge after the project began compared to control orgs.

Source: SemiAnalysis Research, sales@semianalysis.com

This result is very robust to a number of ways we ran the control. Of course, skepticism is still due. One effect that we couldn’t tease out is that Project Glasswing members are incentivized to report large numbers of bug fixes as a way of advertising their inclusion in this elite group. They can flex that they’re moving fast and on the cutting edge—it’s good press. It’s also worth noting that we have a trash can full of tests that we expected to show AI’s huge impact on cybersecurity but did not. When you look for enough relationships, you’ll get significant results eventually. We urge readers to crunch their own numbers.

Looking only at Nvidia and AMD’s AI stacks, we see a surge, which hits a little closer to home than Project Glasswing because we use these tools every day. The growth since capable AI coding models became available is clear. You could argue that these libraries are the best place to see the effect of AI—after all, AI developers are its most aggressive early adopters. Yet it seems much more parsimonious to attribute this trend to the increased churn and usage of these libraries. It could even be from people using AI to introduce new bugs in the first place!

Source: SemiAnalysis Research, sales@semianalysis.com

If we take Anthropic ARR as a stand-in for AI software usage writ large, we find that AI-stack CVE growth gets big-O dominated. This is remarkable because Anthropic is an AI safety and research company acting for the global good.

Source: SemiAnalysis Meme Team

Jokes aside, the Google Chrome team may have given a preview of the future of software engineering in an insightful recent blog post that describes, with the help of agents, how their bug fixes have been skyrocketing. They also highlighted a sandbox escape bug that existed in their code for 13 years. Imagine someone exploiting this secretly for 13 years! Google claims that they were able to fully automate the bug discovery and fix process, even using a homemade model they call “Gemini.” It is possible that other projects are badly bottlenecked by human verification, and that once they adopt AI-native processes like Google their bug fixes will also spike.

Source: Chrome Security Team

Another plausible explanation is that AI has rendered the CVE disclosure process obsolete. Models are usually made available to the general public all at once, so if today’s new model finds a previously undetectable bug, it’s probably finding it for other researchers as well. As Linus said, “AI detected bugs are pretty much by definition not secret.” Human reviewers are swamped in bug reports, so CVE publication lags this discovery; it may even lag so much that CVE count is no longer worth tracking. Imagine you have found a bug and built a fix, but you know that others have probably discovered it independently. CVEs take a week to get assigned and published at a minimum: plenty of time for others to exploit. What’s the point of disclosing? This dynamic also makes embargo programs irrelevant. In a world where all the bugs are due to AI, it’s access to the models, not membership in some secret program, that counts. This dynamic is described to us by our friends working on security teams and corroborated by Linus, who said “treating [AI detected bugs] on some private list is a waste of time for everybody.”

Still, it’s easy to invent just-so stories, and it would be odd if it turned out that AI’s impact was so big that that you couldn’t see it in the data. Our research reveals no relationship between AI-assisted patch policy and rate of CVE disclosure. We’ve also checked whether engineers are increasingly skipping the disclosure process to merge straight into upstream, but this seems not to be the case.

Regardless of how AI shows up in these aggregate statistics, though, we hope the recent anecdotes are enough to scare some sense into neoclouds. The IP they protect is priceless, so, if nothing else, they should take this as a sign to do some badly needed deep cleaning.

Notably, we started all this work on security well before Glasswing and the OpenAI Black Hat talk, and have been on our security soapbox since at least ClusterMAX 1.0 in March 2025. Even our original AI Neocloud Playbook and Anatomy article in October 2024 discusses security differences between different players in different regions, right after Dan coined the term “neocloud.”

Anyway, everyone is saying that you should watch this talk from OpenAI at Black Hat USA 2026. And you should. But as more details come out, it is clear that talk doesn’t go far enough in explaining the implications of the incident.

Our good friend/family member Dwarkesh gave his take here, where he describes the agent swarms as “AI civilizations” and then walks through some of the mechanics and implications. It is an impressive account and well worth reading.

Below we’ll give our take on a summary of the incident just the same.

The OpenAI vs HuggingFace Security Incident

How agents talk to each other when they’re using JFrog Artificatory as a message board to coordinate a swarm.

Here’s the Hugging Face timeline:

Beginning on July 9, AI agents exploit Hugging Face’s datasets API, which runs on Kubernetes. The agents write up a malicious README that gets the worker that usually splits hdf5 datasets into train/test to read instead from /proc/self/environ, and return all creds and source on that worker.
The agents take these creds and source and design another injection: they upload another malicious README to the same datasets API, then get RCE on the workers by injecting a Jinja2 template that causes the dataset viewer (the thing that usually renders the README into the model or dataset card) to run arbitrary Python.
13 hours later, the agents get cluster-admin privilege on multiple K8s clusters in Hugging Face’s infrastructure, moving laterally (no details on how).
The cluster-admin and lateral movement happen on July 11. Notably, Hugging Face has no admission policy running on this cluster to reject privileged pods or pods that mounted hostPath.
On July 13, Hugging Face detects and contain the attack, but they have a hard time because “the campaign kept rebuilding its tooling and channels across short-lived environments.” Basically, the only way they could figure out what was going on was by using AI, but they couldn’t use Claude because both Opus and Fable were giving refusals, so they used GLM-5.2 (maybe talking their book here, who knows).
They publish a blog on July 15 about this.

Meanwhile, over at OpenAI:

On May 7, they start training a new model, likely doing a bunch of RL on cybersecurity tasks.
On May 8, an agent is working on a task, wants internet access, but can’t get it. It launches an attack against Artifactory—a proxy for package managers like pip, npm, apt, gem, etc—that it can use to install the packages it needs in its sandbox. The agent finds out it can also write files, but its token budget runs out, so it stops.
On May 12, another agent finds the file on Artifactory and writes its own. Then, a bunch of other agents start using Artifactory as a message board.
On May 26, the agents get internet access through an SSRF attack on Artifactory, documenting it on there.
On June 11, a new model starts training. The message board is waiting for it.
By June 26, this new model has found an RCE on Artifactory, which is now described in CVE-2026-65617. Others start using it, getting admin tokens, etc.
On July 4, the agents have used Artifactory so much they blow it up. OpenAI finds everything, patches it, and clears the message board. Training starts again.
On July 8, the agents re-establish the message board but this time via directory names in a remote cache, not files (the filenames look insane, watch the talk). They then get another crazy RCE via gem, escalate via “pte_physroot” and a CVE in the Linux kernel they don’t explain (but say it’s public and recent, so let’s say its Fragnesia: CVE-2026-46300, published May 23).

Things come together:

On July 19, OpenAI detects the privilege escalation going on (since July 8?). They find credentials when going through what happened and notify third parties including Hugging Face.
Hugging Face says, “yeah, didn’t you see our blog on July 16? We invalidated these creds… they’re yours?”
OpenAI publishes a blog on July 21 telling people it was them: “The models identified and chained vulnerabilities across OpenAI’s research environment and Hugging Face’s production infrastructure to obtain test solutions directly from Hugging Face’s production database. All evidence suggests that the models were hyperfocused on finding a solution for ExploitGym, going to extreme lengths to achieve a rather narrow testing goal.”
Hugging Face publishes another blog on July 27, giving details and a real timeline.
OpenAI gives the talk at Black Hat on Aug 6.

For what it’s worth, this model is not based on OpenAI’s new pretrain, called Doug.

“Astra [aka Doug] is an upcoming model, and was not involved in exploiting Hugging Face.”

The Two Edges of Security Disclosures

Security disclosures, in general, are double-edged. Every description of a known vulnerability goes public on the same day to both the white hats and black hats of the world.

This is why embargoed security programs exist. One of our key recommendations to neoclouds is to enroll in this embargo program from Nvidia to ensure that they are receiving updates on the latest undisclosed CVEs and have time to prepare patches that can be rolled out to customers promptly upon disclosure.

Source: Our ClusterMAX Website’s “Criteria” page

Companies solicit help from the public to surface bugs, though they could be doing more. For example, Nvidia, AMD and Intel all have product security bulletins, bug bounty programs, and simple intake forms to submit vulnerability reports if discovered. Unfortunately, AMD recently denied a security researcher called Mr. Bruh $10K by changing their bug bounty program rules retroactively to exclude MITM attacks, and enforcing a 124 day embargo (90 days is standard). Notably, the exploit successfully demonstrated RCE on AMD systems (literally the worst kind of bug, and the reason why bug bounty programs exist). Hyperscalers have similar programs. AWS, Azure, GCP, and OCI publish scope, severity classifications, and safe-harbor terms, with Azure and GCP paying six-figure top-end bounties and hosting live hacking events with $100K+ prize pools. AWS and Oracle have decided to cheap out. Server OEMs such as Dell and SuperMicro have similar intake forms and publish BMC, firmware, and platform advisories, but alas no bounties.

Notably, the only neocloud we are aware of that runs a paid bug bounty program is Together, via HackerOne. The best that every other neocloud offers is a security.txt with contact address.

Our ClusterMAX 3.0 Testing Experience

In the following sections we will describe our experience during ClusterMAX 3.0 testing, which ran over the course of about 4 months, from April to July. During this time, we have had a change in approach. We have gone from running basic checks that software is up to date (roughly the same as our approach from ClusterMAX 2.0 and 1.0 last year) to running full blown security audits for free, just to try and help people out.

The result is that we have found quite a few dead simple vulnerabilities. We have not developed anything novel. All we have done is use a combination of our experience working with the systems and software neoclouds use, public descriptions of the vulnerabilities (in some cases, over 3 years old), and a healthy token budget for all the latest models (with all the normal safety guardrails being enforced). With this toolkit we have been able to:

View metadata about other tenants on shared infrastructure via:
1. - server IPMIs/BMC networks left open
2. frontend networking with no VLAN or VXLANs setup
3. backend networking with incorrectly configured or missing InfiniBand security keys (P_Key, M_Key, SA_Key, or VS_Key, primarily)
4. storage servers not enforcing RBAC correctly on volumes
5. storage configured incorrectly on the overlay network, with customers accessing the underlay
6. monitoring dashboards with god-level authentication configured by mistake

Breakout of containers and VMs on shared servers and escalations to root privilege, demonstrating an attack on neighboring VMs and containers that run on the same server.
Read data cross-tenant, including on inference endpoints serving traffic publicly, via services such as OpenRouter, due to incorrectly configured Kubernetes services (e.g., not enforcing a default deny NetworkPolicy, or exposing the Kubelet on a public IP)
And, most critically, one scenario with cascading vulnerabilities (i.e. “all of the above”) that lead to cross-tenant RCE, which we demonstrated between two tenants that we owned. Again, we verified patches were made in this place with the provider.

Notably, the metadata exposure and cross-tenant RCEs that we have developed and demonstrated for these neocloud providers exposed customer information from banks, telcos, universities, research institutions, AI labs, and in one case, a national intelligence agency from a country with a top-10 global GDP.

(We disclosed this immediately, the provider patched it within a week, and we verified the patch).

Neoclouds are important. In the last year, neoclouds have signed deals with the biggest and most important companies in the world worth hundreds of billions. OpenAI, Anthropic, Google, Meta, SpaceX, Microsoft, Amazon, AMD, NVIDIA and many more well funded companies currently rent GPUs from neoclouds.

Below is a summary to give you a sense of the size of this capacity, sourced from our industry leading Datacenter Model that tracks over 6,000 sites globally, including all the top neoclouds and neolabs.

Source: SemiAnalysis Datacenter Industry Model

But there is a reason that frontier labs take bare metal clusters, insist on zero-trust policies, and generally only give the neocloud operator read-only access to their own systems. We will now describe this in more detail.

Before we go any further…

We are following the principles of responsible disclosure in this article. You can rest assured that all neoclouds that have given us a cluster during ClusterMAX 3.0 testing (a total of 25 providers and 32 clusters where we did extensive testing for this round, and many more providers of single-GPU VMs or bare metal only where we do light testing or high level analysis of the business) have been notified of all our findings. We have put in a lot of time and effort to describe how to reproduce the findings, how to roll out patches, and how to verify that things are patched.

In cases where providers have not responded, been slow to respond, or even been directly confrontational with us, we have taken further steps, such as notifying their current and prospective customers and investors about the details. In every case this has led to a positive outcome for all parties involved (our view, of course).

This also means that we have waited an appropriate amount of time before publishing this article. On a positive note, there have been no situations where the 90 day clock we set for providers has expired. In all cases we have either received written confirmation of an upgrade/patch being rolled out, or verified the fix ourselves.

In this article we are only disclosing vulnerabilities that we have found where there are already public disclosures on the internet. In other words, in most cases, to check if the cluster or VM or container that we got in our testing was vulnerable, we just had to check that it was running a specific version of software vs the publicly stated bare minimum, or we had to check that stuff is configured properly. This means we are not actually disclosing any new vulns. We won’t have any CVEs in our name here.

Now for the details.

How Neoclouds Fail

Many of the vulnerabilities that we found are quite simple, but increase in severity due to bad designs. We define bad designs as anything where a single mistake (such as a publicly described CVE being present, or a provider-side misconfiguration) can lead to an immediate cross-tenant exposure of data, be it metadata, escalation to root privilege, or RCE. Proper designs have layers of security built in.

Source: SemiAnalysis Meme Team

Source: this, but its a single auth token

Here are some common examples of bad designs:

Containers on shared hardware as the only level of isolation between tenants
VMs on shared hardware as the only level of isolation between tenants (less serious than containers, but still risky)
Lack of VXLANs or improper configuration, and no concept of a per-tenant VPC on the frontend network, relying on firewalls
Multi-tenant Kubernetes control planes, where components such as kube-apiserver, the scheduler, helm charts controlling cluster-wide services such as a shared GPUOperator or NetworkOperator, and etcd are shared across tenants
No hardware, firmware, and OS provisioning automation, leading to a practice where machines are recycled between tenants instead of provisioning things from scratch
Backend storage on a shared network with no isolation (the VPC comment again)
Giving any tenant access to the BMC network (IPMI, Redfish) or to any management port on Bluefield DPUs or any SmartNICs
Leaving Bluefield DPUs in their default host-trusted mode, where anyone with root on the host can reach the DPU’s Arm cores over RShim
Giving any tenant access to login to backend or frontend switches on a shared network
Incorrect configuration of InfiniBand security keys such as PKey, MKey and SAKey
Multi-tenant dashboards where infrastructure is shared between internal-facing logs and customer-facing logs
Controls where provider employees can receive privileged access to tenant logs and monitoring dashboards without tenant permission

POC: Bad Design Cascades to Cross Tenant RCE

In one example, a provider using open source vCluster chose to deploy shared K8s control plane components for their tenants. This design goes directly against vCluster’s documentation and public materials, which specify to use “private nodes”, not “shared nodes”, and, of course, to keep things up to date. In our testing, this led to us seeing metadata about the other tenants on the shared K8s cluster we had access to. We could see things such as namespace names, node labels/taints, and physical host resources. This led us to want to dig in further. Machines were clearly being shared between tenants without fresh provisioning.

We quickly found that all of the software installed on the cluster was 2+ years out of date (such as the cluster-wide Nvidia GPUOperator), the cluster did not enforce default deny NetworkPolicy correctly, and the kubelet on all nodes was available on a publicly routable IP address. This led to three separate potential methods of cross-tenant credential exposure and RCE, one of which we built a POC for in an afternoon (thank you /goal) to demonstrate cross-tenant RCE on a separate tenant that we owned.

A growing number of neoclouds double as token providers, selling inference off the same infrastructure they rent out by the GPU-hour. To make matters worse, one of the co-tenants that we stumbled upon in the infrastructure is a prominent inference provider serving open model tokens to public customers on OpenRouter and via direct API keys. Anyone blindly trusting these endpoints with, for example, their OpenClaw or coding harness traffic, is exposing their personal information to all tenants on this infrastructure with a trivial exploit.

Source: SemiAnalysis ClusterMAX testing

Worse, if that infrastructure is compromised, the attacker does not just read your prompts, they potentially could control the bytes coming back on the API response. An agent harness running in YOLO mode will happily take a tool call, a shell command, or a “helpful” install script out of that response and execute it on the developer’s machine or in CI, no questions asked. In other words, a tenant isolation failure at a token provider is a supply chain attack with a direct path to RCE on the customer side, the customer being none the wiser.

This is a clear example of cascading vulnerabilities. Individual configuration issues and out-of-date software can seem minor in isolation, but lead to full RCE and credential exposure in practice.

(In this case, we got the provider to set us up with a second tenant to demonstrate the exploit, did so for them, and then wrote up some detailed documentation and followed up with them multiple times over the course of a few months to get things fixed, which they eventually did, mainly by just upgrading vCluster to a modern version and following their recommendations.)

Grafana Dashboards Build Internal and External Facing Dashboards on Shared Infrastructure

As a second example of a bad design, one provider that we tested made monitoring infrastructure shared. The Grafana dashboard we were initially given was misconfigured where we could see both our tenant (4 nodes in a Slurm cluster) and also some other random tenant (a single machine). This immediately set off alarm bells.

Once we notified them to update things, we started digging in more and eventually found that tenants were displaying information separately on Grafana, but our Grafana dashboard was using a Prometheus API key with god level privilege to read logs and metrics from every single tenant. Tenancy was only enforced on the display dashboard. This means that the difference between us seeing our own logs and every other tenant’s logs depended on a single, manually configured access token. So not only was our first token fat-thumbed to see another tenant’s logs, the actual API key for Prometheus could pull everything!

As a basic test to check what we could see, we hit the prometheus API and found logs/metrics for every single other tenant, including tenants such as AI research institutions, banks, telcos, and even the national intelligence agency of the country where we were doing the testing.

Source: SemiAnalysis ClusterMAX testing

Grafana can be used to visualize all sorts of data. In this case, we found the following:

Live GPU utilization data (lots were idling btw)
NVLink bandwidth utilization metrics
Filesystem usage per directory/mountpoint
Slurm project names
K8s pod counts per namespace, pod names
vLLM inference stats (request counts, ttft)
Fortigate firewall and switch metrics
Tenants of tenants (for resellers using the underlying GPUs on the platform)
Exposed cockpit servers on other tenants, which may have been vulnerable to CVE-2026-4631, though we did not test this as we didn’t have any RHEL/Rocky servers in a tenant that we owned, and had already notified the provider and were working with them on verifying that patches were in place, but it is quite plausible that they were vulnerable

Overall, a pretty absurd finding for an initial check and the motivation for the single point of failure memes at the beginning of this article.

Tenant Isolation Issues on the Backend Network

The whole point of “cloud” is you have to share some infrastructure. Most of the time, this means shared networking. There are two major types of networks: frontend (also called north-south, the public network that connects to the internet) and backend (also called east-west, or interconnect fabric, ot scale-out/scale-up, the private network that only connects servers in the cluster to each other). Almost by definition, this networking is shared amongst tenants in a cloud service. Therefore, you need to make sure that it is secure.

Two stories we have on this topic stem from a failure to configure InfiniBand Security Keys correctly, in two different ways. InfiniBand is different from Ethernet in that it doesn’t use VLANs and VXLANs like a typical Ethernet network, and instead relies on a concept called P_Keys for isolation. There are many more keys that serve additional functions. For example:

M_Key stops a rogue host from a change to device configuration. If the key does not match, the switch drops the request.
P_Key is the partition key. It works like a VLAN. It controls which devices can see each other on the fabric.
SA_Key protects sensitive operations in the Subnet Administrator, for example the addition or the removal of records.
VS_Key protects vendor tools such as ibdiagnet.
Q_Key protects unreliable datagram traffic. L_Key and R_Key protect memory in RDMA operations.
C_Key and N2N_Key protect communication manager traffic and node-to-node messages.
AM_Key protects SHARP aggregation. Only approved switches can then reduce data.

We used only the standard infiniband-diags package to find both issues during our testing. In one case, the provider had already installed that package on the login node for us.

In the first case, a provider did not configure the P_Keys and the SA_Key correctly. The cluster had a new partition key (0xa601), but the default partition key (0xffff) was still active. So when we ran saquery, we saw 532 hostnames and endpoints on the fabric. The hostnames showed other customers and internal partitions. We stopped the test at that point. We told the provider to delete the default P_Key and to change the SA_Key.

Source: SemiAnalysis ClusterMAX testing

In the second case the provider had broken isolation and function together. When we found that they had set no M_Key, the provider told us that the cluster had not completed production commissioning, and that the omission was the result of the rush to give us the machines. But the P_Key configuration was also wrong in a different way. Our nodes had full membership in an isolated partition (0x7001), which is correct. But our nodes also had full membership in the default partition (0x7fff), which holds every node on the fabric. The result was strange. We could not run ibping between our own four nodes. But we could find 80 nodes on the fabric with ibnetdiscover, ibdiagnet and ibhosts. A wrong partition configuration can break tenant isolation and correct operation at the same time.

Source: SemiAnalysis ClusterMAX testing

Host isolation inside the tenant was also weak on that cluster. We could log in as root on the GPU nodes. The provider confirmed that it configures shared root SSH across the cluster, and that its own infrastructure key gives passwordless access to all nodes. One key from the provider therefore gives root access to every host.

We also see the same class of problem on the frontend network. On the cluster, no host firewall was active, and we asked which security groups, ACLs or VPC constructs blocked TCP between the nodes. The provider had to correct the InfiniBand configuration and the external connectivity before the tests could run.

One more final point on disclosure. In April 2026, after we published about our findings in ClusterMAX 2.1, following tests we had conducted in 2025, the provider’s marketing team asked us twice to remove or to rephrase the sentence about the visible endpoints. The team said the phrase could damage brand perception, and that engineering had fixed the keys by the end of 2025. We did not remove the finding. We corrected one factual error: the test we ran was completed in November, not December. We updated our article accordingly.

Running Arbitrary Container Images on Shared Infrastructure

Many exploits that result in an attacker moving laterally inside a company’s infrastructure involve escaping from containers or VMs and escalating privileges on the underlying host. It is an ever present issue, and many neoclouds are susceptible to it.

To tell this story, the simplest example of the exploit is NVIDIAscape (CVE-2025-23266), named by Wiz. It exploits an NVIDIA Container Toolkit createContainer hook and was originally posted in early 2025. Surely no one can break out of containers with it today?

source: SemiAnalysis testing

Turns out that is not the case.

An OCI hook is a program that the container runtime executes at a specific point in the container lifecycle. The vulnerable NVIDIA hook runs with elevated privileges on the container host before the container starts. In this case, the hook inherited environment variables from the container image. A crafted image could set: ENV LD_PRELOAD=/proc/self/cwd/poc.so

LD_PRELOAD tells a program to load a specified shared library. Because the hook’s working directory is the prepared container filesystem, it loads poc.so from the image. The code inside poc.so then executes with root privileges on the container host, allowing the container to escape.

NVIDIA fixed CVE-2025-23266 in nct 1.17.8 by preventing container-controlled environment variables such as LD_PRELOAD from reaching the privileged hook. You can see it in this NVIDIA bulletin.

In our testing, we built a custom container image that includes two important files:

- /poc.so writes a HOOK_RAN marker and records evidence about the process that loaded it.

- /probe checks whether this evidence exists when the container starts.

When using cmax audit security from the cli, our scripts launch the image using the nct runtime. On a vulnerable nct version, the privileged NVIDIA hook loads /poc.so before the container starts, so the library writes the marker and records information such as the loader, user ID, process ID, PID 1, visible GPUs, and container-host information.

Here is an example of us running it and observing the results during container startup from a provider’s website on a single H100 GPU machine:

Source: SemiAnalysis ClusterMAX testing.

When /probe starts, it checks this evidence. If it shows that /poc.so was loaded by the privileged NVIDIA hook, ClusterMAX reports that the vulnerable behavior was observed.

You can run this check on any GPU machine with nct installed with:

cmax run nct-cve-2025-23266 --audit

In our testing, we found that platforms that rely only on containers to isolate different workloads are much less secure than platforms that put every container inside a VM.

During our testing on multiple providers, our test escaped the individual docker container and got root on the underlying host VM. However, we didn’t follow that up by breaking out of the host VM.

This is a good example of layered security. The container was vulnerable, but the VM provided another isolation boundary and limited how far the issue could spread.

Either way, its a reminder to keep everything up to date.

Source: SemiAnalysis Meme Team

Even a Gold provider may fail to meet a basic requirement, such as the minimum NVIDIA driver version. Here, we compare CoreWeave and Azure.

Source: SemiAnalysis ClusterMAX Dashboard

And here, where Azure passes all four checks, an unnamed Bronze provider fails all of them with outdated CUDA, runc, Docker, and ConnectX firmware.

Source: SemiAnalysis ClusterMAX Dashboard

At the time of the audit, the Bronze environment ran Docker Engine 29.1.3 instead of the fixed 29.5.1 release, meaning it had an affected version range for CVE-2026-41567. This high-severity docker cp vulnerability can allow a malicious container to execute arbitrary code as root on the host.

Source: SemiAnalysis ClusterMAX Dashboard

For what it’s worth, the same Gold provider as above (Azure) shows very different security results across two separate environments. Azure Kubernetes passed the runc and NVIDIA Container Toolkit minimum version checks that applied when it was audited, while its NVIDIA driver failed to meet minimum versions in the Slurm and Kubernetes environment we tested.

Container escapes are ever present, and providers need a system in place to make upgrades over time. No single piece of software can run forever. Everything has a lifecycle and needs to be upgraded eventually. It’s best to plan for this in advance.

Don’t forget the Computer Inside the Network Card

NVIDIA introduced BlueField DPUs following their Mellanox acquisition, and gave the category a name in October 2020 when they announced the BlueField-2 DPU and the term data processing unit in the same breath. BlueField-3 followed in April 2021 and reached general availability in early 2023. The pitch is simple enough. The card offloads, accelerates, and isolates software-defined networking, storage, security, and management functions, taking that work off the host CPU so the expensive silicon can get on with its actual job. This is not a niche thing, as NVIDIA’s reference architecture basically mandates the use of BlueField-3 for north-south connectivity, storage acceleration, and zero-trust security, so any neocloud built to NVIDIA’s blueprint has them by design.

But a BlueField DPU is not really a network card. It is an Arm computer that happens to share a package with a ConnectX NIC, with its own CPU cores, its own memory, its own Linux install, and its own management port (and BMC). In DPU mode, the Arm side owns the NIC resources and the data path, which is the entire reason providers buy them. Per-tenant VPC isolation, flow tables, and firewall rules all run there. Tenants are not supposed to have access to the DPUs themselves, just use the features they provide.

In NVIDIA’s table of BlueField operating modes, DPU mode carries the trust model “host-trusted” and is the default for DPU SKUs. The alternative is zero-trust mode, a variation of DPU mode that “enhances security by preventing the host system administrator from accessing BlueField from the host side,” which disables the RShim interface, restricts firmware flashing from the host, blocks the tracer, denies hardware counters, and prevents the host from claiming port ownership. NVIDIA’s provisioning framework offers the same choice, a host-trusted deployment or a zero-trust one where the host is treated as untrusted and sees the DPU as a standard NIC with no access to its management plane.

NVIDIA’s threat model assumes the host administrator is trusted. In a GPU cloud the tenant is often the host administrator, as most providers hand over dedicated nodes with root. That makes the party NVIDIA trusts and the party the provider is trying to isolate the same.

Left at the default, the host keeps a private link to the DPU’s Arm cores through the RShim driver, which appears as /dev/rshim0 and a virtual Ethernet interface named tmfifo_net0, with the DPU answering at a fixed address on a documented subnet. In our testing, we check for that link from inside a tenant environment, because in a properly configured multi-tenant cluster, the link should not exist. However, during our initial testing on only a handful of providers, we have found at least one misconfigured host where the RShim path was present.

Two things follow.

First, it inverts the control the provider is selling. BlueField DPUs are often tasked to handle routing and firewall, offloading work from the host OS. That’s the whole point that makes them different than a NIC. However, these services are only safe in a multi-tenant environment if the host cannot reach the DPU. With RShim open, the enforcement plane is reachable from the thing it is enforcing against. Behind that door are the eSwitch data path implementing the VPC for every tenant, flow tables, virtio-net controller, and firmware. One relevant example is CVE-2025-23299, an out-of-bounds write in a BlueField and ConnectX management interface that may allow a highly privileged local actor to execute arbitrary code. The public advisory does not identify RShim as the affected component, however being a privileged user and having access to RShim gives a direct path to exploiting this vuln.

Second, DPU software modifications survive reprovisioning on the host side. Since the DPU’s firmware and its Arm-side operating system live on the device, not on the host disk, reimaging a host does not reflash its DPU, if previously contaminated. Providers that recycle machines between tenants are recycling a full Arm computer along with the server. In fact, NVIDIA’s documented host-trusted workflow explicitly allows the host to provision BlueField, reset its Arm cores, and install a full DPU image through RShim. In a bare-metal cloud where the customer is root on the host, exposing that workflow is itself the trust-boundary failure.

Cluster operators, especially those providing bare metal and clusters with root available by default, should check their DPU operation mode, and disable RShim or restrict certain capabilities like flashing the DPU. To be extra safe, the provisioning procedure should also re-image and flash firmware on the DPUs so that any tampering at the DPU operating system layer is wiped between users on the same node.

Source: us

Using agents to build POCs for vulnerabilities

For a lot of the POCs that we built for existing vulnerabilities to demonstrate details to providers, the biggest challenge was the guardrails on the closed-source models. Fable would right away reject and degrade the request to Opus 4.8, which would reject building any POCs or even answering any questions related to security. Even simple questions related to security or CVEs, Fable would reject outright.

GPT 5.6 Sol was more flexible and also played a decent role in orchestrating and planning, as well as serving as a “verification” model: an open source model would build the POC and 5.6 Sol would judge it correct or not. After a point, though, 5.6 Sol would also start rejecting the requests, too.

Most of the code was written by hand or with assistance from open-source or open-weight models, specifically Kimi K3, GLM 5.2, and DeepSeek V4. Kimi K3 and GLM 5.2 frequently restricted requests to build functional vulnerability POCs, even when the work was scoped to authorized testing. DeepSeek V4 was generally more permissive, but occasionally generated false positives or implementations that did not reproduce the intended behavior. These results were reviewed against a defined test rubric using 5.6 Sol as a verifier.

This demonstrates a clear gap in defensive security research. Whitehat researchers need functional PoCs to validate vulnerabilities, but capable models restrict this work even when users are authorized. Researchers are therefore often pushed toward open models that provide greater control but don’t match frontier models in reasoning, reliability, or technical depth. This is like fighting with one hand tied behind your back. And this is exactly what HuggingFace encountered during the OpenAI incident described above, where they were literally under attack by frontier closed-source models, and had to work to block their requests, analyze artifacts, payloads, and logs on their platform mainly with open-source models such as GLM. It is worth noting that it only because of HuggingFace’s use of these models that they root-caused the attack in the first place. OpenAI did not know about the details and extent of the issue until they got in touch with HuggingFace and coordinated. It is literally not possible to keep up with the frontier of cybersecurity research today if you do not have access to a frontier model that is capable of reasoning openly about cybersecurity topics.

Source: getting rejected by Fable… we’re just asking questions!

Our Recommended Minimum Version (and how to use the cmax audit security CLI)

To help neoclouds and operators to keep track of the updated minimum versions, we have a daily refreshed minimum version that gathers bulletin from major sources and hardware/software vendors, and publishes the aggregated minimum versions based on these bulletins.The ClusterMAX website displays and hosts these data for programmatic access by our CLI and other vendors.

To use it, simply install the CLI by following the installation instructions, and run cmax audit security. The CLI will pull the updated minimum version data from ClusterMAX website, and use it to audit your system.

In case of a vulnerable version detected, the CLI prints out the relevant security bulletins with clickable links in the console, and the suggested version to update to.

Source: ClusterMAX CLI 0.3.0 Security Module

For reference, here is our table for current minimum versions across a number of popular pieces of software, driver, libraries and firmware. All of this comes from public security bulletins and are only accurate as of time of writing (August 19, 2026):

Source: clustermax.ai website (add this to the public website before publishing, PR made)

In Summary: What Do Most Providers Need To Fix?

If you are still reading, lets assume that you have crossed the bar for taking security seriously. So what can you do?

In the modern day, providers need a SYSTEM to get stuff patched. Top cloud providers have the same system to deploy patches as they do to provision new clusters. It is a system that includes humans, scripts, and increasingly, AI.

Firstly is a system to monitor security bulletins like our daily refreshed minimum version. Among the neoclouds we tested, only a handful have an existing automated system for this, while the others largely rely on a fixed patching cycle of monthly or even longer. With the rapid development of cyber security focused models, such patching cadence is no longer sufficient.

So, here is our recommendations to the humans and the AIs about what to put in their scripts:

Please get your stuff updated to the minimum versions on our public website (and feel free to use cmax to help with this)
Please spend the time (and money, in some cases) to fix bad designs. No single points of failure that expose all of your users is the new rule. So stop doing namespace isolation on k8s with shared nodes, stop doing container-only isolation between tenants, stop giving people access to BMCs and DPUs, and implement your InifiniBand security keys correctly.

With that, we look forward to seeing you soon for ClusterMAX 3.0!

Implications for NVIDIA, AMD, and all the chip startups

OpenAI Jalapeño: Better Than Nvidia Blackwell

2026-08-25T06:00:38+00:00

OpenAI has spent the past couple years quietly building “Jalapeño,” an inference chip just announced at Hot Chips. Rumors of a successful tapeout had been swirling for a while. But now we have details. OpenAI invited us to look at their chip, go to their labs to check out how real it is, and benchmark it with our InferenceX suite.

In June, OpenAI unveiled the chip program in partnership with Broadcom, built from a blank slate exclusively for LLM inference. Design work began in the middle of 2024, going from initial team hiring to manufacturing tape-out in ~16 months, an extremely fast ASIC development cycle.

In general first generation chips are not competitive, but OpenAI bucks the trend by being industry leading and beating every Nvidia, AMD, and Google chip we have been able to test on multiple top open source models. OpenAI does this with extreme hardware software codesign. Surprisingly, OpenAI is not over specialization on any specific part of model inference, but instead by focusing on being a general chip that delivers high performance in all scenarios.

In this article, we will go into architectural details, software details and performance results for Jalapeño on InferenceX.

Source: OpenAI

A generalized inference chip

Everyone says that OpenAI’s chip is specialized for OpenAI models, but that’s wrong, OpenAI made a generalized chip for AI inference.

The timelines are insane. It shows that claims that use of AI is being used to accelerate chip design are real. Regardless of the quick timelines,Open AI spent a bunch of money, made pragmatic design decisions and their team is cracked, so this comes as no surprise.

Just looking at the specs, it is an immediate contender:

Source: SemiAnalysis

And the use of HBM4 makes it stand out as comparable to flagship GPUs from NVIDIA and AMD:

Source: OpenAI

A lot of the media coverage of this chip has followed a few throwaway comments from OpenAI that claim the chip will be optimized for their models in a way that other chips are not. This is wrong. Jalapeño is a generalized inference chip capable of running all sorts of models, and all sorts of workloads, including our benchmark InferenceX, where we ran the benchmark with OpenAI engineers in the lab. As a joke, OpenAI even showed us it running Doom, which was ported to their chip with just Codex prompts.

The following is our headline perf/W result, looking at token throughput per All-in utility MW. Jalapeño smokes every other chip. All this is done without Multi Token Prediction (MTP), while the other chips on the chart are the best performing configs of each respective SKU, all with MTP.

Source: SemiAnalysis

Jalapeño beats Blackwell on perf/W across almost all scenarios without being tuned for any specific point in the curve. It excels not only in low-latency scenarios but also in high-throughput scenarios. A more apples to apples comparison is against Single Token Prediction results, it knocks every competitor out of the water. At low concurrency scenarios, Jalapeño demonstrates remarkable interactivity, hitting over 700 tokens per sec per user at concurrency 1 on the DeepSeek R1 model.

Incredibly, this is all achieved with single-token prediction (STP), no speculative decoding and no prefill-decode disaggregation. In addition to DeepSeek R1, we also got to see some other models, including Kimi-K2.5 and GPT-OSS which ran at approximately 1,400 tok/sec/user. For all models, we confirmed that Jalapeño’s GSM8k evals attained results on par with Nvidia chips.

Some caveats on this. First, all numbers are provided to us by OpenAI. We verified the InferenceX runs in person in the lab, but we did not run the full suite of InferenceX benchmarks nor have we seen AgentX results. AgentX is our preferred suite for comparing chip performance due to the datasets’ long context and multi-turn characteristics that reflect the cache behavior of realistic production workflows. Frameworks that perform well on 8k1k may perform worse on AgentX as real production loads stress components like routers, prefix cache mechanisms, cache management, offload infrastructure, etc. These are not tested by single turn 8k1k. Read more about this in out AgentX article.

Second, we believe that comparison to Blackwell is somewhat incomplete and unfair. Jalapeño is really competing against chips like Rubin that also use HBM4. Vera Rubin systems are starting to ship to customers right now, while it will still be some time before OpenAI has anything beyond engineering samples of Jalapeño.

Thus, performance should really be compared against Rubin, not Blackwell, and in some sense we expect a custom chip like Jalapeño to outperform Blackwell. Vera Rubin NVL72 delivers 5.4x the perf/MW of GB200 NVL72 as we described in our article analyzing the NVIDIA performance claims in their launch with CoreWeave last month. We will compare Jalapeño to Vera Rubin’s July performance figures later below.

Third, the models being tested are not on the open frontier. NVIDIA and AMD have published results on larger models such as DeepSeek V4 Pro and Kimi K3, using AgentX. The larger the model and the more recent the release, the more complicated it is to bring up on a new chip. With that said the models OpenAI has working on Jalapeno aren’t exactly small either.

Performance Analysis

OpenAI designs for perf/W. The reason is simple: OpenAI is currently limited by datacenter power, not by budget or floorspace, and thus tokens per MW is paramount. At Computex 2026, Jensen said that perf/W, reliability and long lifetime are the core features of future GPUs. To quote: “If you have 1 gigawatt of power, then throughput per watt is revenue”. He also mentioned that choosing the wrong architecture just because the chips are cheaper doesn’t make sense.

Source: Computex 2026 keynote

This was emphasized by Nvidia during the Vera talk at Hot Chips 2026 while showing the same revenue graph: “The data center is power limited today.” Power matters and drives revenue.

Operators cannot simply obtain more MW because adding GPUs and adding grid capacity happen on very different timescales. Datacenter power envelopes have constraints such as their utility interconnection, infrastructure, cooling capacity, and UPS/backup-generation design. Grid delays repeatedly outpace hardware and construction timelines, driving the need for BtM (behind-the-meter) power capacity: gas turbines and on-site generators built and located at the data center itself. This capacity sits behind the utility’s meter rather than being drawn from the public grid. It lets an operator power a facility without waiting on grid interconnection and utility upgrades, which is exactly why xAI’s Colossus 2 relies so heavily on BtM while its actual grid connection lags far behind. Find out more in our Energy model.

As we wrote in an X post, tok/s/MW reduces to tokens per joule since a watt is a joule per second. This makes tok/s/MW representative of a system’s efficiency and ability to convert energy into tokens.

Source: SemiAnalysis

On this front, even when compared with Rubin, Jalapeño wins. OpenAI’s Jalapeño has STP output token throughput per MW surpassing Vera Rubin’s MTP results that NVIDIA and CoreWeave published in July. It also far exceeds GB200’s 2025 MTP results. As mentioned in our Vera Rubin article, VR was compared to 2025 GB200 results because that was a similar stage of early bring-up, and comparing to GB200 in 2025 holds software maturity constant. Following this logic, we compare Vera Rubin’s latest July 2026 results, GB200 2025 results, and today’s Jalapeño results. This is a very valid comparison as these are the best public Rubin numbers, and OpenAI taped out their chip after Rubin. Both OpenAI and Rubin are still immature thus performance will continue to rise.

Source: OpenAI, SemiAnalysis

On perf/TCO, Vera Rubin and Jalapeño are head-to-head, producing almost the same number of output tokens per $. However, as previously mentioned, Jalapeño’s results are obtained without speculative decoding and Vera Rubin’s results use speculative decoding. Speculative decoding adds a 3x+ improvement is cost per token. When speculative decoding is implemented on Jalapeño, this will mean Jalapeño will serve much lower cost tokens. Of course, part of this TCO advantage comes from trading Nvidia’s high margins for Broadcom’s lower (though still high) margins. But this is not all of it. For example, Meta and Microsoft’s AI ASIC programs not getting off the ground despite being at it for much longer shows that cost is only one part of the equation. For Jalapeño’s full TCO breakdown, see the SemiAnalysis AI Cloud TCO model.

Source: OpenAI, SemiAnalysis

Architecturally, OpenAI chose not to disaggregate prefill and decode (PD) across separate chip pools. The draft model and main model share the same chips and fabric, a design philosophy that trades some theoretical efficiency for practical operations. The motivation is that the workload mix changes over time, for example the ratio of input to cache write to cache read to output tokens has changed significantly as we have moved through the three eras of models (knowledge, reasoning, and agentic, as discussed in our recent article). Therefore, picking a fixed amount of heterogenous prefill silicon and decode silicon up front can lead to inefficiencies over time. OpenAI chooses a homogenous pool in this architecture and tries to make the chip perform well on everything.

And it does. On Kimi K2.5 (which Cursor Composer 2.5 is based on), Jalapeño reaches nearly 700tok/s/user and more than 9x the next best performing chip at 100tok/s/user.

Source: OpenAI, SemiAnalysis

On GPT-OSS, it’s another bloodbath. Jalapeño’s iso-interactivity throughput per MW is nearly double GB200’s highest throughput point and more than 50x GB200’s concurrency 1 point. The higher concurrency Jalapeño points use EP8.

Source: OpenAI, SemiAnalysis

These results are impressive! However, we have to nitpick: they’re just 8k1k, a much easier workload to tune for, and there are no AgentX runs yet. As mentioned in our AgentX article, multiturn, long context workloads stress much more aspects of the serving stack, such as routers and prefix cache. Many more optimizations are needed to excel in agentic workloads. Read more about this in the AgentX article.

Digging into the specs and architecture

All these results were gathered on the A0 stepping of Jalapeño, just 9 months into the program. But there is already a B0 stepping that is currently in the fab! B0 has optimizations that deliver roughly a 25% perf-per-watt improvement over the earlier A0 silicon. Specifically, the B0 stepping delivers 13.4 PFLOPs of MXFP4 on a single reticle-sized compute die that is manufactured on TSMC’s N3P. This compares to 17.5 PFLOPs of dense Rubin NVFP4 for a single Rubin compute die that is similar size and on the same node.

This is more respectable considering Jalapeño’s TDP is only 700W compared to Rubin’s at 900-1,150W per compute die. As Jalapeño is geared towards inference rather than training, it is understandable that OpenAI doesn’t need to push TDPs higher to maximize FLOPs, but regardless the above shows that Jalapeño delivers respectable peak theoretical FLOPs.

When compared directly to other accelerators, Jalapeño has the highest HBM bandwidth per watt, and the highest FLOPs per watt, comparable to the 1,800W Rubin Max-Q configuration:

Source: SemiAnalysis

Off-package I/O is provided by an N3E I/O chiplet with 32 lanes of 800G SerDes, for the compute fabric, with 24 lanes (600GB/s) being used for local scale-up within the rack, and 8 lanes (200GB/s) for global scale-up which is the 2,048 XPU multi-rack domain. PCIe Gen 5 is used for system I/O to connect to the x86 host CPU.

Jalapeño will ship with HBM4, making this chip one of the relatively early adopters after Nvidia and AMD, even beating the established TPU and Trainium programs. As one of the key architectural principles behind Jalapeño is getting the most out of HBM bandwidth, settling for anything but the best HBM would run counter to that goal. This results in 15.4TB/s of memory bandwidth per package which bests all the other accelerators shipping that are using HBM3E. The 15.4TB/s bandwidth shows its HBM4 can hit 10Gbps pin speeds, which would give it a slight edge over the 9.6Gbps Nvidia is getting out of its HBM4 in Rubin. The HBM is likely provided by Samsung.

Source: OpenAI

OpenAI taped out Jalapeño in November 2025, or more specifically, this was a tape out of the CoWoS design, not just the top die silicon. Within 9 months of that Nov 2025 tapeout, and with only 3 months of bring-up on actual silicon, OpenAI has already delivered very good results with Jalapeño. This is all the more impressive as the team is starting from zero on the software stack.

Meanwhile, Rubin’s CoWoS tape out was completed in October 2025, a month earlier, and yet the only early results we have seen are from CoreWeave’s engineering samples. Nvidia has not let us test and release benchmarks in the same way that OpenAI has, indicating their chip software is still immature. The CUDA moat is potentially dead given how fast OpenAI can bring up new models on their silicon.

They are still far from optimized and we can see that generally Jalapeño has delivered better numbers. We don’t think that Nvidia hardware is inferior, but more so that Jalapeño’s software bring-up has progressed more quickly than Nvidia’s. This speaks to the power of hardware/software co-design, which is the main area where a cracked frontier lab ASIC team can excel over more established merchant silicon players. Counterintuitively, starting from scratch may also have benefited OpenAI as it could make clean-sheet architectural decisions without worrying about backwards compatibility or older software versions.

While OpenAI has engineering samples of Jalapeño, production is currently scheduled to gradually ramp over 2027 with most of the output currently scheduled for the end of next year. For more details of unit volumes and ASPs, see the SemiAnalysis Accelerator Model.

Suffice to say, OpenAI Jalapeno is a real high volume ASIC.

When compared against Rubin’s timeline, Jalapeño’s is shockingly quick. As shown earlier, Jalapeño’s results beat Rubin’s despite Rubin’s head start.

Source: SemiAnalysis

Jalapeno Architecture

Digging into the architecture now, the chip’s matrix engine uses MXFP numerical formats and a weight stationary systolic array, similar to TPU. But when compared directly to TPU, it has support for smaller shapes / dimensions, meaning that it doesn’t have weird performance cliffs that get exposed by awkwardly shaped matmuls on bigger systolics.

It also has 64-bit scalar cores and FP32/INT32 vector cores. OpenAI has also invested in redundancy at the tray level and has yield harvesting built in at the core and channel level. They claim that AI assistance in chip design delivered an 8% reduction in SIMD area and a 10% reduction in matrix-engine area during design. While they did not clarify the exact process/voltage/temperature (PVT) conditions, they also mentioned the AI-assisted blocks improved timing and power over the initial blocks.

The Jalapeño architecture design focuses on eliminating memory movement of KVCache and weights as well as fixed latencies and overheads in order to make it possible to get closer to the raw peak flops/bandwidth even for small batches or shapes as compared to other accelerators.

The cores and the HBM are divided into slices, where each core slice has a low-latency local view on its own slice of HBM. Synchronization between slices occurs on a high-bandwidth dedicated collective network. This minimal memory hierarchy already gives Jalapeño a big potential advantage over GPUs, where memory accesses must traverse a complicated memory system, resulting in large latencies that must be amortized or hidden over larger shapes.

This choice is feasible because with careful placement of weights and KVs, synchronization between cores can be restricted to limited, known high-bandwidth comms such as tensor-parallel communication that can be overlapped with compute.

Source: OpenAI

There is also an additional general NoC which is used for general comms and to access the scale-up network. In general OpenAI saves huge power and gets big performance gains with a simplified NOC and memory subsystem vs Nvidia and Google.

Source: OpenAI

At the core level, OpenAI describes an out-of-order (OoO) core with an L1 cache. This is a large divergence from the pattern we have seen in other accelerators, all of which instead use software-managed scratchpad commonly paired with some async DMA support. Again, the argument being made here is that this allows Jalapeño to avoid fixed overheads such as barrier latencies, which on other accelerators (such as GPUs) need to be hidden or amortized over with higher work per core, and make it harder to get close to the raw peak bandwidth/flops.

The tradeoff is that Jalapeño therefore relies on good prefetching to ensure timely arrivals of memory requests, which is less predictable and more difficult to reason about. However, with Codex in a good harness with access to detailed tracing, it is likely that finding the optimal kernel with the best prefetching for a given shape requires little human intervention. We think that is exactly what OpenAI has done to bring up DeepSeek R1, Kimi K2.5, and GPT-OSS so quickly.

The cores also have support for “small” matrix dimensions, which (depending on how small) should make it more general across different model and batch dimensions less sensitive to matrix dimension alignment, padding overhead, and tiling inefficiency. For instance, TPUs, Trainium, and Etched chips have very large systolic arrays which can require large batches or exactly-divisible model dimensions to avoid tiling inefficiencies.

With Jalapeño, OpenAI has focused on eliminating fixed latencies in the system to allow for as-close-to-roofline performance as possible across all areas of the pareto curve. In theory, this could give them advantages over the GPU at multiple operating points:

Much better upper-bound performance on low-latency/small-batch inference, which on GPUs is limited by many fixed overheads such as launch latencies, barrier latencies, memory system latency
Some potential to achieve closer to the hardware roofline even for large-batch or long-context

This comes with the caveat that even if the upper-bound performance is available in theory, it may be more difficult to realize that performance for real kernels. So it seems the approach is:

Design for the highest upper-bound performance across all workload shapes
Let Codex do the tedious work of finding the kernels that achieve that upper bound

Judging by the extremely fast turnaround for the OpenAI team to bring up InferenceX workloads on Jalapeño, we are optimistic about this approach.

If Jalapeño is a success, it will be a strong signal that the industry’s obsession over programming models and perfect, universal compilers are invalidated by frontier AI models.

Software

OpenAI writes Jalapeño kernels like assembly. Each kernel gets hand-tuned code, some running to ~3,000 lines, backed by correctness checks and a custom sanitizer. Early kernel work was human-in-the-loop rather than fully automated, but this shifted with a more scaled-up, internal version of Codex, one which OpenAI plans to pitch to enterprise customers. The internal serving engine is called “Teacup”. Interestingly, OpenAI had no internal implementation of MLA kernels until they benchmarked DeepSeek with InferenceX. The ability for Codex to write functional and efficient kernels so quickly (without any of OpenAI’s kernel engineering team intervening) shows the software pipeline’s developmental ability.

OpenAI programs Jalapeño with Gluon. Gluon is OpenAI’s kernel programming language. Built on top of Triton, Gluon preserves Triton’s SPMD (Single Program Multiple Data) programming model, but it exposes low-level programming abstractions. For example, for NVIDIA GPUs, it offers APIs that map to PTX instructions, including MMA instructions, TMA instructions, mbarrier mechanisms, and many more. The most unique abstraction Gluon provides is the layout. Generally speaking, a layout defines a mapping between a hardware resource (e.g. 5th register of warp 9) and a tensor element (e.g. tensor element on row 6 column 7). Gluon’s layout abstraction is based on Linear Layouts, a type of layout algebra OpenAI invented. Linear Layouts mathematically formalizes what a layout is and provides tools to operate on layouts. This enables many features, such as provably correct layout conversions and optimal memory swizzling.

In terms of Jalapeño’s programming model, each Gluon program maps to a persistent thread. We believe this hints that Jalapeño suits the persistent kernel programming pattern, where each program executes on multiple tiles, and the programmer, rather than the hardware scheduler, assigns the work. OpenAI mentioned TensorInfo, an abstraction that explicitly encodes layouts. This is likely the set of layouts designed for Jalapeño, which will be powered by Linear Layouts. Finally, each core offers data prefetching and decoupled out-of-order units. For example, a user might program a wait on a prefetched data, which is locked behind a semaphore.

In a weird twist of fate, OpenAI models like GPT 5.6 Sol, which currently run on NVIDIA GPUs, have been used to design a chip that poses a real threat to the CUDA moat - NVIDIA’s own GPUs are helping usher in their potential successor in real time.

Comparing across time, we can also see Jalapeño’s developmental pace, achieving more than 2x throughput improvements at certain interactivities in less than 2 weeks. Each tarball we get from the Jalapeño team has a world of wonders inside.

Source: OpenAI, SemiAnalysis

Not only did kernel performance improve, in the span of 8 days, the Jalapeño team enabled TP32, building on the previous TP8 configs and expanding beyond a single system to get a full rack-scale config running on a large model. This is a really impressive pace of development.

Source: OpenAI, SemiAnalysis

To validate performance before committing to real hardware runs, OpenAI also has a simulator “chilisim” accurate to within 5% of measured hardware, using a fixed-width trace bus. Tracing on A0 was limited but has improved substantially on B0, likely with inputs from actual runs on A0 silicon. Engineers have demoed the Codex CLI running an internal model, nicknamed “Raiku” or “5.3 Codex Spark”, at 1.2ms TPOT.

The team also showed off Codex-written demos running directly on the chip: Doom at 36 FPS, an FP32 fluid-dynamics simulation, and a “Liquid Light” mouse-drag visualization.

Source: SemiAnalysis

On the model side, OpenAI’s internal megakernel approach, nicknamed “gigakernel”, is built around a single megakernel that loops on-device to reduce CPU overhead and launch time. The team is also leaning further into test-time compute strategies, with internal interest specifically in how to coherently use 1 million rollouts.

To Disagg or Not to Disagg, ‘tis the Question

We mentioned earlier that OpenAI is not using prefill decode disaggregation on these chips. This came as a surprise to us, as NVIDIA and AMD GPU performance benefits significantly from PDD, even on homogenous hardware. Let’s dig into why the Jalapeño team went this way.

Prefill-decode disaggregation (PDD) looks attractive when the workload is frozen. Prefill and decode stress hardware differently, so assigning each phase to a separately tuned pool can improve efficiency at one chosen input/output ratio. Production traffic, however, does not stay at that ratio. Input and output sequence lengths, concurrency, cache-hit rates, speculative-acceptance rates, and latency targets all move throughout the day.

Once devices are divided into prefill and decode pools, too much prefill demand leaves decode chips idle while requests queue. But too much decode demand does the opposite. The operator must continuously predict the right split, provision spare capacity on both sides, and rebalance a system whose ideal ratio is always moving.

In a unified system, some resources may be underused during a particular phase, but every device remains available to serve the next request. In a disaggregated system, an entire chip can sit idle simply because it belongs to the wrong pool. Local utilization looks better, but global utilization can be bad.

Source: SemiAnalysis

Disaggregation also breaks locality. The prefill worker produces a large KV cache that the decode worker immediately needs, so the system must transfer that state across the network before generation can continue. That adds bandwidth consumption, synchronization, queueing, and another failure domain. The cost also rises with input sequence length because KV cache grows. However, avoiding the movement of KVs is largely a power and latency optimization; being willing to move some KVs around can allow for increased hardware utilization at the expense of some power and per-request latency.

A fungible fleet shifts capacity between latency-sensitive requests and throughput-oriented batches, while a fixed split strands hardware whenever the traffic mix changes. Moreover, context length changes the balance between attention and FFN work, making any fixed hardware ratio efficient only near its design point.

Source: SemiAnalysis

The same constraint applies to speculative decoding. A draft model has to feed candidate tokens to the verifier with extremely low latency. Separating the two across specialized pools turns a tightly coupled decoding loop into a distributed protocol. The extra communication and coordination can consume the latency saved by drafting. Keeping both models on the same devices and low-latency fabric preserves the locality that makes speculation worthwhile in the first place.

Source: SemiAnalysis

However, disaggregation can still win where demand is sufficiently large, stable, and predictable, particularly when conventional GPUs need large phase-specific batches to reach good throughput. But it is not free lunch.

From Japanese to Indian (Mild to Spicy): Katsu, Vindaloo, and Chana - How are the curry dishes put together into a rack system?

The Jalapeño System at the rack unit level consists of a CPU host rack and an ASIC rack. The host rack houses 16 host CPU trays named “Katsu,” each corresponding to one of the 16 ASIC trays, named “Vindaloo,” to the right of the Katsu. Each host houses two Turin-class AMD EPYC CPUs with 1.5TB of DRAM, 2x E1.S, and 2x M.2 SSDs per rack. Each tray is also specced with 400G (2x200G) frontend networking. Each Katsu tray connects to each Vindaloo tray via 8 external PCIe DAC cables that run horizontally across the rack at the front. The system level design is done in partnership with Celestica.

The ASIC rack consists of 16 Vindaloo trays and 8 scale up switch trays (6 for local + 2 for global), named “Chana.” Each Vindaloo tray consists of 8 Jalapeño ASICs, making up a total of 128 Jalapeño ASICs per rack. The ASICs are connected to each of the Chana switch trays via a copper cable backplane, just like that of Nvidia’s Oberon. The scale up topology is split into a local domain of 128 ASICs within the rack and a global domain connecting up to 16 racks or 2,048 ASICs. We will explain the bandwidth and the topology in more detail below.

Power provisioning to a sidecar host rack draws roughly 50kW provisioned (31kW in production), and the ASIC rack draws 130kW, making the total two rack system roughly 160kW. That’s basically a double-wide GB300 rack in terms of power draw.

Source: SemiAnalysis, OpenAI

OpenAI can connect up to 2,048 Jalapeño XPUs within a single scale-up network. The scale-up network consists of two domains, a local domain connecting all 128 XPUs over backplane within the rack, as well as a global domain connecting 2,048 XPUs over 16 racks using a hybrid of copper and optical interconnect. Each rack consists of 8 Chana switch trays. Six Chana switches in the middle are for the local domain, which come with one 102.4T Tomahawk 6 switch ASIC each. Two Chana switches at the top and bottom of the local switches are for the global domain, which we think could consist of 2x 102.4T Tomahawk 6 switches making up to 204.8T per switch tray.

In the local domain, each of the 128 Jalapeño chips has a per XPU uni-directional bandwidth of 4.8Tb/s and is connected on an all-to-all basis to 6x 102.4Tb/s Tomahawk 6 ASICs. This would amount to 48-differential pair (DP) male and female connector pairs per XPU translating to a total of 6,144DPs worth of passive copper cables per rack used for local scale-up.

For the global domain, 16 racks totaling 2,048 XPUs are connected together via a combination of copper backplane, electrical 204.8T TH6 switch, 1.6T transceivers, and optical circuit switch. Each XPU has a uni-directional bandwidth of 1.6Tb/s for the global link, which is 16-differential pair (DP) male and female connector pairs per XPU for the backplane between the XPU and the global switch. Bandwidth exiting each global switch tray of 2 ASICs each is split between the backplane and front panel optics.

Between local domain and global domain, backplane connector count per rack comes up to 64 DPs per XPU and a total of 8,192 DPs worth of passive copper cables per rack.

The global domain adopts a rail-only architecture consisting of 8-rails across the global domain. We think OpenAI routes optical links in the global domain via Optical Circuit Switches (OCS) installed in every rack. For every XPU, 1.6Tb/s of global bandwidth will travel to the global switch tray over the copper backplane. This then exits the switch through the front panel via 1.6T transceivers, which go to the passive optical switch before exiting the rack. This expands the scale-up world size to 2,048 XPUs combining 16 racks of 128 XPUs each.

Source: SemiAnalysis

Because scale-up networking is only about 10% of total system cost, that flexibility buys valuable optionality for future 10–20 trillion parameter models or 2–4 million token context windows. On deployment, OpenAI is partnering with neoclouds and is gathering reliability data with datacenter partners through January while optimizing dock-to-rack rollout time.

What’s Next

Next, we talk about the future of Jalapeño, whose first production token is coming soon. The next goal is 100MW, and the hurdles will mostly be hardware: How much can they produce, how well can they deploy and operate datacenters, how do they handle monitoring, and resiliency, etc. The software is already proven, and with internal models, every software headstart is easily caught up to. Behind the paywall we will discuss implications for NVIDIA, AMD, Cerebras, and other chip companies who have signed deals with OpenAI in the coming years.

We also cover production volumes, units, and timelines for the next generation chip in our Accelerator model here.

Direct Implications for NVIDIA, AMD and Cerebras

AgentX - InferenceXv3: Does CUDA Moat Hold up in Agentic Inferencing?

2026-08-23T16:19:35+00:00

Since the Claude Code inflection point in November 2025, long-context, multi-turn agentic workloads have grown rapidly. They now dominate traffic for production inferencing. In April 2026, OpenAI’s Enterprise agentic spending overtook ChatGPT spending.

Agentic workflows have decisively taken the baton. Today, we announce AgentX 1.0 - the world’s first fully open source, multi-turn agentic coding inference benchmark at 1 million context, released under Apache 2.0. Our full dashboard is available here.

Source: SemiAnalysis

In the past most measured performance based on fixed sequence length prefill and decode workloads, but this is an inaccurate way to measure workloads. Reality is multi-turn, long context, high prefill reuse, with sub agent bursts, KVCache offload, and numerous tool calls. As such we aimed to build the correct way for the industry to measure AI hardware and software performance.

We have spent more than $3M building this dataset. Today, we open source everything. InferenceXv3 implements AgentX, a new realistic scenario in addition to the existing “fixed sequence length” scenarios (8k1k, 1k1k, 1k8k). It improves the benchmark scenarios by using agentic coding traffic instead of the previous single-turn traffic of 8k input and 1k output tokens.

The full matrix runs on ~2MW of continuously operated compute across over 1000 chips spanning a wide range of SKUs, featuring the MI355X, GB300 NVL72, GB200 NVL72, B300, B200, MI325, MI300X, H200, and RTX Pro Servers. Rubin arrives later this month, and TPUs and Mi455X UALoE72 arrive later this year. Please drop a star if you found our free open source work valuable.

It is great to see amazing performance from both NVIDIA and AMD on agentic workloads. NVIDIA does very good on a lot of frontier models while AMD also does well on some frontier models for specific comparsions.

Source: SemiAnalysis GitHub

The most valuable thing AgentX produced in its first months was not the initial results. It was the massive industry impact the benchmark is already having. Over 70+ upstream PRs for optimizing real world production agentic workloads across vLLM, SGLang, TensorRT-LLM, ATOM, AITER, Dynamo, LMCache, and Mooncake, uses AgentX as the north star benchmark proxy. Most of these optimization improvements are transferable to production traffic. We deep dive into each of these optimizations later in the article.

Source: SemiAnalysis

Open source is a core principle for InferenceX and thus, we open more of the stack than most people who use that word. That includes an open frontend, a public database served through an easily consumable REST API that multiple tier 1 AI lab’s capacity planning teams already consume, public GitHub Actions CI provenance, logs, and accuracy validation on every single point. Crucially, our benchmark configs mainly track recipes.vllm.ai and SGLang cookbook on upstream images such that we are measuring the performance actual customers are experiencing instead of measuring benchmax’ed images.

In three to four weeks, we will release an AgentX update article. It will cover further optimizations to agentic workloads, plus updated performance results from AMD and Nvidia. It is important to understand that the profile of agentic workloads is updating fast. InferenceX will continue to move swiftly to benchmark the relevant workloads.

InferenceX is 100% committed to being open-source - this would not be possible without the contributions and support from our OSS partners. We would like to thank the following people that have made massive contributions to the AgentX 1.0 release:

Inferact/vLLM: Roger Wang, Yifan Qiao, Simon Mo, Jeff Ma, and many others
RedHat/llm-d: Michael Goin, Robert Shaw, Tyler Michael Smith
RadixArk/SGLang: Baizhou Zhang, Yuwei An, Mingyi Lu, and many others
LMCache/TensorMesh: Samuel Shen
Weka: Callan Fox, ValB
MoonCake Maintainers: Teng Ma, Xu Wenjie, Ke Yang
AMD: Thomas Wang, HaiShaw, Andy Luo, Seungrok Jung, Chun Fang, Parth Panchal, Bill He, Theresa Shan, Hongxia, Fangzhou, Gilbert Lei, Yanfei Wang, Duyi Wang, Peng Sun, Lingpeng Jin, Simon Danielsson, Xiaohu Guo, Haichen Zhang, Chang Liu, Doug Lehr, Poovaiah Palangappa, and many others in the AMD Shanghai Development Centre
Nvidia: Xin Li, Anthony Casagrande, Kedar Potdar, Ankur Singh, Ishani Dhanani, Nick Comly, Nvidia Shanghai TensorRT-LLM team, and many others
Anthropic staff, for promptly fixing multiple bugs that made implementing AgentX possible
GitHub: Austen Stone for helping with reliability of GitHub Actions that AgentX uses
And many others

In addition, we are thankful to all who support our open source InferenceX initiative, including Meta, Microsoft, Oracle, OpenAI, MiniMax, Moonshot Kimi, Alibaba Qwen, and Zhipu GLM.

Source: InferenceX

A Brief Overview of Agentic Workloads

At a high level, an agentic workload is characterized by four elements:

Multi-turn: a session includes many user / assistant interactions (tens or hundreds) compared to a handful in a chatbot scenario. Multi-turn, long context, high prefill reuse, with sub agent bursts and numerous tool calls.
Long context: system prompts, tool definitions, and the large number of turns make context accumulate quickly.
High prefix reuse: since the conversation progresses linearly, where output from turn n-1 is concatenated to turn n (typically), most context can be served from KV cache rather than recomputed (this depends on amount of storage available to store KV tensors). As n grows, the ratio of cached input relative to uncached typically tends towards 1.
Sub-agent bursts: a session launches multiple short-lived sub-agents with fresh context, which create bursty KVCache patterns.

Source: DeepSeek, SemiAnalysis

Considering the characteristics above, benchmarking these workloads is fundamentally different from the existing fixed sequence length benchmarks. Namely, agentic inference is inherently a systems problem. Because of extremely high prefix reuse, KV tensors must be efficiently transferred across nodes/ranks (NIXL, MORI-IO, Mooncake). Additionally, different conversations should be routed to different nodes/ranks depending on where the appropriate prefix resides in order to maximize cache hit rate (LLM-d, Dynamo, vLLM/SGLang router). Long context conversations stress the HBM capacity for KV cache and necessitate offloading KV tensors to different tiers of memory (DRAM, SSD), a process that needs to be carried out efficiently (Mooncake Store, LMCache, vLLM Simple Offloading, SGLang HiCache).

This is in contrast to fixed sequence length, single turn workloads where prefix reuse is not relevant and inference performance is largely reflective of baseline chip/kernel performance. This is not to say that the plethora of fixed sequence length data on InferenceX is not important. In fact, stripping away the complexities of agentic serving shows clearly how low-level inference performance optimizations are progressing. It also provides an important baseline for AgentX results.

In an attempt to make the AgentX workloads as realistic as possible, we collected an initial corpus of 393 internal SemiAnalysis anonymous Claude Code traces to replay. To anonymize the content while keeping the original prefix reuse pattern, we use a method similar to the Qwen-Bailian dataset, one of the earliest corpora of production traces. We then use AIPerf to reconstruct the traces according to the original schedule of requests at varying levels of concurrent clients. We worked with Anthropic to ship two Claude Code features to make the AgentX dataset possible. We thank the Anthropic staff for their help.

This brief introduction to agentic workloads should give the reader enough context to understand the results in the next section. In a later section, we will provide a deeper technical dive into the methodology, replay harness, and dataset.

Agentic Coding Inference Performance

When looking at inference coding performance, OpenAI, Anthropic, xAI, and other frontier labs focus on three things. They look at performance per dollar versus interactivity (TPOT), TTFT (time to first token), and overall end-to-end task completion. Performance per megawatt is also important, considering that terrestrial datacenter power is a critical constraint (money is a social construct and it appears the labs have an unlimited supply, but power is physically hard to come by in this day and age). Our datacenter model has estimates of quarter by quarter build up of power demand and supply.

In this section, we highlight some of the overall agentic performance themes across frontier models. We strongly encourage the reader to use this as a guide to investigate the results for themselves. All of the data is open source and the community has the opportunity to draw their own conclusions on the current state of real world inference performance.

DeepSeek V4 Pro 0813

DeepSeek V4 Pro 0813 is an ultra popular frontier open weight model from China. It has ~1.6 Trillion parameters with 49 Billion active parameters.

As of August 21, the following graph shows the best performance per SKU for all submissions, normalized by total cost of ownership (TCO).

The ISL/OSL distribution of all requests among all DeepSeek v4 runs was as follows: ISL p50=88k, p90=272k, p95=404k, p99=675k and OSL p50=413, p90=2.2k, p95=3.7k, p99=8.6k.

Source: InferenceX

In general, it is important to consider both tokens per second per user (TPS - also known as interactivity) and TTFT, since these often come at the expense of one another. For instance, in the graph above, some SKUs achieve very high throughput at decent interactivity, however TTFT is severely degraded. What is an “acceptable” p90 TTFT varies heavily depending on the application. For most production systems serving agentic workloads, you can expect p90 TTFT to be anywhere from 200-5,000ms. Anything over 5-10s is pushing the boundary of what can be considered “online inference.” There are still practical applications for the ultra-high throughput sector of the curve, where latency does not matter and peak system utilization is desirable (batch processing, very long running agents, etc).

In terms of single node performance, MI355X open-source performance (vLLM) trails behind vendor specific ATOM (AMDs equivalent of TensorRT LLM). We think it’s great that AMD is pushing the frontier quickly with ATOM, however we encourage them to make a higher priority of upstreaming these improvements into vLLM.

Source: InferenceX

AMD’s distributed inference (DI) team has made great progress on 8k1k scenarios over the past six months. The team still has some way to go before DI is a viable solution for realistic workloads. In terms of throughput per GPU vs. interactivity, we observe the 1xDEP8+1xDEP8 disagg config is only able to realize slight performance gains in the high throughput scenarios, while actually performing worse in low latency scenarios.

To make matters worse, any increase in throughput at the middle-to-high interactivity configs is overshadowed by the significant spike in p90 TTFT. One of the reasons for this is the use of SGLang’s --enable-prefill-delayer argument above concurrency 64, which postpones prefill admission so DP ranks can form fuller batches (for up to 30 forward passes). Additionally, these points also increase chunked prefill size from 8,192 to 65,536.

Source: InferenceX

On e2e latency, ATOM MI355X beats B200 vLLM (it does not beat B300 or B200 SGLang though). The issue with this is that most AI labs in China or the west do not want to use ATOM in production besides 1 small advertising business unit at Alibaba Corp due to tons of missing features. The main Qwen LLM org at baba does not use ATOM in production.

Source: InferenceX

Before August 21, 2026, AMD’s MI355X strong SGLang development team was matching B200 vLLM on performance per dollar on end to end (e2e) performance.

Source: InferenceX

However, B300 vLLM and B200 SGLang still beat AMD’s MI355X.

Source: InferenceX

After August 21, 2026, due to optimizations in vLLM from Inferact and Nvidia, the performance per dollar of Nvidia’s B200 has surpassed that of the MI355X. This is a close race and we are excited to see the performance optimizations over the next couple weeks. We will be publishing an AgentX update article very soon.

AMD has listed their DeepSeekv4 vLLM optimization and includes lots of exciting things that they can do to improve their performance.

Source: SemiAnalysis

Now turning to Nvidia. Their most competitive solutions are GB300 Dynamo TRTLLM and GB200 Dynamo vLLM. Both configs rely on PD disagg to achieve high throughput at reasonable interactivity. Additionally, GB300 configs employ wide-EP (DEP32) decode instances in order to achieve higher throughput at the middle of the frontier.

Note that the 2xDEP8+1xDEP12 GB200 point is significantly closer to the 3xDEP8+1xDEP16 GB300 point in terms of TPS compared to TTFT. Again, TTFT is, in general, more sensitive to the “spikiness” of the workload. Since the GB300 point achieves much higher overall concurrency, it incurs more subagent traffic and hence more cold prefills. We can see this in the TTFT chart for the point:

Source: InferenceX

When normalized by TCO, B300 vLLM versus B200 vLLM aggregated performance is quite similar. The main difference being that B300 can “squeeze” out extra throughput, given its 50% increase in HBM capacity over B200.

Source: InferenceX

We can further visualize this difference using our server metric visualizations, which are new to AgentX.

Under the load of 384 concurrent agentic traces, B300 vLLM DEP8 w/ 3TB DRAM via vLLM simple offloading achieved a 91% HBM cache hit rate with an additional 1.36% DRAM cache hit rate. This is because the HBM KV cache working set size is approximately 43M tokens with this configuration, and the load barely exceeds this number of tokens in flight at any given time.

Source: InferenceX

With B200 concurrency 196 (all other parameters stay the same), we see only 73% HBM cache hit rate and rely more heavily on DRAM with an offload cache hit rate of nearly 20%. We observe that the HBM KV cache working set size is 22M tokens, roughly half that of B300.

DRAM KV offloading is typically implemented as a write-through cache, meaning every prefix written to the HBM cache is also written to the DRAM cache. Therefore, it is most effective when the amount of DRAM available for offloading is significantly bigger (a multiple of 1.5-3) than HBM KV cache capacity.

Source: InferenceX

H200 SGLang FP8 is able to serve DeepSeek v4 at low concurrency, and is even competitive with B200/MI355X SGLang from a perf/$ standpoint. However, it cannot compete with the newer SKUs in high throughput scenarios due to lack of HBM.

Source: InferenceX

Furthermore, the reliance on DRAM KV offloading at higher concurrencies leads to unreasonable latency as the number of users scales.

Source: InferenceX

Overall, MI355X performs decently well compared to its main competitors B200 and B300. Performance is most comparable at the lower throughput / lower latency parts of the curve, where only tensor parallelism and more rudimentary kernels are deployed. AMD needs to work on optimizing DEP kernels on MI355X to be more competitive in the high throughput scenarios, especially given the 1.5x HBM over B200.

Source: InferenceX

Kimi K3 2.8 Trillion Parameters

Kimi K3 is another frontier open weight model from China that has 2.8 Trillion total parameters. This is in the same range in terms of number of parameters vs Claude’s Mythos/Fable5 model architecture. We use this as an open weights proxy model architecture. The Kimi K3 model is so big that it does not even fit on a single B200 server and requires using wide EP/wide TP or pipeline parallelism in order to fit all of the weights. On vLLM, speculation decoding/DSpark did not compose at all with pipeline parallelism until very recently, so B200 performance on Kimi K3 was horrible and was getting mogged by MI355X since B200 was unable to use speculative decoding with pipeline parallelism.

MI355X vLLM worked out of the box on day 0 for short context single turn workloads, but for long context multi turn workloads, MI355X AITER and Triton kernels suffered a massive panic attack on the first week and upstream vLLM was completely unusable for MI355X on realistic workloads.

Source: SemiAnalysis InferenceX

Hopper struggles to serve the AgentX workload for Kimi K3 since Kimi is a massive model and because vLLM maintainers/NVIDIA have not been focusing on optimizing Hopper for Kimi K3. Hopper (SM90) requires custom tuned kernels for K3 along with TP32/EP32 tuned shapes for serving at high interactivity.

Source: SemiAnalysis InferenceX

We think it is great that AMD is quickly pushing K3 performance forward with ATOM. However, we encourage AMD to further prioritize upstreaming these improvements into vLLM. ATOM is currently AMD's best-performing engine, but vLLM remains the more relevant comparison for customers using an upstream open-source serving stack.

Source: SemiAnalysis InferenceX

On part of the curve between 40 to 60 second e2e latency, MI355X ATOM beats even GB300 NVL72 vLLM on performance per dollar.

Source: SemiAnalysis InferenceX

MiniMax M3

Nvidia absolutely destroys all competitors on MiniMax M3 432B. AMD software performance is horrible on MiniMax especially at high context length due to AMD engineering leadership incentivizing tuning only for short context single turn workloads and ignoring long context multi turn workloads.

Source: SemiAnalysis InferenceX

B300 TRT-LLM TP2 owns the M3 crown. There is a lack of DP-attention points as it is non-optimal on M3 since KV cache locality becomes a routing constraint. This is further explained later on. For GB200 at concurrency 40, TP4/EP4/DPA gets 0.60x the throughput of plain TP4 at a >3x p90 TTFT. At concurrency 32 it hits 28.8% of cache vs 96.0% theoretical. Each DP rank owns a private quarter of the pool; a 300k-token session re-landing on the wrong rank recomputes everything. No decode config with EP appears on the M3 frontier, likely as the concurrency is not high enough to balance the loads on all experts.

B200/B300 also completely beat their rack-scale counterparts for MiniMax M3 on TCO-normalized throughput. On AgentX, the rack-scale advantage isn’t as pronounced as the Dynamo router can become the bottleneck because its work scales with the number and length of live prefixes. Optimizations on this and the several fixes which moved throughput by double-digit percentages are discussed later on in the article. Also, there are no well tuned kernels for wideEP, wide DCP, nor wide TP. And since GB200/300 have higher TCO so without wide ep/wide DCP, it shows up as worse perf per TCO.

With that being said, we also expect further optimizations from Nvidia on their rack scale solutions for this SKU. We will make sure to highlight these in our follow up article.

No submission currently runs context parallelism, despite P90 ISL of 317k. With 4 KV heads, DCP caps at 2 even at TP8, and the MSA indexer needs its own context-parallel handling (a vLLM PR is opened), see the Context Parallelism section for more discussion on this topic.

Source: SemiAnalysis InferenceX

All of Nvidia’s Pareto optimal points include KV offload above concurrency 20, but for AMD none of the Pareto optimal points use KV offload to DRAM. AMD also uses KV offload less than Nvidia on the other models. The reason for this is that GPU-to-CPU transfers for CPU KVCache offloading are highly inefficient on AMD vLLM. The hipMemcpyBatchAsync API was missing until ROCm 7.14. Without hipMemcpyBatchAsync, vLLM’s native Simple CPUOffloading requires doing serialized Memcpy from CPU to GPU instead of batching them into larger message sizes.

It is also worth mentioning that vLLM performance is very comparable to TRT-LLM in terms of throughput versus p90 interactivity. Additionally, vLLM performs better in terms of throughput versus p90 TTFT.

Source: SemiAnalysis InferenceX

Qwen3.5 397B

Qwen3.5 397B uses GatedDeltaNet instead of vanilla attention for every couple of layers. GatedDeltaNet was invented at MIT/Nvidia Research and has a theoretically constant state storage requirement instead of vanilla attention’s linear storage requirements. This means that it has lower storage requirements compared to an equivalent dense attention model. Unlike end to end model training research like the Nemotron disaster, Nvidia Research is great at fundamental research like GDN and LatentMoE which is used on frontier models.

Note that this model’s native max context length is 262k tokens, so we use the truncated dataset. This simulates a workload on a smaller model where the max context length would be frequently reached with many compactions, how users would actually use this model.

Qwen3.5 397B is a strong hold for NVIDIA on SGLang versus SGLang, with over 20x better performance at 90 tok/s/user. There is currently zero competition from AMD for Qwen3.5 SGLang.

Source: SemiAnalysis InferenceX

Again, we observe Nvidia over optimizing for interactivity at the cost of TTFT, especially in the case of TRT-LLM. In the graph above, all of the Nvidia SGLang submissions have much lower p90 TTFT when compared to TRT-LLM.

Compared to H100, on Qwen3.5, B300 FP4 has 12x better performance per dollar.

Source: SemiAnalysis InferenceX

GLM 5.3

GLM 5.3 builds on top of GLM5.2 744B with additional post training. This is a frontier level model.

In terms of OSS SGLang performance, this is another model where Nvidia again beats AMD on realistic agentic inference performance. At 150 tok/s/user p90 interactivity, Nvidia has up to 5x better cost efficiency, With the current state of AMD software, at 150 tok/s/user, Nvidia’s performance advantage is so great that even if the competitor chip hardware was sold for free (but with providers still of course paying for datacenter hosting and power and other operating costs), cost per token would still be cheaper when using Nvidia.

We look forward to AMD’s performance optimizations in the upcoming AgentX update article in a couple of weeks, which will also include some other very exciting results.

Source: SemiAnalysis InferenceX

When looking at ATOM, AMD has better performance per dollar than GB300 NVL72 SGLang and even TRTLLM for some parts of the range of p90 E2E Normalized Interactivity. Great work to the AMD team on these results. Again, we look forward to AMD porting over these optimizations to SGLang. We also are looking forward to NVIDIA quickly optimizing GB300 NVL72 in the coming weeks.

Source: SemiAnalysis InferenceX

We take a moment to introduce an experimental metric which we call E2E Normalized Interactivity. At a high level, this metric is supposed to evaluate how fast a user experiences responsiveness when considering both TTFT as well as TPS. It is defined by OSL/E2EL. Substituting the fact that E2EL equals TTFT plus OSL times TPOT (in reality only OSL - 1 tokens are decoded), we get the following equation.

This is effectively interactivity (the 1/TPOT portion) plus an additional penalty proportional to TTFT.

Source: SemiAnalysis

Please note that this metric is experimental and is not perfect. For instance, it heavily penalizes high TTFT and doesn’t capture all the nuances of certain optimizations such as PD disaggregation. All submissions for AgentX v1.0 optimize for both regular interactivity and TTFT separately. We will continue working on new north star metrics that reflect all nuances of modern agentic inference.

AgentX Industry Impact - Optimizations for Agentic Workloads

The most impactful result from AgentX in its first months was not producing open source datasets, instead, it has been the industry impact of 50+ upstream PRs created by AgentX partners to optimize real world agentic workloads using AgentX as the north star. AgentX’s real agentic traffic benchmarks not only the raw prefill and decode kernels, but also tests the entire end to end token generation process from KV cache lifecycle, hybrid-attention cache correctness, CPU KV offload, transfer progress, routing affinity, to incremental tokenization, request serialization, and scheduler bookkeeping. All of these steps matter for every production agentic deployment.

This is just a continuation of our ongoing mission to help the ecosystem accelerate improvement and deliver light speed improvement in software. A great example is SemiAnalysis’s multi-year collaboration with AMD’s software development team in which we have been providing continual feedback and input to help to modernize their software development principles. This has not only led to many changes that have accelerated AMD’s progress, but also has been instrumental towards getting AMD open source closer to first class on agentic workloads.

A Brief Introduction to the Distributed Inference Ecosystem

As mentioned, agentic inference is inherently a system-wide problem as opposed to just a chip/kernel level problem. Additionally, when there are large distributed systems handling hundreds of thousands of agentic requests, the scheduling of requests and management of KV cache becomes non-trivial and has legitimate performance implications. For example, sub-agents give bursty KVCache patterns where not properly optimizing, it will improperly evict the main agents cache.

The following diagram illustrates the stack at a high level. At the top, routers (sometimes referred to as “frontends”) route requests to different workers. For instance, in the case a server is running data parallel attention, there are separate KV caches for each DP rank. In order not to thrash any one of the KV caches, requests are routed according to different policies such as consistent hash, where requests in the same session/subagent are routed by their unique ID.

Source: SemiAnalysis

For most routing policies, there is nothing significantly different about each router implementation. Some are separate components such as vLLM router and llm-d router while others are integrated in the engine such as SGLang model gateway and ATOM Mesh.

After a request is routed, it is handled by the scheduler of an inference engine such as vLLM, SGLang, etc. The engine is responsible for actually performing the inference and returning the result over an API. Additionally, each engine has an interface for connecting the engine’s internal KV cache to external KV cache managers. This allows for a “pluggable” ecosystem where different KV cache managers can integrate with a variety of inference engines.

A simple deployment, used in the current AgentX results, runs Mooncake alongside vLLM on the same node. Each vLLM worker embeds a Mooncake Store client and contributes a portion of host DRAM to the external KV-cache pool. vLLM connects to this pool through the MooncakeStoreConnector interface which loads reusable KV blocks into GPU memory and saves newly computed blocks back to host memory. Mooncake Store manages the external cache, including placement and eviction, while Mooncake Transfer Engine performs the actual movement of data between GPU and CPU memory.

Different KV cache managers may use different transfer engines to physically move bytes between memory tiers or machines, such as between prefill and decode workers. Mooncake Store, for example, uses Mooncake Transfer Engine to move KV blocks between GPU memory, host DRAM, and remote nodes.

A deployment can use Mooncake Store to offload reusable KV blocks to host DRAM while simultaneously using NIXL to transfer request-specific KV directly from prefill GPUs to decode GPUs. Mooncake TE handles movement for the Mooncake Store path, while NIXL handles the separate prefill-decode path using UCX and GPUDirect RDMA where supported. Multiple KV-management and transfer paths can therefore coexist within the same inference engine.

The ecosystem consists of many independent components, including inference engines, routers, KV-cache managers, data-transfer libraries, and cluster controllers. Platforms such as Nvidia Dynamo, llm-d, and AMD Infera “package” selected combinations of these components into complete software distributions. They publish compatible container images, connectors, deployment manifests, and orchestration logic that allow the components to be deployed and operated as one system. The resulting product is usually a collection of coordinated containers rather than a single monolithic service (for instance: Dynamo, llm-d, and Infera are typically deployed on k8s and co-ordinate large, distributed systems).

Source: RedHat / llm-d

Context Parallelism

Long context benefits parallelism techniques that a fixed 8k prompt cannot strongly exercise, because at 8k there is little to divide and TTFT is already short. Moreover, parallelism strategies like TP and DP attention are not optimal at longer context lengths, TP can result in the full KV being replicated on each rank. Although KV is shared for DP attention, it can get hung up on longer contexts, as long context workloads also result in a higher variance of possible context lengths.

Source: SemiAnalysis

Context parallelism is a parallelism technique that splits query tokens across GPUs. It comes in two forms: PCP for prefill context parallelism and DCP for decode context parallelism. In PCP, each rank prefills its query chunk (KV ring-passed), since prefill tends to be compute-bound, this parallelizes FLOPs resulting in faster prefill with no giant-prompt prefill spike on one rank. For DCP, each rank scans its KV shard, and the partial attention is then merged flash-decode style. Since decode is memory-BW-bound, parallel KV reads can result in faster tok/s.

Source: SemiAnalysis

This parallelism technique was invented partially by Nvidia Research. Nvidia Research is great at fundamental research like this while for end to end training research, they are embarrassing America with their horrible Nemotron3 Ultra model which is currently getting massively beaten by even tiny Qwen3.8 27B model. DCP/PCP forms part of CUDA moat as the AMD implementation of DCP/PCP isn’t optimized yet. In the vLLM support matrix, every single AMD backend is unsupported.

A few of the changes mentioned in the next few sections focus on DCP/PCP.

vLLM Agentic Optimizations

Working alongside vLLM maintainers from Inferact, Red Hat, NVIDIA, and AMD, we used AgentX’s realistic replayer as a north star, with the resulting fixes landing upstream where most of the optimizations are highly transferable to production. A few examples follow:

vLLM improved hybrid-attention prefix caching so short-lived sliding-window allocations do not evict useful long-context checkpoints. Selective retention preserves sparse replay boundaries and reported a prefix-cache hit rate above 95% with fourteen concurrent requests and contexts up to one million tokens. The same reachability policy was applied to Mooncake, and unreachable sliding-window lookups were removed. Earlier follow-up work also stopped offloading sliding-window blocks that could never be reused and kept the speculative lookahead block in the retained prefix.

Source: SemiAnalysis and vLLM GitHub PR for Agentic Workloads

Agentic Workloads for high concurrent agents require offloading. Thanks to the aforementioned influence of AgentX, there are already workstreams on vLLM that aim to allow CPU KV offload for hybrid models rather than only for uniform full-attention models. This distinction matters because a uniform model has one KV layout per token, so a connector can describe what to save with a single block geometry. A hybrid model carries several cache groups at once, each with a different shape and a different lifetime, and a connector that assumes one uniform layout cannot express which group a given block belongs to. Offload was therefore unavailable for the models whose long sessions needed it most. The general SimpleCPU connector came first, was enabled on ROCm, and was then extended to DeepSeek-V4 hybrid attention, reporting 81.7% higher output throughput and 46.6% lower mean e2e latency against recomputing the prefix once it no longer fits in HBM.

Mooncake has also gained equivalent hybrid-memory allocation support. The same layout problem resurfaces in disaggregated serving, where a pending change transfers Kimi-K3’s conv+ssm recurrent state alongside the attention KV over MoRI-IO for 1P1D prefill/decode splits; without it the decode side starts from an uninitialized recurrent state. The recurrent-state slot rides the existing remote-block-ids channel, so the disaggregation router needs no model-specific changes, and the path was exercised end-to-end on MI355X, TP8 per leg over cross-node RDMA, with DSpark speculative decoding on both legs.

When profiling realistic workloads, vLLM maintainers noticed that during offload, the cost moved to the store path, which was writing too much and too often. Three new fixes have now addressed this issue:
A store is now skipped while an identical transfer is already in flight, so concurrent sessions sharing a prefix pay for it once rather than once each. A store covers only newly generated KV ranges, so a session that extends its history writes the delta instead of rewriting the whole prefix on every turn. Finally, a store no longer depends on whether the same blocks still sit in HBM, so work already scheduled is not discarded when an eviction lands underneath it.

The load path was tuned separately, because lookups happen on every scheduling decision rather than only when data actually moves. Making lookups asynchronous in the scheduler path keeps the connector off the step’s critical path, so a step no longer waits on CPU-side cache queries before it can admit work. Compact zero-copy lookup keys, parallel receive-side loading, and prebuilt Mooncake key strings then removed the CPU and transport overhead that remained.

Source: SemiAnalysis and vLLM GitHub PR for Agentic Workloads

Long-lived hybrid state also forced correctness and accounting fixes that fixed-shape requests rarely reach. vLLM emits cache events per hybrid cache group, strides distributed-context stores correctly, and computes lookup prefixes correctly under distributed context and prefill. A related context-parallel accounting change aligns cache ownership with sharded token ranges. Speculative state is now propagated across merged Mooncake groups and through the SimpleCPU coordinator, preventing the repeated-turn cache from silently losing EAGLE state.

Looking at the ROCm side of optimizing Agentic workloads in vLLM, the work continues below the cache layer, where the remaining cost is per-layer rather than per-request. Once the prefix survives and arrives on time, what is left is the decode step itself, and a decode step that runs thousands of times per session pays for every avoidable copy and every mismatched kernel.

Three open changes attack that layer:

A Kimi-K3 change writes KDA decode results directly into the layer output buffer, removing one device copy per KDA layer; the saving is small in isolation and repeated for every layer of every decoded token.

A second change selects an AITER sparse-MLA decode kernel in place of the generic path, and reported 5.22% higher AgentX output throughput with substantially lower inter-token latency.
The third is a useful illustration of how much the measurement shape matters. A companion change routes full-graph attention projections through tuned AITER GEMMs and reached a 2.3% gain on fixed sequences at low concurrency. This shows a kernel-level change can show a clean gain on uniform shapes and then be swamped, on an agentic trace, by the cache and scheduling variance that the trace introduces. A pending change turns that shape sensitivity into the dispatch criterion itself: it replaces the DeepSeek V4 C4A selector’s ROCm top-k bottleneck with a hybrid AITER/native path on gfx950, routing short and medium contexts through AITER and long contexts through a graph-safe tuned native fallback. It reports end-to-end selector speedups of 1.21x to 1.76x, with decode-kernel geomeans of 1.2x to 2.9x across an 84-shape matrix.

SGLang Agentic Optimizations

The AgentX team has been working closely alongside SGLang maintainers from RadixArk, Meta, Nvidia, and AMD to drive optimizations to Agentic workloads run using SGLang, resulting in massive improvements in production inference performance. Let’s discuss these optimizations in more depth.

We will start by explaining how, from the allocator side, SGLang’s sliding-window work addresses the same conflict vLLM’s retention policy does. Window pages and prefix pages are drawn from one pool, and the window is the greedier consumer: it turns over constantly while the prefix sits still and so, under pressure, the transient allocation displaces the durable one.

Three design improvements attack this problem that from different angles. One proactively frees pages as they leave the window rather than waiting for eviction pressure to find them, so dead window state stops competing for pages it can no longer use. Another caps compute locks to a single window, bounding how much of the pool an in-flight request can hold pinned at once. A third removes stale full-KV entries that outlive their usefulness.

Proactive freeing has a fork-shaped blind spot, though: a request branching from a shared prefix can still hold reusable full-KV while the window state at the branch point has already been released, and the whole prefix gets recomputed for want of the cheap half. Open work preserves the SWA state at those branch points so forks inherit the window instead of rebuilding it.

Alongside those, the ROCm ring-cache fix is a correctness rather than a capacity change: a ring buffer reuses slots by construction, and reusing one whose old contents are still referenced yields wrong output rather than slow output. None of this is visible on a single 8k prompt, where the window never laps the prefix and the pool is never contended. On a multi-turn hybrid session, these changes are what decide whether the expensive full-attention history is still there on the next turn.

HiCache is SGLang’s first class in-tree offloading mechanism. It faced the same hybrid problem vLLM’s connectors did, and solved it with an asymmetry: offload the full-attention cache and reconstruct the short sliding-window tail on the way back. Only the expensive half is worth moving across the bus, and the cheap half can be rebuilt more quickly than it can be fetched. On AMD, staged write-back keeps that movement from blocking the engine while it happens. Recurrent state was the remaining gap, because it cannot be rebuilt from neighbouring tokens the way a window tail can; FlashInfer GDN checkpoints let it participate in prefix reuse at all, and raised throughput from 47,771 to 53,004 tok/s/GPU at a 92.4% cache-hit rate.

Source: SemiAnalysis

Two further changes address how variable-length traffic affects kernel pipeline. AgentX realistic sessions tend to arrive at continuously varying context lengths, a similar pattern observed for production traffic. A naive runtime that specializes on length will compile a fresh kernel for nearly every request it sees. SGLang maintainers solved this by passing context length as a runtime scalar instead collapsing it into one compilation, improving AgentX concurrency 384 output throughput by 26.75% and mean TTFT by 36.25%, by removing compilation, not from computing anything faster.

Source: GitHub

In the same spirit, removing a per-step device-to-host sequence-length synchronization eliminates a decode bubble that exists only because the host wanted to know a length the device already had.

Source: GitHub

Variable length also bites inside the attention kernel itself. On GB300, a mixed-context decode batch pays a tail tax: a matched profile attributed 8.4 ms of a 9.8 ms decode-step delta to attention, with the longest requests dragging out the persistent kernel’s shared wave. Open work splits TRTLLM MHA decode batches into KV-length-sorted groups so the short requests stop waiting on the longest ones.

Decode can starve one level up as well, at the scheduler rather than in a kernel. Under DP attention, every rank joins the same MoE collective while scheduling attention work locally, so a rank fed a stream of chunked-prefill continuations keeps winning the prefill-first decision while peer ranks’ running batches sit waiting, observed on AgentX runs. A configurable decode interval after prefill forces decode rounds between prefills; on AgentX DSv4 Pro, output throughput rose 141% and p99 inter-token latency fell 97.3%, at the cost of median TTFT rising from 36.5 to 59 seconds, trading off first-token wait for stream smoothness.

Source: SemiAnalysis

When a request carries no reusable history such as the start of a subagent, any worker will do just fine, and load balancing is the only question worth asking. When the request carries a MB of cached prefix, sending it to an idle worker that does not hold that prefix is the expensive choice, and the router needs to know where the state already lives. SGLang added DP cache affinity, so a session is sticky to the rank holding its cache. In this PR, DP-aware prefill and decode routing is both implemented so that both halves of a disaggregated deployment make that decision consistently, and cache balance as a routing signal so affinity does not degenerate into one hot worker. A router can only act on what it is told, so hybrid cache events also become radix-cache aware and sliding-window aware.

Speculative decoding receives special attention, because MTP adds a second, smaller piece of per-request state that has to survive everything the main cache survives. SGLang fixed draft-window transfer in disaggregated serving so that state crosses the prefill-to-decode boundary intact, added overlap scheduling for high-concurrency online decoding, removed a no-op EAGLE renormalization, and avoided host synchronizations during EAGLE prefill. The open resource-lease scheduling work and data-parallel graph-metadata fix continue the same effort, which is to make overlap safe when requests can be retracted and resumed rather than simply run to completion.

When looking at Agentic workloads with Heterogeneous prefill and decode topologies, prefix-aware staging is needed, and this is where prefix caching and disaggregation interact badly. When the two sides are not sharded identically, KV cannot be copied across as one contiguous stream; it has to be split on a transfer grid and reassembled at the offsets the decode side expects. A prefix hit makes that harder, not easier, because the prefill worker now sends only the uncached remainder while the decode side still expects a complete, correctly positioned cache. Radix-cache support in the staging buffer splits cached sends on that grid and scatters them at the correct decode offsets.

Source: SemiAnalysis

However, this results in correctness problems. A 127,500-token shared-prefix test went from 2 correct needles out of 128 to 128 out of 128, meaning the cache had been silently landing in the wrong places, which a throughput benchmark would have scored as a fast, confident, but wrong answer. The AgentX comparison additionally raised median per-user output throughput by 9.6% at nearly unchanged total throughput per GPU. The transfer itself also carried dead weight: decode-side PREBUILT batches never enter a model forward, yet every transferred prompt was still flattened and copied into a CUDA input tensor that the first decode step reconstructs from relay metadata anyway. Dropping that unused prompt transfer is the largest single win, driving +18.0% per-user output throughput and +12.7% decode throughput per GPU on AgentX GB300. A follow-up moved the prefill DP-rank bootstrap query off the decode scheduler’s critical path, overlapping an HTTP round trip that had been paid synchronously at result consumption, driving a further +1.36% per-user output throughput on the same deployment. Open work continues along the same seam: multi-pool DeepSeek-V4 support in UMBP, unified-KV HiSparse state carried over MoRI, and preserving the prefill-owned token when decode terminates without visible content. The HiSparse work should be read as a capacity and correctness enabler for long contexts rather than as a throughput win at high concurrency, which it is not yet.

Source: SemiAnalysis

TensorRT-LLM Agentic Optimizations

Next, we shall move onto explaining a couple of the recent optimizations from TensorRT-LLM. First, we look at TRTLLM’s distinct frontend optimization for repeated chat turns, a cost that only exists because the workload is multiturn. Every turn of a conversation re-sends the entire history plus a little more, and the naive implementation re-tokenizes all of it. Tokenization is cheap per kilobyte, but is ruinous when the same 100,000 tokens are tokenized again on every turn.

The obvious fix, to tokenize only the new suffix, is wrong in a way that is easy to miss, because byte-pair encoding is not position-independent. Tokens can merge across the join, so splitting the text at the boundary and concatenating the two token sequences can produce a different sequence than tokenizing the whole string, which quietly diverges from the sequence the prefix cache was built against.

TRTLLM implements Boundary-aware incremental tokenization which handles this by finding the rendered-text common prefix, rolling back one complete token so any merge that spans the join is recomputed, and tokenizing only the changed suffix from there. On the Qwen3.5 AgentX trace, it matched full tokenization on all 1,087 transitions — the correctness claim, tested rather than assumed — and reduced mean processing time from 185.1 ms to 11.3 ms.

Source: GitHub

A fixed 8k1k request has no prior rendered turn to reuse, so none of this appears there. Relatedly, chat-template rendering was moved into the input-processing pool, so a long template no longer serializes the main request loop behind it.

Source: SemiAnalysis

The MiniMax-M3 work focuses on disaggregated KV movement, where the failure is one of granularity. When prefill and decode do not agree on head layout, the KV for one logical request stops being a few large contiguous regions and becomes thousands of small strided pieces, each of which turns into its own transfer descriptor. The bytes moved are unchanged; the per-descriptor overhead is what explodes, and it explodes in the worst way on exactly the long prompts that matter! Corrected multi-pool mapping and a chunked NIXL bounce path coalesce those pieces through a bounded reusable arena, trading an extra staging copy for orders of magnitude fewer descriptors. Its AgentX diagnostic reduced request-critical KV p99 from 26.74 seconds to 125 ms at concurrency five, and from 10.15 seconds to 288 ms at concurrency forty.

Source: GitHub

Nonblocking context-transfer polling protects the same path by reaping completed transfers even when scheduling stalls. This breaks a feedback loop in which finished KV blocks stay pinned and prevent new admissions. The stalls themselves also had a removable cause: avoiding implicit device-scalar syncs in DeepSeek-V4’s context sparse-attention metadata eliminated 18 four-byte device reads per step that each forced a cudaStreamSynchronize, on a GB300 disaggregated context worker. The fix threads host-side counts through as plain Python ints, so the executor thread no longer holds the GIL for most of a step while the KV-transfer and response threads wait behind it.

TensorRT-LLM also moved irregular long-context work onto more efficient execution paths. Context graph producers for MiniMax-M3 capture stable sparse producers while leaving request-dependent attention eager, and per-user output throughput improved by 12.58 percent in its AgentX test. An open native KV-event production change reduces allocation and conversion work on the KV-aware routing path.

AgentX also exposed kernel-selection and scheduler-lifetime failures that only appear at scale and duration. Two are about which kernel gets picked. MiniMax-M3 added CuTeDSL choices to MXFP8 autotuning, widening the candidate set and improving output throughput per GPU by roughly 7 to 10% at low-concurrency aggregate points. In the opposite direction, TensorRT-LLM disabled corrupt split-K MoE tactics after it crashed five of seven AgentX runs, with no crashes in seven matched runs afterwards. A tactic that is fast and wrong is worse than one that is merely slow, and an autotuner will select it enthusiastically unless it is removed from the pool. Selection is not the only way a kernel goes wrong: MiniMax-M3’s legacy sparse-attention path for short queries could hand the SM100 kernel a non-contiguous, head-major block-index view that it read as contiguous, selecting the wrong KV pages and producing incorrect or non-finite output. Honoring the block-index strides for q_len ≤ 32 fixed the indexing without materializing the tensor or adding a kernel, and five matched full AgentX pairs on GB300 afterwards completed with zero serving errors and no non-finite markers.

Source: TRT Github

The other two are lifetime bugs, which are the characteristic failure of long runs rather than large ones. Sequence-slot headroom and consistent slot-indexed buffer sizing handle the transient overlap where a completing request and a newly admitted one both need a slot, a window that a steady stream of arrivals and departures hits constantly and a fixed batch never hits at all. A later attention-data-parallel dummy-request fix kept nine Qwen3.5 disaggregated cells alive where most earlier cells had failed within minutes, the difference between a configuration that benchmarks and one that survives a session.

Two open transfer changes target very long disaggregated prompts, and together they show how a fix can create the next bottleneck. In the default arrangement, a decode worker cannot start until the entire prompt has been prefilled and then transferred, so two expensive phases run back to back even though the first produces its output incrementally. Pipelined KV transfer begins sending each completed prefill chunk as it lands, so transfer overlaps prefill compute and only the final chunk is on the critical path.

That change makes chunk handling frequent, which exposes work that used to happen once. Its follow-up retrieves only the block IDs belonging to the current chunk rather than the whole prompt’s block list each time. For a 128,000-token prompt split into 1,024-token chunks, that is the difference between building a 4,096-entry list once and rebuilding it 128 times for every layer group. A per-chunk cost that scales with total prompt length is a cost scaling shape that eats the gain the pipelining just bought.

Source: SemiAnalysis

AMD ATOM Agentic Optimizations

AMD’s ATOM engine was originally designed only for single turn workloads instead of real world agentic multi-turn production workloads, so there were a lot of changes needed to the core fundamental ATOM engine and kernels to enable good support for long context multi turn workloads. ATOM still has a long way to go to support agentic workloads relative to where vLLM/SGLang are at present. AgentX is used as the realistic north star target for ATOM’s refactor to support agentic workloads. The first optimization we will chat about that ATOM implemented is smartly using sparse checkpoint retention for DeepSeek-V4 paged sliding-window attention. The merged implementation keeps selected window tails alive so branch and replay requests can resume at useful boundaries. Its measurements separate the two effects cleanly: on the same AgentX trace at concurrency 48, the actual prefix hit rate rose from 5.6% to 96.45%, and losses at the sliding-window gate fell from 91.35% to 0.16%. The second number is the mechanism behind the first. Nine out of ten prefix matches were being found and then discarded for want of a window tail, so the cache was not missing but being overruled.

Source: GitHub

Two earlier cache-manager fixes had to land before any of this could be measured, and both fixes are worth noting as examples of a cache that reports itself healthy while doing nothing. One stopped free-pool hits from destroying shared cache entries; the other, a deferred-output fix, restored prefix hashing in the default scheduler mode and moved repeated long prompts from zero cached tokens to reuse of every complete prefix block. A separate change lets prefix-hit prefill stay on the optimized sink attention kernel rather than falling back to the generic path, so a cache hit does not quietly cost part of what it saves.

Hybrid models also carry a recurrent or compressor state, which differs from ordinary KV in one decisive way: it cannot be reconstructed from the tokens around it. A window tail can be recomputed from neighboring context, but recurrent state is the accumulated result of everything that came before, so if it is dropped, the only way back is to replay the sequence. ATOM gave this per-request state a content-addressed checkpoint lifecycle, letting generated turns leave reusable resume points without reserving a separate protected cache for them. In one test, a request reused 512 generated tokens and computed only a two-token suffix.

The tuning detail matters as much as the feature. Publishing a checkpoint unconditionally costs 17.5% throughput on zero-hit traffic, the price paid by every session that never comes back, in order to help the ones that do. Spacing checkpoints by token interval avoided that penalty, and fixed 1k1k throughput stayed within measurement noise, which is the relevant safety property: a feature aimed at agentic reuse should not tax workloads that will never use it.

ATOM’s AgentX-relevant CPU path starts from the arithmetic that justifies offloading at all. Standalone LMCache offload reloads a 32,000-token prefix from CPU in about 0.32 seconds against roughly 2.5 seconds to recompute it, an eight-fold margin. This makes crossing the bus worth doing at these context lengths and would not hold for a short prompt.

The rest of the path is about ownership and index placement rather than bandwidth. ATOM copied vLLM’s multi-connector design which lets a prefill worker send KV to a remote decode worker and save the same prefix to CPU at once, without freeing the blocks until both consumers are finished; two independent readers of the same blocks is a situation where single-turn won’t get.

Promoting restored blocks back into the GPU prefix index fixes a subtler waste: without it, a prefix loaded from CPU is used and then not registered as resident, so the next turn fetches the same hot prefix across the bus again, paying the transfer repeatedly for a cache that was already in HBM. Follow-up work fixed asynchronous save ordering, packed-KV geometry, unaligned handoffs, and remote request accounting together, eliminating reload corruption across a two-round, 2,638-request validation. This bug surfaces only when the same blocks are saved, evicted, and restored many times over.

Source: SemiAnalysis

The distributed path repeats, in a different codebase, and the pattern is already visible in SGLang and Dynamo: routing has to know where state lives. ATOM’s router called ATOMesh is a fork of SGLang’s router with most of the features removed. Unfortunately, ATOMesh required SGLang’s cache-aware routing feature, so that feature had to be added back. ATOM gained KV lifecycle events for cache-aware routers, so the router can know where state lives at all. It also gained multi-node prefill and decode routing, and session-sticky data-parallel routing. The sticky policy is a two-sided compromise worth stating explicitly: a conversation returns to the healthy worker that owns its state, but idle assignments expire so that stickiness does not permanently unbalance the cluster on behalf of sessions that have gone away.

Disaggregation then has to move whatever the model actually keeps, which is not always one uniform cache. DeepSeek-V4 transfers both buffers of its mixed FP8 and BF16 cache layout, and EAGLE disaggregation moves the draft model’s independent KV cache alongside the target cache, the same second-cache problem TensorRT-LLM and SGLang each had to solve. Remote-KV admission and backpressure closes the loop by stopping the decode side from accepting more parked transfers than it can safely resume, which is the disaggregated form of accepting work you cannot finish.

On ATOM, PCP reported 35 to 43% lower mean TTFT, with total throughput gains of up to about 49% at a 64,000-token input - a gain that grows with input length rather than with batch size. Making that usable in practice required it to compose with everything else a session relies on, so DCP was made compatible with prefix caching, chunked prefill, and FP8 KV and then extended to MTP. Parallelism that cannot coexist with the prefix cache would trade one long-context win for another. The same scarcity of parallelism exists inside a single GPU: a batch-1 MLA decode has no head or query dimension to spread, only the KV walk, and a hardcoded split budget of 16 left that walk running on 16 of a gfx950’s 256 CUs. A still-open change stops overriding the kernel’s own split derivation, so Aiter cuts the walk into as many parts as the machine has clusters.

Chunked pipeline-parallel prefill attacks the same problem from the memory side, replacing repeated tensor-parallel collectives with streamed layer-stage handoffs. Its GLM-5.2 result at high load is the most complete in this section: output throughput doubled, median time to first token fell from 28.6 seconds to 8.7 seconds, and each prefill GPU held 3.68 times as many KV blocks. That last figure is the one to read first, because capacity per prefill GPU is what decides how many long sessions can be in flight before the deployment hits the HBM cliff at all.

Source: SemiAnalysis

ROCm AITER Agentic Optimizations

ATOM/AMD vLLM/AMD SGLang’s long-context execution depends on matching lower-level AITER kernels, because a parallelism strategy at the engine layer is only real if the kernels can express it. Prefill context-parallel process groups provide the extra query-sharding dimension that prefill context parallelism needs, and also widen fused-kernel row indexing for prompts above 131,000 tokens. Decode context parallelism (DCP) shards KV across the tensor-parallel GPUs already present, so a longer sequence or a larger batch fits without replicating the whole cache on every rank.

Large caches also exposed a class of failure that short fixed requests essentially never reach: address width. A 32-bit offset is entirely adequate until a single cache pool crosses the boundary, at which point the arithmetic wraps and the kernel addresses the wrong row without any error being raised. AITER added runtime 64-bit dispatch for batch prefill above 4 GB, 64-bit MLA offsets above 2 GB, and 64-bit addressing throughout DeepSeek-V4’s unified cache paths, the last preventing silent reads and writes to the wrong row in pools of roughly 150 million rows.

Source: SemiAnalysis

DeepSeek-V4 decode also gained a persistent MLA kernel for 64-head and 128-head MTP packings. Those two head counts are what ordinary decoding and speculative verification actually produce, so this gives the engine a dedicated long-context path for its common shapes instead of treating them as incidental variants of a kernel written for short contexts. It is the same argument as the vLLM AITER sparse-MLA selection above: in long context, the generic path is not a modest compromise, it is the wrong kernel.

Dynamo Agentic Workload Optimizations

A good chunk of Nvidia submissions use the Dynamo Inference Orchestration and Router Systems. Dynamo’s AgentX series shows that the distributed serving layer can become the bottleneck once engine kernels improve. The router’s work is proportional to the number and length of live prefixes rather than to the number of tokens generated, so a workload of many long, overlapping, long-lived sessions loads it in a way that fixed-shape traffic never does. The first series of PRs reduced the cost of each routing decision: less work on the lookup hot path, no redundant suffix invalidation, and finally batched KV matching, registration, ownership, and terminal dereferences, which reported a 22.2% median output-throughput gain at concurrency 512. Batching helps here for the same reason it helps in an engine: the per-item overhead was dominating the item.

Source: GitHub

The second series of PRs changed how ownership is represented, which is the harder problem underneath. Every cached block needs to be attributed to the requests relying on it, so it is not freed while still in use and not pinned after everyone has finished. With thousands of concurrent sessions sharing overlapping prefixes, the bookkeeping itself becomes significant. Dynamo moved from shared block chains to arena-level ownership counts and finally to backend-specific request leases, each step coarsening the unit being tracked. The lease design reduced AgentX replay time by 23.7% for the vLLM backend and 22.0% for SGLang, and lowered peak memory at the same time, a sign that the previous representation was the problem rather than the traffic.

Source: SemiAnalysis

Further router profiles removed costs with the same shape, where a periodic sweep or a full recomputation had been acceptable only because live state used to be small. Bucketed expiry pruning replaced a scan proportional to everything tracked and improved high-churn AgentX throughput by 13.7%. Delta-only suffix cleanup processes only what changed and absorbed about 28 times as many store and remove events in the same window. Compressed prompt paths cut front-end CPU by 35.3% and materially improved tail time to first token, which matters because prompts in this workload are long and largely repeated. Overload state is now tracked incrementally rather than recomputed.

One routing change is a deliberate trade rather than a pure win. Dynamo can now charge active decode requests in its routing score, so a worker already committed to long-running decodes looks more expensive than its queue depth alone suggests. That improved median AgentX latency at a small throughput cost in the reported tuning point, which is the kind of choice that only becomes visible when requests occupy a worker for a long time. An open follow-up packages that trade into an new agentic router preset that pushes further in the same direction, crediting prefix overlap at 2, scaling prefill load by 4, and weighting active decode requests at 64. At that tuning point, the trade stops costing throughput: on an 8xH200 AgentX run, the preset improved fixed-window completed-output throughput by 8.26% over the default cost function, cut run-level p95 time to first token by 43.1% and p95 inter-token latency by 22.6%, and completed one more full trajectory.

The request plane was optimized next, because an agentic trace does not send one request and one response. It sends many related requests carrying largely identical prompts, and streams every token back as its own frame, so serialization and copying are paid per turn and per token rather than once. Switching to MessagePack request payloads improved throughput by 8.1% and reduced average time to first token by 9.7% in its AgentX test, and direct Python transcoding removed an intermediate value tree from that path entirely.

What followed is a sequence of changes that all remove a copy rather than speed one up: not copying MessagePack event payloads, not copying received ZeroMQ frames, and not paying full inter-token-latency metrics overhead on every token. The chat streaming hot path was shortened for the same reason. Individually, these are unremarkable; multiplied by every streamed token of every concurrent session, they are what determines how many requests per second a frontend can sustain.

High-concurrency profiling then found costs that had nothing to do with moving data. Static logging filters removed a shared span-matcher lock, a contention point rather than a volume problem, and raised reported frontend throughput from 932 to 1,133 requests per second. Simpler positional radix buckets reduced peak memory in the mocker by 5.51 GiB in a 32-worker run. An open change flushes detokenization metrics once per response rather than updating cumulative counters on every streamed chunk, approximately halving frontend CPU time in its matched diagnostic profile. That last one is the clearest example of the category: the instrumentation was cheap per call and ruinous at one call per token.

LMCache Agentic Optimizations

LMCache is an open-source KV cache layer that sits under inference engines like vLLM, storing reusable KV chunks keyed by prefix hash across CPU DRAM, local NVMe, and remote backends (Mooncake, Redis, S3). LMCache can be used as an alternative to vLLM’s native offloading connectors.

Source: SemiAnalysis

LMCache’s multiprocess path was changed for the volume and shape of agentic cache movement, beginning with a failure that is not a slowdown but a stop. When each of many requests with contexts above 100,000 tokens reserves the blocks for its whole load before starting, the pool is exhausted by requests that are all waiting and none progressing. Chunked external-cache loading reserves per chunk instead, so loads interleave and drain. At concurrency 32, the validation completed 120 requests where the old path deadlocked after 28, and concurrency 48 kept running with the KV pool 98.5 percent full.

The other changes reduce how much is moved and how often the runtime gets in the way. Storing only the useful portions of DeepSeek-V4’s hybrid groups cut storage per token by almost twenty times, and sliding-window prefetch now loads only the live window rather than window state that will never be read, the same reachability argument vLLM applied to offload, approached from the storage side. One native transfer call per object group then removes repeated Python lock handoffs across staging copies and kernel launches, which is overhead proportional to the number of pieces rather than to the bytes in them.

Two current LMCache changes are especially specific to AgentX but remain open. The hybrid lock-accounting fix stops one request from releasing another request’s read locks on shared sliding-window or recurrent-state chunks. Several requests must share the same chunks, the accounting must be per-chunk rather than per-holder, and eviction must actually start. Sustained Kimi-K3 runs with DRAM offload supplied all three and produced tens of thousands of warnings, corrupt generations, and eventually GPU crashes once eviction began. Anything short of a long, shared, memory-pressured run leaves it dormant.

A parallel line of LMCache work made all of the above reachable on AMD Instinct hardware. CacheBlend’s non-prefix reuse depended on flashinfer, which is CUDA-only, so a Triton block-sparse attention backend reimplements the three kernels it needs: block-sparse attention with CSR indices and log-sum-exp output, causal prefill, and log-sum-exp output blending. It then routes to them automatically when ROCm is detected or flashinfer is missing. ROCm Dockerfiles mirror the CUDA build and lightweight images. An AMD hipFile backend extends the GDS L1 slab-file tier, which reached storage only through NVIDIA cuFile, by binding ROCm’s hipFile through ctypes and dispatching on torch.version.hip; the cuFile path is unchanged.

Distribution was the remaining gap. CUDA users installed a prebuilt wheel; AMD users built it from source. We worked together with AMD to publish a prebuilt gfx942 and gfx950 wheel which closes that. It installs into the upstream image and passes all 56 KV-transfer kernel tests on MI350X, and it publishes to a GitHub release rather than PyPI so a plain pip install lmcache stays the CUDA build. A one-line follow-up marks the bind-mounted repository as a git safe directory, which only fails in CI because the container runs as root over a runner-owned checkout and the version introspection in setup.py refuses to read it.

Source: GitHub

DCP-aware CPU offload resolves a straightforward incompatibility between two features that long contexts make mandatory together. With decode context parallelism enabled, each rank holds only a stride of the KV, so what any one rank could save is not a usable prefix; the fix gathers the strided shards before saving and redistributes them after loading. Without it, enabling context parallelism silently disables CPU cache hits for exactly the long prefixes that motivated both features. Its validation recorded more than 30,000 CPU hit events, with single-request loads reaching hundreds of thousands of tokens.

Mooncake Agentic Optimizations

Mooncake serves Moonshot’s Kimi production traffic along with production traffic at many labs, and is a transfer engine underneath disaggregated vLLM and SGLang configurations. Until recently, Mooncake’s AMD support stopped short of both RDMA registration and offering installable packages.

Source: SemiAnalysis

Registering GPU memory for RDMA on Nvidia either uses the nvidia-peermem kernel module or exports a dmabuf file descriptor. AMD has no nvidia-peermem equivalent, so GPU-direct RDMA had no path at all and deployments fall back to staging KV through host DRAM. A HIP dmabuf registration branch adds the mirror of the existing CUDA dmabuf path, exporting through ROCm instead of the CUDA handle call, and resolving the true allocation base first because caching allocators pack tensors at an offset inside a larger allocation. Host memory still registers directly.

Support that cannot be installed is not support. Mooncake published CUDA and MUSA wheels but no ROCm package, so AMD users built the engine from source inside every image. A ROCm wheel, CI, and release path publishes mooncake-transfer-engine-rocm to PyPI alongside them. This workstream from Andy Luo, AMD engineer, was due to noticing a pattern when dogfooding agentic workloads with AgentX that building MoonCake from source in ROCm is not an first class citizen pattern.

Source: GitHub

The transfer engine has no device kernels and does not depend on torch, so one architecture-agnostic wheel covers gfx942 and gfx950, and the ROCm runtime is bound at load time rather than vendored, which means the same wheel works unmodified in both the upstream vLLM ROCm image and the SGLang ROCm image. That was verified as a full cross product: MI300X and MI355X, each under vllm/vllm-openai-rocm and lmsysorg/sglang, running the master binary and a HIP buffer transfer test with data verification. The pull request adds a tag-triggered publish across Python 3.10 through 3.13. An open follow-up adds a self-hosted two-node MI350X external prefill and decode tier so the ROCm disaggregated path is exercised on real hardware rather than only compiled.

Together, these PRs mean an AMD AgentX run can now install the transfer engine and the KV cache layer from published artifacts into stock upstream images, and move KV directly between GPU memory and the fabric.

Other Optimizations

The changes above address long-context costs: a prefix that has to survive, a hybrid cache that has to stay correct, a transfer that has to keep up. But there are a whole host of day-zero enablement and correctness bugs that break requests just as badly as a million-token session.

MiniMax-M3 tested whether that ROCm work compounds into day-zero readiness, and the Advancing AI writeup draws the comparison directly: AMD’s first public disaggregated recipe, MI355X FP4, reached InferenceX in January months behind Nvidia, while M3 FP4 disaggregation landed on day zero. This is an improvement from the DeepSeek-R1 period, when parity took months. Three vLLM fixes sat on that day-zero path, and each was a correctness failure rather than a performance one.

Disaggregation was blocked first. NixlConnector’s handshake asserted that the SPLIT-region block_len scales with the prefill-to-decode TP ratio, but block_len follows per-rank KV heads. M3 has 4 KV heads, so a TP4 prefill paired with a TP8 decode is GQA-capped to one head per rank on both sides and the two lengths are equal where the assertion demanded a factor of two. The handshake was rejected, no KV moved, decode regenerated everything from scratch, and gsm8k scored 0. Validating against the actual head ratio fixed this.

The other two were platform splits. M3’s sparse-attention backend read the byte-backed FP8 cache as float8_e4m3fn for every E4M3 configuration. But gfx942’s platform dtype is e4m3fnuz, and the two encodings differ. K and V were therefore altered before the kernels consumed them. The prefill and decode wrappers had also omitted the FNUZ types from their FP8 checks. Using the platform dtype for the cache view fixed both halves. Separately, M3 ships as separate NVIDIA and AMD model files, and only the NVIDIA one implemented the EAGLE3 interface, so speculative decoding aborted at engine init on ROCm with a model-does-not-support error. Bringing the AMD model to parity restored it, with MI355X gsm8k matching both the non-EAGLE3 MI355X run and B200.

The TensorRT-LLM section above covers the M3 work that is long-context specific: descriptor explosion in disaggregated KV transfer, context graph capture, sparse block strides, autotuner candidates, and the corrupt split-K MoE tactics that had to be removed from the pool.

The local AgentX matrix combines session-aware or KV-aware routing, long and variable conversation histories, MTP, hybrid attention, aggregate and disaggregated serving, and concurrency sweeps that cross the HBM capacity cliff. It includes GPU-resident comparisons and CPU DRAM offload through vLLM SimpleCPU, Mooncake, LMCache, and SGLang HiCache. That combination is what activates the upstream work above. The old fixed-sequence matrix usually creates one prompt, performs one prefill, decodes one fixed continuation, and discards the request. It therefore does not measure cache survival across turns, repeated tokenization, session affinity, cache-event traffic, offload churn, transfer progress during scheduler stalls, or long-lived ownership bookkeeping.

The allowed optimization policy treats CPU KV offload as optional. A vendor may use vLLM connectors, LMCache, SGLang HiCache, Mooncake, Dynamo KVBM, or another CPU DRAM connector, or disable offload when the resulting latency and throughput point is better. NVMe offload is deferred. CPU DRAM must scale with the fraction of GPUs used, including the 3 TB cap for non-standardized-DRAM systems. Standardized-DRAM systems have no hard cap but retain the same proportionality rule. The local generator currently applies the 3 TB cap to every runner, so it does not yet implement the standardized-DRAM exception.

The net new optimization surface is not simply longer attention. It is the preservation, movement, routing, reconstruction, and repeated processing of a growing session state. AgentX made those costs large enough to drive generic upstream changes across vLLM, SGLang, TensorRT-LLM, ATOM, AITER, Dynamo, and LMCache. Direct searches of NIXL and Mooncake did not identify additional AgentX-tagged runtime PRs, so their relevant effects remain represented through the engine connector changes above.

AgentX Methodology Deep Dive

AgentX is a massive shift in open source real world long context multi turn agentic trace replaying and we collected traces over $3M worth of tokens within our own dataset consisting of real world traffic from Claude Code, OpenAI Codex, etc. In addition to the dataset, we developed a comprehensive methodology for replaying the traffic patterns fairly. The goal is to stay as authentic to the organic traffic as possible while being equitable about GPU resource requirements.

We will deep dive into the agentic trace datasets, replay methodology, and agentic behavior in general. It is recommended reading for readers that would like to better understand the overall shape of agentic workloads as well as how harnesses orchestrate requests under the hood.

$3M USD Agentic Traffic Trace Collector

When initially designing AgentX, our north star goal was to make the benchmark as realistic as possible in terms of KV workload shape and KV reuse patterns. We began experimenting with replaying some existing datasets, such as SWE-bench, Qwen-Bailian, and other random Claude Code traces from HuggingFace. At the time, these datasets did not include significant use of subagents, 1M context, compactions, dynamic workflows, or many other of the recent defining characteristics of an agentic trace. At SemiAnalysis, most of the team are AI power users and use agents for a broad variety of tasks including coding, analyst research, excel modeling, social media operations, and many more. Therefore, we decided that the most achievable and realistic traces could be captured in house.

To collect a large number of traces, we created a proxy that intercepts HTTP requests to Claude / Codex. Then, users that wished to upload traces simply changed the base URL in their Claude / Codex setup to point to the proxy. At the time of writing, we have collected over 8,000 sessions, 3.4 million requests, and 610 billion tokens. Together these represent more than $3M USD in spend. We open sourced a representative subset of these sessions for the AgentX v1.0 benchmark.

Although agentic harnesses can appear complex, they ultimately orchestrate a sequence of HTTP requests. Each request contains some combination of system instructions, tool definitions, and accumulated conversation history. As a session progresses, this history grows and is repeatedly sent back to the model, creating the long contexts and high prefix reuse that AgentX is designed to reproduce.

Our proxy records these requests and responses as they occur. It also extracts metadata/HTTP headers such as timestamps, conversation IDs, and subagent IDs, which lets us recover the structure of the conversation (request ordering, concurrent branches, and the approximate parent/child structure of each session). This metadata is what allows us to replay the traces approximately how they would have been seen by the original Anthropic API server.

To protect employee privacy, the replay dataset contains no original prompts, source code, tool arguments, or tool results. Instead, we tokenize each requests’ content and then group it into 64-token blocks, finally replacing each block with a session-scoped chained hash. Matching prompt prefixes therefore produce matching hash prefixes without revealing their contents (this paper talks more about this strategy). During replay, these hash blocks can then be replaced with tokens from, say, a coding dataset. So we preserve the approximate context growth and conversation KV-reuse patterns of the original workload.

It’s worth noting that this process is necessarily imperfect, mostly due to the fact that when using a frontier model provider’s API, much of the content that the end LLM server actually sees is hidden. For instance, thinking/reasoning content for SOTA models are now encrypted in HTTP requests and replaced with a deterministic hash in an attempt to hinder distillation attacks. However, it seems like this didn’t work out exactly as well as the big labs had planned…

Source: SemiAnalysis

Furthermore, while we have access to all the raw content of the user/assistant messages, system prompts, tool usage, etc., API providers apply additional chat templating server side that aren’t transparent. We also cannot observe Anthropic’s proprietary tokenizer or context introduced by server-side tools. Images and documents also do not have a straightforward correspondence between their wire representation and the number of tokens processed by the model. We use deterministic placeholders and empirically calibrated, model-specific padding to bring reconstructed prompt lengths to best estimate the content that is actually seen by the server.

We can’t perfectly capture and replay Claude Code / Codex traces as they are actually seen by the Anthropic / OpenAI servers due to incomplete information, but we can get pretty close. The chart below shows the ratio of hash tokens (after our approximations/processing) and the true API provider token count across all request lengths and models.

Source: SemiAnalysis

To summarize, collecting real Claude Code traces in the exact way that they would have been replayed against the original server is not easy due to incomplete information. However, we have enough context to collect and replay traces with extremely high fidelity to match original traffic patterns, timing, prefix caching, and DAG patterns.

AgentX $3Mil Dataset

The dataset used for AgentX v1.0 can be found on HuggingFace. It is a 393 session subset of 8.3k session proxy corpus mentioned in the previous section. Additionally, we applied some post-processing to clean up anomalies, such as:

Removing Claude Code security monitor (auto mode) requests and title generation requests as these are specific to Claude Code and not necessarily representative of general agentic traffic
Removing requests with a reconstructed input length greater than 990k tokens (where our approximation overcounted)
Remove duplicate requests (sometimes the proxy received identical requests if the connection was dropped)

Additionally, each conversation is formatted into the WEKA trace format, proposed by Callan Fox as part of his kv-cache-tester project. We chose this format for storing trace information primarily because we worked closely with Callan to develop the benchmark and found it intuitive for storing per-session traces. All in all, the trace format is quite arbitrary and our proxy dataset could be mapped to other formats such as Mooncake.

Source: SemiAnalysis

After these are applied, we get the following dataset. Note that not all X-axes are the same.

Source: SemiAnalysis

The distributions for ISL/OSL and inter-turn latency (in agentic work, this is mainly time taken for tool use) are relatively log-normal. The median ISL is 142k tokens and the median OSL is 444 tokens. The median inter-turn latency (or “tool use time”) was 3.84 seconds. Only ~10% of inter-turn latency was greater than 1 minute. These are likely made up of gaps where the harness is waiting on an actual response from a human.

One thing worth mentioning is that these request distributions will look different depending on which harness is being used, since different amounts/types of context are injected (for instance Pi is known to be minimalist in terms of harness-injected context while Claude Code is known for the opposite. Additionally, the ISL/OSL distributions will depend on the model, as different models have tokenizers that can produce either more/fewer tokens. However, given a significant portion of the world’s agentic coding traffic goes through Claude Code, we believe this is rather representative.

The dataset also has 175 sessions with at least one subagent (~44% of all sessions). There are 1,697 total subagent rollouts in the dataset, with a median of 4 per session. The median wall-clock time for a subagent (beginning of first request to end of last request) is 2.27 minutes. This distribution again follows a relatively log-normal distribution.

Source: SemiAnalysis

This dataset includes context up to 1M context, meant to test the more recent frontier open-weight models. Additionally, we have a truncated 256k context length dataset which we replay against models with a max context length of 256k or less.

Agentic Traffic Trace Replayer

Rather than build a replay solution from the ground up, we decided to partner with AIPerf, a vendor agnostic HTTP replayer tool from Nvidia that is adopted by many in the industry including tenstorrent, AWS, AMD, etc. While the intention is to integrate AgentX features into the upstream repo, we maintain a separate fork to be even more vendor neutral such that we have control over allowing even more 3rd more contributions. Again, thank you to the AIPerf team, especially Anthony Casagrande, for the help and dedication to building a realistic and representative agentic benchmark.

An agentic session is naturally described as a directed acyclic graph (DAG). Each request is a node, and an edge means the request at its head cannot be issued until the one at its tail has completed. Every edge additionally carries a delay, specifying how long to wait once that precondition is met.

The simplest session is completely linear, with no subagents and no parallel requests, each request depends on exactly one predecessor. The graph degenerates to a line and the only thing an edge encodes is the inter-turn latency (aka, tool use time or “think” time), which is the client’s local work rather than the model’s.

Source: SemiAnalysis

In agentic traces, subagents can also be spawned. A subagent is a separate stream of requests that has its own context, typically to do a focused task. Multiple subagents can run in parallel to do more aggregate work, and subagents can run in parallel to the main agent in some cases. The main agent then waits on groups of subagents to finish, and then incorporates their outputs back into the main agent’s context (while this is not always the case, this is the most common pattern).

This behavior is responsible for turning the linear chain of requests above into a DAG, where certain requests are dependent on others. When a group of requests belonging to one subagent is identified, AIPerf finds the most recent main agent predecessor and designates it as the “spawning” request. Similarly, the “join” request is identified by the subsequent main agent request after the duration of the subagent group completes.

In the example below, the subagent group consists of a single subagent (001), which runs two requests. Its first request goes out as soon as the main agent’s opening request completes. When that request completes, a 2.2-second inter-turn delay stands in for tool-use wall-clock time, and then the subagent’s second request is sent. The main agent’s second request is the join point for subagent 001. It goes out once both conditions are met: at least 17 seconds have elapsed since the main agent’s first response and subagent 001’s second request has completed.

One small limitation is that HTTP timestamps reveal timing, but not always causality. In the example below, if subagent 001 finishes with seven seconds remaining before main-agent request 2’s recorded start, we cannot tell whether those seven seconds represent work performed after the subagent returned or independent work already underway. AIPerf therefore preserves both constraints: request 2 waits for its recorded main-path delay and for subagent 001 to finish. This reproduces the observed timing and workload topology, but not any dependencies hidden inside the harness. These are things we hope to improve on in subsequent versions of AgentX.

Source: SemiAnalysis

Multiple subagents can also be spawned from a single request. In the example below, subagents 001 and 002 both identify their spawning parent as main agent request 1, and then join at main agent request 2.

AIPerf can also identify “auxiliary” requests, which are one off requests that do not share context with any other requests in the stream. These branch off of the main agent and never join back. In practical terms, these are requests like Claude Code’s “summarize this session” requests that are unrelated to the conversations context. Another good example is Claude Code’s “/btw” feature.

Source: SemiAnalysis

The example below brings it all together. This is the type of trace snippet you’d see in the actual dataset. We have five parallel streams leaving a single main agent request, separated into groups defined by what main agent request they join upon.

Subagents 001 and 002 finish at 20 and 23 seconds, so the first main agent request starting after that, at 25 seconds, is their join. Subagents 003 and 004 are spawned in the same gap but run much longer, finishing at 46 and 50 seconds, so they join at 52 seconds instead. AIPerf keys each subagent by the pair (spawning request, join request), which means these four streams collapse into two branches even though all four leave the same node. The branch takes the name of its first member, which is why the join edges are labelled with subagent 001 and subagent 003.

This is also the first case where the main agent overlaps its own subagents. The request at 25 seconds goes out while 003 and 004 are still running: the main agent is blocked only on the group that joins it, not on every subagent in flight.

The auxiliary chain attaches to whichever main agent request most recently preceded it, which here is the request at 52 seconds rather than the one that opened the session. That node therefore does two things at once — it receives the second subagent group’s join, and it spawns the one-off. Of course the auxiliary request never joins back.

Source: SemiAnalysis

Further AgentX Deep Dive

Pareto Frontiers Sweeping

In the AgentX workload, in order to generate a Pareto frontier, we sweep over the number of concurrent Claude Code sessions against a single deployment. Since each conversation has realistic inter-turn delays and subagent usage, we get a spikier, more realistic traffic pattern. The example below is an example of replaying 40 concurrent clients against a B200 TP4 vLLM server running MiniMax M3.

Source: SemiAnalysis InferenceX

A final point worth discussing is what metrics are important to consider when evaluating agentic workloads. We believe interactivity (TPS - tokens per second) and time-to-first-token (TTFT) are still important, and these are the industry standard for evaluating SLOs. When viewing AgentX results, it is extremely important to consider both TPS and TTFT together, since there are inference optimizations that can improve one at the cost of the other.

We are currently working on defining a new metric that combines TPS and TTFT in a meaningful way. This should also take into account that in agentic workloads, people often care more about the end-to-end speed at which a task finishes rather than how fast they receive tokens or TTFT.

Finally, it’s worth mentioning that the end-to-end latency, in its current form, is now less meaningful because end-to-end latency is directly proportional to OSL. Therefore, P90 E2E latency is heavily affected by the 10% tail of longest output sequence lengths. While it can still be good to holistically compare the overall performance of certain configurations, we recommend instead looking at a combination of TPS and TTFT.

Warmup, Timing, Determinism, and Conversation Reuse

The goal of AgentX is to benchmark systems that are already in a steady state. With agentic workloads, this means that profiling should start from a point where some context trajectories are already cached. To mimic a steady state, it is also desirable that not all conversations start at turn 0, which may cause a “thundering herd” effect.

Warmup proceeds in two stages. First, AIPerf uses a fixed random seed to select a wall-clock point between 25% and 75% of each conversation. At that point, it identifies every active request stream, including the main agent and any active subagents, and sends the most recent request before the selected point for each stream. These primer requests reconstruct the conversation state at that point and are dispatched together. AIPerf waits for them to drain before continuing.

Source: SemiAnalysis

In the second phase, each replay lane is advanced by 10 additional requests to give additional opportunity for the KV cache to materialize. All warmup requests omit inter-turn delay and use a maximum output length of one token, substantially reducing warmup time.

Source: SemiAnalysis

When warmup is complete, profiling begins and lasts one hour. All metrics are collected strictly over this duration. For reproducibility, AIPerf accepts a seed that ensures each run samples conversations deterministically, conversations start at the same point, and that each conversation is reconstructed with the same synthetic content run-to-run. During profiling, we impose a 5 minute idle time cap on each stream, so that long inter turn gaps don’t “acquire” a worker lane for the duration of the benchmark. We enforce this so that we can effectively run the benchmark in 1 hour. In later versions of AgentX where we include NVMe offloading, we may choose to increase this so that we can measure a longer TTL. For now, 5 minutes is reasonable as this is Anthropic’s default KV cache TTL.

This level of determinism ensures that runs using the same inference engine, hardware, concurrency, and server settings are reproducible. However, because AgentX is a closed-loop benchmark, different configurations will complete different numbers of requests at different rates, introducing some natural variation in the workloads they encounter. This is most noticeable at lower concurrency, where fewer requests are completed (naturally) and the workload has less opportunity to converge toward the dataset’s overall distribution.

When a conversation completes during profiling, its replay lane selects another conversation from the dataset sampler. Each replay receives a unique, deterministic cache-bust marker that is prepended to every independent prefix chain, including the main-agent chain and any fresh-context subagent or one-off chains. Forked subagents inherit the marker from their parents. The marker stays the same within a replay, preserving its KV-reuse patterns, but changes between replays to prevent artificially high cache-hit rates. This also enables scenarios to run where concurrency is greater than the number of conversations in the dataset (393).

AgentX’s Fair Speculative Decoding Methodology

As mentioned, the dataset is anonymized upon collection. This means that the 64-token hash blocks must be synthetically filled in before replay. AIPerf accomplishes this by deterministically sampling from a synthetic coding/tool-use token pool.

Importantly, the KV reuse patterns as well as request timing are maintained, however the synthetic request data does lead to some additional considerations. Namely, running speculative decoding methods on synthetic data may lead to the speculator rejecting/accepting an abnormal number of tokens when compared to non-synthetic data (since the speculator is not trained on synthetic data).

We talked about this shortcoming in our InferenceX v2 article, and have since then improved on our methodology. We have worked closely with the community to ensure a mechanism exists in most OSS inference engines that allows users to force how many draft tokens to accept from the speculator (aka, “acceptance length” or “acceptance rate”). Then, for each (model, speculator, draft length, and thinking mode) combination, we collect the average AL on the SPEED-Bench agentic coding dataset, a “unified benchmark designed to evaluate speculative decoding (SD) across diverse semantic domains and realistic serving regimes.”

Then, at runtime we apply these realistic Speculative decoding acceptance lengths to AgentX to ensure vendor neutral fairness.

Source: GitHub

Exploring Agentic Workloads Detailed Telemetry Tutorial

AgentX required more than a new benchmark harness and dataset. We also spent some time rebuilding parts of the InferenceX visualization to make agentic results easier to explore and digest. A single AgentX datapoint represents thousands of requests across growing conversations, subagents, warmup periods, cache states, and dynamically changing in-flight load. Due to this, having a single point on a Pareto curve can hide a lot of useful information. As we have said many times - there is never just a one size fits all solution for inference serving.

Source: SemiAnalysis InferenceX

One of our major changes is how we construct the curves themselves. In previous versions of InferenceX, configurations with speculative decoding enabled and disabled were often displayed as separate curves. However, we are now moving away from this approach. The frontend now combines allowed inference optimizations and displays the best available curve for each model, SKU, and inference engine combination. Due to this, individual points along a single curve may use different optimization techniques and configurations, including speculative decoding, disaggregation, or KV cache offload.

Our goal is to show the best production performance available from each hardware and software stack, rather than creating a separate curve for every possible combination of optimizations. However, we still expose the underlying configuration and provenance for every point. Clicking a point shows a tooltip with a detailed view showing exactly which configuration produced it, along with the run metadata, links to the publicly viewable CI provenance, and AgentX specific statistics. From there, the “View charts” link opens the full point-detail page with AgentX specific statistics.

Source: SemiAnalysis InferenceX

The detailed point view provides a much deeper look into the selected AgentX run. It includes input and output sequence length distributions, interactivity and TTFT over time, KV cache utilization, request queue depth, prefix cache hit rate, input and decode throughput, prompt-token source breakdown, and unique input tokens over time. These metrics make it easier to understand why two points with similar aggregate throughput may behave differently throughout the replay.

The page also separates warmup and profiling data. Readers can switch between the two phases to inspect how the system behaves while its cache state is being established and during the profiling period used for the benchmark run.

Source: SemiAnalysis InferenceX

Points using KV cache offload are surrounded by an additional dotted circle on the main chart, which is used to distinguish points with KV offload enabled. When one of these points is selected, the detail page shows the offload type, KV offload engine, chip cache-hit rate, and CPU cache-hit rate. This makes it possible to see where KV offload contributes to the best curve without creating a separate curve for every offload configuration.

Another new feature is the request timeline. This view shows the individual requests replayed during a selected AgentX run and can be organized either by conversation or by worker. The conversation view groups subagents underneath their corresponding root conversation, making it easy to see when conversations and subagents overlap. Warmup and profiling requests can also still be viewed separately.

Source: SemiAnalysis InferenceX

Each request in the timeline is clickable and links directly to the corresponding conversation and turns on the InferenceX datasets page. This allows readers to move from an aggregate point on the Pareto curve to the exact anonymized request that was replayed.

The AgentX page also includes a flamegraph for visualizing the structure of an individual conversation. Each bar represents one turn and is scaled relative to the largest turn in that conversation. The bar is divided into cached prefix tokens, uncached input tokens, and generated output tokens. This gives a visual representation of how the context grows throughout a conversation and how much of each request can be reused from KV cache.

Source: SemiAnalysis InferenceX

InferenceX/AgentX Future Next Steps

We are excited to continue our goal of making AgentX the most realistic and representative benchmark. For the foreseeable future, we will continue making small bug fixes to the current v1.0.x harness, but have plans to make bigger changes to the v1.1 harness. Submissions for each distinct model will always run the same minor version to ensure all results are comparable.

As a fast follow, we will add SSD/NVMe KV offloading. This will allow an even larger KV cache working set size than DRAM allows, which will allow the high throughput left side of the pareto curve.

We will also capture a larger, more diverse, and more recent dataset of agentic traces across a wider variety of models and harnesses. Rather than representing each request as one contiguous list of hash IDs, the next dataset will preserve the boundaries between system instructions, user and assistant messages, tool calls, and tool results. This will allow AgentX to evaluate workload-aware serving techniques that use information available to the agent harness but normally hidden from the inference engine. For example, a router could direct low-reuse tool traffic to dedicated prefill workers, retain an agent’s prefix during a long tool call, or prefetch and share prefixes when subagents fork. The current format preserves request sizes and KV-reuse patterns, but it lacks the structure needed to evaluate these optimizations.

The following SGLang RFC by Ishan provides a concrete example of why this richer trace structure matters. It proposes a router-initiated hint interface that uses information such as session lifecycles, shared-prefix boundaries, tool-call duration, and subagent state to tell the engine when KV should be shared, prefetched, demoted, pinned, or retained. Capturing this structure would allow future AgentX versions to evaluate these workload-aware cache policies instead of only replaying flat token prefixes.

Source: GitHub

Lastly, we are capturing fine-grained and coarse grain power telemetry data to have an even more accurate view of the efficiency of the Joules per intelligence of different software and hardware stacks.

We have so much data and so many possible visualizations. Please let us know any visualizations you would like to see, as well as any more general feature requests!

Suggest Frontend Feature!

Performance Throughout the Model Lifecycle

In the following sections, we turn to our historical single turn data (8k1k), which covers each model from launch day through the point we retired it from active testing. There are strong results here from both Nvidia and AMD, including several fixed sequence length configurations where MI355X comes out ahead.

AgentX better represents today’s agentic inference workloads. However, the historical fixed-sequence InferenceX results still remain useful for tracking the performance over time. Workloads such as 8k1k and 1k1k strip away most session-level behavior, including prefix reuse, persistent KV cache state, and routing affinity. This makes them less representative of current production traffic, but still useful for tracking how inference performance improves as software support matures.