It’s been a little over 1 year since the disastrous Llama 4 release spurred Zuck to rebuild his entire AI org. Highlights include the shocking $14.3B Scale AI “investment” just to poach Alexandr Wang and the best people from his Safety, Evaluations, and Alignment Labs (SEAL) team, the multi-hundred million dollar (sometimes $1B+) pay packages offered to top AI researchers/engineers, and the expedited compute ramp enabled by their new “Tent” datacenter design. For more details, see our original post on MSL.

Since then, frontier AI has increasingly felt like a two horse race between OpenAI vs Anthropic. Google had a brief moment in the spotlight with Gemini 3 Pro and Nano Banana, but they’ve since faded dramatically. Despite their Windsurf acquisition, they’re far from a compelling agentic coding product, and 3.5 Flash is a benchmaxxed prop that performs far worse than GPT 5.6 and Opus 4.8 in real world scenarios (much less Fable and 5.6). 3.5 Pro is not even Opus level on coding. Microsoft has completely blown their early lead with GitHub copilot and failed to effectively leverage their access to OpenAI IP. SpaceXAI is selling $26B a year worth of GPUs to Anthropic/Google, and the Chinese labs are simply too compute poor to truly reach the frontier.

Meanwhile, MSL made their public debut this April with the launch of Muse Spark. You could argue this model represents a relative regression for Meta. Llama 3 70B and 3.1 405B were both SOTA open-source on release, whereas Muse Spark, despite also being closed source, lagged both DeepSeek v4 Pro and Kimi K2.6—open source models released around the same time—on most benchmarks.

However, evaluating Muse Spark in isolation is missing the forest for the trees. What matters for MSL is the slope, not the intercept. Rebuilding your entire team from the ground up obviously comes with some short term setbacks, and it appears Meta has finally finished paying down this debt. Thus, the interesting question is not where MSL is today, but trying to predict where they’ll be in the next 6 months. We think it's very possible they are better than Google by then due to the team’s focus.

At the simplest level, there are three things you need to build a true frontier model: data, talent, and compute. We believe Meta is the only hyperscaler/neolab on track to be world class at all three and therefore has the best chance at catching up with Anthropic/OpenAI. We’ll explain why in full detail below, but as a teaser, here are the AI compute projections from our new Tokenomics Model.

Lastly, behind the paywall, we’ll discuss what this all means for Google—the company most people today still believe rounds out the AI big 3.

Data is the new oil (for real this time)

We’ll start with data because it’s Meta’s newest advantage and probably the most underappreciated of the three.

In 2024, Ilya famously said that “data is the fossil fuel of AI.” While this analogy correctly highlights the importance of data for training AI models, it incorrectly assumes that the amount of good data is finite. In reality, if demand is strong enough, market forces will find a way.

This time, the invisible hand created a new human data/RL environment supply chain. The three incumbements—Mercor, Surge, and Handshake—are all at $1B+ ARR, and many new entrants who are barely a year old (e.g. Fleet, Mechanize, and Afterquery) are sitting around $100M ARR.

Reinforcement learning (RL) is the most important scaling law for improving AI capabilities today. The general idea is that instead of just predicting the next token, you teach the model how to complete entire tasks (e.g. fixing a bug in a codebase). Besides the tasks themselves, RL requires you to provide an environment for the model to complete the task in, tools the model can use to interact with the environment, and verifiers to check if the model’s answer is correct or not. For more background, see our previous posts.

It is worth emphasizing that many AI insiders believe more RL environments/tasks are all we need to automate virtually all white-collar work. Here’s a quote from my roommate Sholto Douglas, a prominent Anthropic researcher, on the Dwarkesh podcast last year:

Generally, all of these data companies create new RL tasks by hiring expert contractors from the relevant industries (hence why it’s called “human data”). However, there’s another thing every data company is desperately searching for today: real recordings of white-collar work.

Screen recordings are extremely valuable for making RL tasks

Naively, you might think screen recordings are primarily useful for older training paradigms like supervised fine-tuning (SFT) which teach the AI to mimic humans. After all, the whole point of RL is that you let the AI figure out the steps itself and assign rewards based solely on outcomes.

In reality, however, screen recordings can still be extremely useful for making RL tasks as long as you’re willing to do a little extra work.

The first benefit is realism. RL data is only good if it’s 1) representative of real economically valuable work and 2) at the right difficulty level for the AI. Too easy and there’s nothing for the AI to learn, too hard and it’ll never achieve any reward.

The only way to calibrate task difficulty is to just have the AI try solving the problem a bunch of times and then iterating accordingly. It’s actually quite hard to make an RL task that’s sufficiently difficult for frontier models today, so your expert contractors normally spend their time making tasks more challenging. Mechanize, for example, only expects the software engineers they pay $400k+ per year to make 1 good task per week.

Under these constraints, if you ask the average PE analyst to come up with a new financial modeling task from scratch because you want to RL your model to be better at Excel, you often end up with a contrived task that’s optimized for difficulty over realism. This is the main problem with benchmarks like OpenAI’s GDPval and Mercor’s Apex suite (a good benchmark and a good RL env are roughly equivalent). If you read the tasks, you’ll see that most are unnaturally over-specified and don’t sound like something a human would actually ask an AI to do.

For example, here’s a GDPval task that asks the AI to make an itinerary. In the real world, the difficult part of this task would be fetching all the context from disparate sources (email, text messages, random websites, etc) to build the schedule, but GDPval ignores all of that by just giving the AI super detailed step by step instructions instead. A human would never write this 1k+ word prompt to an AI. It would’ve been easier to just make the itinerary themselves at that point!

On the other hand, if you base your task on a screen recording of someone doing their day-to-day job, then it’s guaranteed to be representative of real knowledge work by definition.

Relatedly, workflow best practices and correctness definitions can change a lot over time, so the only way to ensure your data stays realistic is to have a regular stream of real recordings. For example, a good coding RL task today would involve orchestrating subagents, and this simply didn’t exist just 7 months ago.

Screen recordings are also extremely useful for making verifiers. These days, most verifiers are rubrics. Since white-collar work is generally subjective—unlike math and coding where you can just check if the final number is correct or run a suite of integration tests—your verifier ends up being a set of rules/preferences encoded in a rubric that a human or LLM will then use to grade the AI’s output. As the tasks themselves become longer and more complex, so do their corresponding rubrics.

If you collect enough traces (typically thousands) of many people doing the ~same task in slightly different contexts, then you’ll eventually capture the entire action space for said task. This ends up being sufficient information for an LLM to more or less one shot the rubric. Of course, you still need a human to do final review and crucially assign weights to all the different criteria, but this is obviously much more efficient (and typically also higher quality) than asking experts to create the rubrics from scratch.

Recordings can be super useful for deterministic verifiers as well. Knowing the underlying state of the application after the task is completed is often step 1 of creating a deterministic verifier for knowledge work, because the whole point of the verifier is to check whether the AI successfully reached the desired end state. A sufficiently detailed recording gives you this info for free.

Meta just created a top tier RL environment startup

With this context, we can better appreciate the recent news about Meta starting to track their employees’ screens, keyboards, and mouse movements. This is quite literally some of the most valuable data in the world today! Of course, it’s also poetically apt that the Scale AI man is the one spearheading the transformation.

Whereas all the data companies are desperately trying to partner with investment banks, law firms, and advertising agencies to record their workflows, Meta is one of the few companies in the world that has a sufficiently large workforce dedicated to each of these industries in-house.

The fact that Meta is still nimble and aggressive enough to do this despite the PR hit and initial employee backlash is already quite impressive. Yes, they’ve since walked it back slightly by strengthening privacy protections and giving employees the option to pause the tracker for 30min, but we think these are very minor concessions.

Furthermore, they took their data efforts to another level in late May by announcing a new “applied AI engineering org” as part of their most recent round of layoffs/restructuring. ~3000 engineers, which includes 70% of their new grads and a significant number of seniors, will now be making RL tasks/environments full-time.

We think this is an extremely underappreciated advantage for MSL. Anthropic has been the most aggressive lab by far when it comes to buying coding data from RL environment startups, and it’s one important reason why their models are so good at coding today.

Mercor recently disclosed that they logged 2,517,000 expert hours on their platform in 2Q26, which is equivalent to ~4800 people working 40 hours a week. Meta is already in the same ballpark, and their average quality is likely higher. Additionally, they have another ~70k people to pull from if this experiment ends up being as valuable as we think.

It’s also worth briefly dispelling the myth that these 3000 Meta engineers will be doing mindless, low-level data labeling. The days of undereducated contractors from third-world countries drawing bounding boxes or classifying text as NSFW are long gone. At this point, the models are sufficiently smart such that creating a good piece of training data is a real intellectual challenge. Deeply understanding failure modes, ensuring your environment is robust to reward hacking, and scaling task creation without quality degradation are all non-trivial engineering problems.

Besides purchasing from data companies, Anthropic engineers have also been making coding tasks themselves for over a year. Frontier labs are willing to pay $5000+ for a single decent coding task. Mercor’s average pay rate recently surpassed $100/hr, and the rate for SWEs is significantly above average. The top expert contractors at all of these data companies are making over 7 figures a year.

Not only is making RL data clearly more economically valuable than some big tech jobs, it just might be more intellectually stimulating too.

Instagram ads can fund a lot of compute

Compared to OpenAI and Anthropic, Meta has a balance sheet befitting a hyperscaler, and compared to Google, Meta doesn’t have a cloud business that’s aggressively trying to rent out as much compute as possible. Add in the fact that Zuck is willing to take free cash flow negative, and Meta should be able to bring up more internal AI compute than anyone else in the world.

This is exactly what we see happening. Our new Tokenomics Model projects that Meta will have more AI compute than both OpenAI and Anthropic by the end of this year.

It is important to note that a meaningful portion of this compute will be used for recommendation systems (RecSys) and generative ads. However, even if we remain conservative and only assign specific, high-profile Meta DC sites to MSL, their training compute is still comparable to OpenAI and Anthropic through 2026 and 2027. See our recent newsletter for a more detailed breakdown on Meta’s compute strategy and our Tokenomics Model for concrete numbers.

An unprecedentedly large compute ramp led by 5 titans

Much has been written about Colossus and Elon’s ability to bring enormous amounts of compute online quickly, but what Meta’s doing today is arguably even more impressive, as our Datacenter Industry Model subscribers have known for over a year.

Meta is simultaneously building 5 1GW+ “titan” clusters. There’s Prometheus in Ohio, Hyperion in Louisiana, and 3 unnamed campuses in El Paso, Iowa, and Indiana.

Never in the history of humanity have we ever seen a full 1GW campus under construction simultaneously—the closest was AWS building 800MW in Indiana for Project Rainier—yet Meta has two of them right now! Hyperion and Iowa.

For Hyperion, Meta is building the world’s largest single buildings at 400MW each. A total of 1.5GW is under construction today: 3x 400MW monsters plus 3 more standard 100MW buildings.

In Iowa, Meta signed a 1GW lease with a leading datacenter operator (as we flagged in June 2025 in our Datacenter Model). As you can see from the satellite images below, they’ve gone from nothing to the full GW under construction in just 1 year.

At Prometheus—which is already partially operational today—Meta doesn’t quite have a full GW currently under construction, but they are fully embracing the scrappy tent DC design we called out in our previous MSL article. The Prometheus cluster also keeps expanding—from an initial ~1GW to now >3GW within two years. To understand how Meta achieves this, check out our Datacenter Model. We have building-level tracking of all five Titans, and are the first to call out any expansions with precise monthly timelines. Some of these facilities use behind-the-meter power and we track the exact type of system and flag permitting risk.

Connecting the titans: Meta’s solution to scale-across

Since Meta primarily builds and runs their own datacenters, they have more flexibility to customize their infrastructure to fit their actual needs. Prometheus is a good example of this. Instead of a single datacenter or campus, Prometheus is a constellation of 27 datacenters spread across 6 campuses. 5 of these are within 6km of each other and the 6th is 75 to 80km away from the rest.

The reason behind this design choice is that scaling to hundreds of megawatts is currently very challenging, especially regarding thermal and power management. Training the next frontier AI models requires a lot of compute—way beyond 100MW—which is why spreading the workload across multiple datacenters is the path forward to overcome current limitations.

However, with this design choice comes a new challenge: networking. The whole point of a larger cluster is to be able to train larger models in a reasonable timeframe. If the accelerators are not able to communicate efficiently, then there is no point in a gigawatt scale campus. This is where scale-across comes into play.

To address this Meta has introduced a solution called AI-Backbone (or AIBB for short). This is an evolution of their 10X Backbone, specifically designed for AI and massive cluster needs. This network architecture consists of multiple L3 Superspines (or Backend Aggregation - BAG for short) that interconnect up to 5 DSF (Disaggregated Scheduled Fabric) or 7 NSF (Non-Scheduled Fabric) scale-out regions (those can be mixed as well). The L3 Superspines are then aggregated on a single L4 Inter-BAG hub which is set to provide around 22 petabits per second of bi-directional bandwidth across the whole Prometheus cluster.

While the DSF and NSF regions fit into a single datacenter room, the L3 and L4 are distributed across all the datacenters. The L3 layer sits in a single campus, while the L4 is primarily used to interconnect the campuses together. The connection between L3 and L4 is a mix of LR optics and Dense Wave Division Multiplexing (DWDM) systems with ZR optics, depending on the length of the fiber to reach the other campus.

While this seems to be the perfect solution, this network architecture and the distance between the datacenters inherently introduce latency. Inside a DSF or a NSF region, the latency usually sits between 1 to 10µs, but the latency to reach a site that is 100km apart cannot be lower than 500µs, just due to light propagation in the fiber, which forces Meta to use asynchronous strategies for training workflows. The pretraining can be in 1 region synchronously, but the RL can be spread globally quite easily.

Overall, while Prometheus offers a solution to current limitations, the other Titans will go even further with scale-across, connecting campuses up to 2,000kms apart.

Assembling the MSL superteam

Last year, Meta became famous for offering AI researchers compensation packages that would make Patrick Mahomes jealous. After spending $14B to acquire Alexandr Wang and another $1B+ to buy out Nat Friedman and Daniel Gross’ venture fund, it was reported that Meta had poached at least 14 researchers by the end of June 2025. Most of these people were ex-OpenAI, but there were a handful from Anthropic and Google as well. Some big names include Shengjia Zhao, Trapit Bansal, Joel Pobar, and Jack Rae.

Since then, MSL has continued their recruiting frenzy. Notable research/engineering hires include

MSL’s not just hiring members of technical staff either. This January, they brought on Dina Powell McCormick—a well connected finance bro and former advisor to both Trump and W Bush—as President and Vice Chairman to help build out their compute fleet. In the same vein, they also poached the 3 musketeers from OpenAI’s compute team (Pete Hoeschele, Anuj Saharan, Shamez Hemani) in April, though 1 of them has already quit because of Meta’s culture issues on the infrastructure org.

Much like when a sports team suddenly starts lavishingly spending on superstars, only time will tell if this newly assembled Meta superteam can actually win a championship. However, Zuck has made his intentions clear and is pooling together all available resources to take a true shot at the frontier. As of today, you can’t say the same about any other hyperscaler. This could be good or bad for Meta’s business depending on your priors, but we believe they have the best shot at catching up to Anthropic/OpenAI.

But just to be clear, success is far from guaranteed

We’re overall bullish on the future of MSL, but it’s worth emphasizing that they are still basically at step 1. We commend them for marshaling the resources and balls necessary to take a true shot at building RSI, but now they have to do the actual work. Catching up to Anthropic is easier said than done, and at the end of the day, Meta is still a big tech company with lots of competing opinions and priorities.

As we explained in our recent piece on Meta compute, there are many ways Meta can temporarily monetize compute in the short term while still giving themselves the optionality to full send MSL if their research org can hit the right milestones. However, if they do anything that demonstrates a true weakening of resolve, like signing a long term deal to sell compute with no clawback, disbanding their new RL task creation org, or letting their top researchers walk away, then we would be materially less bullish. You could even argue that any one of these would be tantamount to a death sentence for MSL.

We briefly tested Muse Spark 1.1 before the official release and believe it’s roughly on par with Opus 4.6 or GLM 5.2 for general agentic use cases. The fact that Meta chose to price the model just under GLM 5.2 feels intentional. Some of our engineers noticed it has a bad habit of ignoring warnings instead of fixing them and doesn’t properly use the edit tool.

None of our internal token volume will be moving to Muse Spark 1.1, but that’s still to be expected at this stage. Even in the bull case, we don’t expect them to be on par with Anthropic or OpenAI until the end of this year.

What this means for Google

Lastly, behind the paywall, we’ll discuss what this all means for Google—the company most people today still believe rounds out the AI big 3.

Data is the new oil (for real this time)

We’ll start with data because it’s Meta’s newest advantage and probably the most underappreciated of the three.

Screen recordings are extremely valuable for making RL tasks

In reality, however, screen recordings can still be extremely useful for making RL tasks as long as you’re willing to do a little extra work.

On the other hand, if you base your task on a screen recording of someone doing their day-to-day job, then it’s guaranteed to be representative of real knowledge work by definition.

Meta just created a top tier RL environment startup

Not only is making RL data clearly more economically valuable than some big tech jobs, it just might be more intellectually stimulating too.

Instagram ads can fund a lot of compute

This is exactly what we see happening. Our new Tokenomics Model projects that Meta will have more AI compute than both OpenAI and Anthropic by the end of this year.

An unprecedentedly large compute ramp led by 5 titans

Meta is simultaneously building 5 1GW+ “titan” clusters. There’s Prometheus in Ohio, Hyperion in Louisiana, and 3 unnamed campuses in El Paso, Iowa, and Indiana.

For Hyperion, Meta is building the world’s largest single buildings at 400MW each. A total of 1.5GW is under construction today: 3x 400MW monsters plus 3 more standard 100MW buildings.

Connecting the titans: Meta’s solution to scale-across

Overall, while Prometheus offers a solution to current limitations, the other Titans will go even further with scale-across, connecting campuses up to 2,000kms apart.

Assembling the MSL superteam

Since then, MSL has continued their recruiting frenzy. Notable research/engineering hires include

But just to be clear, success is far from guaranteed

What this means for Google

Introduction

When Dario Amodei left OpenAI to start Anthropic in early 2021, the viral release of ChatGPT was over 18 months away and the commercialization of LLMs was practically zero. Just a few short years later, Anthropic and OpenAI combine for ~$100B of ARR and a clear winner emerged in the profitable monetization of AI models in 2026 as Claude Code took the software development world by storm.

Anthropic confidentially filed for IPO on June 1^st. Over 1 month later, equity raises from hyperscalers loom, and a reported OpenAI push out of their own IPO until 2027 has led some to question the ability of the labs to raise. However, Anthropic is the clear clubhouse leader in capturing the B2B market today and is doing so in a profitable manner against an unfocused and money-burning competitor.

With this lead, we expect Anthropic to take advantage of their superior business model and margins to invest in further in new models that help extend their lead and monetization over closed and open source competitors. Anthropic has the ability to truly make OpenAI dance and we see Anthropic as the first $6T company as a base-case possibility if they continue to execute. Pricing power, gross margins, business model, and profitability are all reasons for Anthropic to IPO first and put the impetus on OpenAI to open their financials and raise the necessary capital to compete and fund the massive AI buildout still to come.

We’ve already seen 2 AI Labs IPO this year (Zhipu and Minimax from China), but Anthropic would be the first AI lab of this scale to do so. A confidential filing means there are no public numbers disclosed. Fortunately, the Tokenomics team at SemiAnalysis works to build the financials from the bottom-up by SKU, tier, and customer type. Recently, a WSJ article on Anthropic’s financials confirmed the accuracy of the work our Tokenomics team does across labs and hyperscalers to help investors, corporates, and other stakeholders understand the economics and financials of the AI Ecosystem.

In this article, we’ll dive deep into the work our team has done in the Tokenomics Model and walk you through the financial details of Anthropic, how that compares to OpenAI, where we see the market moving, and implications for the value chain and broader AI market. While other SemiAnalysis work focuses on the technical aspects of AI Labs, this piece will focus solely on the current and future financials, margin economics, and long-term outlook for Anthropic.

Introduction

Up until now the majority of AI buildouts have been primarily cashflow funded by the hyperscalers such as Google, Amazon, Meta, Microsoft, Oracle. Over the last year, that's started to turn with Oracle then Meta, and now even Google turning to debt. Nvidia revenue is skyrocketing, and even 3 years into the build out, the general market is still materially lower on shipment volumes and revenue estimates for Nvidia in the 2nd half of this year versus our through supply chain tracking in the Accelerator Model.

AI Debt Financing will become a multi-trillion-dollar credit market, with over $7T of debt outstanding by 2029 driven both by AI IT Capex and AI Datacenter Capex needs. This will make it the second largest asset backed debt market after the US mortgage-backed financing market at just over $13T.

Annual AI Capex – including GPUs, networking, storage and attached CPU compute as well as for the Datacenters to house AI compute – will be well north of $2T in 2028. Cumulative AI Capex from 2024 to 2029 will reach ~$11.1T, and credit markets will be the main funding source for this buildout.

The surge in borrowing requirements to date has driven by briskly growing demand from AI labs and hyperscalers, and the construction of AI clusters to service this demand was made financeable by long-term take-or-pay compute contracts backed by hyperscalers, with the most common offtake period being 5 years. Though this article will mainly focus on financing AI IT Capex - that is, GPUs and related capex like storage, networking and CPUs attached - there is much that also needs to be done to support the growth of datacenter capex financing. A much more detailed breakdown of both AI IT Capex and AI Datacenter Capex can be found in our AI TCO Model and AI Datacenter Model respectively.

Executing on any AI Compute buildout requires assembling all three legs of what we call the AI Project Trinity – Capital, Offtake, Datacenter:

Yet this is far from an impossible trinity, and many Neoclouds have been executing on cluster build-outs. Deals have been happening, but they come together thanks to clever structuring of one or more legs of the Trinity, close sponsorship of or matchmaking by capital providers like private equity firms that have been serial providers of capital to Neoclouds and Datacenters, and of course, some good old-fashioned risk taking on the part of everyone involved.

Our forecasts for GPU shipments and datacenter capex for the next few years imply that the total outstanding AI debt financing needs will quickly surpass the size of all other US asset backed markets.

But growing this debt market from hundreds of billions in 2024 and 2025 to ~$7.1T by 2029 is no mean feat – there are a number of significant obstacles that must be overcome for the debt market to reach this size and for the compute market to serve more than just hyperscalers and large AI labs:

The current Neocloud market structure has a few other issues. The most pressing problem to solve is broad-based access to compute for renters other than hyperscalers and large AI labs, as well as the limited supply of shorter-term rentals given that most lending is for the 5y backstop template.

For example, VC-backed AI startups and inference providers may be cash rich, but they want large clusters on short term contracts so they can quickly get to their next round of funding and reload on compute. With most Neoclouds thus far preferring to stick with doing large 5y offtakes, these startups are forced to take on larger prepayment or longer contracts than they want, rent fewer GPUs than they need, end up using different GPUs than they would prefer, and often have to settle for start dates far into the future.

Inference providers in particular are very contracted time sensitive compared to training focused AI Labs. While AI Labs are able to commit for longer periods like 3 years and beyond, inference providers are completely unwilling to sign for longer than 1y and would rather forego access to compute than take the risk of committing for any extended period of time.

When it comes to shorter-dated rentals for everyone else, it is still a seller’s market. At this point, we are only aware of a few Neoclouds that are still offering 1y rentals, and they are setting aggressive terms for rental – sometimes requiring prepays of up to 100% of the total contract value. Neoclouds have so much demand for their GPUs that they are able to solve for a prepay amount that can entirely fund the cluster capex, meaning a theoretically infinite IRR as they can stand up a cluster with no cash out the door on their part!

Enter: The Nvidia Backstop

In 2025, we wrote extensively about how datacenter capacity was the bottleneck for AI compute growth. By early 2026, the datacenter supply situation improved considerably, but it became clear that chip production was now the limiting constraint. Now – mid-way through the year – it is clear that financing will now be one of the most significant obstacles to ramping large scale compute broadly available to everyone.

This is why Nvidia has stepped in and has started backstopping GPU rental offtakes themselves. In the backstop program, Nvidia provides a take-or-pay commitment to Neoclouds – a minimum revenue guarantee on the underlying GPU capacity. In exchange for this backstop, Nvidia also shares in a portion of the Neocloud’s revenue earned above the backstop level.

The Neocloud is of course free to rent to any other customer they would like for any term length they deem commercially reasonable and indeed the intent is for the Neocloud to never actually have to invoke the backstop.

With this backstop in hand – the Neocloud can much more easily assemble the AI Project Trinity:

The incremental revenue from the backstop program will be significant, but Nvidia stands to gain far more from these backstops than this additional revenue. They aim to do nothing less than entirely reshape the structure of the GPU market itself. We have already discussed how the TAM they can sell into will become bottlenecked very soon if a 5y hyperscaler backstopped offtake is the only viable deal structure.

In January 2026 for institutional subscribers, we first discussed how Nvidia was becoming the Central Bank of AI. A central bank exists to supply liquidity when others in the banking system are unwilling to step in, supporting economic activity until others are ready to take over.

Most in the neocloud ecosystem are unable to raise enough debt for large GPU buildouts unless they lease to the big hyperscalers directly. Nvidia doesn't want the market to be the same handful of concentrated buyers. The small buyers want GPUs, can pay for them, but they can't offer credit ratings to creditors funding the build out. In mid-2026, Nvidia is clearly showing it stands ready to offer this support as the central bank.

How Nvidia’s Backstops are Structured

Nvidia’s backstop program is typically six years in length, and during these six years, Nvidia stands ready to purchase compute at pre-agreed price levels that vary over the time period. Each Neocloud is expected to negotiate the backstop terms individually, and we can expect that different Neoclouds may end up with different revenue share and backstop schedules. The below table is an illustrative example to explain the basic terms of the program – we present a backstop pricing curve that we believe to be on the lower end of the backstop range – in this case at an average of $2.36 over the six-year period, but we expect that most Neoclouds will negotiate higher backstops.

Let’s start by exploring Neocloud economics under the backstop program by considering a few different scenarios. In the first scenario, the Neocloud focuses on a customer base that rents at a 1-year or lower tenor, so we model the 1-year rental price for a GB300 starting at $6.75/hr for the first year before decaying over time as the overall market cost of compute declines. As with any business that is operating a “curve trade” – that is investing long and renting out on shorter tenors – we should expect that in theory the average realized rental price over the project lifetime should be slightly higher than the current 6y fixed price of ~$4.00. This is to compensate the Neocloud for taking on future price risk.

In any given year, the Neocloud will earn 100% of the rental price up to the backstop amount, but in respect of the amount which the rental price exceeds the backstop, the Neocloud will share a portion of this revenue with Nvidia.

In the table below, we show workings for this scenario. Let’s walk through an example of how to calculate the Neocloud’s revenue share. In the case where Nvidia earns 40% revenue share, we see that in year 1, the Neocloud will charge its customers $6.75/hr on average. Of this, the Neocloud earns the full amount up to Nvidia’s year 1 backstop of $3.68/hr, but for the difference between $6.75/hr charged to customers and the $3.68/hr backstop amounting to $3.07/hr, Nvidia will earn $1.23/hr and the Neocloud will earn $1.84/hr. In total, the Neocloud will realize $5.52/hr for the first year, which of course is lower than the $6.75/hr that they would have earned without the backstop. Over the six years of this scenario, the Nvidia take rate works out to be about 18% on average, but without the backstop, there would be no cluster!

Our second scenario analysis illustrates how the backstop math would work with a 6-year fixed price offtake. This scenario is admittedly an oxymoron – if you can get a 6-year fixed price offtake, then you of course do not need an Nvidia backstop and would thus would certainly not want to pay the revenue share out to Nvidia. Turning around and renting out compute on a 6-year contract would also be against the spirit of Nvidia’s backstop program as this deal would do nothing to further the objective of supplying compute for shorter tenors to a variety of different buyers, especially emerging workloads.

It is important to highlight that while Nvidia does have meaningful internal compute needs, neither Nvidia or the Neoclouds have any intention of using the backstop, and thus it is not surprising that the backstop levels would result in a typical Neocloud earning a zero or slightly negative project IRR.

In the table below, we illustrate the economics of renting to Nvidia under the backstop. This scenario would only materialize if the Neocloud is unable to find sufficient third-party customers at market-clearing rental and must activate the backstop.

That said, this scenario is exactly why the structure is financeable. Though the IRRs earned under this scenario will turn out to be near zero or even slightly negative, the Neocloud will at least still be able to cover debt payments, making lenders comfortable with this structure. Loan underwriting for clusters with an Nvidia backstop will evaluate debt service coverage ratios in the scenario in which the backstop is triggered to determine how much funding can be provided.

The table below summarizes the return profiles under the various scenarios discussed above as well as displaying the economics of a Neocloud renting at the same prices but without any backstop.

We see that the short-term 1y rental book scenario clearly offers the best IRR among the backstopped scenarios at 25.4%, and these IRRs could realize even higher if the rental market turns out to be strong enough such that 1y and shorter rental rates decay by less than we modeled in the curve below. As expected, the IRRs are higher without a backstop – reaching as high as 40.7% when renting for 1y tenors with no backstop. But if Neoclouds activate the backstop and rent to Nvidia at those prices, they can expect to earn zero or negative IRRs.

How to Price a GPU Loan

Many lenders are already being sounded out on the Nvidia backstop structure. They use the typical hyperscale backstopped compute deals as a starting point for their credit assessment but instead substitute Nvidia’s credit rating.

Let’s briefly review how the GPU financing market works before we explain how this market is evolving to cater to the Nvidia backstop. Pricing for Neocloud financing with a long-term offtake is heavily driven by credit spreads for the hyperscaler or other company that is providing the offtake as opposed to following the Neocloud’s credit spreads. Instead – the Neocloud’s execution risk is what drives the spread above the underlying hyperscaler’s credit spread.

Readers may notice the huge gap between CoreWeave 5y unsecured bonds at ~10% and the 5.9% that CoreWeave paid for the fixed-rate tranche of their DDTL 4.0 $8.5B delayed-draw term loan, which was backstopped by Meta. This 5.9% yield works out to about 90 basis points wider than Meta’s 5y bond yield of ~5.0%. This can be interpreted as the market pricing CoreWeave’s execution risk with respect to that particular project at 90bps.

We breakdown the main type of financing we see today in the table below from lowest credit risk to highest credit risk.

A natural question to ask is why has financing not evolved outside the four walls of 5-year IG offtake take-or-pay contracts? Neocloud financing has been challenging because it involves financing a complicated and novel stack of infrastructure and IT equipment where end demand for tokens is still new, not well understood by lenders, and lacks a long history from which to draw insights on risk management from. For lenders, it is much easier to focus on the hyperscaler offtake or backstop as there is no need to dive deep into tokenomics or rapidly evolving AI models and delve into the minutia of which scale-up network topologies perform best on the all to all collective – at the end of the day, it is hyperscaler risk.

Neoclouds have also had a strong incentive to stick with this recipe. To fund a cluster that they intend to rent on shorter terms with a broad book of customers, a Neocloud would have to fall back on unsecured funding. For a top Neocloud, this would mean paying an additional 4% interest, which would significantly impact the economics of the deal given the 70-80% loan to value ratios used to finance GPUs. In our modelling, we show that increasing funding from an all-in cost of 5.62% to an all-in cost of 10% drops PBT margins from 14.8% down to only 5.4%. Unsecured lending to a top Neocloud would probably be around 10%, but expect even more expensive unsecured financing for smaller Neoclouds with a shorter track record.

The most important debt ratio that lenders look at is the debt service coverage ratio (DSCR). DSCR is calculated as the ratio of the project’s cash generation divided by total debt interest and principal payments in each period, and it is a measure of how well covered the debt payments are by the cash flow of the business.

When it comes to the Nvidia backstop, banks look at the DSCR ratios assuming revenue earned under the scenario where the backstop is activated. They are looking for a DSCR of at least 1.3x, particularly for the first few years of GPU financing agreement and size their loans to ensure that this bar is met. This typically corresponds to a 70-80% loan to value (LTV) ratio.

Great Expectations – How the GPU Financing Market will Come of Age

GPU financing will have to rapidly evolve if it is to meet the staggering capital needs we outlined earlier in this article, and financing Nvidia backstopped clusters is the vanguard of this huge shift. Lenders in the past were not comfortable financing projects with price risk and have wanted a long-term offtake from a strong credit, a residual value cushion, a parent guarantee, a vendor backstop or some other floor under the asset. However, lenders are now starting to look deeper into Neoclouds’ business models, with our assistance and due diligence consulting services. Even though they have the Nvidia backstop in their back pocket, the highest quality lenders are now utilizing SemiAnalysis to scrutinize Neoclouds’ operational quality, go to market plan, customer book and pricing strategy.

This is a healthy development – and the Nvidia backstop provides the support that lenders need to get up the curve and prepare for the era when they will finance Neoclouds as a standalone platform basis without any external backstops or guarantee, just like they would understand any other business they lend to that invest in the long term yet are exposed to shorter-term price risk – be it petrochemical companies, memory chip manufacturers and the like.

We think that initial lending will be at a wider spread than the current 5y backstopped hyperscale deals which come in at around SOFR + 225bps (roughly equivalent to a Z-spread of 195bps, or about 5.9% total yield) but at a tighter spread than CoreWeave’s 5-year unsecured corporate bond which has a yield of ~10% (roughly equivalent to Z-spread of 600bps).

Tools that GPU Lenders Will Need

Lenders will also need tools to price and manage risk in the new GPU lending world order. They will need a good GPU rental price index, a way to forecast GPU rental prices into the future and a model for estimating and tracking GPU residual value over time. They will need some way to differentiate among operator quality and a framework for understanding the differences in operator capability. They will also need to be able to model end demand, and how token demand maps into GPU demand.

Luckily – at SemiAnalysis, we have already launched products that address many of these needs!

Our GPU Rental Pricing Index tracks bilateral contract prices across the entire term structure, from on-demand to 5-year deals, across all major GPU SKUs, giving lenders and investors a transparent, transaction-validated benchmark where none previously existed. The H100 1y index is freely available for public use!

Meanwhile, our AI TCO Model has been forecasting GPU rental prices since 2023 and provides a complete three statement financial model complete with return metrics like IRR, return on invested capital, debt service coverage ratio and more. In our consulting practice, where we have provided technical and consulting due diligence to clients that have provided tens of billions in capital to Neoclouds, we have used the AI TCO model as a tool to validate many of their commercial assumptions and help underwrite the investment cases. The AI TCO Model also includes a framework for calculating GPU residual values and much more.

ClusterMAX is the only Neocloud rating system in the industry. Our ClusterMAX rating system evaluates GPU cloud providers across 10 criteria including reliability, networking, and pricing, giving capital providers a standardized quality framework to assess counterparty and collateral risk. We do significant custom diligence as ClusterMAX cannot be used for commercial purposes without our authorization.

Our tokenomics practice tracks end token demand and consumption closely across hyperscalers and AI labs with a focus on SKU and model level ARRs for all key labs, model performance, token pricing, and more.

Lastly, our InferenceX benchmarking product continually measures and tracks real-world GPU inference throughput and token efficiency across inference systems, models and software configurations, enabling lenders and operators to quantify the productive output, and therefore the revenue-generating capacity, of a cluster they are financing. It is an important resource for mapping token demand into GPU demand.

Current and Future Backstop Activity

Nvidia’s backstop program is only getting started, and we see announcements coming faster every quarter. Many Neoclouds are already deep in negotiations with Nvidia, and there are still others that are considering engaging with the backstop program. Let’s briefly discuss the deals that have been announced to date – all of which are in the Asia Pacific region so far.

The first is SharonAI’s 72MW AI factory in Australia. Announced in June 2026, the deal has SharonAI scaling up to as many as 40,000 GB300s under a six-year backstop. Sharon disclosed a total backstop value of $4.88B, which works out to an implied floor of about $2.33/hr/GPU on average over the six year period. Sharon AI’s footprint is now set to grow to 132MW with 102MW contracted, and the company guides to reaching more than 55,000 Nvidia GPUs in total (inclusive of the 40,000 GB300s backstopped) by mid-2027.

Firmus’s 360MW AI Cluster in Batam, Indonesia is the next major project announced with an Nvidia backstop. It will likely be housed in a DayOne facility in Kabil Industrial Tech Park (KITP). Announced just recently on June 29^th, 2026, this project shows the scale at which Nvidia is rolling out the backstop program. Firmus’ first clusters were based in Singapore and focused on H100s deployed using immersion cooling. The next key cluster launched was the 18,000 GB300 cluster in Melbourne, Australia housed in a 42MW self-built datacenter. This project was financed using a $10B USD facility led by Blackstone (Tactical Opportunities, Credit and Insurance) and supported by Coatue.

The Nvidia backstop allows Firmus to grow their business by an order of magnitude. As per Nvidia’s key objectives that we outlined above, the focus of this cluster will be to open up compute to a variety of AI Natives, Enterprises and Inference Providers, with many different rental tenors to be offered. Firmus announced that they expect $25B to $30B of customer revenue over the six years. We expect a portion of the revenue above the backstop level will be shared with Nvidia as per the structure we outlined earlier today.

Beyond this – Firmus also announced a 600MW firm-energy deal with Gunvor that will also underwrite 1.2GW of new renewables development and 1.5GWh of storage in South Australia by 2032. Firmus will still need to identify a datacenter provider or build their own datacenters before they can deploy GPUs.

We have seen many other neoclouds having backstops, but many are not public yet.

Nvidia is not the only merchant GPU vendor to utilize backstops. AMD has already been providing backstops starting last year. We wrote in our AMD article from June 2025 that AMD offered AWS, OCI, Digital Ocean, Vultr, Tensorwave, Crusoe and other Neoclouds a backstop deal, whereby in exchange for customers’ willingness to buy more AMD GPUs, AMD will stand ready to rent back a significant chunk of this capacity in the form of long-term contracts for internal AMD software development purposes if the Neocloud isn’t able to full sell their capacity.

There are many more deals in the works and we believe that many of the clusters announced to be built in partnership with Nvidia will later be confirmed to be built atop an Nvidia backstop. In the next section, we will explore how this revenue backstop will affect Nvidia’s balance sheet and income statement.

What this Means for Nvidia’s Financials

Executing on any AI Compute buildout requires assembling all three legs of what we call the AI Project Trinity – Capital, Offtake, Datacenter:

Enter: The Nvidia Backstop

With this backstop in hand – the Neocloud can much more easily assemble the AI Project Trinity:

How Nvidia’s Backstops are Structured

The table below summarizes the return profiles under the various scenarios discussed above as well as displaying the economics of a Neocloud renting at the same prices but without any backstop.

How to Price a GPU Loan

We breakdown the main type of financing we see today in the table below from lowest credit risk to highest credit risk.

Great Expectations – How the GPU Financing Market will Come of Age

Tools that GPU Lenders Will Need

Luckily – at SemiAnalysis, we have already launched products that address many of these needs!

Current and Future Backstop Activity

We have seen many other neoclouds having backstops, but many are not public yet.

What this Means for Nvidia’s Financials

With Bloomberg headlines suggesting Meta could become a Neocloud, the market’s reaction was immediate: aggressive sell-off of Neoclouds like Coreweave & Nebius, and debates of “overcapacity” coming back. Let’s set the record straight – we believe that both takes are erroneous and that Meta’s datacenter & compute procurement will accelerate, not slow down. Capex in 2027 will be shockingly high. In just the first six months of the year, Meta has contracted over 5GW of capacity across Cloud & Colo, and that doesn’t even include all their accelerating self-build activity. Everything is computer and everything is a neocloud.

Meta’s capacity under construction just keeps accelerating. We show below Meta’s two largest campuses – these two pictures represent 2.5GW of capacity under construction! By the way, if you believed the laughable headlines that “half of US datacenters and delayed and only 5GW are under construction”, we show you here that just two datacenters are half of this. Read our piece “Stop Saying Half of US Datacenters Are Delayed” for more on why these headlines are completely off.

Of course, this naturally raises the questions of what Meta will do with this compute, and whether they’re going to flood the market with all of this supply if they turn into a Neocloud. Broadly speaking we see four major high-value use-cases, which are all differentied and very different relative to what traditional Neoclouds do:

This high optionality, with four high-value-add options, makes it easy for Meta to keep aggressively contracting compute. Meta Superintelligence remains the core engine, but if it doesn’t work out, there are many high-margin alternatives to sell compute. It is essentially a CFO’s dream and makes it very easy to go all-in on compute – we bet Susan did a 180° flip when she saw the pricing of SpaceX compute deals! Meta won’t be a normal bare-metal IaaS vendor with ~30% gross margins – all its options are high value, and enable to easily afford paying a margin to other Neoclouds in order to accelerate their fleet buildout - even if MSL doesn’t work out.

Our Datacenter Model breaks down quarter by quarter their capacity additions across selfbuild, datacenter leasing, and cloud renting. After nearly 10GW of deals signed since early 2024, the bulk of their capacity additions are now through 3^rd parties. We expect this to continue and believe that Meta will be a huge source of RPO growth for the likes of Coreweave, Nebius and others.

Our model also breaks down, quarter by quarter, Meta’s capacity into MSL, other AI, and non-AI. While MSL’s huge growth is understandable (they need to keep up with Anthropic & OpenAI!), the surge of Other AI in 2026 suggests plenty of optionality to monetize compute across RecSys, “SpaceX equivalent”, and Bedrock-type token as a service. If RecSys scales a bit less than expected, they have other options. Of course, if MSL fails, Cloud capacity will skyrocket.

Let’s now dig into these four options. We start with the SpaceX-like deal, and the Bedrock-type ambition. Then, behind paywall, we discuss the outlook for MSL and for RecSys.

SpaceX dreams – Elon’s genius deals

Elon Musk shocked the AI infrastructure world when he announced the first Anthropic deal. He double-shocked the world when he came up with the Google deal. The reason is simple: the revenue per MW of these deals are respectively triple and quadruple what peers are charging, as shown below. The profit per MW gap is, of course, even larger given the cost structure is roughly the same.

SpaceX’s pricing on the Google deal is even higher than what we observe on the on-demand or short-term rental market. Elon essentially invented a new market segment. Our AI Cloud TCO model team tracks hundreds of GPU cloud deals every year, including all contract terms like SLA, pricing, contract length etc. That’s why we have, by far, the world’s best information on GPU pricing by contract length – see a preview in our free dashboard here.

We have never ever seen a deal that large, and that short: the deal is three years, but with option to cancel from both parties within 90 days – so it’s effectively a 3-month deal with automatic renewal.

The reason this has never happened is because very few companies can do it. The financing burden excludes all the Neoclouds from participating in that market – for large clusters, they need to secure an offtaker over multiple years. The top 3 hyperscalers could’ve done it… but they all see higher-value long-term options, such as Microsoft getting OpenAI’s IP in exchange for equity investments & compute, Amazon focused on increasing the adoption of Bedrock and Trainium, Google similarly with TPUs and Vertex (now Gemini Enterprise Platform).

This effectively leaves only two companies that can truly capitalize on this new market: Oracle and Meta. For the former, we see this as a massive blow. It’s yet another piece of evidence that they could’ve done a much better job at monetizing all of their gigawatts of compute. A look at the valuation trajectory of Oracle vs SpaceX illustrates the massive divergence. For both, Gigawatts are an increasing portion of their valuation, and a large driver of that evolution within the last year.

Enter Meta. Given their laser-focus on building Superintelligence, they probably didn’t put enough thought into the Cloud option. SpaceX and Elon paved the way for them. At $50B/Gigawatt of annual revenue, it becomes an easy call. Just allocating 200MW of compute to an external customer drives $10B/yr of revenue, at sky-high margin. And the ability to cancel that contract within 90 days makes it even easier – if they want to give MSL more compute, they can, in short notice.

That’s also a perfect match for Meta’s datacenter construction strategy. A year ago, we were the first to track their new “tent” ultra-fast datacenter design. Since then, Meta tents have been popping up all over the US! By bringing datacenters online fast, even if “lower quality”, it’ll be easier for Meta to monetize compute in a SpaceX fashion.

As such, we expect Meta to announce such a deal soon. It’s a great way to get the flywheel started. Anthropic is our prime suspect, but others could step in as well, such as OpenAI or Google.

Will Meta build the new Bedrock?

Another great option for Meta is to strike a deep partnership with frontier labs to sell their models with Meta’s own compute. As said earlier, we believe they are in final talks with Anthropic to get private access to their LLMs, just like Amazon has through Bedrock (agreement explained in detail here). This means that for Meta, another way to monetize their compute will be to sell Claude. We see three main paths:

There is, of course, also the opportunity to distribute models to free social media users, and to the broader Meta ecosystem including things like connected glasses. It appears more likely that they will prioritize use of their own models, but it’s good to keep the optionality, and they likely have a great path to monetization. The huge distribution potential and network effect means that the likes of OpenAI and Anthropic are likely seeing this as highly strategic, and ready to make concessions to get a piece of the pie.

Scaling Ads Recommendation systems: how much can Meta accelerate?

For Meta, the key AI story is how RecSys is contributing to revenue acceleration at massive scale. At the end of 2022 and early 2023, the market widely saw the company as entering a stage of maturity and low growth amongst an investment cycle. Today, it’s crystal clear that Meta has dramatically re-accelerated revenue growth, in large part due to GPU investments. They have been a key trigger, both on training and inference: RecSys models for ads have been getting bigger and more expensive to run, but much smarter, driving better yields for advertisers as seen in the ability for advertisers to pay higher prices while still seeing strong ROAS. At the same time the RecSys models for content have led to more time on platform across the Family of Apps, improving the monetizable surface area and leading to strong ad imprssions growth.

The key question on everyone’s mind is now – is this sustainable? How much can Meta accelerate core revenue growth? While Meta is playing catching up on the frontier lab side, the non-Meta SuperIntelligence AI chip fleet is producing outstanding ROI. We cover this split compute split between RecSys AI, Meta SuperIntelligence, and other Meta LLM products in our Tokenomics Model.

Behind paywall, we discuss the outlook for RecSys scaling, and then for Meta Superintelligence Labs.

Let’s now dig into these four options. We start with the SpaceX-like deal, and the Bedrock-type ambition. Then, behind paywall, we discuss the outlook for MSL and for RecSys.

SpaceX dreams – Elon’s genius deals

As such, we expect Meta to announce such a deal soon. It’s a great way to get the flywheel started. Anthropic is our prime suspect, but others could step in as well, such as OpenAI or Google.

Will Meta build the new Bedrock?

Scaling Ads Recommendation systems: how much can Meta accelerate?

Behind paywall, we discuss the outlook for RecSys scaling, and then for Meta Superintelligence Labs.

As transistor density scaling has slowed, advanced packaging has become the primary scaling vector. However, AI accelerators have grown so large and require such fast interconnects that the package itself is now hitting limits. Circular interposers constrain package size and wafer utilization, HBM4E doubles the I/O count while increasing speed, and multi-kilowatt packages are overwhelming conventional cooling architectures.

ECTC is the industry’s premier conference for all things packaging. This year’s disclosures were closely aligned with upcoming commercial products. Intel provided an overview of EMIB-T integration, package scaling and future roadmaps. Marvell showed how custom HBM can move interface logic off the accelerator while shortening package routing. TSMC and Microsoft pushed coolant directly into silicon, while Marvell and Lightmatter brought optical interconnects onto the package.

This roundup covers ECTC 2026 technologies most likely to shape AI accelerator packages over the next several years.

Intel EMIB-T

Intel was the largest corporate presenter at ECTC. The key disclosure was EMIB-T. This is the next generation of EMIB with through-silicon vias (TSVs). After the initial announcement, Intel has filled in more of the architecture and roadmap, including tighter bump pitch, larger packages and on-bridge capabilities. Their disclosures show why EMIB-T is expected to be used in Google’s TPU v9, and why it is the most credible alternative to TSMC’s CoWoS platform for large-package AI accelerators.

EMIB-T scaling test vehicle with 2× reticle silicon content. Top-down SEM images show 110, 55 and 36 µm mixed bump pitches. Source: Intel, “Scaling the EMIB-T Advanced Packaging Technology to Address the Future HPC/AI Demand,” ECTC 2026

Intel has validated EMIB-T at a 36/35 µm bump pitch on a package with 2× reticle-sized silicon. This is a reduction over the 45 µm pitch used in Granite Rapids, and a 65% increase in bump density. Granite Rapids-AP is a large package, measuring 70 mm × 105 mm, or slightly less than 9 reticles of area. Validation for 36/35 µm bump pitches is now expanding to 4.5× reticle silicon packages, with certification targeted by the end of 2026.

The next pitch step is also underway, with Intel testing 25 µm bump pitch on a vehicle with two 1-reticle silicon dies connected through a single 3 mm × 18 mm EMIB-T bridge.

Further scaling gets harder. Below 25 µm, the solder volume in each bump becomes very small. Shorts, opens, and assembly-driven yield loss become much more likely. EMIB-T can keep scaling, but the limiter shifts from bridge routing density to bump formation, placement accuracy, and assembly yield.

Intel also showed how large EMIB-T packages can become. Full panel-sized packages are possible, but Intel framed quarter-panel packages as the practical target. It showed a 240 mm × 240 mm test vehicle, equivalent to roughly 67 reticles of area. However, we observed that the sample at the booth exhibited severe warpage. At that size, the bridge is only one part of the problem. Substrate handling, warpage, overlay, and panel-level patterning become first-order constraints. Intel is also evaluating advanced lithography approaches to keep overlay tight enough for these large substrates at quarter-panel and even full-panel scale.

While bump pitch and package size are important, the bridges matter just as much. EMIB-T is much more complex than the EMIB used in current products. It adds TSVs, more metal layers, a power mesh, and MIM capacitor layers so the bridge can carry both high-density signals and vertical power delivery. Intel showed a cross-section with 10 metal layers, including 4 routing layers, and MIM capacitors between M1 and M2. Intel focused on its improvements for HBM4E.

The “T” in EMIB-T is for TSV. Their job is power delivery. In conventional EMIB, power is delivered vertically through the substrate in non-bridge regions, but power delivery near the bridges must spread laterally through the package and die-side routing. With TSVs in the bridge, power can be delivered directly through the bridge, reducing the current path distance significantly. Intel claims that DC voltage drop can be reduced by 68-80% with these TSVs.

HBM4E is difficult because the interconnect must scale both signal density and power delivery at the same time. HBM4 doubles the pin count versus HBM3, and the PHY needs additional power rails such as VDDQ and VDDQL. These rails take routing area away from signals, raising signal density in the remaining space.

To combat this, Intel did not route every HBM channel the same way. It placed the longest signal paths on cleaner routing layers. On M9, only ~28% of the longest channel length runs through the tightest routing region, while on lower layers such as M3, that number rises to ~84%, but those channels are shorter. This keeps crosstalk and insertion loss from being dominated by the worst routes.

Power delivery is also moving into the bridge. EMIB-M introduced metal-insulator-metal (MIM) capacitors between M1 and M2, and EMIB-T builds on that approach. Intel disclosed a capacitance density of 500 nF/mm², roughly comparable to Intel 18A MIMs. Intel claims these on-bridge capacitors improve power delivery network (PDN) AC impedance by more than 82% versus an EMIB-T package without bridge MIM capacitors.

Intel also simulated EMIB-T with HBM4E. At 12 Gb/s, Intel showed ~67% UI eye width without receiver equalization. With a one-tap decision feedback equalizer (DFE), that improves to ~72.5%. A DFE is a receiver-side circuit that reduces interference from prior bits after the signal passes through the package channel.

Intel also simulated higher speeds of 12.8 Gb/s, 14 Gb/s and 16 Gb/s. The UI eye width remained above 60% at all tested speeds, with modest pad capacitance reduction.

Intel’s EMIB roadmap goes beyond passive bridges with only routing and capacitors. Future versions include higher-density on-bridge MIM capacitors, larger high-aspect-ratio bridge dies, sub-25 µm bump pitch, active bridges, and embedded voltage regulators inside EMIB dies. Intel has also disclosed substrate-core embedded deep-trench capacitor (DTC) concepts and >2500 nF/mm² eMIM-T capacitors embedded below the base die, although we have not seen either in shipping EMIB products.

EMIB-T is still behind TSMC’s CoWoS platform on several vectors. TSMC has already deployed DTC/eDTC integration and is further along on integrated voltage regulators and active local silicon interconnect (LSI). EMIB-T narrows the gap, but Intel is still catching up to an ecosystem that has been executing in volume for years.

Marvell Custom HBM

At Marvell’s 2024 Industry Analyst Day, Marvell announced custom HBM. At the time, it was a vague claim with few technical details. HBM had been built around JEDEC compatibility: standard DRAM stacks from the memory vendors, standard HBM PHYs on the accelerator, and a fixed wide interface between them. At Hot Chips 2025, Marvell showed the floorplan of the custom base die. At ECTC, it finally provided the package-level details for custom HBM4E.

The JEDEC specification fixes the interface between the HBM stack and host. That is good for interoperability: any memory vendor’s HBM can be paired with any compliant host. However, it is bad for power, performance and area. The host ASIC must implement the standard HBM PHY and route a very wide, parallel interface with standardized pad placement and breakout rules. As packages get larger and HBM speeds rise, that fixed boundary makes it harder to optimize shoreline, routing density, power delivery and signal integrity.

Custom HBM keeps the DRAM core dies unchanged. Instead, a custom base die, with an optimized die-to-die interface, is fabricated on an advanced logic process. The custom base die can integrate the HBM controller, management and monitoring capabilities, custom logic, and expansion interfaces.

Marvell claims this reduces the host ASIC footprint dedicated to HBM PHYs and associated logic by ~60%, directly freeing up area for more compute, cache or I/O. The custom interface moves much of the memory-side interface into the HBM base die.

Marvell’s example uses 1024 channels at 32 Gb/s, reaching 4.1 TB/s, equivalent to a 2048-bit JEDEC HBM4(E) interface at 16 Gb/s.

The package also gets easier to route, with the custom interface shortening the interposer channel length from 6.5 mm to 1.5 mm, allowing Marvell to keep the same 9 routing layers and 2/2 µm line/space (L/S) while increasing bandwidth.

Marvell’s example uses an organic redistribution layer (RDL) interposer instead of silicon, reducing packaging costs. Organic RDL is limited to much coarser L/S than the silicon interposers in CoWoS-S or the silicon bridges in CoWoS-L and EMIB-T, which makes layout harder. Marvell relies on customized shielding and routing patterns in different sections to maximize bandwidth density while controlling crosstalk.

At GTC, Nvidia announced that Feynman would use custom HBM. Nvidia’s reasons are likely similar to Marvell: higher bandwidth, lower power, and less accelerator die area dedicated to HBM. We estimate that ~16% of Rubin GPU die area is dedicated to HBM-related logic and PHYs. Custom HBM allows Nvidia to offload much of that burden onto the HBM base die.

Custom HBM also enables expansion interfaces beyond the standard HBM link. Instead of forcing all memory traffic through the limited accelerator die shoreline, the base die can act as a secondary memory controller and fan out to additional memory. That memory could be higher-capacity, lower-bandwidth LPDDR, or even a second layer of HBM. This gives the accelerator a path to add capacity without consuming the valuable die shoreline needed for external I/O. This is relevant to AMD’s upcoming MI450 and future MI500 GPUs, which will support LPDDR for increased memory capacity.

Samsung HBM Interposer

Samsung also presented its solution for HBM4E on interposers. HBM4E pushes data rates to 12 Gb/s and above, and doubles the number of I/O pins, increasing routing complexity. HBM4E could require 2× the interposer layers versus HBM3E and 5× as much as HBM2. Due to the increased I/O count and higher data rates, power consumption is also expected to increase by 86% versus HBM3E, and 5.6 times versus HBM2.

Samsung proposed an 8-layer silicon interposer which it says reduces the layer count by 20% versus the estimated requirement. The interposer stack uses a repeated two-signal / one-ground staggered arrangement to shield high-speed signals, with 75% of the layers being allocated for signal routing.

Another key aspect of the interposer is the ultra-high-density capacitors (UHCs). Samsung does not spell out the exact capacitor structure, but they are likely similar to Intel EMIB-T MIM capacitors or TSMC CoWoS DTCs. The UHCs can only be placed on the M1 layer, which is also heavily used for signal routing. Thus, the available area is limited.

If routing is unbalanced, the capacitors get pushed to one side of the interface, creating uneven PDN behavior between the logic and HBM sides. Samsung’s layout redistributes the routing across M1 and the other layers so that UHCs can be placed more evenly across the entire interface. This reduces PDN impedance and voltage noise while keeping routing density manageable.

Samsung HBM Hybrid Bonding Thermals

Samsung also discussed HBM thermals, particularly with hybrid bonding. HBM stacks are getting faster and taller, while the logic die underneath is also consuming more power. With 16-hi HBM, the thermal resistance is still acceptable, but with future generations moving to 20-hi and 24-hi HBM, new approaches are needed.

Samsung compared thermal compression bonding (TCB) and hybrid copper bonding (HCB) for HBM on 2.5D GPU packages with 2 GPU dies and 8 HBM stacks, similar to Nvidia Blackwell. The internal HBM thermal resistance drops by 12.2% with air cooling and 12.9% with liquid cooling. Total HBM thermal resistance drops by 3.5% with air cooling and 7.7% with liquid cooling.

The improvement is uneven as HCB only addresses part of the thermal network. Samsung separates the path into internal resistance, system-level resistance, and GPU-to-HBM crosstalk. The internal resistance and crosstalk drop by ~12.5% and ~9.8% respectively, but the system-level resistance, including thermal interface materials and cooling, increases by ~2.3%.

As more power moves into the HBM base die, such as in memory-bound workloads, the thermal bottleneck shifts. This is especially relevant for custom HBM, where the memory controller and more logic move into the base die. GPU-to-HBM crosstalk becomes a smaller share of the total thermal resistance, falling from 13% at 1× base-die power to 5% at 3× base-die power.

According to Samsung, moving to HCB could allow for higher inlet temperatures or higher package power. In its estimate, moving to HCB allows inlet temperatures to rise by 1-2 °C at constant package power, or package power to increase by ~4% at constant temperature. Samsung also estimates cooling power would fall by ~7%.

Samsung also isolated the impact of HCB at the stack level. The improvement is much larger here: Baseline HCB reduces stack thermal resistance by ~19% versus TCB. Increasing the HCB pad count raises the reduction to 22.3% at 2× pad density and 29.1% at 4× pad density.

Microfluidic Cooling

TSMC showed direct-to-silicon cooling on a large GPU-like test vehicle on CoWoS-R. CoWoS-R differs from CoWoS-S by using an organic rather than a silicon interposer. CoWoS-R was chosen as it provides better warpage tolerance and process compatibility. The test vehicle uses a 3.3× reticle interposer with 4 SoC dies and 8 HBM stacks. Each SoC die consists of 4 SoC heater groups, which together cover roughly half the interposer area.

TSMC compared three approaches: a conventional lidded cold plate package, a lidless cold plate package, and its micropillar direct-to-silicon design. The lidded and lidless cases still use traditional cold plates and thermal interface material (TIM). For the last variant, micropillars were formed directly onto the backside of the SoC dies.

With conventional cooling at 1-2 liters per minute (LPM), a lidded package dissipated 1.9-2.3 kW while the lidless package dissipated 2.5-3.0 kW, using relatively warm 40 °C DI water. Both solutions saturate beyond 4 LPM because TIM becomes the bottleneck.

The micropillar test vehicle matched the lidless cold plate result at 2 LPM, then pulled ahead at higher flow rates, dissipating 4 kW at 4 LPM and 5.3 kW at 8 LPM. Across the full test vehicle, TSMC reported uniform power dissipation above 5 kW. The micropillars brought the liquid coolant into much closer proximity to the heat source, facilitating this improvement.

However, micropillars are not without downsides. TSMC had to form the micropillars after chip-on-wafer (CoW) processing without damaging the CoWoS-R structure and develop new sealant materials to keep the coolant contained despite package warpage and thermal expansion mismatch. The test vehicle passed moisture sensitivity level 4 (MSL4) with no helium leakage or sealant delamination.

Unfortunately, this was one of only 3 papers by TSMC at ECTC as compared with Intel’s 12 and Samsung’s 11.

Microsoft’s approach differs from TSMC’s in the cooling structure. While TSMC used silicon micropillars, Microsoft used straight microchannels etched into the GPU silicon. Instead of using a thermal test vehicle, Microsoft tested on an actual Nvidia GH200 GPU. This likely allowed Microsoft to capture real thermal distribution and hotspots more accurately. Microsoft tested a variety of workloads on the GPU such as HPCG and HPL, each with different compute and memory stress characteristics.

Across these workloads, Microsoft reports 51-60% lower junction-to-inlet thermal resistance for the GPU at a 1 LPM flow rate. The HBM improved less, at only 27-37%, because it was still cooled through a cold plate and TIM. Overall, this results in a 50% reduction in thermal resistance for the package.

Microsoft also showed some preliminary reliability data. While thermal performance is important, datacenter deployments also require high reliability and low downtime. Over a period of 6 months, Microsoft recorded only 9 potential clogging events across ~4370 observations. The rate declined over time, suggesting early instability after installation, followed by a more stable operating period. Even after 6 months, there was no measurable silicon erosion in the microchannels. At the node level, the GH200 successfully completed 3 weeks of repeated benchmarking followed by a 1-week continuous run at a stable package power. Microsoft is still testing cluster-level mean time between failures (MTBF) and availability.

Marvell Optical Interconnects

Another prevalent theme at ECTC was optical interconnects and co-packaged optics. Marvell had several papers in this area, primarily on its Optical Multi-Chip Interconnect Bridge (OMIB) and Photonic Fabric, both of which it acquired through its purchase of Celestial AI earlier this year.

While photonic interposers have been discussed for a few years, Marvell showed off a more localized version at ECTC. Fabricating a multi-reticle photonic interposer will likely prove challenging for yields, especially due to reticle stitching. Marvell also noted that a photonic interposer may lack features available in conventional silicon interposers such as high-density capacitors.

Marvell’s approach has a photonic integrated circuit (PIC) embedded in the organic RDL interposer only where it is needed. In other regions where optical interconnects are not required, an electrical bridge can be used instead.

As the PIC is embedded in the RDL, its grating couplers would normally be obstructed after overmolding. Marvell places a silicon/glass optical block over the grating region before molding to maintain an optical path to the top, where the fiber array unit (FAU) can be attached.

Marvell’s OMIB test vehicle has one primary XPU die and six EIC dies on top. Embedded in the interposer are six PICs (big blue rectangles), six electrical bridges (thin blue rectangles) and 12 DTC dies (small blue squares). The roughly 2×-reticle RDL interposer uses four layers at 2/2 µm L/S.

Marvell also showed off a conceptual multi-die XPU with optical chip-to-chip interconnects to reduce latency and hop count. Marvell claims that OMIB removes the shoreline limitations, since the same bridge can route both on-package die-to-die links and external optical interconnects. Marvell cites a bandwidth density of 1.8 Tbps/mm² with this approach.

The process flow shown by Marvell is chip-last, similar to TSMC’s CoWoS-L. Marvell builds an organic RDL interposer with embedded bridges, OMIB PICs, DTCs, and other components, with C4 bumps connecting it to the package substrate. Bridge TSVs and tall copper pillars connect to the RDL, and the ASIC dies and EICs are attached last.

In the near term, vertically stacked optical engines like TSMC’s COUPE are more achievable than OMIB-style connections or a full photonic interposer. Marvell connects the EIC and PIC using microbumps at 50 µm pitch, then mounts the resulting engine to either the package substrate or an interposer.

The substrate configuration can use a UCIe-S-like parallel bus at a coarse 130 µm C4 pitch, while the interposer configuration can use a UCIe-A interface at a tighter ~40-45 µm pitch. Marvell favors the substrate approach due to its simplicity and better thermal isolation.

Marvell tested an optical engine using a 5 nm (likely TSMC N5) EIC, with four 56 Gb/s TX-RX pairs for 224 Gb/s in each direction. The design uses electro-absorption modulators (EAMs) instead of the micro-ring modulators (MRMs) preferred by other companies, citing better thermal stability and a wider operating wavelength range. While these advantages are real, we believe that EAMs will prove difficult to manufacture at scale.

Marvell also compared the thermal characteristics of an optical engine connected via UCIe-S on substrate and UCIe-A both on a silicon interposer and over a silicon bridge. Under full XPU load, the PIC temperature rose by less than 5 °C on the substrate, versus ~25 °C on the interposer and ~20 °C with the bridge. The organic substrate’s low thermal conductivity and relatively large millimeter-scale air gap isolate the PIC. In both UCIe-A configurations, the fine-pitch silicon close to the XPU provides a low-resistance thermal path. There is also a much smaller air gap, at only ~100 µm, which contributes to the heating of the PIC.

The thermal transients occur within ~30 ms of an XPU power-state change. The PIC heats at ~10 °C/s on the organic substrate, versus ~100 °C/s with the bridge and ~120 °C/s on the interposer. Marvell argues that the EAM bias voltage can be adjusted electronically fast enough to track these changes, while ring modulators require heater-and-feedback loops, which are constrained by slower time constants.

Lightmatter Passage M1000

Lightmatter has previously detailed the Passage M1000’s architecture and optical interconnects, and has outlined its manufacturing approach. At ECTC, it provided a much deeper look at the assembly process, fiber attachment, and packaging results for integrating the multi-reticle photonic interposer with ASIC chiplets.

The test vehicle uses chip-on-wafer assembly to attach 15 ASIC chiplets to a four-tile M1000 interposer. We estimate the interposer measures ~2100 mm², about half of the 4000 mm² eight-tile configuration shown at Hot Chips 2025. The M1000 is highly configurable to different sizes and tile counts depending on the needs of the product.

Each of the four tiles has 32 optical waveguides at 127 µm pitch. Electrical signals and power travel from the substrate to the ASIC chips through the C4 bumps at ~176 µm pitch, two backside RDL layers, and 126 µm deep TSVs with a 10 µm width before reaching the ASIC chiplets through microbumps. The ~2100 mm² interposer occupies less than one-third of the 7200 mm² organic substrate. This ratio is closer to the logic area to package size ratio than the package-spanning interposer area of an equivalent silicon or organic RDL solution. It is unclear whether this is a consequence of the optical architecture or a design choice specific to this test vehicle.

Attaching a silicon interposer of this size to an organic substrate creates severe warpage. The module reached ~59 µm of warpage at the 260 °C reflow temperature, and ~56 µm after cooling back down to room temperature. With a 118 µm-thick interposer and C4 bumps at ~176 µm pitch, this is enough to compromise joint formation. Lightmatter used a magnetic fixture to hold the substrate flat during attachment and reported >95% electrical assembly yield, with healthy microbump and C4 joints across the package.

Lightmatter used a thermal test chip with four independently powered quadrants, each dissipating 170 W. This resulted in a power density of 1.47 W/mm² across the 369 mm² active area. At this power, the photonic interposer reached approximately 100 °C using a 25 °C coolant flowing at 1.8 LPM/kW. This validates cooling 680 W from a concentrated test-chip area in a package designed for more than 900 W across nearly three reticles of ASIC silicon.

Other Highlights

Hybrid Bonding

Hybrid copper bonding still delivers the finest pitch and highest I/O density for HPC applications. The open problem is keeping the interface extremely flat and clean while reducing the bonding temperature. Two material approaches stood out this year.

The first uses organic dielectrics, whose mechanical compliance increases tolerance to particles and surface roughness while reducing bonding stress. Mitsui Chemicals and ASE demonstrated pressure-less Cu/polymer bonding at 200 °C and 10 µm pitch. TOK and NYCU demonstrated a 10-second bonding process at 150 °C. Samples bonded at 200 °C maintained stable resistance through reliability testing.

The second approach uses fine-grain copper. Its higher grain-boundary density accelerates copper diffusion at lower temperatures, and subsequent grain growth increases conductivity. Intel combined fine-grain copper with a low-temperature dielectric stack, achieving uniform wafer bonding after 175 °C and 200 °C anneals. Electrical yield was around 60% in two of three samples, although Intel described these results as a lower bound due to test vehicle and probing limitations. The experiments used wafer-to-wafer (W2W) test vehicles rather than the die-to-wafer (D2W) process targeted by the technology.

The most aggressive pitch came from Applied Materials and EV Group, which demonstrated 450 nm pitch wafer-to-wafer bonding at 98% yield across a chain of 20 million links. Failure analysis associated open links with carbon-rich benzotriazole (BTA) residue at the copper interface. A PVD TaN/Ta barrier stack significantly improved yield. CEA-Leti separately achieved greater than 97% yield after a 100 °C anneal without plasma activation.

Together, these results show that reducing pitch and bonding temperature requires the copper, dielectric, CMP, surface preparation, and annealing to be co-optimized to achieve hybrid bonding with low warpage and no cracking. We expect continued refinement by material suppliers and equipment vendors to improve post-bond yield from 2027 onward.

Interposer Alternatives

As package sizes outgrow the practical limits of circular silicon interposers, more vendors are proposing interposer-less integration schemes beyond Intel’s EMIB-T.

Intel and SPIL placed an SRAM chiplet in the embedded-bridge layer of a fan-out embedded bridge (FO-EB) package, connecting it vertically to the logic die through 25 µm pitch microbumps. The test chip reached more than 265 GB/s/mm² at 0.24 pJ/b.

Panel-scale organic interposers provide another route around the size limits of silicon. Resonac demonstrated the individual process modules for a dry-film embedded-bridge interposer on a 320 mm × 320 mm panel, including 5 µm microvias and 2/2 µm L/S. ASE fabricated an RDL on a 600 mm × 600 mm panel, then divided it into four 300 mm × 300 mm panels for assembly with existing equipment. Its test vehicle used three RDL layers at 5/8 µm L/S.

IBM took a more localized approach with Direct Bridge Multi-die (DBrM). Chiplets are first joined along their edges, creating a mechanically rigid subassembly around a 30 µm pitch silicon bridge. The subassembly withstood more than 30 N in bending tests, versus 0.2 N for IBM’s previous underfill-only structure. This is a positive result for reducing warpage.

Unimicron modeled an even simpler structure requiring neither an interposer nor embedded bridges. Two chiplets connect through a thin silicon bridge mounted beneath them while attaching directly to the substrate. Unimicron’s simulations show that underfill between the chiplets and bridge is necessary to control microbump strain.

While TSMC’s CoWoS-R and CoWoS-L remain limited by the circular wafer on which their RDLs are fabricated, constraining package size and wafer utilization, these alternatives shift integration to panel-level or reconstituted formats or eliminate the interposer entirely. We expect similar architectures to begin appearing in ASICs over the next several years.

Thermal Interface Materials

In advanced thermal solutions, TSMC and its partners continue to dominate the show. TSMC’s direct-to-silicon cooling removes TIM1 entirely, but most near-term systems will still need better materials between the silicon and heat spreader. TSMC’s OSAT partner, SPIL, tested gallium-based liquid-metal (LM) composites, a silicone-based HS-TIM and a carbon fiber HCF-TIM, in a 55 mm × 55 mm FO-EB package. Their measured conductivities were 5.7 W/m·K and 10 W/m·K respectively, and both produced lower thermal resistance than a commercial silicone TIM at 4 W/m·K.

The two TIMs exhibited drastically different reliability outcomes, with HCF-TIM maintaining 95% coverage after 1000 hours at 150 °C while HS-TIM fell to only 75% as its silicone matrix hardened and partially delaminated. Both LM-based TIMs reduced thermal resistance relative to conventional S-TIM, with HCF-TIM delivering the best performance and reliability.

Purdue, the University of Aveiro, and UCLA took a different approach, embedding Cu/Sn microbumps in nanocrystalline diamond. The resulting interconnect layer reached an effective in-plane thermal conductivity of 500 to 600 W/m·K, around twenty times that of conventional microbumps with underfill. This is not a TIM1 replacement, but a way to spread heat laterally through the interconnect layer of a 3D stack. The process remains early, with the paper only using a single-sided test structure rather than an assembled 3D stack.

On the other hand, we saw little discussion on SiC as TIM1 or part of thermal solutions, implying a relatively long way till maturity.

Glass Substrates

Glass’s aurora dimmed somewhat this year, with fewer innovative papers presented at ECTC. However, the unsolved problem is still SeWaRe, the lateral crack that begins at a diced glass edge under RDL stress. Georgia Tech characterized the failure experimentally. Corning used finite element analysis (FEA), peridynamics, and analytical fracture mechanics to model its propagation, showing that stiff copper layers drove cracks towards the glass midplane, while compliant polymer layers changed the crack path. Corning also found that low-CTE polymers combined with appropriate glass selection could reduce the failure risk.

STATS ChipPAC investigated the assembly and reliability of large glass-core packages. Its 74 mm × 74 mm glass-core packages failed every test segment without edge coating, while the edge-coated packages completed assembly and reliability testing without abnormalities. The edge coating also reduced warpage by 33.5% relative to the uncoated glass-core package. Build-up pull-back and edge coating increasingly look like requirements for reliable glass-core substrate assembly.

On a positive note, Intel demonstrated an industry-first 510 mm × 515 mm, 24-layer glass-core panel with fully copper-filled through-glass vias (TGVs), two embedded EMIB bridges, and optical waveguides co-formed between the TGVs. The large prototype was displayed at Intel’s booth. It was processed on existing organic-substrate lines, while singulated units showed no SeWaRe after thermal shock testing.

As OSAT adopters, Amkor and STATS ChipPAC measured 30-40% lower substrate-level warpage with a thinner glass core than their organic references, although assembly defects and TGV filling problems show that the process remains immature.

Glass is making real progress, but this year’s data still support manufacturing development rather than high-volume adoption.

RDL Scaling

Finally, RDL L/S continues to shrink even as package sizes grow. The primary driver is UCIe 3.0, which supports speeds of up to 64 GT/s for future ASIC-to-ASIC and ASIC-to-HBM links. These high-speed die-to-die interconnects tighten signal integrity requirements as organic interposers grow larger and denser.

The roadmap has progressed from 10/10 µm line/space around 2015 to 2/2 µm today, with 1/1 µm emerging as the next target. Reaching the submicron era will require major changes to both RDL routing architectures and manufacturing processes. The process is shifting from semi-additive plating towards damascene for sub-2 µm copper, where CMP planarization and low-shrinkage dielectrics become the key gating steps.

Resonac used polymer damascene and panel CMP to form 2/2 µm wiring on a 320 mm × 320 mm glass panel, including a four-layer via-and-trench structure. imec and Fujifilm pushed damascene to 1/1 µm on 300 mm wafers. Ushio resolved 1.5/1.5 µm L/S over an 18-reticle field without stitching, with 16 exposures covering a full 510 mm × 515 mm panel. Sumitomo Bakelite and Georgia Tech showed a fully imidized liquid dielectric with only 4% cure shrinkage at a relatively low temperature of 200 °C and a fine 2/2 µm L/S.

As the most advanced RDL manufacturer, TSMC also collaborated with GUC to present their work on 8-layer RDL scaling, which is believed to be the near-term limit of the CoWoS-R platform. GUC demonstrated an STCO-based design and validation flow for integrating a 64-bit UCIe-A interface fabricated on TSMC N3 and integrated on an 8-layer CoWoS-R RDL.

Its STCO framework uses ground-signal-ground interleaved transmission lines to control crosstalk and skew, while simulations show that C4-side IPDs provide localized decoupling and reduce voltage fluctuation at the chiplet microbumps.

The design targets 16-36 GT/s with a 64-bit, 10-column UCIe-A interface at 45 µm bump pitch. Signal traces were routed at 2/2 µm across six layers, with the seventh reserved for power delivery. The test chip achieved a measured on-die eye width of 0.77 UI at 32 GT/s, while simulations showed an eye width of 0.74 UI at 36 GT/s. The results show that organic interposers can meet the signal and power integrity requirements for heterogeneous chiplet systems.

Stacked Memory

Samsung showed an approach to DRAM stacking that avoids TSVs entirely. Vertical Cu post Stack (VCS) connects four stacked memory dies through <56 µm-pitch, <30 µm-wide high-aspect-ratio copper posts embedded in molding compound, rather than etching vias through silicon, while also replacing the package substrate with an RDL. Shortening the interconnect cut power consumption by 41% versus a conventional wire-bonded stack, from 0.646 W to 0.384 W at iso-speed. Samsung also reported better signaling, with maximum data rate rising from 8.6 Gb/s to 11.8 Gb/s with only 8% higher power consumption.

The form factor also improves substantially, with package height and footprint each reduced by 40% while bandwidth increases 2.6× and I/O count 6×. Although Samsung’s focus is on mobile platforms like smartphones, this approach is also promising for higher-power workloads. We believe VCS and similar approaches could help future AI accelerators reach higher bandwidth at lower power and a smaller footprint, while also enabling higher-density memory modules like SOCAMMs for server CPUs.

This roundup covers ECTC 2026 technologies most likely to shape AI accelerator packages over the next several years.

Intel EMIB-T

The next pitch step is also underway, with Intel testing 25 µm bump pitch on a vehicle with two 1-reticle silicon dies connected through a single 3 mm × 18 mm EMIB-T bridge.

Intel also simulated higher speeds of 12.8 Gb/s, 14 Gb/s and 16 Gb/s. The UI eye width remained above 60% at all tested speeds, with modest pad capacitance reduction.

Marvell Custom HBM

Marvell’s example uses 1024 channels at 32 Gb/s, reaching 4.1 TB/s, equivalent to a 2048-bit JEDEC HBM4(E) interface at 16 Gb/s.

Samsung HBM Interposer

Samsung HBM Hybrid Bonding Thermals

Microfluidic Cooling

Unfortunately, this was one of only 3 papers by TSMC at ECTC as compared with Intel’s 12 and Samsung’s 11.

Marvell Optical Interconnects

Lightmatter Passage M1000

Other Highlights

Hybrid Bonding

Interposer Alternatives

As package sizes outgrow the practical limits of circular silicon interposers, more vendors are proposing interposer-less integration schemes beyond Intel’s EMIB-T.

Thermal Interface Materials

On the other hand, we saw little discussion on SiC as TIM1 or part of thermal solutions, implying a relatively long way till maturity.

Glass Substrates

Glass is making real progress, but this year’s data still support manufacturing development rather than high-volume adoption.

RDL Scaling

Stacked Memory

It’s been reported that token consumption inside of enterprises is hitting a budgeting wall after unhinged consumption earlier this year. The SemiAnalysis team talked with over 50 customers by slack, phone, and at the Databricks AI Summit to understand trends within the enterprise.

Rise and Fall of Tokenmaxxing

Tokenmaxxing started earlier this year when companies like Meta and Salesforce began encouraging their employees to consume as many AI tokens as possible to boost productivity. At Meta, an employee even built a “Claudeconomics” dashboard that ranked the top 250 power users in the company. The results showed that Meta employees consumed over 60T tokens over a 30-day period, with the single highest individual accounting for roughly 280B tokens. Employees started competing for rankings like “Token Legend” and “Cache Wizard” by having agents do research for hours simply to burn tokens.

The dashboard was shut down 2 days later after The Information reported the spend.

That episode was just one amongst others in the enterprise tokenmaxxing trend in 1H26. Companies are now shifting focus from tokenmaxxing to token budgeting. Most recently, Uber made headlines for burning through their Claude Code and Codex annual budget in four months. In response, the company imposed a $1,500/month/employee limit, with over-limit requests allowed and approved on a case-by-case basis. To see if the news reports on early 2026 tokenmaxxing and now tokenbudgeting were true, the SemiAnalysis team conducted on-the-ground conversations at the Databricks AI Summit and talked with large enterprises to understand the trends.

Our View of the Data & Narrative

There are many news stories out there on tokenmaxxing and resulting budget blowouts. However, in our work in the Tokenomics Model, we estimate that 90^th+ percentile customers make up most of the revenue and are at very little risk to API revenue cuts through the rest of the year. Even Meta, who was burning through 70T tokens per month in February and is spending close to at least $50,000/year per employee (at list price) is only a 3-5% customer for Anthropic per our estimates. Ramp data shows a similar trend amongst the top customers. 99^th percentile customers spend almost $90,000/yr per employee while 90^th percentile customers spend ~$7,300. This is a stark contrast to the median Ramp customer spending just $136. Note Ramp customers are generally way more tech forward so it's already a high spend skewed distribution. The media fortune 500 is well below $100 per employee still.

Our conversations with enterprise customers, including many Fortune 500s, followed this split. Many tech forward Fortune 500s spend well under $2,000/year per employee in AI, with the larger spend mostly in the engineering and data science departments. This suggests that the s-curve of growth in enterprise usage still has plenty of runway. Today’s market is driven by the coding vertical explosion and other VC-backed AI companies whose products build on top of Anthropic or OpenAI models (90^th-99^th percentile customers who are seeing their revenue accelerate too).

What the coding market has done to AI Lab ARR will be repeated with Cyber (Mythos re-release dependent) at an even faster pace than Claude Code and again with white-collar knowledge work as Cowork, CoPilot, Codex, and Computer type products penetrate the enterprise.

The Tokenomics Model estimates coding related spend at the AI Labs on a 1^st and 3^rd party basis and ARR and margins at the application layers (companies like Cursor, Loveable, GitHub CoPilot, and more) to help investors and corporates track growth of and within the vertical. We believe that over 70% of ARR today across OpenAI and Anthropic can be attributed to coding use cases with Anthropic spend higher than OpenAI’s given the difference in B2B vs B2C mix (Anthropic 90%+ B2B vs OpenAI 60%).

Still, cheap tokens are in widespread demand. We see massive growth in spend across both the for the Token-as-a-Service (TaaS)/API Endpoint Market for both frontier models and open-source models. Our estimates for AWS Bedrock this quarter drive our total AWS growth rate number well above street. The Tokenomics Model also forecasts massive demand for TaaS providers like Together, Fireworks, Baseten, and others who make up over $4B of ARR today.

Thus, to say, our work suggests that headlines are overblown, enterprises continue to spend, and new use case/verticals for demand and token consumption are keeping the AI train moving forward at an aggressive pace.

Budgeting: Pick a Number, Any Number

Most companies we talked with (n>50), also impose a hard cap on AI usage - though there doesn’t seem to be a token number that companies are converging towards. At the lower end, we spoke with the head of AI at a top 3 US aerospace and defense manufacturer and one of the world’s largest pharmaceutical companies which cap employees at $250 and $500 a month, respectively. At the higher end, our conversations with companies like Workday and Stripe revealed that their employees’ budgets are about $2000 a month.

A subset of companies has not yet imposed any limit at all, for what seems like two possible reasons:

The financial industry is a clear example of the former. Much of it has been slow to adopt AI, and those that have moved have done so minimally. In conversations with various data scientists, analysts, and AI leads across a range of asset managers, regional banks, and auto-finance firms, the same pattern kept surfacing: employees boxed into the tools provided through Microsoft’s platform.

The most mature companies are implementing a soft limit, which should be viewed by employees as guidelines rather than a hard rule. At a public cybersecurity company, the Director of Analytics, who oversees all developers and data scientists, said they set a “limit” of $800 a month for juniors and anywhere from $1,600-$4,000 a month for more senior staff. Data scientists are given the largest budget, as they tend to work with larger datasets requiring more tokens, a pattern that recurs across companies with flexible budgets. Should employees exceed their allowance, managers are alerted to have a conversation rather than cutting them off until the count resets.

The Art of Token Conservation

With rising token spend and increasing executive consciousness towards that spend, employees are starting to learn to adapt. When that same aerospace and defense manufacturer first introduced its $250/month limit, some of the power users burned through it in four days. Employees can now request a higher budget, but that was not always an option. Employees now must get creative to conserve tokens. The company has disclosed that although employees have access to Claude, it has “turned-off” Opus 4.8 and Fast-Mode deeming it unnecessary. Management believes that handing employees larger token budgets would push them to automate tasks that shouldn’t be automated at all, like writing emails. We believe this anti-automation view by management teams is naive. Email, much like Slack with connectors from the AI Labs, will become more automated and AI native over time enabling greater productivity, visibility, and collaboration throughout the organization.

One global travel-tech company took a slightly lighter-touch approach. With 800 engineers out of 1,500 total employees who collectively spend a little under $10M a year on AI, it recently switched the default Claude model for all staff from Opus to Sonnet. Opus remains available but using it now requires a conscious choice. Most employees have a $200/month budget by default, though it can be increased to tens of thousands of dollars depending on seniority and role, with expected budget increases coming soon.

As more companies impose limits, many employees are unbothered as they don’t come close to the cap. Our conversations with customers at the Databricks AI Summit and in research calls showed that at many organizations most employees do not come close to the limit.

This mirrors the power usage we see here at SemiAnalysis where a handful of employees spend 4 to 5 figures per day, and others spend close to 0. Those who approach or exceed it have found ways to stretch their tokens. We have not impossed limits as the top performers are the ones utilizing a lot of tokens.

Employees of companies on the Microsoft 365 Enterprise subscription receive free, unlimited access to the standard Copilot chatbot. Because that usage isn’t tracked against the monthly budget, employees can game the system by using Copilot’s 365 chat to draft and synthesize ideas first, before spending metered tokens on Claude or Codex.

AI As Headcount Lever

The cost of AI tools sits with the company, but so does the benefit. A big 3 US airline stands apart from the others we met with in how it budgets. Its token allocation is tied to the specific project and the revenue that project is expected to bring in. When a project comes in, the financing team decides what percentage of revenue is set aside for expenses like travel and contractor fees, and now that same budget must cover token usage as well.

This reframes AI spend entirely. Companies that hand out generous allowances do so on the expectation that employees will work faster. Output expectations rise to match spend, and many workers have found themselves putting in even longer hours than before. Across over 50 conversations with medium to large enterprises, one thing became clear: the headline Uber, Meta, and other Fortune 500 tokenmaxxing stories were a result of poor incentives and lax oversight rather than an absence of high ROI activities/projects to spend those tokens on. The clearest expression of the ROI dynamic is at Amazon. Despite its widely reported layoffs, the company is hiring at a much faster pace due to efficiencies unlocked by AI tools.

Budgeting Here to Stay

Caps, soft limits, and conservation tricks are all ways of managing the bill at the end of the month. In contrast to the high numbers of token spend in the news flow, Anthropic’s own documentation says the average Claude Code usage per developer is between $150-$250 per month, and only 10% of users spend over $30 per day. One thing is clear after our on-the-ground work talking with customers: there is not a material risk present to 2H26 AI budgets and we expect Anthropic and OpenAI’s API business to continue to grow at their current net new rates m/m for the foreseeable future.

Summary of Selected Conversations

Rise and Fall of Tokenmaxxing

The dashboard was shut down 2 days later after The Information reported the spend.

Our View of the Data & Narrative

Budgeting: Pick a Number, Any Number

A subset of companies has not yet imposed any limit at all, for what seems like two possible reasons:

The Art of Token Conservation

AI As Headcount Lever

Budgeting Here to Stay

Summary of Selected Conversations

Today, the US grid is serving most datacenter load in the US, but we’re reaching a tipping point. As the insatiable demand for power of AI Labs and hyperscalers keeps accelerating, the grid simply can’t add capacity fast enough. That leaves Behind-The-Meter as the only way for the largest players to secure the power they need. Nearly a year ago, our Onsite Gas deep dive was the first to predict the fast rise of new entrants in the BTM gas equipment market. Since then, companies like Bloom Energy, Bergen Engines, Wärtsilä and many others have been remarkably successful. Overcoming GEV and Siemens turbine capacity constraints proved far easier than many had feared.

Today, we go deeper and model US Grid capacity to understand the shortfall that must be filled by Behind-The-Meter solutions for datacenters.

Our research suggests that BTM will power well over half of new US datacenters in 2028+, and the Total Addressable Market (TAM) for DC BTM equipment to cross 50GW/year by 2029. New Grid Capacity isn’t growing fast enough, and also needs to serve non-datacenter load growth.

The chart above shows the three core building blocks of our forecast: Expected Datacenter US Gross Power Demand, available US Grid Capacity, and New Grid Supply. We use the best of SemiAnalysis industry-leading insights to build this forecast.

The second building block of our Energy Model, grid headroom, analyzes supply & demand dynamics in each major part of the US grid. Our model follows the methodologies of all ISOs & RTOs and models UCAP/ICAP reserves, supply & demand growth, reliability risks, etc.

The third block forecasts new grid supply, through a bottom-up forecast produced by our new Energy Model. We track 40,000 generation assets in the US and forecast quarter by quarter Commercial Operation Date (COD) for all fuel types. We then estimate the “true” capacity value of power plants via our proprietary ELCC model, adapting to the specifics of each ISO and major non-ISO region.

Our forecast points to barely 15GW of net-new ELCC capacity being added annually, with a rising trend towards 20GW+ by the end of the decade. This is effectively all the firm capacity being added to the system that a grid operator can recognize to serve firm datacenter load - as well as other firm load like industrial plants, semiconductor fabs, etc.

Netting that accredited supply against peak demand and required reserve margins is what yields headroom itself, the spare accredited capacity a market has left to host new load after covering its own peak demand and required reserve margin. On this basis, available headroom is already approaching zero and turns negative by 2027, based on our analysis of required reserve margins across the country.

As such, we expect power generation to be a major bottleneck to grid-connected datacenter load growth (transmission is another one and will be the topic of a follow-up deep dive). All datacenter operators have already seen this firsthand: a utility tells you it can meet a 2027 load-ramp schedule, e.g., 500MW, and later comes back saying they can only deliver that load only in 2029 because the long-lead equipment grid interconnection requires, from main power transformers (MPTs) to high-voltage breakers and, in some cases, network upgrades, simply is not available any sooner. Worse, the burden increasingly falls on the buyer: securing grid-connected power now often requires developers to post substantial letters of credit, security deposits, or sign take-or-pay commitments to fund the generation built to serve their load. Switch Datacenter, for instance, closed a multi-billion-dollar performance letter-of-credit facility in 2026 to back exactly these obligations. And in many cases, despite billion-dollar commitments, the utilities don’t even face any penalties for not delivering on time.

These generation and transmission constraints, combined with inadequate market incentives, makes Behind-The-Meter often the most attractive solution for GW-scale newbuilds. And we’re already observing that many top-tier developers are planning 5GW+ behind-the-meter facilities in Texas, where permitting onsite gas is easier. Our Datacenter Model covers this in extreme detail, separating the few «real» 5GW+ campuses from those that show no sign of credible development under the hood.

Our new Energy Model is the best tool to forecast and understand relative market positioning by company: we track manufacturing capacity, equipment lead times, installation lead times, major deal announcements, and much more. We’ve also covered recent secondary-market developments in depth, with notably surging turbine availability.

Let’s dig in. Our argument runs in three steps, and this report walks through them in order: grid supply is structurally constrained, that constraint pushes the marginal buyer behind the meter, and that shift reshuffles the winners and losers across equipment OEMs and IPPs as the market grows across fully islanded and hybrid BTM setups.

First, we establish the supply constraint: drawing on our new Energy Model, we explain how we forecast power-plant additions, how we estimate the true capacity value of each fuel via ELCC, and how we measure the grid headroom that results.

Next, we turn to the buyer’s decision: the relative attractiveness of BTM versus grid for the largest buyers of power and compute, broken down by use case.

Finally, we examine the practical bridge between them: the hybrid co-location structures emerging in ERCOT — codified by the Batch Zero process — that blend on-site generation with continued grid access, and where we think the early winners are taking shape. We explain more on this later.

Note that this article focuses on grid constraints and behind-the-meter as a solution, but there are alternatives, like Load Flexibility. These will be the focus of future deep dives, and out of scope for this one. We briefly explain later in this article why we think BTM will be the prominent solution over alternatives like Load Flexibility.

Forecasting Power Plants: No Gas Until 2028

Our analysis of 40k generators reveals one of the core challenges for the US grid: the lack of firm, dispatchable capacity added to the market over the next two years. Our nameplate capacity forecast shows that the US industry will add less than 10GW of gas per year in 2026 and 2027, with additions only picking up in 2028 and beyond.

To achieve this forecast, we replicate our Datacenter Model methodology: it starts with an extensive analysis of construction timelines, with hundreds of thousands of empirical datapoints analyzed, backed by real-time satellite imagery. We simply don’t see enough power plants in advanced construction stage that would enable a 2026-27 delivery.

The 2026–27 shortfall isn’t the product of one bottleneck but a stack of them. First, institutional and queue friction: the intrinsic slowness of key grid stakeholders like utilities, and overcrowded generation interconnection queues. In PJM, the queue itself is no longer the binding constraint — conversion is: roughly 57GW has cleared studies and been offered or executed interconnection agreements, yet since 2020 about 24GW of projects with fully executed agreements (including 13.5GW of gas) terminated before reaching commercial operation, largely due to permitting denials, supply-chain delays, and financing that could not close. Permitting alone accounted for 29% of project milestone changes between January 2023 and January 2026, versus 23% for supply-chain delays (Powering Reliability through Market Design).

Second, the technology mix works against speed: the bulk of the gas capacity ordered by utilities is made up of CCGTs and combustion turbines. As we’ve explained in depth in our Onsite Gas deep dive, CCGTs are the most efficient way to burn gas for power, but their build is the slowest of any generation technology — construction and time from planning to COD run 4–6 years in some ISOs. Faster technologies like Fuel Cells and RICE are being aggressively secured directly by datacenter operators for onsite generation.

Third, the supply chain stretches every timeline further. Gas turbine and generator step-up transformer lead times have each extended to three to four years, versus a historical norm of roughly 18 months. That pushes total gas-plant development from a ~24-month baseline to at least four years even under optimistic assumptions, with CCGTs at the long end of the 4–6-year range.

Finally, a project may also slip for reasons beyond the plant itself such as increased equipment costs, community pushback, availability of labor, financing concerns, etc.

Renewables, Storage and ELCC

Of course, gas isn’t the only fuel that matters for the US grid. Renewables and storage are taking on an increasing share, as our Energy Model tracks in great detail. We see solar and BESS each adding over 20GW nameplate per year in coming years, as shown in the chart above. However, from a grid perspective, their value is significantly lower. As shown below, on an ELCC basis (Effective Load Carrying Capability), their contribution is minimal. We’ll explain this concept in more depth just below.

For renewables, the cause is simple. Not only are these energy sources intermittent, they’re also highly correlated. Solar’s famous Duck Curve is the best example. As you add more solar to the system, the marginal value of adding new solar declines very sharply because all plants generate electricity at roughly the same hours. In ERCOT, the rapid build-out of solar and wind has compounded this by pulling the two price signals in opposite directions. The 4CP window — the four 15-minute intervals across June-September that set transmission charges for the following year — used to sit in the late-afternoon load peak. But as renewables flatten daytime net load, it has drifted later into the evening, closer to sunset. The curtailment peak has moved into the later hours of the day, when solar output falls off and the steep net-load ramp leaves the grid short of supply. The result is a wider, steeper evening ramp that batteries have to carry until gas can pick up.

Quantifying this declining, weather-dependent value requires a dedicated metric. Utilities and grid planners rely on the concept of ELCC to do exactly that. ELCC estimates the ’true’ capacity value of a power plant to the overall system. It is also a useful gauge of available capacity headroom in a market: where excess ELCC capacity exists, the system has room to absorb a new large load, so a datacenter can interconnect — whereas a market without it does not. These entities think of their resources from a system perspective, and because each system is unique, each ELCC methodology is unique as well. Our Energy Model shows below how diverse solar ELCC is across regions.

Let’s pick a few examples to better understand how ELCC works and how it differs across regions.

PJM uses a complex formula called Marginal ELCC to account for the value of different energy resources. FERC accepted PJM’s shift to marginal ELCC accreditation in early 2024, replacing the prior Equivalent Forced Outage Rate (EFORd) method for thermal units and average ELCC for intermittent resources, alongside an hourly risk model that better captures correlated outages during extreme weather. For more details on PJM and some of their specificities, read our deep dive on PJM vs ERCOT market designs and their impact on household power prices.

Methodologies diverge sharply across markets. MISO accredits resources seasonally under its newer seasonal resource-adequacy construct, reflecting how scarcity risk concentrates differently in summer versus winter; ERCOT, an energy-only market with no centralized capacity market, relies on a reliability standard and its own effective-capacity measures rather than a centralized ELCC accreditation. The practical result is that the same solar or storage asset can be credited very differently depending on the ISO. ERCOT illustrates the extreme: in some congested zones and segregated transmission corridors, planners now leave solar’s capacity contribution out of local reliability modeling altogether — a de facto ‘no-solar scenario’ that credits incremental solar at essentially zero when sizing firm capacity needs.

Everything so far has concerned solar and wind. Storage is the other half of the renewables build-out — so what about BESS? Our forecast points to a rising penetration of 2hr and 4hr BESS, which naturally have a higher ELCC value than 1hr. Longer duration BESS has more room to provide relief to the system in times of constraints.

However, Storage also fundamentally suffers from declining marginal ELCC, and increasing duration doesn’t change that fact. The reason is that a given duration of BESS solves a specific pain point of the grid: for example, 4hr BESS increases the reliability of the grid for events with a duration of 4hrs or below. But as more 4hr BESS is added to the system, the <4hr grid risk becomes nullified, which reduces the value of incremental BESS. The grid reliability risks shift to >4hr events, which must be addressed by other fuels (or by 8hr+ BESS, which will initially have high ELCC, but will again see its ELCC decline as 8hr events get derisked). As a result, on most grids today incremental 4hr BESS adds little marginal ELCC.

Our point is not that renewables don’t matter, they are, and will remain, an important part of the AI buildout, just not its most important one. It is that nameplate badly overstates what they add to firm capacity: on an ELCC basis, each incremental GW of solar, wind, and storage is accredited at a steep and widening discount as the risk it addresses gets saturated. That gap between nameplate and accredited capacity is exactly what determines how much new load a market can actually host — which is the subject of the next section: grid headroom.

US Grid Headroom goes red by 2027

Grid headroom is the capacity a market has left to absorb new load once it has covered its own peak demand and required reserve margin. We model it subregion by subregion — each ISO and large non-ISO market — from local accredited supply and reserve requirements. Headroom goes “red” when a market’s reserve margin falls below its required target: at that point there is no spare accredited capacity to host an incremental large load like a datacenter without eroding reliability. Across a growing set of subregions, our analysis shows that threshold being crossed by 2027.

Fundamentally, the headroom calculation is simple: total accredited supply, minus peak demand, minus required reserves. As a live example, PJM’s 2027/2028 Base Residual Auction cleared roughly 134,478MW of unforced capacity (UCAP), yielding a 14.4% reserve margin against a 20% target — a physical deficit of about 6,517MW UCAP. The strain is not PJM-specific: NERC’s 2025 Long-Term Reliability Assessment flags 13 of 23 North American assessment areas as facing resource-adequacy shortfalls over the next decade. Required reserves are a regulatory artifact, set by either grid planners or vertically integrated utilities, depending on the territory, and can change year by year but tend to be fairly stable.

Note that estimating true headroom is more complicated than that. Utilities themselves often struggle to understand their headroom, and the smartest datacenter developers can successfully navigate the environment by finding “hidden” headroom. We‘ll dig much deeper into this in the future deep dive.

Why BTM wins over grid-connected: speed and certainty of timelines are key

With grid supply structurally constrained — too little firm capacity, too little effective capacity from renewables and storage, and vanishing headroom — the decision now sits with the buyer. And we think BTM is now the most attractive option. This statement might shock some stakeholders, but we base our view on the fundamental drivers of demand: AI labs. Companies like OpenAI, Anthropic now make up the bulk of demand, directly but also often indirectly as they represent a significant portion of the datacenter capacity added by companies like Amazon, Microsoft, and Oracle. Our Tokenomics and Datacenter models both quantify this quarter-by-quarter and have successfully called out major deals and shifts ahead of the headlines, such as the Anthropic TPU rise and the Oracle/OpenAI RPO surge.

The key advantage of BTM vs Grid is speed and certainty on the timeline of power. On speed, onsite generation can be energized in a fraction of the grid-interconnection timeline — requested BTM in-service dates cluster around 2027–28, against grid timelines that routinely slip toward 2030. On certainty, the schedule sits in the buyer’s hands rather than the utilities. As explained earlier, timelines provided by utilities are notoriously unreliable and they often push back or revise down the load that they promised datacenter operators… with little to no penalties at all.

This doesn’t work for AI Labs, for which access to large scale compute is the lifeblood of their business. They need power both to generate revenue (inference) and to fuel future revenue growth (training). In addition, as we cover in extreme depth in our AI Cloud TCO Model, power as a percent of total TCO is mostly insignificant, meaning that any amount of power secured by an AI Lab is actually worth billions. For example, our Tokenomics Model susbcribers know very well the margins Anthropic is currently making on API, and the implied tens of billions of dollars of annual revenue per GW. Given the cost of GW-scale DCs and revenue potential, it simply doesn’t work if it risks multi-year delay, or if the load-serving entity faces no symmetric dollar penalties for being delayed.

In the grid vs BTM debate, another key factor is redundancy and uptime, historically a major advantage provided by the grid. As explained in our Datacenter Electrical Systems deep dive, the playbook to reach five nines of uptime was simple, and came in two steps: first, connect to a substation with N+1 redundancy, which on its own delivers roughly three to four nines depending on assumptions; then add backup power covering more than 100% of nameplate load demand — backup generators and batteries — to close the gap to five nines.

However, AI labs and some hyperscalers have relaxed those requirements as there is now a lower uptime tolerance applied to both inference and training, not just training. Many of Meta’s self-built AI datacenters, for example, target just two nines of uptime and forgo backup generators entirely, as detailed in our Industrials Model.

This removes historical cost barriers to BTM adoption. As we alluded to in our onsite gas deep dive, the main challenge and driver of cost overrun is redundancy and reliability. The grid manages this at the macro level, while an onsite power plant must do it for the much narrower purpose of serving one (large) datacenter alone. Providing four or five nines of redundancy at a BTM site is a recipe for unbearable costs. But now that customers are willing to accept lower redundancy, the economics of grid vs BTM are much more balanced.

Nevertheless, practice varies — some operators add gensets to their BTM-to-FTM bridge builds, while many omit them altogether on a grid connected datacenter.

In our view, it can make sense to add gensets if you plan to connect to the grid in the near future and can redeploy the generation equipment elsewhere, or if you’re exploring a grid-connected BTM solution (more on this later). Most BTM sites are not planned to be islanded forever; they are bridges while the interconnection queue clears. However, if the queue is expected to clear — say 5 years from today — it’s hard to justify ~$1mm/MW in capex for a backup genset fleet, especially since the realized uptime gain is on the order of a single nine once human-factor and recovery-time failures (not power-source outages) are what actually dominate downtime.

Ultimately, the call comes down to the tenant’s design requests, SLAs, and how long the site expects to run off grid.

We’ve seen the following: Where BTM is a short bridge — on the order of two years or less to interconnection — operators tend to spec the gensets at initial build and project financing package, since grid energization and the backup role that follows are close at hand. Where a site expects to stay off-grid for far longer, or effectively permanently, that rationale weakens: primary on-site generation already carries the load with its own redundancy, so a grid-backup stack that sits idle for years is capex worth deferring. Meta is the clearest case — its Ohio campus is designed never to connect to the PJM grid, and its El Paso and Louisiana sites are built around dedicated, utility-scale gas rather than a near-term interconnection. We understand Cipher’s AWS-leased Black Pearl site and several of Fluidstack’s build-outs follow similar logic, deferring or skipping the genset-as-grid-backup route altogether.

Now, none of this means the grid is being written off and that the uptime and reliability requirements of a datacenter are no longer of importance. Rather, it reflects our view of how market participants are adapting to meet the rampant power needs of AI DCs – and it’s our view that tenants are becoming more lenient with these requirements as they have nowhere to go.

Also, there are many promising solutions to enable the grid to absorb more datacenter load growth, and we believe that would be the best outcome for all: lower power costs and higher reliability for everyone. Here are a few solutions outside of today’s article’s scope:

All these topics will be covered by future SemiAnalysis energy deep dives. Stay tuned.

The Hybrid Structures Bridging BTM and Grid Interconnection in ERCOT and a likely Winner

To deal with the limited grid headroom, the market is settling onto a spectrum of hybrid structures — particularly in ERCOT — that blend on-site generation with continued grid access, and this is where we’ve been observing a large share of deal activity and interest. Regulators are racing to write the rules, with FERC at the federal level and ERCOT and the PUCT in Texas updating their frameworks quickly.

In ERCOT, the Batch Zero process is codifying these structures. At its core is a single number for every large load — a maximum withdrawal limit, the amount the site may draw from the grid independent of any on-site generation. The governing framework (NPRR1325 and PGRR145, approved by the ERCOT board on June 1, 2026 and effective July 11) added two new co-location constructs alongside the long-established Private Use Network (PUN). The cleanest way to make sense of the landscape is to separate the two questions these structures answer: how a site interconnects and meters with the grid (as a PUN, WLPUN, or PCLR), and how it may sources its power (from existing generation under an NMA, newly built generation under BYOG). We take each in turn.

Powering your Datacenter

NMA (Net-Metering Arrangement) — sourcing from existing generation. An already-operating generator co-locates with a new large load and nets that load against its on-site output behind a single meter, so only the residual — what the load draws above what the co-located unit supplies — settles against the grid, with any surplus exported. The defining feature is vintage: the generator was already operating before September 1, 2025, the statutory trigger under Texas’s SB6. That brings the arrangement under regulatory review through ERCOT’s net-metering arrangement process (PUCT Project 39169) and a 120-day transmission security assessment. ERCOT studies the system impact — including the effect of pulling that capacity off the grid to serve the load — and the PUCT can approve, deny, or condition the arrangement, for instance by requiring the load to curtail during grid emergencies or the generator to make capacity available when the system is tight. Because the generation already exists and was already supporting the grid, the central regulatory question is whether re-pointing it to a private load leaves the rest of the system whole.

The bulk of announced ERCOT co-location activity to date sits in this existing-generation bucket. Below are a few datacenter examples — roughly 2,885 MW of reported projects in the co-location segment, with likely more beyond these.

BYOG (Bring Your Own Generation) — sourcing from new generation. Rather than wait on grid upgrades to serve its full load request, a large load builds, co-develops, or contracts new co-located generation to secure a near-term path to energization. ERCOT evaluates the project on three parallel tracks: a Batch Study sets the grid withdrawal limit (what the site may draw independent of on-site generation), Generation Interconnection sets the export limit (what the on-site units may push back to the grid), and Transmission Planning identifies any network upgrades required. The site’s achievable load is therefore its withdrawal limit plus available on-site generation, subject to system-stability limits, and energization is staged accordingly: the site draws up to its withdrawal limit (say, 100 MW) from day one and ramps toward full capacity as each generation unit comes online. The distinction from an NMA is one of category and vintage — BYOG describes how a site sources its power, by adding its own newly built generation, and because that generation is new it sits outside SB6’s net-metering review.

Metering & connecting your datacenter

A Private Use Network (PUN) places an entire campus — load plus its co-located generation — behind a single point of interconnection, so ERCOT meters only the site’s net exposure with the grid rather than each individual resource. Building on that base, Batch Zero introduced two new constructs:

Winners and Losers of the rise of Behind-The-Meter

Let’s now turn our attention to winners and losers. We’ll discuss turbine manufacturers like GEV, fuel cell and RICE vendors like Bloom and INNIO, IPPs like NRG. We first called out Bloom in December 2024 for Core Research subscribers as the biggest beneficiary.

Today, we go deeper and model US Grid capacity to understand the shortfall that must be filled by Behind-The-Meter solutions for datacenters.

Next, we turn to the buyer’s decision: the relative attractiveness of BTM versus grid for the largest buyers of power and compute, broken down by use case.

Forecasting Power Plants: No Gas Until 2028

Finally, a project may also slip for reasons beyond the plant itself such as increased equipment costs, community pushback, availability of labor, financing concerns, etc.

Renewables, Storage and ELCC

Let’s pick a few examples to better understand how ELCC works and how it differs across regions.

US Grid Headroom goes red by 2027

Why BTM wins over grid-connected: speed and certainty of timelines are key

Nevertheless, practice varies — some operators add gensets to their BTM-to-FTM bridge builds, while many omit them altogether on a grid connected datacenter.

Ultimately, the call comes down to the tenant’s design requests, SLAs, and how long the site expects to run off grid.

All these topics will be covered by future SemiAnalysis energy deep dives. Stay tuned.

The Hybrid Structures Bridging BTM and Grid Interconnection in ERCOT and a likely Winner

Powering your Datacenter

Metering & connecting your datacenter

Winners and Losers of the rise of Behind-The-Meter

We were the first to describe the memory shortage coming from AI’s insatiable usage in reasoning and agentic flows in late 2024 on the newsletter. We have since previously published multiple in-depth pieces on memory, as well as detailed coverage of CXMT and China’s compute. With CXMT set to IPO in the coming months, we believe a dedicated deep dive on them specifically is warranted. The company is likely to become the largest semiconductor IPO in China and mark a major milestone for the country’s leading memory manufacturer, which is also destined to compete only more fiercely with the leading memory suppliers of Samsung, SK Hynix, and Micron from here.

CXMT, established in 2016, is going to be listed on China’s STAR Market. Now the leading DRAM player in China, its history shows an interesting path of technology transfer, talent flows, and the patience of state-venture capital, together of which are turning the company toward indigenous innovation.

The Silicon Valley Returnee

Zhu Yiming, the founder of CXMT, earned his undergraduate degree in physics at Tsinghua University in 1994 and came to SUNY Stony Brook for his graduate study in electrical engineering. He then worked in Silicon Valley and became a project lead at MoSys (Monolithic System Technology) around 2001. In 2005, he went back to China with a set of SRAM patent and US$100,000 in seed money, founding GigaDevice, which later became well known for its SPI NOR flash and microcontrollers, a fabless design house that grew into one of the world’s top NOR flash suppliers. But the global NOR flash market is much smaller compared to DRAM or NAND flash. Zhu Yiming always dreams big, and not surprisingly he chose to march into the DRAM business.

DRAM, however, is not a design-house game in which one can stay fabless. DRAM is capital-devouring, IP-fortified, and manufacturing-bound, and by 2016 the whole industry was dominated by three survivors, Samsung, SK Hynix, and Micron, fortified by four decades of patents and capital that no newcomer had breached. Zhu’s SRAM patents and GigaDevice’s NOR franchise gave him neither a DRAM cell, nor a DRAM process, nor any cover from the incumbents’ patents. Thus, when Zhu and the Hefei municipal government launched the DRAM venture in 2016, the “506” project that became CXMT, the core technology had to come from somewhere else entirely.

The DRAM Foundation: Inheriting Qimonda

The dead company was Qimonda. Though the company went bankrupt in January 2009 due to the 2008 Global Financial Crisis and the following dramatic memory downcycle, it was then the leading DRAM player in Europe. A subsidiary of Infineon, which originated from Siemens, it offered a rare alternative: a deep DRAM patent base and a cell architecture, both originating outside the dominant Samsung-SK Hynix-Micron triangle. In June 2015, Polaris Innovations, a subsidiary of WiLAN, a Canadian patent-monetization firm, bought roughly 7,000 Qimonda patents and applications from Infineon for about 30 million euros. In December 2019, Polaris and CXMT signed an agreement: a license to a large set of DRAM patents. CXMT leaders once publicly claimed that they obtained roughly 2.8 TB of Qimonda technical documentation, which became the basis for CXMT’s DRAM business.

A major technology CXMT inherited from Qimonda and then developed its 46nm-class BWL (buried wordline) cell, which CXMT scaled toward the 10nm class. BWL is the load-bearing idea. Instead of routing the access transistor’s gate across the wafer surface, BWL sinks it into a trench beneath the bitline. It pulls the gate off the surface so the cell collapses to a 6F² layout (versus 8F²). It also lengthens the channel without spending surface area, suppressing the short-channel leakage that wrecks retention. And it cuts gate-to-bitline parasitic capacitance. Buried wordline plus stacked capacitor is the architecture all three leading players run today. The trench holdout died holding the stacked/BWL escape hatch, and that is exactly what CXMT picked up.

Besides patents, the more durable thing CXMT pulled out of Qimonda’s collapse was its engineers. Qimonda’s Xi’an R&D center, with 400-500 engineers, was one of the largest Qimonda built outside Germany. After Qimonda’s collapse, though the whole Xi’an R&D center was acquired by Tsinghua Unigroup, the wider diffusion of individual talents benefited CXMT. In addition, CXMT successfully attracted Karl-Heinz Kuesters, a senior engineer, from Qimonda’s German site to Hefei, China. Kuesters had spent 24 years as a VP of technology and pre-development across Siemens, Infineon, and Qimonda. The pre-development line Kuesters ran was the stacked-capacitor work, the architecture CXMT actually builds on. He joined CXMT as a technical consultant, for which the EE Times called Kuesters the company’s “ace in the hole.” What Kuesters brought was the part of Qimonda’s legacy that neither the patents nor the 2.8 TB of documents carry: the tacit know-how. Having led DRAM development for two decades, Kuesters could tell CXMT’s engineers which of Qimonda’s design choices to keep and which to discard, and how to move a cell that worked only in the lab into volume production, the integration and yield judgments that no patent records.

The American side shows the same pattern. Ping Er-xuan, CXMT’s VP for future-technology assessment and the public face of its roadmap (the “46nm to the 10nm class” framing), came not from Qimonda but from a US career at Micron, SanDisk, and Applied Materials, where he ran memory and materials technologies. Ping brought process-and-materials depth and an emerging-memory view: individual mobility.

CXMT has also recruited heavily from Korea and Taiwan. Korean prosecutors have charged former Samsung employees with leaking technology to CXMT, and dozens of Korean engineers are reported to have worked there. Similarly in Taiwan, there is constant talent poach with attractive compensation package for top-tier engineer across equipment and process development.

This is the part that bears on where CXMT is heading. The Qimonda patents were always a finite, expiring asset. What lets CXMT keep moving, from G4 to G5 and now into HBM, is the assembled capability of domestic talent, Chinese nationals who worked at foreign companies and returned, as well as some experts from foreign firms, not the documents. The inheritance only got it started. The talent turned a foreign legacy into in-house R&D juggernaut. However, it took nearly a decade for CXMT to generate profits. The question is who has patiently funded CXMT’s development and borne its near-decade losses?

The Patience of State-Venture Capital

It is difficult to not attribute at least part of CXMT success to the strong support of local and central government in China. Hefei municipal government is one of the great examples. Hefei is one of the tech innovation hubs in China, well known for its patient state-venture capital nurturing successful companies over the past two decades, from BOE (the world’s leading display panel maker) to NIO (a leading EV maker) and now to CXMT. In particular, the Hefei municipal government did two things for CXMT. First, the Hefei government helped CXMT build a local supply chain around its fab. Hefei’s playbook is to take a large equity stake in an anchor “link-leader” and then draw the rest of the chain in around it. The city did that for BOE in displays and NIO in EVs, and from 2016 it replicated the same playbook for CXMT. Surrounding CXMT’s plant in Hefei’s airport-zone industrial park, the government produced a dense local cluster. Two packaging-and-test houses, Peyton and Xinfeng, are located within a street or a wall of CXMT’s fab, with Xinfeng booking over 99% of its revenue from CXMT. An on-site bulk-gas plant run by Guanggang supplies most of CXMT’s needs, while wafer-reclaim capacity from Zhiwei Semiconductors a subsidiary of Zhichun Technology (至纯科技), is located in Hefei’s Xinzhan Hi-Tech District. The state-venture-capital vehicles have also taken outright control of an upstream chip-molding equipment maker, Wenyi Technology. Such a local supply-chain cluster gives CXMT a localized industrial base.

In addition, Hefei’s state-venture capital could afford to lose money for a very long time. Unlike a private venture-capital fund answerable to LPs that expect a return on a fixed timetable, Hefei’s state-venture capital, ultimately backed by the city’s municipal and development-zone state entities, faced no such clock. They kept funding a company that, even after turning its first annual profit in 2025, still carried an accumulated deficit of roughly RMB 36.65 billion built up over nearly a decade. The original “506” project, launched in 2016, began with Hefei’s state-venture capital funding about 80% of the project’s first phase (RMB 14.4bn of 18bn). Across successive rounds Hefei’s vehicles were diluted, but they never sold down and never walked away. By the IPO, the largest holder, Hefei’s Qinghui Jidian, at 21.67% and state-venture capital vehicles together holding over 30%. That willingness to treat a fab as a decade-long bet rather than a fund-cycle return is the catalyst that the technology and the talent both depended on.

From Inheritance Toward Independence

Put the three threads together and CXMT’s first decade resolves into a single arc. Qimonda supplied the foundation, with a licensed patent base and a cell architecture from outside the incumbent triangle. Talent supplied the motion, with key figures like Kuesters and Ping as well as the returnees from the American majors, and the contested hires out of Korea. Those people turned a frozen blueprint into a process that could keep scaling. Then the Hefei government supplied what the other two needed but could not generate on their own: capital, patience, and a localized supply chain. None of the three would have produced a DRAM maker alone; together they did.

In the following section, we will be discussing CXMT’s financials, technology, equipment ecosystem.

Next Step after 10 Years: An IPO in the Supercycle

The past decade of CXMT’s history, while impressive, may prove to be only an early chapter in the company’s longer-term story. The company is now preparing for what will become one of China’s largest semiconductor IPOs of the past decades, and potentially the most closely watched semiconductor listing globally this year. In December 2025, CXMT officially entered the IPO filing stage when the Shanghai Stock Exchange accepted its STAR Market listing application, following a prolonged period of market reports throughout 2024 and 2025 suggesting that the company was preparing to go public. More recently, CXMT’s application advanced further through the review process by submitting the formal CSRC (China Securities Regulatory Commission) registration on May 27 and currently under final review, moving closer to a landmark listing.

While understanding strategically important, unlisted Chinese companies such as CXMT and YMTC has never been easy, CXMT’s move toward a public listing has been helpful for us to understand more details about the company given its IPO prospectuses that have shed meaningful light on the company’s historical performance, future trajectory, financial profile, market positioning, and technology roadmap. By combining these disclosures with our Memory Model, we can develop a more accurate view of CXMT’s current position and a more robust forecast of its future performance.

At a high level, CXMT is clearly the fourth biggest DRAM maker globally by almost every metrics while expanding its lead over legacy memory suppliers. For the full year, CXMT revenue increased 156% YoY to ~$8.6 billion, up from ~$3.3 billion in 2024 and ~$1.2 billion in 2023. Net income also turned positive for the first time, reaching $1 billion, underscoring the company’s rapid scale-up and improving profitability. Even with such impressive results, CXMT’s CY25 revenue is still materially behind Samsung (~$72.3B), SK Hynix (~$52.1B), and Micron’s (~$37.2B) DRAM revenue.

In 1Q26, CXMT reported revenue of $7.3 billion, representing roughly 700% YoY growth and already approaching the company’s full-year revenue in 2025. Operating margins also expanded sharply, reaching approximately 70%.

But we think this is just the beginning. We estimate the company to do even better if not explosive in the next two years at least. Just based on its filing, the company’s 1H26 revenue is expected to be 7x y/y and reached to more than 16 billion. For full-year 2026, we believe CXMT’s revenue could exceed $50 billion. If achieved, this would suggest the company has more than doubled revenue every year since 2023, and more than 6x its revenue in 2026 year over year basis, with its earnings trajectory accelerating meaningfully as both scale and profitability improve.

In our view, such significant upside to CXMT’s earnings is clearly driven more by the cycle itself than company’s technology or market positioning. When we take a closer look into the correlation of CXMT’s ASP trajectory and bit shipment. Company’s bit shipments increased by only 11% in 1Q26, while ASPs rose by roughly 57%, following QoQ ASP increases of 63% and 68% in 3Q25 and 4Q25, respectively. In other words, what really drove up company’s earnings is really the explosive ASP growth rather than significant market share gains over its peers in global DRAM end-market. By bit shipment, we model CXMT’s market share will increase from 9% in 2025 to 12% in 2027. While a 3% market share gain sounds minimal, it is significant for a market we size at close at $1T in 2027.

Surprising enough, the strong pricing uplift coming to CXMT is not an exception. We have been observing similar dynamics across both leading-edge and legacy memory suppliers in the DDR5, DDR4, and even DDR3 markets over the past year or so. As noted in our February memory piece, where we described the memory market as entering a “once-in-four-decades shortage,” we believe DRAM pricing remains on track to double again this year driven by sustained supply-demand imbalance across these product categories. Since February, we have become even more confident in this view and believe it could potentially exceed our expectations by year-end.

What may be more interesting to readers who have not yet followed CXMT or the memory market closely is how the company’s pricing compares with industry leaders. Based on our Memory Model, CXMT’s DRAM ASP challenges the common misconception that Chinese memory is structurally cheaper and will flood the market, thereby pressuring global pricing. While this may have been true in some cases in the past, we believe it is somehwat inaccurate in this cycle, and the latest company data points support the same conclusion.

Taking 1Q26 as an example, CXMT’s DRAM ASP was only slightly below Samsung, SK Hynix, and Micron — by roughly 5–10% — in the same quarter. And as we model it out throughout 2026, we think this will still to be the case directionally although the gap will gradually expand. We believe this widening gap over the coming quarters will be driven less by inherent pricing differences but more by the change of product mix. Leading suppliers continue to benefit from a higher server DRAM mix in their bit shipments, as well as a more favorable pricing outlook for server DRAM compared with consumer DRAM.

As such, we expect this mix to increase further over the coming quarters as server becomes a larger share of DRAM end-market demand. By the end of 2027, we expect server DRAM and HBM account for well more than 50% of the total DRAM end-market. Given that server DRAM and HBM carry higher $/GB than other memory end markets, this should allow leading memory suppliers widen their ASP gap versus CXMT as server DRAM mix increases, especially considering the expected material price increase for HBM in 2027 (we have 2027 HBM pricing in our Accelerator & HBM Model and Memory Model). We also have details on LTAs for major memory buyers such as hyperscalers and Nvidia.

Strong ASP tailwind has materially improved the company’s margin profile. CXMT’s FY25 gross margin reached 37.8%, moving closer to Samsung at 39.4% and Micron at 39.8%. However, it remains far below SK Hynix at 60.4%, as SK Hynix benefits from a much higher HBM mix, which carries higher ASPs and margin last year. CXMT’s ~38% margin is a significant swing from -113% in FY23 and -4.7% in FY24. Last year is not only the year CXMT reaches a record-high gross margin, but also the first year that the company achieves a positive margin profile.

With DRAM ASP continues to increase in 2026, CXMT’s margin profile has improved further. Its operating margin reached 70% in 1Q26, compared with SK Hynix at 73%, Samsung at 81%, and Micron at 84% in the same period. Besides the strong ASP growth, firm’s margin improvement also needs to thanks to company’s near-total exposure to commodity DRAM, which effectively has higher margin now vs. HBM. Based on its filing, nearly all of company’s bit sales are conventional LPDDR and DDR products. HBM is still a very minimal contributor to the company’s revenue and earnings.

This becomes clearer when we run a simple cost-per-bit analysis on DDR5 products across four memory suppliers. For DDR5, we find that CXMT’s cost per bit remains meaningfully higher than that of the three leading suppliers, by more than 30%. However, because DDR5 pricing was already exceptionally strong in 1Q26, we believe this still lifted CXMT’s gross margin to over 70%.This suggests that the improvement in CXMT’s margin profile is primarily driven by pricing, rather than by a material improvement in product competitiveness or cost structure.

In addition to printing record-level earnings, we believe company is gaining ground from the capacity perspective. By the end of 2026, we expect CXMT to reach roughly 350 kwspm, which is only modestly below Micron’s estimated ~385 kwspm. This would position CXMT close to becoming the industry’s third-largest memory supplier, if ranked only by wafer capacity.

However, CXMT remains materially behind the two leading DRAM suppliers, Samsung and SK Hynix, which we estimate at approximately 720 kwspm and 595 kwspm, respectively. Next year, with the initial ramp of Shanghai Phase 1 and the full ramp of Hefei and Beijing, CXMT capacity could reach the 420kwspm range if capacity in year-end, representing ~17% of global DRAM capacity, up from ~13% in 2025. In terms of bit shipments, CXMT’s share of global bit shipment is expected to rise from 9% to 12% in 2027.

CXMT’s global capacity share could increase further as its Hefei site reaches full operation and the two phases of its Shanghai site continue ramping through 2028. We believe the company will reach 500kwspm of wafer capacity by the end of 2028, accounting for ~17% of global DRAM supply, up from 11% in 2025.

Given CXMT’s expanding role in global DRAM capacity, as in past cycles, investors are concerning about potential supply-demand disruption from Chinese players. While these concerns are understandable, we believe they are likely overplayed at least for the next two years. We factor in incremental wafer capacity and bit shipments from CXMT and other memory suppliers—and assuming utilization rates in the high-90% range—we continue to see DRAM as extremely supply constrained. We constantly update our wafer add numbers, demand numbers, and pricing assumptions in our memory model, much faster than trendforce or sell side banks.

Looking solely at CXMT’s wafer additions, we do see meaningful capacity expansion when compared with other suppliers. We expect CXMT to add roughly 85kwspm, 70kwspm and 80k each year from 2026 to 2028, versus Samsung at 15k/50k/110k, SK Hynix at 60k/60k/90k, and Micron at 30k/90k/115k. Even with these wafer additions, we expect DRAM to remain undersupplied by a high-single-digit percentage this year, widening to a low- to mid-teens in bit undersupply next year. We addressed in detail in our previous piece why DRAM is likely to remain undersupplied through potentially 2028, even with these incoming incremental wafer additions.

We see little ability for the company to irrationally accelerate capacity expansion beyond its current pace in a way that would meaningfully disrupt a market now providing an extremely favorable pricing environment because fab construction timelines are so long. This pricing backdrop has been the primary driver of the company’s explosive earnings growth–which the company hopes to see it continues. Based on the fab buildout we are tracking, we also not yet see the sign of this possibility although we would like to stress that the total wafer capacity of Shanghai site could have over 400kwspm of wafer capacity in a full-ramp status.

Specifically on its wafer capacity, we see quite limited wafer allocation toward HBM. By reconciling with CXMT’s IPO-related filings to date, we find that wafer allocation to HBM has been very limited, even until today. By the end of 2025, we believe only ~5 kwspm of CXMT’s ~265 kwspm of capacity is allocated to HBM. We think this figure will increase to nearly ~30 kwspm and ~55kwspm by the end of 2026 and 2027 respectively. This capacity trajectory seems better align with CXMT’s filings indicate that roughly 99% of revenue consists of DDR and LPDDR products in 2025, as discussed earlier.

This wafer allocation dynamics could change, however. We believe China’s broader push for self-sufficiency in AI compute could conflict the company’s strategic priorities, and we believe this push could intensify over time, given the HBM supply constraints discussed earlier and government’s determination to address this issue.

To that end, in our estimates, we factor in government influence for CXMT to allocate more wafer capacity to HBM overtime. As a result, we expect CXMT’s HBM wafer capacity to accelerate materially in 2027 and 2028 with its HBM technology improvement, supported by continued growth in China’s domestic compute market. We estimate CXMT’s HBM wafer capacity will reach 55kwspm and 100kwspm in 2027 and 2028, respectively. This would increase the company’s share of global HBM wafer supply from 1% in 2025 to 12% in 2028.

It is important to remember that CXMT, unlike other memory suppliers, is not only an economically and technologically important company for China, but also a strategic asset that the country can leverage to advance prioritized policy objectives.

Strategically though, it does make sense for CXMT to allocate more DRAM wafer capacity to commodity DRAM over HBM in near term. The commodity DRAM currently offers materially higher margins than CXMT’s HBM products, while also delivering more than 3x the bits per wafer on a like-for-like basis.

Given that CXMT has not yet fully matured its HBM technology, allocating significant wafer capacity to HBM would likely generate limited profit while consuming scarce DRAM wafer capacity that could otherwise support higher-margin commodity DRAM at greater volume. In this context, prioritizing commodity DRAM is both economically rational and better aligned with CXMT’s current manufacturing capabilities and pricing environment. China must allocate to HBM though as sales of HBM to China are somewhat limited besides some loopholes that allow Korean vendors to keep shipping to China.

On technology readiness, we believe CXMT is still struggling to stabilize supply for HBM3 8-hi, with even greater challenges in 12-hi. On the front end, the company appears to have made progress stabilizing production of its Gen 4 (G4), or 1z-equivalent DRAM. We believe the majority of CXMT’s DRAM output this year will be fabricated on the company’s G4 process node. Yet, front-end wafer-sort yield should still be materially lower for the core DRAM die used in HBM, given its larger die size, more demanding cell performance, overall performance requirements versus commodity DRAM. We think the front-end wafer sort yield remain a major challenge for the company, and its gap versus its peers here is still large. While we believe yields on CXMT’s G4 node have improved, we suspect it still lower than industry standard of 85-90% mature yield level for 1z given the lower margin we saw throughout 2024 and 2025. This might suggest that equipment limitations and manufacturing know-how remain persistent obstacles that CXMT will need to overcome.

For its next process node, the company’s G5, or 1a-equivalent DRAM node, while can theoretically continue advancing without EUV akin to Micron in 1a process node, but it will face increasing fabrication and design challenges. These challenges will only add further manufacturing and design pressure when the node is applied to both DRAM dies for HBM. The lower yield and more challenging ramp-up schedule could impact company’s bit output as well despite the company can compensate the yield loss by adding more wafers.

On top of this, we believe die stacking remains the major obstacle for CXMT’s HBM. HBM stacking usually introduces significant technical hurdles, including thermal stress, die cracking, warpage, bonding defects, and yield loss across multiple stacked dies. Our understanding is that these issues become even more severe as the company attempts to move from HBM3 8-hi to HBM3 12-hi, and eventually HBM3E, given company’s yet sufficient enough of know-how and manufacturing experience for 12hi or above HBM.

Die stacking is not a challenge unique to CXMT. Even leading memory suppliers are encountering difficulties. For 12-high HBM4, we understand that suppliers continue to face significant stacking-related issues, including die cracking, thermal management challenges, and yield loss.

These challenges become even more pronounced as memory suppliers seek to manufacture 16-high or even 20-high HBM. For next-gen HBM4E, we note that one reason Rubin Ultra is expected to use 12-high HBM4E rather than 16-high is supply: 16-high HBM requires higher DRAM wafer intensity and involves a more difficult manufacturing process, which can lead to greater wafer loss and lower effective supply of DRAM bits. These conditions put both memory suppliers and customer in a very tough spot given highly supply-constrained environment in DRAM.

We think there is an increasing possibility that CXMT will skip HBM3 and focus instead on HBM3E 8-hi and 12-hi. We believe this potential roadmap change is driven by two factors: 1) customer demand for more competitive HBM products in ’27 timeframe, and 2) mainstream accelerators will equip with HBM3E, HBM4, and HBM4E.

On the back end, while it remains debatable whether CXMT is using MR-MUF or TC-NCF, we believe the packaging challenge should be relatively more manageable, as the company and its back-end partners face fewer export control constraints. CXMT has been working closely with leading OSATs such as Tongfu Microelectronics for some time, and we believe its back-end capabilities should have gradually improved, though a gap likely remains versus leading memory manufacturers.

Given these existing manufacturing challenges, we model CXMT’s HBM3 8-hi’s front-end and back-end yields at roughly 35% and 70%, respectively, implying an overall yield of only around 25%. We think this number should be lower when company attempts to produce HBM3 12hi or HBM3E 12hi given higher difficulty in die stacking and bonding. At these yield levels, CXMT’s HBM output would be even more limited on the same DRAM wafer capacity than leading memory suppliers. More importantly, the resulting HBM would likely carry very low margins, especially compared with commodity DRAM in the current pricing environment.

CXMT’s HBM struggles continue to be reflected in its limited product presence and slow penetration within China’s AI accelerator market. We think only Huawei, Cambricon, and select emerging Chinese AI chip startups are likely to adopt CXMT’s HBM, though we suspect adoption rates will be large. We actually believe domestic AI accelerator vendors would still prefer foreign HBM3, or even HBM3E, if they can secure supply through any available channel or stockpiled inventory before the export controls in December 2024. As China’s domestic CSP capex and broader compute buildout are surging, there is little doubt that domestic HBM demand is also growing rapidly and should continue to increase.

With that said Huawei and CXMT will have custom HBM that is not based on the slow JEDEC standards and phys, so it will be able to close the bandwidth disadvantage.

We believe China could face a more severe HBM supply constraint than what would be implied by slow domestic HBM development alone. This constraint is likely to be further exacerbated by tight supply across all three major HBM suppliers, each of which is already restricted from selling HBM2E-equivalent or more advanced HBM products into China under the U.S. export controls announced in December 2024. Given the tight supply environment, these suppliers are likely to have even less willingness to risk violating export controls to sell into China.

However, HBM re-export and smuggling could complicate this conclusion. We understand that some Chinese companies continue to obtain HBM3 from memory suppliers, a dynamic we reported on last year and that has since been corroborated by other leading media outlets. We believe this remains the case today.

Based on our conversations with industry participants, re-export through foreign offices or partner companies located in third countries remains one pathway through which Chinese companies can access HBM. In addition, some downstream OSATs or intermediaries in third countries appear to facilitate these flows. Some entities may ship partially assembled systems or modules in forms that are not treated as fully manufactured GPUs or ASICs and are therefore still permitted to be shipped into China. The HBM can then be recovered and repackaged onto domestic Chinese GPUs or ASICs.

What the IPO Structure Reveals

CXMT could become one of China’s largest semiconductor IPOs, and its ownership structure matters more than the headline financials. CXMT reported RMB7.14B of consolidated FY2025 net income, yet only RMB1.87B was attributable to parent shareholders, with 74% attributable to minority interests. The reason is the ownership architecture. CXMT holds 30.68% of the economics of Changxin Xinqiao and 31.72% of Changxin Jidian Beijing while controlling 73.01% and 75.32% of the votes through long-term acting-in-concert arrangements. That lets the company consolidate fabs it mostly does not own, so the consolidated figure overstates what public shareholders will actually receive by roughly four times.

That same voting structure undercuts the company’s declaration that it has no controlling shareholder and no actual controller, which the prospectus lists as a formal governance risk. CXMT exercises majority voting control of its fabs through acting-in-concert pacts, and state vehicles including National IC Fund Phase II, Hefei, and Anhui together hold well over 30% even after the listing. The arrangement looks designed to manage export-control and foreign-investor perception at a moment when CXMT’s ties to the Chinese state draw the most scrutiny.

The raise that will get reported badly understates the listing. CXMT plans to deploy RMB29.5 billion, or about $4.1 billion, while issuing 10% to 15% of its post-IPO shares. Fully funding those uses through the IPO implies a price of roughly RMB4.41 at 10% dilution or RMB2.78 at 15%, versus RMB2.63 in the June 2025 financing. The low end represents barely any per-share appreciation despite 1Q26 revenue of $7.3 billion and net profit of $4.8 billion. At RMB2.78, CXMT would be valued at about RMB197 billion, or $27 billion, equivalent to just 1.8 times annualized first-half 2026 parent earnings. This arithmetic floor sits well below a realistic book-building valuation. This is too cheap and should be much higher valuation in our opinion.

The allocation reinforces CXMT’s current priorities. Of the RMB29.5 billion in planned net proceeds, RMB20.5 billion, or 69.5%, funds wafer-production-line and DRAM technology upgrades, while RMB9 billion, or 30.5%, supports forward-looking DRAM research. The prospectus discloses no dedicated HBM project and does not mention HBM. Its project descriptions focus on newer process platforms, product iteration, and the migration of existing lines toward mid-to-high-end DRAM. The IPO therefore primarily strengthens CXMT’s core DRAM manufacturing and technology base, with no disclosed funding commitment to a near-term HBM expansion.

The size of the earnings move deserves a flag on cycle timing. CXMT guided in its December 2025 filing to an FY2025 parent loss of RMB 0.6 to 1.6 billion. Five months later the prospectus reported a RMB 1.87 billion profit, with consolidated income running past double the earlier high-end estimate. It also shows how quickly peak DRAM pricing moves the valuation denominator in either direction.

Finally, Alibaba’s place on the cap table changes how to read CXMT’s demand. Alibaba Cloud is at once the anchor hyperscaler customer, a near-4% holder, and an endorser, sitting alongside GigaDevice, chairman Zhu Yiming’s own fabless house, at 1.8%. Domestic volume is effectively guaranteed in a way the Korean incumbents did not have in their home markets, which matters more than the small percentages suggest.

For paid subscribers, we will take a deep dive into CXMT, China’s broader WFE ecosystem, the impact of export controls, and the implications for China’s memory and compute ambitions. We will also discuss HBM in more detail.

CXMT’s Equipment Ecosystem Under Export Controls: Domestic vs Foreign, and the Domestication Curve

The Silicon Valley Returnee

The DRAM Foundation: Inheriting Qimonda

The Patience of State-Venture Capital

From Inheritance Toward Independence

In the following section, we will be discussing CXMT’s financials, technology, equipment ecosystem.

Next Step after 10 Years: An IPO in the Supercycle

With that said Huawei and CXMT will have custom HBM that is not based on the slow JEDEC standards and phys, so it will be able to close the bandwidth disadvantage.

What the IPO Structure Reveals

CXMT’s Equipment Ecosystem Under Export Controls: Domestic vs Foreign, and the Domestication Curve

The claim that half of 2026 US datacenter capacity will be delayed or canceled has been circulating widely across financial and social media. This traces back to Bloomberg’s April 1, 2026 piece, America’s AI Build-Out Hinges on Chinese Electrical Parts, which framed the 2026 capacity slowdown as a consequence of a fragile, China-dependent equipment supply chain. Bloomberg didn’t lead with that framing, but within days, TechRadar, Tom’s Hardware, The Register, and other news outlets ran sharper, more clickbait versions claiming half of datacenters are cancelled, and that’s the version now circulating.

We find these claims quite amusing. We’ve consistently been first to call-out high-profile delays, like Core Scientific ahead of Coreweave’s Q3’25 earnings in our industry leading datacenter tracking model. We update the dataset by reviewing every site dozens of times a year. However, over the last 6 months, our YE2026 NA Hyperscaler Self-build forecast only moved by ~1%, and NA colocation <5%. What is causing the discrepancy?

In our view, the culprit is obvious: the data sources behind these claims of “50% of 2026 datacenters are delayed” are essentially uninformed vibe-coded datacenter forecasts that take announcements at face value, without any bit of critical judgement. We’ve seen more and more Claude Coded datacenter models and estimates crop up, all of them wrong. Thankfully, that’s not how we built our model, which is trusted for billion-dollar investment decisions by all the world’s largest tech companies in the world, as well as energy and industrials giants, and all the largest investors on Wall Street.

Claude Code pulls press releases, views unfounded GW-scale announcements as ground truth, misunderstands construction timelines and grid complexities, and compiles a terribly inaccurate report. As resident Claude Code users spending $170K+ in just one week, we are very familiar with how to use Claude, and the mistakes others are actively publishing.

Let’s be clear, delays and cancellations are occurring. Our Datacenter Model flagged the STACK Infrastructure / Oracle permitting impasse months before Bloom Energy stepped in. We caught the Nebius New Jersey delays, which are still persistent. We nailed the Core Scientific datacenter delays ahead of Coreweave’s Q3 earnings. However, we then successfully predicted Coreweave’s 1.7GW of Active Power by end of 2026 (exactly the Company’s guide) as we determined that other datacenters were on time (no, not everything is delayed!). Our data also predicted ~35B of RPO to be signed by Q1 2026 via our industry-first method to forecast RPO based on datacenter signals.

And most recently, we’ve determined STACK Infrastructure/Oracle’s site has now been delayed to 2029, due to a lack of gas pipeline and the ensuing burdensome regulation. We’ve flagged many many others, some of which have materialized, but many others which aren’t realized yet and the market is still misunderstanding. Check our Datacenter Model for full information.

First, the headline number is built on a hugely flawed denominator, off by multiples. Bloomberg’s data comes from Sightline Climate, which estimates that of the roughly 12 GW expected to come online in the US in 2026, only about 5 GW is currently under construction. Checking against our Vision Model, trained on hundreds of thousands of satellite images, even their Under Construction figures are off by multiples. Taking ONLY the top-two hyperscalers alone gives us a higher Under Construction capacity than 5GW. And that’s self-build only, it doesn’t include the many GWs of capacity under construction by 3rd party developers.

Sightline’s dataset clearly doesn’t track every datacenter under development; it likely tracks the large, publicly announced projects. That skews the basis heavily toward the kind of speculative megaprojects from unexperienced datacenter developers most likely to slip in the first place. The “50% cancelled or delayed” figure isn’t really a statement about the US datacenter pipeline, but rather about the slice of the pipeline most prone to slipping.

Second, even taking the Bloomberg/Sightline number at face value, almost all of what’s getting flagged sits in the early-stage, or pre-construction “announced” bucket. These are speculative MW that weren’t going to land on a 2026 timeline with any rigorous analysis. We’ve been writing for months that the market is in structural oversupply of early stage projects, with developers and landowners announcing projects on preliminary power estimates, often without financing, interconnection studies, or equipment orders in place.

Instead of falsely declaring them as 2026 projects, they rather show up in 2028+ in our model. In December 2025, we flagged that over half a Terawatt is in the large load queue in the US, and that number is now over a Terawatt. The vast majority are highly speculative, yet some developers make high profile announcements without much to show for. We do not put low probability datacenters in our datacenter timelines.

We will start by covering how common these questionable announcements are, and then display what real delays look like. We then cover the noise fueling the misled delay narrative, and close with why none of it has moved our numbers, for datacenters or for the equipment suppliers.

The Anatomy of a Datacenter Delay

We have three broad buckets to categorize datacenter delays, which we examine in depth below:

1. Aggressive Announcements by Newer Datacenter Developers

New entrants in the space commonly try to give themselves some publicity by announcing massive GW-scale datacenters with aggressive timelines, despite their project being early stage. But it can take more than 4 years from land purchase to datacenter delivery. When we see a brand-new announcement in 2025 saying 2026 operational from a new “unknown” developer, it should raise some red flags. And for flagship multi-GW announcements, we often find that deeper analysis reveals these projects are very early, and sometimes don’t even have long lead time equipment ordered. We are not denigrating these operators, but rather promoting caution on interpreting headline announcements correctly - which vibe-coded forecasts often struggle with. Let’s look at a few examples.

Data City, unveiled by Energy Abundance, boasted a 5GW campus, of which 300MW would launch in 2026. Looking again, the Energy Abundance website has very little except a “Contact Us” button. There has been no news since the announcement, and satellite imagery doesn’t show any evidence of physical progress. While AI-generated datacenter forecasts likely consider this as 2026 delayed capacity, any serious forecast would’ve taken this with some skepticism.

In June 2025, APR Energy announced a potential 400MW datacenter campus in Texas, where “The first phase of the project’s development is slated for early 2026.” At the same time, they were “currently seeking data center developers, hyperscalers, and AI partners to partner on the development of the facility.” No customer yet, a new developer, targeting a <1-year timeline needs to be approached with more scrutiny. There’s been no news since.

Another example is the Cloudburst facility in San Marcos, TX. It was announced as a 1.2GW facility, of which 50MW live by Q3 2026 (then pushed to Q4 2026). When announced in February 2025, that Q3 2026 timeline was unrealistic given many approvals weren’t secured yet. The company is now making good progress, and got the go-ahead by the local county in April 2026 for a >10B project - congrats to Cloudburst. They broke ground in November 2025 for that first 50MW phase, but the Q4 2026 timeline remains too aggressive based on real-time imagery, since shell construction hasn’t started yet. However, it’s easy to see how Claude Coded forecasts could get confused and assume 1.2GW is delayed. Some publicly available estimates out there claim that the site is planning to deliver 1.2GW by 2026 - which is not even consistent with the company’s own statements.

These are just a few examples of headlines, there are many more. If you look closely at the Claude Coded estimates, you will find many of these announcements are taken at face value. This is not a knock on the developers — as these projects may still be realized later - but an egregious error in many estimates’ methodology.

2. Advanced Projects With Overly Optimistic Construction Timelines

Many do not understand what it takes to construct a datacenter, and there’s plenty of new(er) operators with ambitious plans. Our forecast accuracy is the most reliable because we have refined our timelines extensively. We review every single type of datacenter design, both through local filings, and extensive satellite imagery. The vibecoded datacenter models have not learned what true timelines are, and instead mislead everyone.

Nebius’ flagship campus in New Jersey, built by developer DataOne, is a great example of the challenges of constructing a datacenter. In March 2025, DataOne was targeting a 4 month delivery for its first 50 MW phase despite the developer having no prior datacenter track record in the US. The schedule relied on precast materials and extensive prefabrication to compress the build. To be clear, we think this approach was highly innovative and we are big fans, but things are always more challenging than initially expected, especially for a new datacenter developer.

The first challenge arose in the equipment procurement. Per a LinkedIn post of an executive leading the development, late deliveries from a few suppliers pushed the first phase from 4 months to 6, i.e. a 2 month slip on its own.

Then, commissioning and installation proved longer than expected. Satellite imagery shows the shell going up remarkably fast, but setting up the MEP took far longer than the shell itself. In September 2025, the executive still claimed that he would imminently deliver the full facility. But our December 15th capture shows the chillers serving the second 25 MW still not in place, meaning the initial 50 MW building wasn’t complete almost three months after delivery was declared imminent.

Putting the full sequence together makes the slippage easy to see (timeline from DataOne’s own public disclosures, Nebius earnings calls, and our satellite tracking):

To understand whether the site will be on time for 2026 delivery and Nebius meet its ARR guidance of $7-9B by year-end, check our Datacenter Industry Model.

Shown in our graph as one of the newer operators with impressive plans, Core Scientific ran into issues that an experienced operator would have likely accounted for when projecting timelines. Let’s walk through a quick timeline.

Couple this with satellite imagery of a multiple buildings with no chillers installed, this was clear that the 250MW by YE2025 timeline was going to be missed.

Following this, we promptly informed our clients that we see delays coming for CoreWeave.

These challenges are not always avoidable, but experienced operators typically account for them in both timeline and budget.

3. Well-Capitalized Projects Facing Permitting and Local Opposition Issues

Rather than waving our hands about “regulatory challenges,” let’s detail another project in specific: the STACK Infrastructure/Oracle location that we have delayed to 2029. The following information was published to our Energy Model and Datacenter Model clients, of which the Energy Model is available as a $0 bundle for limited time. Oracle’s June 10 earnings call guided 1H2027 customer delivery for this campus: a timeline FERC records do not support, nor our estimated construction timelines.

Three things went wrong here. The project tried to thread permitting limits with a two microgrid design, each sitting very close to the major-source threshold. It required a large infrastructure buildout, including a brand new gas pipeline. And organized local opposition slowed every step along the way.

In our March Datacenter Model note, we flagged the Oracle/STACK New Mexico Stargate campus (Project Jupiter, in Doña Ana County) with a delay due to its onsite power system was neither finally permitted nor solidified. The entire campus depends on two behind-the-meter gas microgrids, East (NSR 10732) and West (NSR 10734), filed with the New Mexico Environment Department by Acoma, LLC. NMED received both applications on November 17, 2025 and ruled them complete on January 22, 2026, but each microgrid was being squeezed in as an individual PSD-minor source rather than one major source.

NMED’s December 2025 incompletion letters found East could emit up to 521 tpy NOx against a requested 248.90 tpy cap and West up to 388 tpy against 249.97 tpy, both too close to the 250 tpy major-source threshold to be enforceable. On March 23, 2026, NMED pushed both reviews from the April 22 target to July 21, 2026, ordered a public hearing, and cited roughly 7,155 comments plus the applicant’s intent to imminently amend the filings, leaving the public schedule of Q4 2025 construction and Q4 2026 operations untenable.

Meanwhile, the onsite gas generation systems had still not been locked in. In the West Microgrid, for example: “These thirty-four (34) turbines will consist of a combination of GE TM2500 and Mitsubishi FT8 natural gas fired turbines depending on availability for purchase.” Depending on availability for purchase with turbine lead times extending multiple years did not make a great case.

Our Energy Model note “Oracle’s New Mexico Stargate Gas Pipeline Delayed: No Bloom Without Gas” published in May found new developments. With the gas turbines unable to clear permitting in time, Oracle withdrew the East and West turbine applications on April 27, 2026 and filed for Bloom Energy fuel cells instead, targeting ~37 tpy NOx. However, this didn’t solve the binding constraint: the 400,000 Dth/d of gas to arrive in one unprepared pipeline.

A September 2025 precedent agreement between Green Chile Ventures (likely Oracle subsidiary) and Transwestern baked an August 15, 2026 in-service date into the contract, and Transwestern filed the Green Chile Pipeline at FERC (CP26-80-000) on January 29, 2026 seeking an expedited blanket certificate. That route ran into compounding obstacles: on March 20, the New Mexico State Land Office denied right-of-way across state trust land, forcing a reroute, and on April 13, 2026 FERC staff filed its own protest over the missing New Mexico State Historic Preservation Office (SHPO) sign-off while multiple environmental intervenors filed to intervene, starting a 30-day cure window.

Oracle and Transwestern's May 1–4 attempt to extend the cure deadline went nowhere, the window lapsed on May 13 with no SHPO clearance, and on May 15 FERC's Office of Energy Projects opened a formal Section 7 review demanding a full alternative-route analysis avoiding state lands. Transwestern's subsequent sworn filings confirmed no viable alternative route exists and no path around Section 7. The contractual August 15 in-service date became infeasible, and our base case moved first power off 2027 entirely and out to 2029.

NIMBYism rarely kills a capitalized project of this scale outright, but adds months to permits, hearings, and rights-of-way, enough to delay a project past strict deadlines and into large consequences.

What Is Fueling All This Panic?

If those are what real delays look like, the headlines driving the narrative look very different. Most of what circulates as evidence of a datacenter pullback falls into three buckets: moratoriums, organized local opposition, and announcements with timelines that were never realistic to begin with. None of them survive contact with the actual pipeline.

A datacenter moratorium is a legislative freeze on new construction permits, designed to give jurisdictions time to assess environmental, energy, and economic impacts before approving more capacity. As of April 2026, at least 12 states have filed moratorium bills, and Indiana alone has at least four counties (Fulton, Marshall, Putnam, and White) that have enacted them. These make for alarming headlines until you look at what they actually cover: none of them had datacenters, and we had nothing in our model beyond early stage.

The most dramatic example is Maine, where the Governor vetoed LD 307 in April, which would have been the first statewide datacenter ban in the US. A statewide ban sounds like ominous for the buildout, except Maine has less than <5MW planned. The pattern holds across the moratorium map: these measures land overwhelmingly in inconsequential areas, or if not, the projects were already too early stage for 2026 anyway.

NIMBYism can be very impactful, as seen with the STACK/Oracle example. The distinction is that opposition slowing a real, capitalized, underway project is rare, and opposition killing a speculative rezoning that existed only on paper is common. The Compass withdrawal in Prince William County, the rejected rezonings in Georgia, the pulled proposals across 20+ states: nearly all of it sits in the announcement layer, where the projects had no equipment ordered, no interconnection agreement, and no 2026 delivery to lose.

The third bucket is the announcements themselves like we showed above. Most pipeline estimates, including the ones underlying the “half of 2026 capacity will be delayed or canceled” framing, rely on announcement-stage data: a press release, a real-estate filing, sometimes a non-binding LOI, none of which is binding in any meaningful sense.

ERCOT is the cleanest illustration of how this can pile up. As of April 2026, ERCOT was assessing more than 410GW of large-load interconnection requests, over 87% of them from datacenters. Texas’s all-time peak demand is about 85 GW, so the queue is nearly five times the size of the grid itself. However, tracking the veritable projects as we do, there’s 311GW of Phantom Datacenter demand.

When Texas Senate Bill 6 introduced a real filter (site control documentation, a $100,000 minimum study fee, mandatory curtailment terms), speculative filings started to slow down, exactly as a sorting mechanism should work. When the press reports that a meaningful share of the pipeline has been “delayed or canceled,” much of what is actually being canceled are megawatts that never had the site control, the financing, or the interconnection work to clear those gates in the first place.

All three of these generate fear in headlines and roughly nothing in delivered 2026 megawatts. They concentrate in the early stage layer we already treat as structurally oversupplied, they were in our numbers long before they reached financial media, and that is why our estimates have not moved.

Why Hasn’t Our 2026 Outlook Moved?

We have walked through the constraints above and continue to track them closely. The question worth answering is why our 2026 outlook holds despite the headlines.

The short version is that the cancellations and delays cluster in a layer of the pipeline we already treat as structurally oversupplied. The veritable 2026 projects, meaning the ones with site control, equipment on order, interconnection agreements signed, and/or vertical construction underway, continue to progress on schedule. Hyperscalers, whether its self-build or colocation tenancies, are routing around the constraints in ways that announcement-stage tracking does not pick up.

Early Stage Projects Are Structurally Oversupplied

The bulk of project cancellations occur at the earliest stages of development, before projects have cleared the hurdles required to move toward execution. These are typically announcements rather than fully formed developments, projects that lack critical elements such as permitting approvals, committed tenants, vertical construction progress, or secured grid interconnection. In our Datacenter Model, this segment of the pipeline is intentionally treated as structurally oversupplied, seen later in 2028.

The volume of announced capacity at the early stage exceeds what can realistically be delivered, and we view that imbalance as a normal feature of the market. It reflects the asymmetry between low barriers to announcing projects and real constraints on building them. A project generally begins to move beyond this higher-risk phase once it has secured at least two, but preferably three, of the following:

Ordering long lead time equipment is also crucial, and that requires deposits and capital.

Leading Operators Are Adept At Navigating All Constraints

Hyperscalers and leading operators know how to operate in these markets. Land banking multiple locations is the baseline move, but development happens via two parallel playbooks: one political, one physical.

The political playbook is really about permits and zoning. Operators with years of history in a place know who to call and what they need to file. They have the local land-use lawyers on retainer, working relationships with the utility VPs and economic-development staff who keep the project moving, and a track record of having delivered prior projects. Those relationships are maintained through ongoing community investment, from PILOT structures and operator-branded workforce programs to capital infrastructure that doubles as public utility, like the $31M Microsoft put into the Quincy Water Reuse Utility before leasing it back to the city for $10 a year. By the time the public hearing rolls around, the developer has heard every concern before and knows how to sell the project around them.

When the political playbook alone is not enough, the project rarely dies. Developers increasingly withdraw proposals ahead of a formal vote rather than hand opposition groups a public rejection that other counties can cite, then relocate to a friendlier jurisdiction, refile a modified design, or shift to brownfield and unincorporated land where the approval steps are fewer. There’s plenty of times hyperscalers don’t progress with sites, not just the ones that are publicly canceled. The site still dies, but it dies quietly, and the project moves somewhere else. Hence the many landbanks.

Grid connection lead times have stretched into 7–10 years in multiple metro areas, and equipment lead times for critical SKUs remain 1y+. The market has evolved adaptations on both axes.

On energy, the playbook has expanded from a single grid-connection path to multiple:

On equipment, three adaptations have moved from edge cases to standard practice over the last 12 months:

The political playbook secures the first right to build, and the physical playbook ensures the build progresses accordingly.

What Do We Do Differently?

With all the above challenges and the varieties of workarounds, how are we able to determine veritable delays? Some projects may die on local opposition, some might pass and see the approving council members ousted. Labor is a challenge, or you can import from across the country. Even behind-the-meter sites are still a novel solution.

Our research agents monitor thousands of municipal, county, state, and utility portals — most poorly structured, many actively resistant to scraping, some still publishing in scanned PDFs and handwritten exhibits. We parse them in parallel and reconcile every signal against live satellite imagery, which is how we generate forward-looking delay calls across the full US pipeline rather than just the highest-profile projects.

The Datacenter Model methodology is coupled with the Industrials and Energy teams tracking the power and equipment/labor markets respectively. When every angle is tracked in parallel, it becomes straightforward to identify which projects carry real cancellation or delay risk, and which apparent “delays” are simply the market culling speculation that was never going to deliver in 2026. Behind the paywall, we discuss another veritable delay coming up, what we are watching, and the players set to benefit.

The Equipment Supplier Narrative Is Wrong Too

We said at the top we would come back to the equipment suppliers. The fear priced into names like Vertiv is that the wave of datacenter delays and cancellations will start eating into record orders and backlog. However, our Datacenter Model capacity forecast shows quarter by quarter delivery, and it’s clear to us that delivered MWs are accelerating.

Equipment lead times remain long. Reinhausen tap-changer bushings, which are a single subcomponent inside a transformer but gate delivery of the entire large HV stack, are now quoted at three to five years. Things have stabilized somewhat over the past year, though most critical SKUs are still running over a year, and several of the major manufacturers, including GE Vernova, Hitachi Energy, and Mitsubishi Electric, are booked out three to four years on their main equipment lines.

Capacity relief is slow to arrive, since production expansion can take two to three years; Hitachi Energy’s new South Boston plant, announced in September 2025, is not expected to be operational until 2028. So in the meantime, the way capacity actually gets allocated is through prepayments. Buyers wire money up front to hold a slot in the queue, which boxes out players with less capital. For behind-the-meter systems, we have heard prepayments in the 10 to 15% range are now standard practice for securing a queue position. AI demand has lifted the pure-play datacenter electricals, Vertiv and Schneider among them, to margins above 20% on the strength of that same imbalance.

Now run the cancellation fear against that backdrop. The projects getting canceled sit in the early stage layer that never placed equipment orders in the first place; a speculative announcement dying in a county commission hearing removes zero orders from anyone’s books. The projects that do sit in OEM backlogs have prepaid for queue position, locked in long-lead SKUs during scoping, and in many cases already broken ground. And in the rare case a real order does fall away, a queue running three to four years deep means the slot is reallocated to the next buyer in line rather than vanishing.

Our Industrials Model tracks more than 550 datacenter equipment suppliers across 75 equipment subcategories, mapped to over 6,000 facilities globally, and maps every major supplier’s exposure project by project, which is how we separate backlog at risk from backlog that is locked.

What we’re watching into 2H26

The Anatomy of a Datacenter Delay

We have three broad buckets to categorize datacenter delays, which we examine in depth below:

1. Aggressive Announcements by Newer Datacenter Developers

2. Advanced Projects With Overly Optimistic Construction Timelines

Putting the full sequence together makes the slippage easy to see (timeline from DataOne’s own public disclosures, Nebius earnings calls, and our satellite tracking):

To understand whether the site will be on time for 2026 delivery and Nebius meet its ARR guidance of $7-9B by year-end, check our Datacenter Industry Model.

Couple this with satellite imagery of a multiple buildings with no chillers installed, this was clear that the 250MW by YE2025 timeline was going to be missed.

Following this, we promptly informed our clients that we see delays coming for CoreWeave.

These challenges are not always avoidable, but experienced operators typically account for them in both timeline and budget.

3. Well-Capitalized Projects Facing Permitting and Local Opposition Issues

What Is Fueling All This Panic?

Why Hasn’t Our 2026 Outlook Moved?

We have walked through the constraints above and continue to track them closely. The question worth answering is why our 2026 outlook holds despite the headlines.

Early Stage Projects Are Structurally Oversupplied

Ordering long lead time equipment is also crucial, and that requires deposits and capital.

Leading Operators Are Adept At Navigating All Constraints

Grid connection lead times have stretched into 7–10 years in multiple metro areas, and equipment lead times for critical SKUs remain 1y+. The market has evolved adaptations on both axes.

On energy, the playbook has expanded from a single grid-connection path to multiple:

On equipment, three adaptations have moved from edge cases to standard practice over the last 12 months:

The political playbook secures the first right to build, and the physical playbook ensures the build progresses accordingly.

What Do We Do Differently?

The Equipment Supplier Narrative Is Wrong Too

What we’re watching into 2H26

The Cost of Capability

Coding assistants are the greatest B2B SaaS application the world has ever seen: a $30B+ ARR market across the six largest players today, on track to clear $100B by year end, per our tokenomics model.

The agentic coding capabilities of those assistants don’t come from pre-training alone. Post-training, and reinforcement learning (RL) in particular, is what elicits these capabilities from such pre-trained models. Concretely, Claude Opus 4.8 scores 69.2% on SWE-bench Pro and 74.6% on Terminal-Bench 2.1, and RL training is a major part of what drives the score.

Dario Amodei, Anthropic’s CEO, has described RL as showing the same kind of scaling pre-training once did, where performance climbs log-linearly with how long you train (link). However, that scaling is enormously expensive, which makes RL system efficiency critical: it sets how much RL you can afford, and with it how far model capabilities can go.

What governs the efficiency of an RL training system? In this article, we conducted RL training experiments on open models with open-source RL frameworks, and compared pricing to hosted RL training solutions such as Tinker. We show that system efficiency comes down to matching trainer and generator throughput.

Acknowledgements

The Three Actors

An open-source RL training system has three actors: the generator, the RL environment, and the trainer. The generator performs inference on prompts from the dataset, generating a rollout: a prompt and a model’s generated response. Unlike in pre-training, the dataset only provides prompts, not the full target. Instead, the system generates the training signal. To generate rollouts, the generator interacts with the RL environment. The RL environment produces a reward based on the rollout. For example, a code environment executes the generated code in a sandbox and gives reward scores based on the test case pass rate. Then, the trainer ingests rollouts and rewards generated by the generator and trains the model, producing new model weights. Finally, the trainer pushes the new weights to the generator, closing the loop of an RL training step.

The RL Algorithm, Just Enough

Unlike pre-training, which maximizes the log-likelihood, RL training maximizes the expected reward. Group-Relative Policy Optimization (GRPO) is the open-source standard, and most open-weight models train with some variant of it.

GRPO samples multiple completions for each prompt, forming a group of rollouts. We assign each rollout a reward, a score of the rollout. We then compute each rollout’s advantage: its reward relative to the group’s average, capturing how much better or worse it did than a typical rollout for that prompt. Rollouts above the group average get reinforced, and those below get suppressed.

If every rollout in a group gets the same reward, each one equals the group average, so every advantage is zero and the group produces no training signal. This is governed by the group’s reward distribution: a group only teaches the model something when its rollouts show different behaviors. The extreme case is a uniform distribution, which happens when a task is too easy (every rollout passes) or too hard (every rollout fails), i.e. when the solve rate is near 100% or near 0%.

From Synchronous to Asynchronous RL

Classic policy gradient algorithms (which GRPO is based upon) assume the rollouts in a group come from the same policy, i.e., the model with the same set of weights. In the context of the training system, this means the generator cannot update weights until it finishes the current batch of rollouts. As a result, if the trainer finishes a training step, it’ll have to wait for the generator. This leads to synchronous execution of training and generation, greatly reducing system efficiency.

PipelineRL introduces asynchrony into the system by allowing the trainer to push new weights to the generator while rollouts are still in progress (in-flight weight updates). This way, the trainer can overlap its execution with the generator’s rollout. However, this comes with a cost. A sample, the unit a trainer consumes, is defined as a rollout and the reward the RL environment assigned to it. With in-flight weight updates, each sample is generated by a mixture of old and new policies. We refer to the phenomenon as policy staleness.

PipelineRL shows that RL algorithms can tolerate policy staleness to an extent. Samples too stale degrade model learning. Synchronous execution wastes too much compute to be practical at scale, and async techniques are essential. In effect, PipelineRL is a throughput-matching scheme with bounded policy staleness. It allows the trainer and generator to run at different speeds, capped by how stale samples can get. This is why PipelineRL has become the de facto implementation in open-source RL training.

The RL Environment and Sandbox

The RL environment provides feedback to the model for the model to learn how to do a task. Practically, the RL environment is implemented as a sandbox, a containerized runtime for code execution. Depending on the complexity of the RL environment, it could range from lightweight Firecracker micro-VMs (micro-Virtual Machines) to full-blown QEMU VMs.

Sandboxes for RL environments have unique system challenges. The interaction latency between the generator and the RL environment is critical to the end-to-end rollout latency, and sandbox startup latency is one of the major overheads. Sandbox service companies like Modal optimize the startup latency with techniques like content-addressed caching.

Serving at scale is also challenging: sandboxes scale along with the number of concurrent rollouts, and the number of live sandboxes fluctuates greatly as rollouts are pruned or completed.

Finally, sandboxes need to be robust against system failures. During learning, a model’s unexpected behaviors may deplete a sandbox’s designated resources. For instance, creating a million files can cause the sandbox to run out of memory. Sandbox orchestration needs to be able to detect and recover from those system failures.

The Throughput-Matching Framework

Quantifying Trainer and Generator Throughput

We view the RL training system as a queue: the generator produces rollouts into a queue, and the trainer consumes from it. When the generator is slower than the trainer, the queue empties and the trainer starves, idling between steps. When the generator is faster, the queue grows and its samples age, causing policy staleness issues.

We model the system efficiency through the lens of matching the producer (generator) and consumer (trainer) throughputs. In an ideal RL training system, the trainer consumption rate should be roughly equal to the generator production rate. The trainer consumes samples, performs a training step, and then broadcasts weights to the generator. We measure the training step time, including the idle time waiting for new samples, the compute time, and the weight broadcast time. We further derive the sample throughput (samples / compute time) to represent the trainer consumption rate. We also calculate FLOP/s per GPU and model FLOPs utilization (MFU) to characterize hardware efficiency.

The generator produces samples through LLM inference and RL environment interactions. Since the straggler, the longest rollout in a group, sets the group's completion time, we measure end-to-end latency to estimate the average sample throughput. We can derive the sample throughput (samples / end-to-end latency) to represent the generator production rate.

We further break down the latency into LLM inference time and RL environment interaction time. RL environment interaction involves sandbox startup latency and sandbox execution time.

Throughput Constraints

Multiple factors constrain the upper and lower bound of the throughput. The trainer consumption rate is samples per step / training step time.

On the generator side, we define the generator production rate as the number of concurrent rollouts / end-to-end latency.

Concurrency, or the number of concurrent rollouts, is limited by KV cache memory space and average sequence length. The aggregate memory capacity of generation nodes - model weights and activations are the space for KV cache. KV cache dictates the maximum number of tokens the generator can hold, and dividing it by the average sequence length gives us an estimate of max concurrency.

Not all generated samples are consumed by the trainer due to training signal quality differences. We further define the effective generator production rate as acceptance rate × generator production rate. Factors that affect the acceptance rate include:

Policy Staleness

Staleness is the gap between the policy version that produced the sample and the one that the trainer uses when it applies the gradient. Staleness is a byproduct of async training: the trainer pushes weight updates in-flight while rollouts are being generated.

Staleness happens at different granularities. At the trajectory level, staleness is the gap between the policy that the rollout starts generating with and the newer policy the trainer updates, e.g. the trainer is at version t+k while the trajectory started from version t. Token-level staleness occurs when in-flight weight updates happen mid-rollout, so different policy versions generate different parts of a rollout. Policy staleness also happens at an environment state-level, which we explain more in the Partial Rollout and Stateful Sandbox Design section.

Policy staleness means the trainer trains the model on off-policy signals, which can destabilize training, so we typically set a policy staleness budget. The policy staleness budget bounds how stale a sample can be, i.e. how far the generator is allowed to run ahead of the trainer before its samples are rejected. This in turn caps the gap between the generator’s production rate and the trainer’s consumption rate, which is what lets the two run at different speeds.

The policy staleness budget limits the max difference between the trainer consumption rate and the effective generator production rate. Concretely, it limits how many steps the generator can be ahead of the trainer, so the RL algorithm can tolerate the max staleness in samples.

The Moving Target: Model Capability and Behavior

Model capability and behavior are first-class system variables in RL training. Model capability determines how well the model can solve a task, measured by solve rate: what percentage of the rollouts in a group passes the verification. When the solve rate is near 0 or near 100%, the reward distribution is uniform, leading to zero advantage, and collapses the training signal. This drives decisions including group size and advantage filtering strategies. It also shapes the curriculum: the order of tasks being presented to the model. The curriculum is chosen to keep the solve rate at a productive middle band, so the tasks are neither too easy nor too hard for the model’s current capabilities.

A model's token usage behavior affects the output length, which indirectly determines the max concurrent rollouts in the generator. RL training typically elicits Chain-of-Thought reasoning, causing a model to generate long reasoning traces. This behavior drives the average output length, which increases KV cache usage. Under the same memory budget, this reduces the max concurrent rollouts and increases the sample generation end-to-end latency.

A model’s tool call behavior affects the sandbox time in the end-to-end latency. Concretely, more tool calls increase the sandbox time, and unexpected tool call operations strain the sandbox infra. We measure the tool call behavior with the number of tool calls and the sandbox error rate.

Both model capability and behavior shift during RL training. Model capability improves, and behavior drifts. The constraints move with them, and the training system has to adapt to ensure efficiency.

Case Studies

We ran experiments to ground our theory, and we shared our experiences with open-source RL frameworks.

Long Model Response and Early Exploration Training Phase

Setup

Analysis

In this case study, we show how two factors influence the system efficiency: how the model produces long responses with long thinking traces, and how well the model learns the task.

The model’s long response behavior is the dominant driver of LLM inference time, which makes up the majority of sample generation time. Long-response variance also drives severe tail latency within each rollout group, which forces a system-side mitigation: oversampling. With oversampling the generator launches more concurrent rollouts than the trainer needs per batch and it discards unfinished or errored ones as soon as the target count is reached. In this run we discard a relatively high 60% of dispatched rollouts, reflecting how skewed the latency distribution becomes when the responses are long.

On the model capability side, the curriculum is slightly too hard for the base model. We observe the number of rollout groups that produce zero solutions jump sharply after the first training step, but average reward and pass@16 are slowly recovering. These are signs that the early RL exploration pushed the policy away from previously solvable problems before it could relearn them.

The combined effect is a generation-bound system: the queue runs dry and the trainer starves. To mitigate this issue, we trade off trainer efficiency for generator efficiency. The trainer consumes samples at 2.75 sample/s, waits 30% of the wall-clock time, and runs at 10.5% MFU. On the other hand, the generator delivers 1.95 sample/s and uses 3x the trainer’s compute. This underscores how much inference efficiency matters during RL training.

Frequent Tool Use and Easy Task

Setup

Analysis

In this run, we see two factors set the system efficiency: model behavior drift and the curriculum being too easy for the model.

We observe both the average response length per turn and the number of tool calls (20 to 51) triple over the run. Together they push sequence lengths up and shift the workload toward a prefill-heavy profile. This supports our choice of disaggregated serving, which improves the worst-case time-to-first-token (TTFT) for prefill requests.

On the capability side, the curriculum is too easy for the model. On average, 55% of problems have a 100% solve rate, i.e., every rollout in the group passes. A rollout group where every rollout has the same reward produces zero advantage and contributes no training signal. As a result, average reward stays flat. The curriculum mismatch reduces the effective training samples and subsequently the effective generator production rate.

All the effects result in a heavily generation-bound system. The queue fills up fast, but a large portion is filtered out, so the output is low. LLM inference dominates end-to-end generation time and keeps growing as training progresses. The trainer spends 74% of the wall-clock time waiting, and its consumption rate is 5× the generator’s delivered production rate.

Sandbox Scaling Challenge

Setup

Training Stages

Analysis

The training dynamics were roughly in line with our other Qwen3 235B training runs. The run was generation-bound: trainer idled 30% to 60% of the time. Generation length grew aggressively: 3 to 8% of the samples hit max sequence length 128K, and for similar reasons, the tail latency blew up to 7500s for a noticeable straggler on step 9.

The unique part of this case is that we pushed the concurrent rollouts and encountered sandbox scaling issues. Scaling concurrent rollouts is critical to sample generation throughput. It enables higher batch size, more reward signal variance, and subsequently provides a better training signal. However, each rollout corresponds to at least one sandbox, and scaling rollout concurrency challenges the sandbox infrastructure reliability. In this case, we experimented with concurrent rollouts ranging from 96 to 960. At 960, we encountered sandbox initialization dead errors and straggler 1-hour sandbox spin up latency, and to mitigate that, we scaled down to 96, but then observed low rollout efficiency. This experiment shows how critical sandbox scaling is to RL training efficiency.

We thank Ao Shen and Kaichao You from vLLM / Inferact for helping conduct this experiment. The detailed writeup of errors and attempts is here, the WandB logs of the final stage training are here, and the code repo is here (verl, uni-agent).

Partial Rollout and Stateful Sandbox Design

Setup

Analysis

Overall, this experiment has system characteristics similar to other Qwen3 235B runs: generation-bound with 60% trainer wait ratio, long 12K response length per turn, a moderate average 12 tool calls per sequence, and a relatively low resolve rate.

We’d like to highlight slime’s partial rollout feature and how the sandbox design has to adapt to support it. Partial rollout allows the straggler rollouts to be saved into a buffer and restarted in a later batch. This mitigates the long tail latency of straggler rollouts, which improves system efficiency.

In slime, partial rollout marks straggler rollouts as aborted, saves them to a replay buffer, and puts them into the queue in future batches. This creates challenges for the sandbox:

Slime triggers partial rollout abortion in two situations: when a target rollout batch is filled (64 samples in our case), and when the trainer pushes a weight update. This leads to an interesting dynamic of partial rollout count in the replay buffer. We see the number increase after steps 5, 10, and 15, since we update weights every 5 intervals. We also see the number spikes at step 7 and 17, and we believe it’s due to the previous step (step 6 and 16) completing as soon as it reaches the target rollout batch, leaving a big backlog.

Unlike PipelineRL, slime’s SGLang configuration evicts the KV cache for aborted rollouts. This means resumed rollouts are essentially large prefill requests, which is exactly the workload type that benefits the most from PD disaggregation.

Partial rollout is where environment state-level staleness shows up. When an aborted rollout resumes in a later batch, the trainer has pushed new weights in between, so the resumed rollout faces not only the policy gap, but a state gap that’s specific to stateful environments. In our SWE-Bench case, the sandbox it wakes up in is not a fresh repo. The sandbox holds the half-applied edits, created files, and working-directory state that the old policy produced over its earlier turns. The newer policy now must continue from a situation it didn’t create and wouldn’t necessarily have created itself. This can corrupt the training signal: during training, the advantage gets attributed to a trajectory the current policy only partially owns. Seeing the high resume rollout count after weight updates, we suspect environment state-level staleness contributes to the run’s low resolve rate.

In summary, partial rollout rescues the stragglers to maintain the queue-filling throughput, but it comes at the cost of stateful-sandbox complexity and explicit environment state-level policy staleness.

Software User Experience

Prime RL

Prime RL has great user ergonomics. Users perform most commands with uv and configure setups in the .toml files. Prime RL also comes with agent skill files, allowing users to use AI agents more smoothly when working with the repo.

Regarding system design, we appreciate that Prime RL has an “orchestrator” actor to clarify who manages the trainer-generator dynamics. We love the hub of RL environments Environments Hub and look forward to it continuing to grow. Prime RL uses Torch Titan for training, vLLM for generation, and their in-house Prime Sandbox for hosting RL environments. Prime RL also supports PD disaggregation for the generator, which greatly boosts performance on agentic RL training. Overall, the codebase is lean and has most state-of-the-art techniques.

However, we also encountered some rough edges. Prime RL’s heavy reliance on uv assumes users have great familiarity with uv. We spent a lot of time compiling and re-installing flash attention 3 because we couldn’t figure out why uv uninstalled it. We encourage Prime Intellect to provide more resources to educate users on how uv works, in addition to the docs saying what uv commands not to execute.

Although Prime RL assumes it runs on a SLURM cluster, it assumes jobs run on bare metal instead of Pyxis. When we ran into CUDA library version mismatch for DeepGEMM libraries, running jobs in containers would solve the issue more easily.

We also had a lot of failed runs that turned out to be Prime Sandbox failures. We had a hard time parsing through Prime Sandbox’s error messages, which usually happens late into the run.

Errors include dangling sandboxes using up sandbox quota, out-of-resource errors, out-of-credit issues, many of which can be detected before launching a run. Prime Sandbox is still in beta, so we expect it to improve in the future.

We thank Prime Intellect for supporting the effort, including software and hardware resources. Prime Intellect provided training recipes based on this PR (Commit hash here).

slime

slime is known for its clean and minimal abstractions, and we believe it lives up to the hype. We particularly like slime’s hook abstractions. They allow us to write customized functions for rollout processing, reward functions, and metric logging utilities. The function signature and contracts are clear, and it’s a breeze to integrate and add new features during development.

We also want to highlight how friendly and helpful slime’s developer team is. Zilin, slime’s core maintainer, has offered guidance on configuration tuning and answered slime design questions.

Our main gripe is slime’s focus on co-located mode, and as a result sparse documentation of asynchronous modes. We figured the difference between async and fully async and the mechanisms of partial rollout mostly through trial and error. We hope slime can provide better documentation on those topics.

Modal

Modal has great API documentation: We referenced Modal’s Sandbox API extensively when implementing the sandbox integration in slime, and the documentation is clear and helpful. Modal services have also been relatively robust throughout our experiments. Under small scale, Modal has been reliable, and creating / terminating sandboxes is easy and responsive. The Modal team has also been very friendly and helpful. Modal offered support early on in our effort with API credits, and Modal’s slime integration team Peyton and Nan offered great advice and feedback on our experiment results.

The main challenges for Modal arose when we ran many concurrent sandboxes. As we reported above, we saw dead initialization errors and long-tail sandbox start-up latencies. We suspect this was our own misuse of the API rather than a hard limit in Modal, but we couldn’t pinpoint the root cause on our end during training. We later reached out to Modal and collaborated on identifying the issue. It turned out to be the resource limit on our account, and we were able to verify the stability of high sandbox concurrency after Modal raised our limits (Reproduction code here). We appreciate the responsiveness of Modal’s developer team, and we hope Modal can offer more sandbox observability tools and more documentation on how to scale sandboxes in the future.

History of Open-Source RL Frameworks

The release of DeepSeek R1 sparked the open-source community effort to reproduce the algorithm and infrastructure. One of the early efforts is OpenRLHF, a large-scale RL training framework that implemented PPO, REINFORCE++, and then GRPO. OpenRLHF profoundly influenced the subsequent RL framework developments. Numerous OpenRLHF maintainers later developed popular RL frameworks, including slime and verl. These frameworks created vibrant Chinese communities of RL training, which we believe positively contributed to recent advances of Chinese models. These open-source frameworks also enabled academic researchers to develop new algorithms and techniques, bringing RL research within reach of academia.

We thank Kaichao, who led vLLM’s early-stage RL integration in OpenRLHF, verl, etc., for sharing this history. We hope this shows that sincere collaboration and sharing, rather than hostile competitions and information hiding, propels science forward.

Conclusion

RL training system efficiency is a matter of queue health. Oversampling, early pruning, and partial rollout controls what enters the queue, adaptive sampling and policy staleness controls what leaves the queue, and in-flight weight updates allow trainer and generator to operate at different rates.

RL training is as much of an infrastructure problem as an algorithm one. In this post, we focused on systems, explaining how system and algorithm designs are intertwined, and grounding them with real-world experiments. We hope this ignites interest in RL infrastructure and bridges RL algorithm researchers with system engineers.

We plan to continue exploring the systems and infrastructure of RL training in our future articles. We will explore in depth techniques such as PD disaggregation and speculative decoding, and system challenges of large-scale RL training such as training Nemotron 3 Ultra. We are also looking into benchmarking sandbox infrastructure and analyzing model rollout traces.

If you’re interested in working on these topics, we’re hiring! Please contact us at letsgo@semianalysis.com . Attach your resume and ask us questions about this article.

For our last section, we present a TCO analysis of RL training, and we compare it to RL training platforms Tinker from Thinking Machines Lab. We quantify how much more cost efficient Tinker is, and we hypothesize the reasons behind it.

RL Training TCO Analysis

The Cost of Capability

Acknowledgements

The Three Actors

The RL Algorithm, Just Enough

From Synchronous to Asynchronous RL

The RL Environment and Sandbox

Serving at scale is also challenging: sandboxes scale along with the number of concurrent rollouts, and the number of live sandboxes fluctuates greatly as rollouts are pruned or completed.

The Throughput-Matching Framework

Quantifying Trainer and Generator Throughput

We further break down the latency into LLM inference time and RL environment interaction time. RL environment interaction involves sandbox startup latency and sandbox execution time.

Throughput Constraints

Multiple factors constrain the upper and lower bound of the throughput. The trainer consumption rate is samples per step / training step time.

On the generator side, we define the generator production rate as the number of concurrent rollouts / end-to-end latency.

Policy Staleness

The Moving Target: Model Capability and Behavior

Both model capability and behavior shift during RL training. Model capability improves, and behavior drifts. The constraints move with them, and the training system has to adapt to ensure efficiency.

Case Studies

We ran experiments to ground our theory, and we shared our experiences with open-source RL frameworks.

Long Model Response and Early Exploration Training Phase

Setup

Analysis

In this case study, we show how two factors influence the system efficiency: how the model produces long responses with long thinking traces, and how well the model learns the task.

Frequent Tool Use and Easy Task

Setup

Analysis

In this run, we see two factors set the system efficiency: model behavior drift and the curriculum being too easy for the model.

Sandbox Scaling Challenge

Setup

Training Stages

Analysis

Partial Rollout and Stateful Sandbox Design

Setup

Analysis

In slime, partial rollout marks straggler rollouts as aborted, saves them to a replay buffer, and puts them into the queue in future batches. This creates challenges for the sandbox:

Software User Experience

Prime RL

We also had a lot of failed runs that turned out to be Prime Sandbox failures. We had a hard time parsing through Prime Sandbox’s error messages, which usually happens late into the run.

We thank Prime Intellect for supporting the effort, including software and hardware resources. Prime Intellect provided training recipes based on this PR (Commit hash here).

slime

We also want to highlight how friendly and helpful slime’s developer team is. Zilin, slime’s core maintainer, has offered guidance on configuration tuning and answered slime design questions.

Modal

History of Open-Source RL Frameworks

Conclusion

If you’re interested in working on these topics, we’re hiring! Please contact us at letsgo@semianalysis.com . Attach your resume and ask us questions about this article.

RL Training TCO Analysis

Almost four years ago, we published that SMIC had started shipping 7 nm (N+1) chips. Now, SMIC is shipping its third-generation 7 nm (N+3) process in Huawei’s Kirin 9030, with a minimum metal pitch of 32.5 nm, about 10% tighter than the 36 nm minimum metal pitch shipping in Intel’s latest Panther Lake CPUs on 18A.

The headline is true, but incomplete cherry picked metric. N+3 reaches the density of TSMC N6 through aggressive DUV multi-patterning and design-technology co-optimization (DTCO), but it pays for that in complexity, efficiency and process control.

We found this and more in our reverse engineering and teardown where we cover SMIC’s N3 process technology, Huawei’s packaging, memory, architecture, and more. SemiAnalysis has been building a state-of-the-art teardown lab in Oregon capable of analyzing the world’s most advanced and important chips over the last year and half. We have already generated revenue on advanced datacenter chip teardowns including our recent reverse engineering of a major TSMC customer’s COUPE CPO optical engine + EIC 3D stack.

This is the first public report from the SemiAnalysis Teardown Engineering & Evaluation Lab, or STEEL for short. The lab is aggressively scaling up and out and we’re excited to announce it publicly. This is a bit of inconvenient timing for TechInsights as they are private equity owned and currently being sold while having enjoyed virtually no credible competition for decades. This has led to TechInsights underinvesting in CAPEX.

SemiAnalysis exceeds TechInsights in revenue despite no venture or private equity ownership and being founded only 6 years ago. Because we have no external investors and are founder led, we move faster, build faster, and we can release client chip teardowns for free regularly, while focusing on datacenter for our major clients.

This report will detail our teardown of the Kirin 9030 and our findings on SMIC’s N+3 process, the most advanced in China. For comparison, we’ll show our teardown of the MediaTek Helio G99, made on TSMC N6. Through this comparison, we can look at the effect of export controls – SMIC N+3 and TSMC N6 are comparable nodes, but one is heavily export-controlled, the other free to use the West’s most advanced equipment.

Here we see both China’s progress and constraints. SMIC N+3 reaches TSMC N6-class logic density, but it requires far more aggressive DUV multi-patterning, so it does not match N6 on process maturity or cost. The Kirin 9030 Pro performs similarly to three-year-old Android flagships, and trails far behind the current flagship SoCs from Apple, Qualcomm, MediaTek, and Samsung. The efficiency gap is even wider.

Export controls have not stopped Huawei and SMIC from shipping advanced silicon, but they have forced a different path. Without EUV, SMIC is leaning harder on DUV multi-patterning, DTCO, and increasingly complex integration. The roadmap continues forward through tighter design rules and backside power, but each step adds cost and process risk. Huawei’s τ scaling and LogicFolding show another path: stacking active logic and recovering density through advanced packaging and system-technology co-optimization (STCO).

Die Shot & Floorplan

To understand the Kirin 9030, we must first understand Huawei’s SoC history. HiSilicon is Huawei’s chip design arm, responsible for the Kirin smartphone SoCs, Kunpeng server CPUs, Ascend AI accelerators, and switch/router networking silicon.

Before export controls, Huawei was TSMC’s largest customer – the only customer on TSMC’s first EUV node, N7+, and among the first on N5, alongside Apple. That ended in late 2020. Huawei switched to Qualcomm SoCs in its flagship smartphones, though export controls limited them to 4G-only variants.

In late 2023, Huawei returned to in-house silicon with the Kirin 9000s, a successor to the Kirin 9000, fabricated on SMIC N+2 instead of TSMC N5. In the following years, they released the Kirin 9010 and 9020 on the same N+2 process. These chips used Huawei’s in-house TaiShan CPU cores and Maleoon GPU.

We have not torn down a Kirin 9020 ourselves, so the predecessor die shot is from Kurnal. The die shots show how Huawei spent its silicon budget: which functional blocks are where, and how their areas compare to the predecessor.

The total die area is nearly identical, but the 9030 uses that area more aggressively. A denser process lets Huawei fit an extra middle CPU core, more GPU and NPU cores, and larger caches into the same footprint.

In contrast, the Helio G99 is a much smaller, low-cost SoC, built for budget smartphones rather than a flagship device. While the Kirin 9030 is ~140 mm², the G99 is only ~29 mm², roughly one-fifth the area. The underlying TSMC process technology, however, is directly comparable as a baseline for analyzing SMIC’s.

Architecture & PPA

The Kirin 9030 is an evolutionary refresh, not a clean-sheet design. Its CPU, GPU and NPU cores carry over the 9020’s families, and the gains come from three levers: the SMIC N+2-to-N+3 process step, DTCO and floorplan work, and incremental microarchitecture. Area is where the first two show up, and the 9030 scales well here. Performance and efficiency are the harder test. Huawei’s design holds up better than its node would suggest, but the chip still trails, both because N+3 sits behind the leading-edge nodes and because its cores, while competent, remain a few generations behind the newest designs.

The new prime core is an incremental update. The main changes are a 10% frequency increase from 2.5 GHz to 2.75 GHz and a doubling in the L2 cache from 1 MiB to 2 MiB. Despite the increased cache, the core size decreased by 7.6%. Excluding the private L2 cache, the core size decreased by 21%. This is a large reduction for an incremental node.

Compared with the TaiShan New V120 core in the Kirin 9020, the Kirin 9030’s middle core is almost unchanged architecturally, yet each core shrinks by ~22%. Most of that comes from the move from N+2 to N+3, with layout likely accounting for the rest.

Visually, the most noticeable change is the increase from 3 middle cores to 4. There is also a 20% increase in the shared L3 cache of the big cluster. This helps to improve multi-core performance without sacrificing much in terms of area.

Even with each core shrinking, the big CPU cluster’s total area is essentially unchanged. The per-core savings went back into an additional middle core and larger caches.

The tiny cores shrank less than the prime core (excluding its L2 cache) and less than the middle cores. This is likely because fixed overhead is a larger share on a small core. We cannot resolve any architectural changes from the die shot alone, but the per-clock and efficiency gains shown below point to more than pure process and layout scaling. The area reduction was offset by a doubling of the shared L2 cache from 2 MiB to 4 MiB, leaving the total tiny CPU cluster area slightly larger.

Area is the easiest improvement to see from a die shot, but it is only one part of PPA (power, performance, area). For modern logic, power and performance matter just as much, and often more. Since Dennard scaling broke down in the mid-2000s, voltage and frequency have not scaled in step with transistor dimensions, so each node has had to fight harder for gains in performance and efficiency.

The starkest comparison is not Kirin 9020 versus Kirin 9030 Pro. Apple’s efficiency cores run circles around Huawei’s prime core. Apple’s low-power core delivers 20% higher integer performance while drawing only 1 W, compared with 4.5 W for Huawei’s prime core. N+3 matches TSMC N6, but N6 is several generations old. Apple and Qualcomm build on N4 and N3P, which are denser and sit on a better voltage-frequency curve, giving them a larger transistor budget and more performance per watt.

The 9030’s own cores did improve. The middle and tiny cores gained 17% and 14% in per-clock integer performance over the 9020, with floating-point flat on the middle core and up 11% on the tiny. The tiny core improves cleanly, with performance rising while power falls and efficiency increases by 45% in integer and 24% in floating point. The middle core is mixed: integer performance rises but power rises faster, cutting integer efficiency by 7%, while lower power lifts floating-point efficiency 16%.

Per-clock gains at the same or lower frequency are microarchitectural, so the cores are tuned, not just shrunk. Both also failed to hold their rated maximum frequencies, pointing to thermal, power, or stability limits. Per clock, the middle core sits around Arm Cortex-A720 and the tiny core near the Cortex-A520; absolute performance trails because Huawei clocks them much lower.

The prime core is roughly Cortex-X2 class per clock, a 2021 design. Apple’s 2020 M1 Firestorm core is still 35% higher per clock and 57% faster in absolute integer performance at a similar 4.5 W. The current leading edge is further ahead again: the Apple M5 P-core is 60% higher per clock and 2.7× faster, the Arm C1 Ultra 45% higher and 2× faster.

Matching older high-end cores per clock is a genuine design achievement. What Huawei cannot match is the voltage-frequency curve and transistor budget of leading-edge nodes, which let Apple, Qualcomm and others spend more transistors in the same area on wider cores, larger caches and deeper buffers while running at lower voltage.

Huawei’s LogicFolding roadmap is one answer, stacking active logic to recover density and shorten signal paths. We return to it later.

The GPU compute units (CUs) changed more visibly than the CPU cores, moving to a more rectangular layout for both the arithmetic logic unit (ALU) clusters and the CU overall. Even with ray-tracing support added, a CU shrank ~28%.

However, that shrink is offset by the increase from 4 to 6 CUs and the area outside the CUs grew 33%. Overall, the GPU cluster is larger by ~10%.

The GPU is where Huawei makes its biggest gains. The Maleoon 935 is not competitive with current flagships, but it is a large step up from the 920 and reaches older-flagship territory. In 3DMark it is 70% faster in Wild Life Extreme (WLE) and 79% faster in Steel Nomad Light (SNL) than the 920; with 11% higher clocks and 50% more CUs, the ~67% theoretical uplift roughly matches WLE and is beaten by SNL.

It edges ahead of the Snapdragon 8+ Gen 1 in WLE and SNL, and the Dimensity 9200 and Apple A16 in WLE, but stays far behind newer parts: the Snapdragon 8 Elite Gen 5 and Dimensity 9500 are ~2.4–2.6× faster in WLE and ~3.2× faster in SNL.

The Maleoon 935 is Huawei’s first GPU with hardware-accelerated ray tracing; there it lands slightly ahead of the Exynos 2200, and on par with the Apple A16, with current flagships up to 3.7× faster.

The Neural Processing Unit (NPU) saw the largest structural changes of any block, moving from a Lite and a Tiny core in the Kirin 9020 to a Lite and two Tiny cores in the Kirin 9030. Both core types also show significant layout changes.

This is a reversal in Huawei’s NPU design. The Kirin 9000 5G, its last flagship chip on TSMC N5, used two Lite and one Tiny core. The series of SoCs on SMIC N+2 moved to one Lite and one Tiny core, likely for area savings. With the Kirin 9030, Huawei has shifted back toward a larger multi-core NPU cluster, but with the additional area going to a Tiny core rather than a Lite core.

We’re diving deep into the most advanced datacenter and AI hardware hitting the market. To learn more about what’s in the pipeline or to commission a custom teardown, contact sales@semianalysis.com. Interested in joining us on this ride and think you can be a difference maker? Check out our Careers page.

Before diving into the process stack, the package and memory are worth separating from the SoC itself.

Memory

The Pro variant of the Kirin 9030 carries 12 GB of Samsung DRAM, with two stacks of four dies each. The dies were identified as the K4L2E165YD, a 12 Gb LPDDR5X-9600 device fabricated on Samsung’s 1a node, the fourth generation of its 10 nm-class DRAM after 1x, 1y and 1z. 1a has shipped in volume since 2022, so this is current memory rather than older-node inventory.

The 16 GB Pro Max variants we obtained were found with both CXMT and Samsung packages. The CXMT package is marked CXDD7JEDM, with two stacks of four dies, packaged in week 45 of 2025. The inferred die dimensions from X-ray computed tomography (CT) are consistent with a known density of ~0.3 Gib/mm² for the CXMT G4 process, roughly equivalent to other manufacturers’ 1z processes.

Packaging

The Kirin 9030 uses a typical integrated package-on-package (iPoP) stack: multiple DRAM dies in a memory package sit above an organic redistribution layer (RDL) interposer, which sits above the SoC and package substrate. The full package is then mounted to the printed circuit board (PCB) through ball-grid array (BGA) solder bumps.

The memory package substrate is a thin bismaleimide-triazine (BT) laminate carrying the LPDDR5X stack. The organic RDL interposer over the SoC routes the PoP signals around the die and carries possible dummy thermal copper pillars. The package substrate, a thicker Ajinomoto Build-up Film (ABF) build-up over a BT core, fans the flip-chip bumps out to BGA pitch and embeds the power planes.

The whole stack is organic. The only silicon is the SoC and the LPDDR5X dies; there is no silicon interposer. Keeping it all-organic brings the package’s coefficient of thermal expansion (CTE) close to the PCB’s, reducing board-level warpage, and avoids the cost of a silicon interposer the SoC’s bandwidth does not need.

In an iPoP stack, the memory package connects to the organic RDL interposer through an array of solder bumps. Underfill fills the gap around those bumps, adding stiffness and protecting the joints from mechanical stress. The Pro and Pro Max variants differ here, which we cover behind the paywall.

Process

The die shot and architecture tell us how Huawei allocated its silicon budget. The process tells us what SMIC can manufacture. We use the Helio G99 as the process reference for TSMC N6. Both SMIC N+3 and TSMC N6 are evolutions of previous 7 nm-class nodes.

We used targeted TEM cross-sections through logic and memory regions, imaged in both fin-cut and gate-cut directions. Each cross-section caption gives its horizontal field width (HFW), the real width of the imaged area. We start at the transistor fins, then move up through standard cells, local interconnect, and SRAM.

SMIC has not overtaken Intel or TSMC. It uses aggressive DUV scaling and DTCO to reach N6-class density, but that density doesn’t translate into comparable performance and efficiency, for two reasons: the node gap to leading-edge nodes, and Huawei’s core designs.

Fin Profile

One of the most important knobs in a FinFET process is the fin profile: the shape of an individual fin and the channel where current passes from source to drain. The ideal fin is tall, narrow, and nearly vertical. A taller fin increases effective channel width, while a narrower fin improves electrostatic control by thinning the body the gate must control. Push either too far, and the process pays for it: weaker drive current, fragile fins, taper, footing and line-edge variation that hit yield and device variability.

The Intel 22 nm, 14 nm, and 10 nm fin cross-sections show how FinFET nodes have improved over time. 22 nm fins were a first-generation structure, relatively short, wide and strongly tapered. The shape limits current density and reduces gate control uniformity across the height of the fin. At 14 nm and 10 nm, Intel pushed the fins taller and narrower while also making the sidewalls more vertical. Rather than shrinking the device, these changes increase the effective channel width per fin and improve electrostatic control. The trade-off is that taller fins at tighter pitches are much more difficult to manufacture.

Now, let’s compare the Helio G99 on TSMC N6 with the Kirin 9030 on SMIC N+3. Both processes are in the same class, with fin pitch of 30-32 nm on N+3 and 34 nm in our N6 cross-section. The pitch for N6 is especially interesting as N7’s HD library is generally listed with a 33 nm fin pitch, and N6 did not shrink pitches directly. Its density gains came from DTCO instead of tighter pitches. The 34 nm pitch was stable across our sampled region and serves more as a comparison against the SMIC N+3 we have not investigated further.

Pinning down N+3’s fin patterning scheme takes more than one core unit. The CPU cores show a dense ~32 nm pitch, with the pitch between N-P fin pairs alternating between 78 and 88 nm. Logic alone may be consistent with dual-pitch mandrels of 120 and 110 nm, but this is a complex and unusual approach. Combining the pitch from the the 8T SRAM, which has more complex repeat unit, with the CPU core sequence allows us to reverse engineer the patterning steps with more confidence.

As both the logic and SRAM should share the same base grid, a single CD mandrel lithography pattern with 128 nm pitch undergoing SAQP produces a die-wide ~32 nm grid (128 nm/4), which supports the pitch sequencing seen in both logic and SRAM cells.

In the sampled cross-sections, N+3 shows a taller, narrower, higher-aspect-ratio fin than N6. The measured fin aspect ratio is ~9.5:1 on N+3 versus 7.8:1 on N6. N+3 also shows less top rounding, with an estimated radius of ~2 nm, compared with 2.8 nm on N6. Even though the fin widths differ, the ratio of top rounding to fin width tells the same story, with N+3 at 0.37 and N6 at 0.44. In a geometric sense, lower is better; a perfectly rectangular fin would have no top-rounding penalty.

These are single-digit-nanometer features measured from a handful of cuts, so treat the absolute numbers as approximate. The important result is the relative gap: N+3’s fins are consistently taller, narrower and less rounded than N6’s.

Standard Cell

A standard cell is the basic building block of chip layout: a fixed-height row pairing one NMOS and one PMOS transistor that share a gate, tiled in a grid to build logic blocks. The key dimensions are contacted gate pitch (CGP), cell height (CH), fin count, and the lower-metal routing grid.

To measure density, we use the Bohr metric: a weighted average of NAND2 gate area (60%) and scan flip-flop area (40%). This represents a realistic mix of combinational and sequential logic. This metric has its limitations, especially for complex cell layouts like TSMC’s FinFLEX, which alternates cells with different fin counts. Even so, it is the best metric for a pure process-level comparison.

Another important measurement is the fin pitch; it refers to the distance between two fins of the same transistor. In a FinFET process, multiple fins are used in each transistor to increase the drive current and thus performance.

TSMC N6 ships both a high-density (HD) library with 2 PMOS and 2 NMOS fins per cell, and a high-performance (HP) library with 3 of each. More fins under the shared gate mean more effective channel width. HP cells switch harder at the cost of area. Designers mix the two on a die, primarily spending HP cells on timing-critical paths, and matching their PPA targets.

In the Cortex-A55 core of the Helio G99, we found a cell height of 240 nm for the HD cell. MediaTek has used HD cells in the G99 to minimize die size and thus cost. As an SoC for budget smartphones at ~$100, this is essential.

By contrast, we found only one library in the Kirin 9030, with 2 NMOS and 2 PMOS fins. This suggests a narrower library strategy than TSMC N6, where both HD and HP libraries are widely used. This likely reflects the smaller customer base and the more constrained domestic design and electronic design automation (EDA) ecosystem.

In all three CPU cores of the Kirin 9030, we found cell heights of 228 nm, 5% smaller than on N6. This is also a reduction of 9.5% over SMIC N+2’s cell height of 252 nm.

SMIC N+3 and TSMC N6’s HD library both feature a CGP of 57 nm. For SMIC, this is a 9.5% shrink over N+2.

In the past, CGP and cell height alone may have been enough to compare transistor density. Now, however, we must consider scaling boosters and DTCO as well. SMIC’s density gain does not come from EUV. It comes from using every available DTCO booster aggressively.

First is fin depopulation: reducing the number of NMOS and PMOS fins in each cell. The first FinFET nodes started with 3 or 4 fins for each transistor. SMIC N+3 and TSMC N6 HD both use only 2 fins per transistor, trading drive strength for density.

Next is contact over active gate (COAG). By landing the gate contact directly over the active gate, instead of out over the isolation region, the cell height drops. N+3 integrates COAG while N6 does not. Our N+3 gate-cut cross-sections indicate COAG, with the gate contact sitting over the active region, while N6 shows an off-gate contact.

Last is single diffusion break (SDB). Diffusion breaks are inserted between cells in the same row to provide electrical isolation, but they also introduce local layout effects (LLE), layout-dependent shifts in electrical characteristics. In the past, a double diffusion break was used, consuming the space of two CGPs. SMIC N+3 and TSMC N6 instead use SDB, saving area but increasing LLE sensitivity. This must be controlled at the process level and accurately modeled in the process design kit (PDK) so that EDA tools can account for it.

Overall, SMIC N+3 has a transistor density of 113.4 MTr/mm², slightly above TSMC N6 at 107.7 MTr/mm². Even without EUV, SMIC has achieved density beyond TSMC’s mature N6 node which utilizes EUV.

Metal Stack

The smallest critical dimension in the teardown is M0; SMIC N+3 uses a 32.5 nm local metal pitch. That is smaller than the 36 nm M0 pitch on Intel 18A in Panther Lake. However, this does not mean that SMIC has a better process than Intel 18A or TSMC N3P. M0 is a local intra-cell routing layer. Its usefulness depends on the full interconnect stack: M1 and M2 pitch, track count, via and line resistance, design rules, mask count, overlay control, and routing flexibility.

The 32.5 nm M0 is consistent with self-aligned quadruple patterning (SAQP), whose four-population line-width loading we read coarsely as alternating widths of 21.5 to 24 nm; M1 and M2, at 38 and 40 nm, are consistent with self-aligned double patterning (SADP), a single A/B split. On TSMC N6, M0, M2, and M3 sit at a relaxed ~40 nm and are consistent with SADP-class double patterning, with no need for quadruple patterning. That said, we measure M2 for example at ~43 nm, likely inflated by sparse routing. We do not assign any specific layer to EUV from our cross-sections; the distinction we can draw is double versus quadruple patterning, not lithography wavelength.

Transistor-level density in the front-end-of-line (FEOL) sets an upper bound, but the design is ultimately limited by what the interconnect stack can route. The lowest metals are the most important for standard-cell density, but the semi-global and global layers determine how usable that density is at the block and chip level.

Two axes are commonly used for chip cross-sections: the fin-cut and the gate-cut. The micrograph above is a fin-cut and shows metals 0 through 3. This axis lets us see the even-numbered metals, with M0 right above the fins.

There are two kinds of M0 lines. The first are the power rails; these are wide wires for the VDD and VSS running horizontally at the top and bottom edges of each standard cell. The wide wires measure 55 nm across, more than double the other M0 lines. Their width minimizes resistance and reduces IR drop. The second kind are intra-cell wires, short segments within the cell that connect terminals to M1. These have alternating widths between 21.5 and 24 nm.

The M0 pitch is 32.5 nm, a 19% reduction versus N+2 and N6. At this pitch, DUV patterning requires more aggressive multi-patterning, increasing mask count, overlay sensitivity, process complexity and cost.

M0 is below what a single DUV-defined spacer (SADP) can resolve, so SMIC cascades a second spacer step (SAQP). The cross-section reflects the cost: the M0 trenches are visibly more re-entrant (narrower at the bottom than the top) than M1 or M2 on the same chip and carry a bright barrier-rich foot where the trench meets the etch-stop layer. That shape is partly the intended damascene profile, as a slightly narrow bottom helps void-free copper fill, but its magnitude at M0 is driven by the tight pitch and the higher trench aspect ratio.

Intel 18A supports an M0 pitch of 32 nm, although Panther Lake has only shipped with a looser 36 nm pitch. This is due to Intel’s heavy usage of HP libraries. Among leading-edge nodes, 18A has the loosest M0 pitches due to PowerVia. With power routing moving to the backside, congestion is reduced, and the entire front-side metal stack can be used for signal routing.

M2 is the first true inter-cell routing layer. It runs horizontally like M0 but spans across multiple cells to carry block-level signals. The M2 pitch sets the cell’s track height – the number of M2 tracks that fit between the VDD and VSS rails, defining what the library calls a 6-track or 7.5-track cell. This layer is the most important, limiting the routing of entire blocks.

SMIC N+3 features a 5.7-track cell. The M2 pitch is 40 nm, a 5% decrease over N+2 and the same as N6. This shrink keeps the pitch at the edge of what is possible with double patterning. Future nodes will need to increase the number of masks for M2 as reducing the number of tracks is much harder due to the limitations in routing.

The micrograph above is in the perpendicular direction, the gate-cut, and shows metals 0 through 4. This allows us to see and measure the odd-numbered vertical metal layers.

The M1 pitch is 38 nm, 9.5% less than N+2 and 33% less than N6. The M1-to-gate ratio matters because it sets local routing flexibility. N+2 and N+3 use a 3:2 ratio, while N6 uses a 1:1 ratio, explaining the huge differences in M1 pitch. The more M1 lines there are compared with the gates, the more flexibility there is for power and signal crossing within the cell. Routing flexibility enables more complex and better cells. Clean fractional ratios are also preferred as a grid is periodic and improves layouts. The 3:2 ratio gives SMIC more local routing flexibility than a strict 1:1 grid, but it also complicates layout and patterning. This is a DTCO choice, with SMIC increasing process complexity to recover density and routability without EUV.

This 3:2 ratio is not very popular in the leading-edge nodes. TSMC has only used it on N7+, the N5 family, and the short-lived N3(B). They have switched back to a 1:1 ratio for N3E. Intel has only used it on the 10 nm/Intel 7 family, with Intel 4, 3 and 18A all using a 1:1 ratio. Samsung is the only one still using a 3:2 ratio at the leading edge, using it in the SF4 and SF3 families. It remains to be seen if SMIC will remain at a 3:2 ratio or move to a 1:1 ratio with its future nodes.

The industry is still actively exploring these local-routing ratios. At VLSI 2026, imec will be presenting work on even higher ratios, including a 2:1 scheme that can reduce area by up to 14%. We will be covering the conference in a future newsletter article.

The final local interconnect layer for N+3 is M3, with a pitch of 44 nm. The M3 pitch is the same as on N+2 and 10% larger than on N6.

The semi-global layers carry the majority of block-level signal routing. They have a coarser pitch than the lower local layers. On leading-edge nodes, they are designed to sit at the limit of DUV single patterning.

M4 through M11 pitches were found divided between 80–82 nm (M4–M6), 128 nm (M7–M10), and 148 nm (M11). Given limited sampling, it is possible these are divided further in dense routing areas. At the top are two giant metal layers, M12 and M13. These have kept the same pitches as N+2 at 1920 nm and 4600 nm respectively.

While the lower layers’ pitches are generally fixed by the process and library, the upper layers vary much more in pitch and count, depending on the design. Even two smartphone SoCs on the same process can have wildly different metal stacks. The Helio G99 carries fewer routing layers, reaching coarse metal pitches of 850 nm by M9, while the larger and higher-performance Kirin 9030 keeps fine pitches until M11.

SRAM

At the leading edge, SRAM is much more difficult to scale than logic. TSMC’s latest nodes have seen little to no bitcell scaling, while logic still has more DTCO levers to pull.

While looking for other logic libraries in the GPU compute units, we stumbled upon the SRAM. The most common type of SRAM has 6 transistors (6T), but this cell had 8 transistors (8T) instead.

8T SRAM adds two transistors to form a dedicated read port. Unlike a 6T cell, where reading disturbs the storage, the decoupled read port removes read-disturb, improving read stability and letting the cell be pushed harder for performance.

At first glance, the cut looked like an unusual logic library, with each cell row having 3 fins of one polarity and 5 fins of another. The rows also alternated in orientation.

Energy-dispersive X-ray spectroscopy (EDS) resolved our confusion. The cut had not landed on the GPU logic, but on the SRAM macro beside it. The unusual fin pattern was due to the SRAM library. We return to EDS in the process flow analysis behind the paywall.

SRAM libraries are not like traditional logic libraries. Due to the unequal number of PMOS and NMOS transistors, they require specialized rules and layout libraries. They do not need the flexibility of logic libraries, so they are hyper-optimized for one purpose: dense, reliable memory.

The SRAM cell we found is a 1:2:2-2:2 cell. This means there is 1 fin per pull-up (PU) PMOS transistor, and 2 fins per pull-down (PD) and pass-gate (PG) NMOS transistor. These 2 PU, 2 PD and 2 PG transistors would usually form a single 6T high-current cell (HCC). An 8T HCC adds a read-pull-down (RPD) and a read-pass-gate (RPG) NMOS transistor, each with two fins.

We measured a cell height of 406 nm, which brings the bitcell size to 0.0463 µm². That is a theoretical peak density of 21.6 Mib/mm². We estimate that a 6T HCC would have a cell height of 292 nm and a size of 0.0337 µm². This is ~12% larger than a 6T HCC on Intel 3 and 4.

We also estimate the 6T high-density cell (HDC) to have a cell height of 228 nm and a size of 0.0260 µm². This is coincidentally the same as the logic standard-cell height measured earlier. The estimate puts the cell near Samsung 7LPP/5LPP and slightly below TSMC N7/N6. That is a theoretical peak density of 38.5 Mib/mm². 6T HDC is arguably the most important cell as it is used for the largest caches in a chip, the L3 caches and system-level cache (SLC).

Both the Kirin 9020 and 9030 have split the SLC into 4 banks to raise total SLC bandwidth. In the Kirin 9030, the SLC increased from 2 MiB to 3 MiB per bank. Correspondingly, the number of arrays within the bank also increased by 50%, from 16 to 24. Each array can store 128 KiB and forms an orderly pattern on the die shot.

From the Kirin 9020 to the Kirin 9030, the area of a 128 KiB SLC array decreased from 0.0477 mm² to 0.0392 mm², an 18% shrink. The achieved density is 25.5 Mib/mm², 66% of the theoretical maximum.

While the SLC was quite similar across both chips, the L3 has seen some major changes, particularly in terms of its layout. The total capacity also went up from 10 MiB to 12 MiB. Much like the SLC, the L3 is also split into 4 banks.

In the Kirin 9020, an L3 bank consisted of 16× 128 KiB arrays and 16× 32 KiB arrays. However, an L3 bank in the Kirin 9030 instead consists of 48× 64 KiB arrays.

In the Kirin 9020 L3, a 128 KiB array was 0.0513 mm² and a 32 KiB array was 0.0154 mm². The size of the 128 KiB array is different on the L3 and SLC as the assist circuitry for the two arrays differs depending on their purpose.

In the Kirin 9030 L3, a 64 KiB array is 0.0210 mm². Although not a like-for-like comparison, normalized for capacity, it is 18% smaller than the 9020’s 128 KiB L3 array and 31% smaller than its 32 KiB L3 array. The achieved density is slightly lower than the SLC, at 23.8 Mib/mm², 62% of the theoretical maximum.

Unlike the L3 and SLC, the prime cores’ private L2 cache uses a 2-bank design. As the prime cores’ L2 is latency-critical, it likely uses 6T HCC instead of 6T HDC. The 9020 has 16 arrays in each bank while the 9030 has 32. Each array has a capacity of 32 KiB.

A 32 KiB array in the L2 shrank from 0.0171 mm² to 0.0142 mm², ~17% smaller. The density is 17.6 Mib/mm², ~59% of the theoretical maximum for 6T HCC.

SRAM scaled well from N+2 to N+3, shrinking by ~19%, close to the theoretical logic shrink. The caveat is that N+2’s bitcells were unusually large, bigger than comparable 7 nm-class nodes, so part of the gain is catch-up rather than true scaling.

With the insights from STEEL’s teardowns, we will be doing a deep dive into SRAM in a future newsletter article.

Everything above came out of a single STEEL teardown: die annotation, block-level area analysis and TEM cross-sections through logic and SRAM. We’re diving deep into the most advanced datacenter and AI hardware hitting the market. To learn more about what’s in the pipeline or to commission a custom teardown, contact sales@semianalysis.com.

Future Roadmap

The same cross-sections that pin down N+3 also show where SMIC can go next. Although N+3 is already close to the practical limits of DUV multi-patterning in several layers, SMIC still has a few scaling levers left.

A theoretical N+4 would likely start with cell height. N+3 uses 5 M0 tracks between its power rails. Moving to 4 M0 tracks, as on SMIC N+2 and TSMC N6, could reduce cell height by roughly 15%. The routing grid is only one side of the shrink; the front end also must fit into the smaller cell.

One possible FEOL lever is reducing the p-to-n isolation spacing from two diffusion grid units to one. Intel used this scaling booster on Intel 4, and TSMC did so on its N3 family. This path trades layout flexibility for density. Fewer M0 tracks reduce local routing resources, while tighter p-to-n spacing raises integration and design-rule difficulty.

M2 is also constrained by the cell height shrink. For SMIC to maintain a ~5.7-track cell, M2 would need to move toward ~35 nm. That would move another layer into SAQP territory.

SMIC could also reduce the CGP from 57 nm to 54 nm. Intel reached a similar CGP on Intel 10 nm/Intel 7 without EUV. The local interconnect is also tougher. If SMIC keeps the 3:2 M1-to-gate ratio, M1 would need to shrink to 36 nm and would likely require SAQP as well. If SMIC moves to a 1:1 ratio, M1 could relax to 54 nm, but it would give up routing flexibility.

Under this theoretical path, we estimate that SMIC N+4 could reach a cell height of 198 nm and a CGP of 54 nm, implying a Bohr density of 137.8 MTr/mm², on par with TSMC N5 or Samsung SF4. However, the difficulty is cumulative. Each step is individually plausible, but together they make N+4 harder than the transition from N+2 to N+3. It will likely take longer, cost more, and carry less process margin.

A theoretical N+5 would require a larger integration shift. One possible path is backside contacts (BSCon), moving power routing and source/drain contacts to the backside, which would reduce front-side routing pressure and enable another reduction in cell height.

The front-side metal pitches could relax to reduce the process complexity. M0 would likely relax slightly to ~34 nm, and M2 and M4 pitches could relax further. CGP is unlikely to shrink much further. Even with EUV, 48 nm has been the practical limit for yield and process control.

This approach would allow N+5’s cell height to fall to 170 nm and its CGP to 53 nm. This implies a Bohr density of 163.6 MTr/mm², on par with Intel 18A’s HP library. However, this would not make N+5 cost-competitive with the leading edge. It would reach a similar density through a much more expensive route. The integration difficulty rises sharply, with new process flows for backside alignment, wafer thinning, contact reveal, and backside metallization.

Past this point, standard density and interconnect scaling become increasingly unattractive. That is where Huawei’s roadmap stops looking like a normal foundry roadmap and starts looking like a packaging roadmap.

Huawei’s τ Scaling Law

At ISCAS 2026, Huawei unveiled its tau (τ) scaling law, reframing process scaling in the time domain. τ is the time cost of data movement and processing: switching delays in transistors, RC signal propagation delays in circuits, compute, memory, and networking latency. Outside Huawei’s terminology, this is called system-technology co-optimization.

This is Huawei’s answer to its lack of EUV lithography. Without EUV, planar density cannot keep pace with TSMC, Intel, or Samsung. If transistor density cannot shrink further, Huawei’s alternative is to shorten wires, reduce buffering, and stack logic vertically.

“LogicFolding”, Huawei’s implementation of this new scaling idea, is, in practice, an aggressive 3D stacking approach. AMD V-Cache places SRAM above or below a CPU die. AMD’s MI350X places active interposer dies (AIDs) underneath accelerator and compute dies (XCDs), with the AIDs handling cache, IO interfaces, the network-on-chip (NoC) and embedded metal-insulator-metal (MIM) capacitors. With LogicFolding, parts of the same logic block are split across multiple active dies bonded face-to-face at ultra-fine pitch. This allows Huawei to shorten some critical paths and reduce buffer overhead, not merely add cache capacity or offload the IO and interconnect.

Shortening wires is where the higher clocks come from. A large share of a modern core’s delay and energy budget goes into driving long interconnects and the repeater buffers along them. LogicFolding distributes a block’s critical-path gates across multiple stacked tiers bonded at a very fine pitch, so the bond interface behaves like an additional metal layer and the longest paths get shorter. That is how Huawei expects to recover frequency and efficiency it cannot get from the process alone.

Huawei’s roadmap shows its intent. Prime core frequency is targeted to rise from 2.75 GHz in the Kirin 9030 to roughly 5 GHz by 2031, far beyond what planar scaling alone could deliver. Prime cores with 3.1 and 3.39 GHz clocks are being tested in its labs, although their power consumption is unknown. Beyond that, chips are in the design, simulation or pathfinding phase, meaning the frequencies are targets. However, the direction matters more: LogicFolding also helps with performance, not just density.

The catch is that Huawei’s density claim is not directly comparable to foundry densities. A stacked design can report more transistors per package footprint by adding active layers, even if each patterned die remains well behind TSMC or Intel in front-end density. This is how Huawei can claim to reach foundry 14A-equivalent density by 2031.

This is not a like-for-like foundry comparison, with Huawei using stacked logic and measuring density per package footprint. On a normalized Bohr-density basis, SMIC N+3 is ~114 MTr/mm², 38% less than Intel 18A’s HD library. Huawei’s 3D roadmap closes the gap by stacking active logic, reaching 215 MTr/mm² by 2030. In 2031, the roadmap density jumps to 295 MTr/mm², implying either a third active layer, partial EUV insertion or aggressive planar DUV scaling.

Huawei’s methodology makes other foundries look much denser as well. Applying it to AMD’s MI450X with an N2 top die on an N3P base die yields a theoretical density of 460.2 MTr/mm² in 2026, compared with Huawei’s 295 MTr/mm² in 2031.

This Kirin 9030 does not use LogicFolding, remaining in a conventional mobile SoC package. Instead, it forms the baseline for how far Huawei and SMIC can push planar scaling. Future teardowns of Kirin and Ascend chips will show both planar logic density and Huawei’s hybrid bonding solutions.

Export Controls & Future Scaling

Export controls changed China’s optimization problem rather than ending it. EUV restrictions raised the cost and complexity of leading-edge manufacturing without freezing it. SMIC reaches N6-class logic density through DUV immersion, SAQP and DTCO, while Huawei shifts more of the burden onto architecture, packaging and system-level integration.

Future nodes will be tougher. N+3 still had room to tighten local metals and reduce cell height and CGP. Further scaling without EUV leaves fewer levers. More aggressive multi-patterning adds masks and overlay error. SMIC can keep pushing DUV, but each step will get more expensive and less forgiving.

The design side is just as critical. Huawei had domestic EDA tools and flows before the Kirin 9030, with the Kirin 9000s, 9010 and 9020 making that clear. Huawei was able to ship multiple consumer SoCs on SMIC N+2 and N+3 while cut off from the Western EDA stack.

US export controls restricted EDA tools for advanced chips in 2022 but did not target tools for more mature chips. In 2025, the US government briefly placed much broader restrictions on EDA software from Synopsys, Cadence, and others, before lifting them less than two months later as part of a trade deal tied to rare earths. Huawei has been unable to access those tools because it remains on a US trade blacklist.

That forced Huawei, SMIC and Chinese academic institutions to build their own tools and flows. Researchers at Peking University recently announced a prototype EDA tool for Huawei’s LogicFolding architecture, which requires a new flow to handle the multilayer layout and floorplan. This is not the same as replacing the full Synopsys or Cadence stack, but it shows where domestic EDA is headed: toward tighter co-optimization between architecture, process and packaging.

These advances are also diffusing into the Chinese ecosystem. SMIC is licensing its N+2 and N+3 processes to HLMC/Hua Hong at the government’s direction rather than by choice. If the same process learning feeds into Ascend accelerators for AI training and inference, the choke point shifts from one named fab to an ecosystem. Alibaba’s T-Head silicon arm and Cambricon, a Chinese AI chip designer that is expected to supply ByteDance, could also be major beneficiaries. Sanctions aimed at SMIC alone become less effective once the manufacturing knowledge has spread to other fabs and design houses.

China is not closing the gap with Intel, Samsung and TSMC. The teardown shows the opposite in several places: no EUV, no backside power, higher process complexity, and visible trade-offs.

But China is still advancing. If domestic chips become good enough for phones, inference, networking and security-sensitive workloads, they can matter strategically without matching TSMC at the leading edge.

Behind the paywall, we show what else STEEL can do, with material and process flow analysis of SMIC N+3, and analysis of the Kirin 9030 package.

We’re diving deep into the most advanced datacenter and AI hardware hitting the market. To learn more about what’s in the pipeline, access the full Kirin 9030 and SMIC N+3 analysis or to commission a custom teardown, contact sales@semianalysis.com.

Die Shot & Floorplan

Architecture & PPA

Even with each core shrinking, the big CPU cluster’s total area is essentially unchanged. The per-core savings went back into an additional middle core and larger caches.

Huawei’s LogicFolding roadmap is one answer, stacking active logic to recover density and shorten signal paths. We return to it later.

However, that shrink is offset by the increase from 4 to 6 CUs and the area outside the CUs grew 33%. Overall, the GPU cluster is larger by ~10%.

Before diving into the process stack, the package and memory are worth separating from the SoC itself.

Memory

Packaging

Process

Fin Profile

Standard Cell

In all three CPU cores of the Kirin 9030, we found cell heights of 228 nm, 5% smaller than on N6. This is also a reduction of 9.5% over SMIC N+2’s cell height of 252 nm.

SMIC N+3 and TSMC N6’s HD library both feature a CGP of 57 nm. For SMIC, this is a 9.5% shrink over N+2.

Overall, SMIC N+3 has a transistor density of 113.4 MTr/mm², slightly above TSMC N6 at 107.7 MTr/mm². Even without EUV, SMIC has achieved density beyond TSMC’s mature N6 node which utilizes EUV.

Metal Stack

The micrograph above is in the perpendicular direction, the gate-cut, and shows metals 0 through 4. This allows us to see and measure the odd-numbered vertical metal layers.

The final local interconnect layer for N+3 is M3, with a pitch of 44 nm. The M3 pitch is the same as on N+2 and 10% larger than on N6.

SRAM

At the leading edge, SRAM is much more difficult to scale than logic. TSMC’s latest nodes have seen little to no bitcell scaling, while logic still has more DTCO levers to pull.

While looking for other logic libraries in the GPU compute units, we stumbled upon the SRAM. The most common type of SRAM has 6 transistors (6T), but this cell had 8 transistors (8T) instead.

At first glance, the cut looked like an unusual logic library, with each cell row having 3 fins of one polarity and 5 fins of another. The rows also alternated in orientation.

From the Kirin 9020 to the Kirin 9030, the area of a 128 KiB SLC array decreased from 0.0477 mm² to 0.0392 mm², an 18% shrink. The achieved density is 25.5 Mib/mm², 66% of the theoretical maximum.

In the Kirin 9020, an L3 bank consisted of 16× 128 KiB arrays and 16× 32 KiB arrays. However, an L3 bank in the Kirin 9030 instead consists of 48× 64 KiB arrays.

A 32 KiB array in the L2 shrank from 0.0171 mm² to 0.0142 mm², ~17% smaller. The density is 17.6 Mib/mm², ~59% of the theoretical maximum for 6T HCC.

With the insights from STEEL’s teardowns, we will be doing a deep dive into SRAM in a future newsletter article.

Future Roadmap

M2 is also constrained by the cell height shrink. For SMIC to maintain a ~5.7-track cell, M2 would need to move toward ~35 nm. That would move another layer into SAQP territory.

Huawei’s τ Scaling Law

Export Controls & Future Scaling

China is not closing the gap with Intel, Samsung and TSMC. The teardown shows the opposite in several places: no EUV, no backside power, higher process complexity, and visible trade-offs.

Behind the paywall, we show what else STEEL can do, with material and process flow analysis of SMIC N+3, and analysis of the Kirin 9030 package.

We have written a lot about Intel. It’s a firm near and dear to our heart, and is the birth of the semiconductor industry. To say we love Intel and their role in the world is an understatement. We also have been very vocally right during their initial product mishaps, as well as supportive and excited for the turnaround. The thing we have believed the most is that Intel’s board is one of the biggest parties to blame for Intel’s death, and we recently got what we want.

Franky Yeary just stepped down from the board after 17 years, and now the new board is filled with people who actually understand the industry, and not financial engineering. The new chair is ex-Qualcomm, Lip Bu Tan is the CEO, and Steve Sanghi of Microchip, Stacey Smith, Eric Meurice of ASML are all there. The board actually understands technology. But while Intel’s turnaround has partially begun, there is a lot more on this long road to a completely revitalized Intel. Today we think that Intel should make another large strategic bet with their fresh board, and it’s instead of buying back shares, it’s time to issue enough shares to fix Intel’s finances for good.

Lip Bu Tan has taken Intel back from the brink, and has raised ~20 billion dollars from the government stake, Softbank, Altera and Nvidia’s strategic investments. They should not stop partially, and capitalize on share prices. They were large net buyers of shares during the bad years, it’s time to issue equity into strength, and if they do it right, it will make Intel’s turnaround that much more successful.

Dilution Here Rewards The Investors Who Already Bet

Look at where that capital came in. The U.S. government took up to 433 million shares at $20.47, a 9.9% stake at signing, with 149 million still in escrow at the end of Q1. SoftBank paid $23.00, Nvidia paid $23.28. Every one of those holders is already above water.

So the instinct that a raise punishes the people who just showed up runs the wrong way. Issuing stock at today’s price, well above those entry marks, lifts book value per share and hands the government, SoftBank and Nvidia a gain. That 10% sovereign anchor is also the reason a large offering clears cheaply in the first place. Intel is one of the very few companies on earth that can sell size into a hot tape with the U.S. government holding the floor, and that is leverage worth using while it exists.

Intel Needs Capital for Execution

Intel has pretty much never traded this expensive on a trailing twelve month basis since the 2000 bubble. While we believe there is a bright future, one of the most critical things it will take is capital, and the share price is not discounting real execution risk.

What’s more, even in the best of cases from the new renewed demand from Agentic CPU, Intel cannot pay for the best of upsides even. We think it’s time for Intel to do a reverse buyback, and issue equity during this equity demand window.

Equity Is Now The Cheapest Money Intel Can Get

The objection will be that Intel has other ways to fund a fab. It tried all of them, and just told us they don’t work. Apollo put $11.2B into 49% of the Fab 34 joint venture. Brookfield structured the Arizona build. Silver Lake took 51% of Altera at an $8.75B enterprise value, netting Intel about $4.3B. Intel sold its NAND business to SK hynix in stages and sold additional Mobileye shares. Smart Capital was the whole identity.

Then, on March 31, 2026, Intel agreed to buy Apollo’s 49% of Fab 34 back, closing April 8 for $14.2B, roughly $7.7B of cash and a $6.5B bridge loan. Management called the unwind accretive, and they were right, which is the whole point. If buying a fab stake back is accretive, then selling fab economics to a partner was expensive money all along. A SCIP hands a sponsor a permanent cut of your best asset in exchange for capital that costs more than it looks, and Intel has now shown with its own checkbook that it would rather own the fab and carry the debt.

So cross the alternatives off. More SCIPs are the option management just spent $14.2B reversing. More debt stacks on top of the $45.0B already on the balance sheet, roughly $51.5B once the Apollo bridge is in. The big asset sales are already harvested, with Mobileye and Altera gone or majority gone. That leaves equity, and against this valuation equity is the cheapest capital in the building.

The foundry business is just beginning with their large Terafab announcement as well as overflow demand from the great N3 shortage. To truly capitalize on this special moment, Intel must be the great supplier of the rest of the industry hungry for leading edge wafers. And that huge bet requires capital well in excess of their ability to fund from operations today.

Just a 4-5% dilution would raise ~$25 billion dollars and easily allow for the most bullish supply capacity story to come true during this critical time.

Agentic CPU Demand Ain’t Enough to Pay for Terafab

SpaceX and Tesla as well as Terafab is the large customer commit that solves the 14A capacity issue. And the initial target of 100k WSPM scaling to 1 million (going to be hard) is going to be extremely rough in terms of capital. This needed to happen, as Lip Bu Tan has publicly told the market he would shut Foundry down if there were no customers. And now that a customer has come, its time to build.

Beyond the Terafab partners, the order book is filling in. Nvidia’s DGX Rubin NVL8 lists dual Intel Xeon 6 host CPUs. Google signed a multiyear deal covering Xeon and custom IPUs. SambaNova came in on inference. The wafer volumes behind these wins are not all disclosed, but capital markets fund a visible book far more cheaply than they fund a turnaround story, and Intel finally has one to point at. An equity raise sold against signed demand prices very differently from one sold against a promise.

Intel has been working it’s absolute hardest to defer capex on lower than expected demand for CPU, but it’s time to bet the farm again ala Gelsinger style. This is the critical moment for silicon sovereignty, and its time to press.

The full multi-phase project for Intel is going to cost up to $119 billion dollars, and while SpaceX is putting up the initial capital, Intel has to contribute meaningfully. Even a marginal amount of capital matching means that this is 10s of billions of dollars that were not in the capex decision matrix just a month ago.

It’s time to undo a decade of financial engineering and issue equity now. Because while the ramp is exciting, it’s going to cost a lot. The equity issuance window is the biggest it’s been in a moment, and if Cerebras can raise 5.55 billion, Intel can raise 25. The point only gets stronger, since Intel’s roughly $498B market cap easily supports a far larger follow-on

How Big The Agentic CPU, Terafab, and 14A Ramp Needs to Be

Dilution Here Rewards The Investors Who Already Bet

Intel Needs Capital for Execution

Equity Is Now The Cheapest Money Intel Can Get

Just a 4-5% dilution would raise ~$25 billion dollars and easily allow for the most bullish supply capacity story to come true during this critical time.

Agentic CPU Demand Ain’t Enough to Pay for Terafab

How Big The Agentic CPU, Terafab, and 14A Ramp Needs to Be

In section 2 of our blog post, we do an comprehensive analysis of DeepSeekv4’s inference on Day 0 Huawei Ascend 950DT. This article serves is the first analysis of Ascend 950DT DeepSeekv4 inference and we breakdown the compute<>communication lap & the different compute streams that Huawei did to optimize performance.

A key goal of InferenceX, especially during a model’s Day 0 release window, is to record each SKU’s performance using open-sourced images and recipes across as many frameworks as possible, regardless of how well these images and recipes perform. This enables us to track improvements over time, which we believe best reflects the real, deployable performance of each chip. The video below shows iterative improvements for non-MTP configs from Day 0 onward for vLLM/SGLang, respectively. visit inference.com to see the MTP configs from day 0 onwards too.

The graphics reflect the thousands of engineering hours that went into tuning DeepSeek v4 inference performance and most of the optimizations are merged into the master branch of SGLang/vLLM. One of the north start goals of InferenceX is to highlight the iterative improvements to performance over time, instead of just snapshots of performance, after all when it comes to engineering, the things you learn along the way are often just as important as the end result.

In the early days of DeepSeek v4 Pro, CUDA vLLM and CUDA SGLang and CUDA vLLM disaggregated prefill worked great out of the box, proving the strength of the vLLM and SGLang open ecosystems. These inference engines are so fundamental to the global ML ecosystem that both teams have started their own company, Inferact and RadixArk, with each raising hundreds of millions of dollars to continue to fuel the growth of their open-source inference engines.

ROCm did not work well either in the first couple of days of DeepSeek v4’s launch. That said, the AMD SGLang engineering team, under the technical leadership of HaiShaw, massively improved performance in the first month - achieving a more than 100x performance by Day 26. We will talk more about the good, & the bad, of AMD software progress in our upcoming, comprehensive State of AMD 2026 article.

The SemiAnalysis InferenceX inference initiative is supported by many in the ML community, including OpenAI, Oracle, Microsoft, Weka, PyTorch Foundation, vLLM, SGLang, and CoreWeave, among others.

The InferenceX team is hugely thankful for the ongoing engineering efforts carried out by the vLLM community maintainers, Inferact, as well as all of the SGLang maintainers around the world at RadixArk, Meta, and elsewhere. We would also like to shout out to and thank Nvidia engineers Kedar Potdar, Ankur Singh, Xin Li, Alec Flowers and many other Nvidia engineers for their Day 0 support for this project. We would also like to extend our appreciation to the AMD engineering team for their Day X support of DeepSeek v4 Pro on the ROCm stack.

If you want to work on low-level benchmarking, InferenceX, or other interesting technical work, then send us your resume to letsgo@semianalysis.com with three bullet points demonstrating your engineering abilities. If available, please attach GitHub repo links, websites, or blogs to show off your projects, work, and knowledge.

Section 1: DeepSeekV4 Pro Day 0 Performance

In this section, we will start by discussing DeepSeek v4 Pro’s out of the box performance on Day 0. We will reference throughput-interactivity curves, how different parallelisms favor throughput versus interactivity, and other inference optimizations such as MTP and disaggregated inference, which were explained in the InferenceX V2 article.

The following two graphs shows all the Day 0 recipes we managed to record, with most recipes using the native model checkpoint utilizing mixed FP4 MoE-FP8 Attention quantized weights (except for the H200 and MI355X SKUs). Because the native FP4+FP8 checkpoint for DeepSeekV4 Pro was not usable on Day 0 for the MI355X, we were only left with the option of using a full FP8 non-native checkpoint.

Unfortunately, AMD SGLang and AMD vLLM distributed inference still does not work on DeepSeekV4 Pro.

Turing to SGLang and vLLM, both supported native DeepSeek v4 Pro on the CUDA platform the moment the model was released publicly. Most advertised recipes, especially for newer SKUs such as B200/B300, worked out of the box without any major issues.

Day 0 Multi-Node Disaggregated Prefill on GB200 NVL72

vLLM and Nvidia were very fast in delivering their GB200 distributed inference Dynamo vLLM recipe in srt-slurm. Disaggregated inferencing and wide expert parallelism (WideEP) are inference optimization techniques that can considerably improve performance per dollar - readers can learn more about these techniques in our InferenceX V2 article. The recipe itself was rudimentary: eager on prefill, using NIXL for KV cache transfer. We independently replicated the recipe achieving up to 5x better results than for a B200 run using lower interactivity configs.

This is a great example of the CUDA moat at work: With CUDA, distributed inferencing tends to be supported near Day 0 for the latest open models.

Day 3 Multi-Token Prediction (MTP) Speculative Decoding

The first MTP support delivered for DeepSeek v4 came on Day 3 from SGLang. Using MTP led to a substantial improvement in throughput at higher interactivities. An explanation of MTP and how it benefits memory-bound small batch size decode can be found in our InferenceX V2 article.

Day 0 ROCm AMD MI355X Disappointment

Turning to ROCm on AMD MI355X, our Day 0 results for DeepSeek v4 were confusing. Most AMD users in the open-source ecosystem were also mired in confusion. The MI355x could only run FP8 on Day 0, and delivered results on the bottom left of the chart below in the overall Day 0 plot. Inference was technically working, but it was an unusable experience given extremely low interactivity levels of only 1-2 tokens per user per second, far below average user reading speeds.

We used the Day 0 WIP recipe provided by an SGLang PR, courtesy of HaiShaw et al at AMD. This was the only working recipe we could find on Day 0. Unfortunately, its performance was disappointing and native FP4+FP8 checkpointing didn’t work - this was likely due to the less mature ROCm ecosystem. However, as we will talk about later in the article, HaiShaw’s team eventually came through, doing an amazing job by improving performance by over 100x from Day 0 through Day 26 through some classic first principles driven engineering work.

AMD ATOM Inference Engine Disappointment

ATOM did somewhat better on interactivity, but it still falls short for a concurrency of greater than 1. We used ATOM #650 during the early days of DeepSeek v4 hardcoded kv_cache[:1,...], and this meant that the KV cache was pinned to a single sequence slot. With only one slot available, a second concurrent request has nowhere to store its KV state. This was the case because the infrastructure that enables batching possible was not yet in place, so we could only run batch size of one user.

ATOM also ran with almost every hot path on a fallback: the FP4 MoE was forced onto Triton because AITER’s fused_moe was broken on GFX950, and the mHC pre-projection patched to Torch because AITER’s kernel crashed, forcing eager execution.

NVIDIA TensorRT-LLM Bugs and Lack of Day 0 DeepSeekV4 Pro Support

TensorRT did not support DeepSeek v4 out of the box because mhcFusedHcKernel.cu had a single hardcoded FHC_HIDDEN = 4096 constant. This is a problem because SHAPE_K, residual/x TMA descriptors, and the MMA kernel template instantiations were all tied to that hidden size. All previous DeepSeek models and DeepSeek v4 flash have a hidden size of 4096, so this worked for the meantime. But attempting to run inference for DeepSeek v4 Pro resulted in a “mhcFusedHcLaunch: hidden_size=7168 not supported (only 4096)” guard error.

Nvidia engineers also encountered this guard error and instead of adding code to support DeepSeek v4 Pro’s 7168 hidden size, they simply removed the guard. To nobody’s surprise, the error then disappeared.

Because of this “fix”, there was a period of over a week where, unless the env var TRTLLM_MHC_ENABLE_FUSED_HC=0 was used, the kernel would compile exclusively for 4096, with nothing rejecting a 7,168 call. Using default settings (fused HC on by default; B300 = SM10x → MMA path), a stock trtllm-serve of DeepSeek v4 Pro feeds 7,168 tensors into the 4,096-wired kernel. Running inference with these settings results in an inference run without triggering an immediate crash, but there are hidden consequences: the engine ends up corrupting hidden states and producing invalid generations. This was fixed in a PR authored by us, and it was surprising that such a simple problem took a week to be noticed and that it took another few days for the PR to be approved.

In the time it took to diagnose the issue and narrow the problem down to the fused HC hidden size mismatch, we had already reached Day 9 post DeekSeep v4 Pro launch. This episode is a good case study that proves the strength of the open native SGLang and native vLLM engine ecosystems. Thanks to these robust ecosystems, Day 0 support will always come to native SGLang and native vLLM first before it comes to TensorRT-LLM or AMD’s ATOM engine (ATOM, by the way, currently has zero production customers).

In the graphs below, we can see how as of today, TRT-LLM’s performance is superior at higher batch sizes, but it tends to fall behind at higher interactivity levels.

Section 1.5: Performance Over Time

As mentioned earlier in the article, we capture a snapshot of Day 0 performance across the inference engines and recipes as it acts as the baseline against which improvements in performance can be measured over time. With this baseline performance - we are able to measure and present the following data analyzing performance improvements over time.

DeepSeek v4 Pro on MI355X - 100x Improvement in less than 1 Month

On Day 0, DeepSeek v4 Pro was technically working on the MI355X, but it was clearly not deployable into any production workflow. However, the improvement since then have been phenomenal - with the AMD team led by HaiShaw delivering an over 100x improvement in throughput in less than a month post DeepSeek v4 release.

The chart above shows how the throughput pareto-optimal frontier climbed when comparing results from the FP8 build on Day 0 released on 25th April to results from the FP4 build released on the 27th of May. The gain came almost entirely from AMD replacing PyTorch-native fallback paths with real AITER, Triton, TileLang, and FlyDSL kernels.

There are two steps that drive the lion’s share of the gains. The largest percentage improvement actually came from the first commit after the baseline Day 0 submission - the team managed to mop up a ton of low hanging fruit and significantly the first iteration from the FP8 baseline. The next largest improvement came a few days later as the AMD team finally got FP4 weight MoE working, allowing us to switch MoE experts from FP8 to native FP4 (MXFP4), improving expert-weight bandwidth. This also moved FlashMLA and the sparse-attention indexer off the torch fallback onto TileLang kernels and enabled HIP graphs.

The next big improvement we saw came from the introduction of AITER mHC kernels, which are used at every layer. This improvement boosted performance such that we were able to see MI355X exceed H200 performance for DeepSeek v4 Pro at lower interactivity levels for the first time.

Before the windowed-attention kernel can run, it needs to know which KV-cache slots each query’s window covers. This is done by SWA-prepare, and its implementation in Triton also helped with the improvement.

The next big jump came on May 19th as the team retired the remaining fallbacks: FlashMLA moved from TileLang to Triton, and as the AITER FlyDSL FP4 MoE kernel landed. The team also enabled fused hash-topk, DSv4 radix attention, fused store-cache, fused WQA/WKV projection, and fused paged-compress, further boosting performance. The concurrency sweep was also increased to 1024, drawing the high-throughput, low-interactivity end of the frontier that did not exist before.

ATOM also improved drastically, expanding from a single conc=1 point to delivering decent throughput along the whole pareto frontier, with some points beating H200. The first gain came from AITER fix #2916, which corrected the device-allocation bug behind the mHC crash and let ATOM restore that AITER kernel. Next, FP4 experts moved onto AITER’s fused MoE kernel (Triton override removed), the sparse-attention OOM was cleared so eager mode and the single-sequence caps could be dropped. Batching support was also implemented, expanding the sweep from conc=1 to conc 1–512 with much better performance.

MI355X MTP

By Week 4, MTP was working for AMD on all frameworks, delivering multiple-fold is-interactivity throughput improvement. One consistent characteristic we noticed however, is that MTP tends to deliver worse results at higher throughput. This is because MTP exploits the compute slack in memory-bound decode, so compute-bound large batch size decode jobs have an MTP cost that outweighs the benefit that draft tokens would provide.

B300

For B300 on SGLang, DeepGEMM MegaMoE, our results showed a 3x improvement over less than a week thanks to the use of a grouped FP4 MoE GEMM that keeps experts resident and does one mega-dispatch instead of per-expert kernels, as well as from tuning to use EP4 instead of EP8.

B200

Performance for the B200 was relatively similar to that of the B300, with TRT being superior at lower interactivity for the B200. But TRTLLM does not work out of the box, compared to CUDA vLLM & SGLang vLLM which works out of the box.

GB300 NVL72

The most dramatic improvements for GB300 SGLang MTP came on June 2nd from the implementation of W4A4 (MXFP4) MegaMoE. Compared to the non-MTP implementation that was in use on May 7th, the main improvements in the June 2nd version came entirely from a rewrite of the GB300 decode topology rather than touching kernels or precision. While the Day 0 recipe ran at most points with narrow EP=8 fed by one or two prefill workers and capped concurrency at 16,384; the May 20th run widened decode to EP=16, scaled prefill to four–twelve workers per decode worker, and pushed concurrency to 21,504.

Based on the graph and analysis above, we can see that, as expected for a larger world size inference system, Wide EP is the main lever for GB300’s excellent performance which is delivered by amortized weight loading across more GPUs. Read the InferenceX V2 article to find out more about Wide EP.

Thanks for reading SemiAnalysis! This post is public so feel free to share it.

B200 Tokens Per Megawatt (MW) Improvements

For the B200 with the vLLM engine, token throughput per all-in provisioned utility megawatt reached 300,000 tokens per second per MW for 50 tok/s/user interactivity on Day 0, improving to nearly 500,000 tokens per second per MW by June 5th.

Tokens per all-in provisioned-utility MW is the best figure of merit for considering return on investment at the fleet scale: It adds more information to raw per-GPU token throughput because it reflects PUE and datacenter overhead. Because a B200's all-in utility power envelope is fixed near 2.17 kW/GPU, that ~1.7x jump from ~300k to ~500k tok/s/MW reflects a pure software gain.

The same class of optimizations that pushed the throughput frontier (MegaMoE grouped-FP4 GEMMs, wider EP, the FP4 weight path, scheduler tuning) drop straight through to power efficiency because the all-in utility power in MW is unchanged.

Many organizations approach inference fleets from a perspective of maximizing a scare quantity of utility power. The question is how one can convert provisioned MW into as many billed tokens as possible at a given utilization and price. This analysis is best informed by metrics like revenue per MW, tokens per all-in utility power, capex per MW among others. This is a business case that our Tokenomics model is built to address.

Current Performance As of June 6, 2026

Let’s round out this section on performance improvements by quickly reviewing the best performance across systems and inference engines. When using SGLang, the GB300 continues to dominates and mogs all other inference systems, demonstrating the advantage of the GB300 NVL72’s rack-scale world size.

When we switch on MTP, serving with the GB300 is unbeatable across all interactivity levels we analyzed. The cost per million output tokens for the GB300 reaches $0.156 at 50 tok/s/user assuming 8k tokens input, 1k tokens output. Find out more about how we calculate Total Cost of Ownership (TCO) in our TCO model.

The rack-scale advantage is fundamentally a scale-up domain story. NVL72 puts 72 GPUs in a single NVLink domain, which lets the serving stack run expert parallelism wide enough to keep DeepSeek V4's MoE dispatch/combine all-to-all entirely on NVLink instead of spilling onto the slower scale-out fabric, all while amortizing expert-weight loads across far more ranks.

B200 and B300 in 8-GPU NVLink islands scaled out over InfiniBand hits that wall much earlier, and the MI355X sits further back on both scale-up domain size and collective-stack maturity. Turning these per-rack throughput wins into deployed serving capacity is a separate question that comes down to how many of each SKU are actually online: shipments and ASPs by SKU, plus installed base and effective FLOPS by customer, quarter by quarter, which we track in our Accelerator & HBM model.

ROCm vLLM DeepSeek v4 Pro Disappointment

When it comes to ROCm, progress has been much slower for native vLLM vs native SGLang. Performance of ROCm vLLM is lagging far behind its CUDA vLLM counterpart. Part of the issue is that AMD is re-focusing on ATOM (an inference engine that serves 0 production tokens) instead of focusing on native vLLM (an inference engine that lots of their major customers uses). We will talk more about in our upcoming State of AMD 2026 article (covering the good, the bad, and the ugly when it comes to AMD inference). One of the positive developments we will be covering is that the functional enablement of distributed inferencing on open source out of the box upstream AMD vLLM has finally happened for non-DeepSeekv4 models. It took many months to get there, and there is still much ground for the AMD vLLM team to cover.

What’s next for DeepSeek v4

vLLM

vLLM’s plan is tracked in the DeepSeek V4 roadmap issue (#40902), with landed code in implementation PR #40860 and a description of benchmarking against SemiAnalysis’s InferenceX dashboard. The FP4 Indexer and initial MegaMoE support has already been implemented, and Hopper is now supported. The remaining work for vLLM for DeepSeek v4 spans five areas:

The focal themes here are on the surrounding systems: a new model runner, pipeline parallelism, KV offloading, and wider hardware coverage.

Upcoming updates to InferenceX, SemiAnalysis’s open source & public’s EcosystemX dashboard will visualize the software evolution & CI coverage & queue times across all major ML open libraries across all major AI chips: NVIDIA, AMD, TPU, Trainium, Huawei, and more.

SGLang

SGLang’s plan lives in performance optimization tracker (#23666) that Nvidia structured around DeepSeek v4’s network diagram, walking it block by block; some items may already be partially covered by the initial support PR (#23600), and community contributions are welcomed.

There are three high-level goals reflected here: CUDA graph support for decode, piecewise CUDA graph support for prefill, and no at-runtime weight processing. In addition, weight prep should happen once rather than per step. Within these three high level goals, the checklist is grouped by V4’s components:

SGLang’s focus is replacing chains of small ops with single fused kernels, making the new attention variants read caches in place, and pulling the decode path fully into CUDA graphs.

Section 2: Huawei 950DT Day 0 DeepSeek v4 Analysis

DeepSeek v4 was the first major open model with first class Day 0 support on the Huawei Ascend, and indeed, part of the DeepSeek official API has been served on Huawei since Day 0. We have Huawei performance numbers on DeepSeek v4 and plan to release a follow up article deep diving into an apples to apples comparison of inference on Huawei vs on the H200 and B200, measuring comparative performance using the same benchmark harness.

Our upcoming public open source SemiAnalysis EcosystemX dashboard will visualize the software evolution and CI coverage across all major ML open libraries, for all major AI chips, including the Ascend stack.

CANN

CANN (Compute Architecture for Neural Networks) is Huawei’s software toolkit for running AI workloads on its own Ascend chips. Starting August 2025, they’ve open-sourced CANN to attract more developers and to “chip” away at Nvidia’s dominance, especially within China given that the US government heavily restricts CUDA chip shipments into China.

On Day 0, CANN released an optimization guide and benchmark figures for Ascend chips. Through it, we can see Huawei’s CANN strategy: To make Ascend competitive through full-stack inference optimization that is aimed at Chinese domestic model releases. Huawei is trying to show the Chinese ecosystem that if DeepSeek releases a new architecture, CANN can ship the kernels, graph path, quantization, serving integration, and deployment recipe.

One interesting methodology of the CANN team that we observed while benchmarking MTP and can’t help but mention was how they dealt with MTP draft token AR(acceptance rate) or AL(acceptance length). Benchmarking MTP is not trivial as the AR/AL of the benchmark may differ from the user’s use case. For example, a benchmark may on average accept two draft token out of three but with deployed use cases being extremely varied, it may end up may only accepting 1.5 tokens out of three on average.

This means that the user may see lower performance compared to the benchmark, and thus wrongly conclude their setup is flawed. We addressed this in our InferenceX v2 article by comparing ARs with MTBench. Future iterations of our benchmark will comprehensively address this gap by using real traces.

To address this quirk, Huawei instead times the full decode step to coincide with the last MTP module, thus recording time per decode step instead of time per token. The final benchmark result published then requires the user to multiply by their usecase’s MTP AL to derive comparable performance metrics, which is a very elegant way to compare performance.

Hey NVIDIA Goliath, There’s a New David in Town - The Ascend 950

Huawei’s internal codename for Ascend 950 chips is “David”, and this codename is referenced multiple times in the CANN codebase. No doubt it is because they believe they are the David to Nvidia’s Goliath.

The SIMT/SIMD 950 chips come in two flavors: 950PR, and 950DT. PR stands for Prefill and Recommendation and are lower cost chips with better cost-performance. DT stands for Decode and Training, and this variant features higher memory bandwidth and higher performance. Both are based on the same Ascend 950 Die, which uses a dual-die UMA architecture, but they are each packaged with different memory. The Huawei roadmap estimates and volumes per quarter of each Huawei chip is available in the SemiAnalysis Accelerator model.

Two major components of the chip architecture that are important to discuss are the AIC(AI Cube) and AIV(AI Vector). AIC is the matrix/tensor core side of Ascend’s AI Core. It is used for dense matrix math: GEMM, matmul, convolution-like tensor ops, attention projections, FFN linear layers, etc. Huawei documentation describes AIC as the matrix-compute core in split AI Core architectures. AIV is the vector core side. It handles elementwise/vector work: activation functions, normalization pieces, masking, reductions, type conversion, layout transforms, post-processing around matmuls, etc.

This is similar to TPU’s MXU. However, Ascend exposes the split between the two functions more directly as separated independent cores, with each able to load its own code segment, and also feature a “dual-master mode,” where AIC and AIV independently run code rather than having the AIV driving the AIC through messages.

The AI CPU is a device-side ARM64 execution unit with direct access to device memory. It is used as a complement to AI Core for the work that maps poorly onto the SIMD/SIMT cores: branch-heavy control flow, scalar logic, dynamic-shape handling, and the value-dependent scheduling/tiling metadata that kernels need before they run. Because the AI CPU lives on the device, Ascend can keep this irregular control-style work local instead of round-tripping work to the host CPU, which is a primary source of latency and pipeline bubbles. The AI CPU is also the unit that historically sat on the older AICore → AICPU → SDMA path for communication orchestration, before the dedicated CCU offloaded that work.

Like the TPU and Trainium, Ascend 950 adds a dedicated CCU communication engine. This engine sits alongside the compute die and handles collective-communication work without consuming AI Core compute capacity by supporting remote-read + reduce + local-write, and local-read + remote-write. The benefit is in lower communication latency, less HBM traffic, fewer user-buffer copies, and freeing compute cores from comm orchestration, avoiding the older AICore -> AICPU -> SDMA path.

Huawei DeepSeekV4 Pro 950DT Profile

The above shows a three step profile of DeepSeek flash v4 on Ascend 950DT, running using a config with a 16-rank DP/EP deployment. It shows 16-rank collective participation plus active MoE dispatch/combine traffic.

As is now standard across most stacks, CANN also uses independent compute and communication operators that can run on multiple streams - performance improves by controlling Cube and Vector core allocation to avoid resource contention. Operations like Prolog, Compressor, and LightningIndexer can be overlapped, C4A Compressor can be completely hidden, and shared expert computation can be hidden under routed expert execution without degrading routed expert performance.

Zooming in into a given decode step, we can see how different components are split into streams. Operations on different streams may run concurrently when the device has free, suitable resources. Models use multiple streams because a layer may not be a single serial chain, and instead can contain branches that only need to synchronize when their results are combined, for instance a shared-expert compute overlapping 100% with routed-expert compute.

Streams 145-148 in the above diagram correspond to metadata streams. These operators fire once per decode pass and precompute value-dependent scheduler/tiling metadata that later kernels reuse. They are the only AI CPU ops in the decode step, the make up a tiny fraction of total time, and are fully overlapped with AI Core compute. The impact is likely larger on longer-context benchmarks, where there is more sequence-length and mask-dependent partitioning to resolve up front.

In DeepSeek v4, Huawei moves the value-dependent scheduler stage for sparse attention and the LightningIndexer onto the AI CPU rather than bouncing it back to the host. These metadata ops build reusable per-core partitioning tensors from runtime sequence-length, mask, and paged-KV information; SparseAttnSharedkv and QuantLightningIndexer then consume them to decide which Batch/Head/Q-block/K-block work each cube core handles, along with the corresponding vector-core reduction tasks. Conceptually, this mirrors FlashInfer’s planning phase on host for paged attention: a cheap, dynamic-shape-aware setup step that runs once and is thus amortized over layers, with the only caveat being that Huawei pushes that same planning work onto the on-device AI CPU instead of the host.

Stream 152 in the daigram above contains the LM head, the last layer, and the second last layer’s o_proj and MoE. This is a decision of the npugraph_ex graph compiler, likely to allow the npugraph_ex runtime to consider the main graph “complete” on stream 144 while the tail work continues asynchronously.

CANN also introduced MC² (merged compute-communication) back in 2024. This is a class of fused operators that are not ordinary kernels nor HCCL collectives. They embed communication and compute into one kernel. In DeepSeek v4 decode, we can see MoeDistributeDispatchV2 and MoeDistributeCombineV2 MC² EP operators being used.

The main takeaway here is that Ascend delivers working, optimized inference infrastructure for DeepSeek v4 on Day 0. The Huawei CANN stack is one of only two stacks with Day 0 Support for DeepSeekV4, the other being Nvidia’s CUDA. As we explained earlier in the article, AMD’s stack unfortunately did not work well on Day 0. This is in stark contrast to last year when DeepSeek v3/R1 released. Back then, only one stack worked on Day 0: the Nvidia CUDA stack.

The biblical story that gave the Ascend 950 its internal codename ends with the giant face-down. But the Goliath in that story stood still and let David sling stones at him, whereas Nvidia’s Goliath is constantly in motion, ships a new architecture every year and improves existing architectures. Huawei has proven it can sling a stone on Day 0; whether it can fell a moving giant is yet to be seen.

DeepSeek V4 Architecture Deep Dive and Co-Design

Inference Optimizations for 1M Context Length

DeepSeek v4 features Compressed Sparse Attention (CSA) and Heavily Compressed Attention (HCA), walking away from Multi-head Latent Attention (MLA). The design is heavily motivated by KV cache size reduction.

In essence, HCA’s KV cache consists of a sliding window of the KV embeddings and a set of compressed KV entries, where each entry compresses key / value into one and across m′ tokens (m′ = 128 for DeepSeek V4 Pro).

CSA uses the same KV cache compression technique as HCA, but with a lower compression rate (m=4). CSA also applies sparse attention on the compressed KV entries by using a lightning indexer to select tokens to attend to. The sparse attention inherits DeepSeek Sparse Attention in DeepSeek v3.2.

By interleaving CSA and HCA, DeepSeek v4 aggressively compressed KV cache size, resulting in 50x KV cache reduction at 1M context length.

However, the novelty of CSA and HCA creates KV cache management challenges for serving frameworks. For example, vLLM’s KV cache memory allocator implements complex strategies to ensure efficient memory loading patterns and support serving features like prefix caching. This includes setting a logical block size that divides the KV compression rates of both CSA and HCA, and a page size bucketing strategy to avoid memory fragmentation due to storing the KV cache, compressor states, indexer KV, each having a different size per entry.

Determinism

To ensure RL training stability, DeepSeek went all in on making computation deterministic. This effort shows when looking intoGPU kernels and their rollout infrastructure. DeepSeek wrote custom kernels for all operations to achieve batch invariance by enforcing a specific reduction order regardless of batch size. This includes batch-invariant split KV attention forward, GEMMs, and MoE backward kernels. Batch-invariant kernels come at a performance loss because using them prohibits many popular algorithmic techniques that don’t guarantee deterministic reduction order. DeepSeek mitigates the performance loss by writing kernels that are tailored for their workload, for instance specializing kernels for matrix shapes. On the rollout infrastructure side, DeepSeek focused on fault tolerance to make all rollouts reproducible. DeepSeek built a token-granular write-ahead log for each generation request, so any request preempted during either prefill or decode can be resumed without recomputation.

MegaMoE

The DeepSeek V4 release also included a new fused MoE kernel that achieves better overlap of all of the operations in the MoE layer. MoE with expert parallelism consists first of a token dispatch all-to-all followed by Linear1, Activation, Linear 2, and finally a token combine all-to-all. Linear 1 and Linear 2 are grouped GEMM operations where each expert in a given rank has its weights applied to its routed tokens. The authors mention in the DeepSeek V4 paper that other implementations overlap/interleave the token dispatch with Linear 1 and the Combine with Linear 2, but there is still a sync across all experts at the operation boundaries, between Linear 1, Activation, and Linear 2. MegaMoE instead splits experts into waves and schedules each wave separately, allowing for finer-grained overlapping of each of the operations and resulting in more of the communication latency being hidden. This is reminiscent of compute-comms fusions such as distributed GEMM, where a compute kernel and a dependent communication kernel are overlapped by breaking the workload into smaller pieces and pipelining to hide communication latency.

The paper claims a theoretical speedup of 1.92x over the naive kernel in the DeepSeek v4 Flash configuration, which means that the naive kernel must spend close to 50% of its time on Dispatch and Combine communication!

Now that we have discussed performance benchmarks in detail, let’s discuss total cost of ownership and cost per token when running DeepSeek v4 on H200 and the GB200 NVL72.

The SemiAnalysis InferenceX inference initiative is supported by many in the ML community, including OpenAI, Oracle, Microsoft, Weka, PyTorch Foundation, vLLM, SGLang, and CoreWeave, among others.

Section 1: DeepSeekV4 Pro Day 0 Performance

Unfortunately, AMD SGLang and AMD vLLM distributed inference still does not work on DeepSeekV4 Pro.

Day 0 Multi-Node Disaggregated Prefill on GB200 NVL72

This is a great example of the CUDA moat at work: With CUDA, distributed inferencing tends to be supported near Day 0 for the latest open models.

Day 3 Multi-Token Prediction (MTP) Speculative Decoding

Day 0 ROCm AMD MI355X Disappointment

AMD ATOM Inference Engine Disappointment

NVIDIA TensorRT-LLM Bugs and Lack of Day 0 DeepSeekV4 Pro Support

In the graphs below, we can see how as of today, TRT-LLM’s performance is superior at higher batch sizes, but it tends to fall behind at higher interactivity levels.

Section 1.5: Performance Over Time

DeepSeek v4 Pro on MI355X - 100x Improvement in less than 1 Month

MI355X MTP

B300

B200

GB300 NVL72

Thanks for reading SemiAnalysis! This post is public so feel free to share it.

B200 Tokens Per Megawatt (MW) Improvements

Current Performance As of June 6, 2026

ROCm vLLM DeepSeek v4 Pro Disappointment

What’s next for DeepSeek v4

vLLM

The focal themes here are on the surrounding systems: a new model runner, pipeline parallelism, KV offloading, and wider hardware coverage.

SGLang

SGLang’s focus is replacing chains of small ops with single fused kernels, making the new attention variants read caches in place, and pulling the decode path fully into CUDA graphs.

Section 2: Huawei 950DT Day 0 DeepSeek v4 Analysis

CANN

Hey NVIDIA Goliath, There’s a New David in Town - The Ascend 950

Huawei DeepSeekV4 Pro 950DT Profile

DeepSeek V4 Architecture Deep Dive and Co-Design

Inference Optimizations for 1M Context Length

By interleaving CSA and HCA, DeepSeek v4 aggressively compressed KV cache size, resulting in 50x KV cache reduction at 1M context length.

Determinism

MegaMoE

Now that we have discussed performance benchmarks in detail, let’s discuss total cost of ownership and cost per token when running DeepSeek v4 on H200 and the GB200 NVL72.

We are witnessing the birth of another Chinese hardware giant. Three years ago, Unitree was a quadruped company. By last year, they parlayed quadruped dominance into creating and leading the humanoid market. This year, their G1 humanoids are finally entering into viable deployments, and three new designs are on the way, including their most-direct Western humanoid competitor.

Tesla first unveiled a humanoid in 2022, and while it and other Western players are now producing early humanoids that remain works in progress, we hear Unitree may ship its 10,000th in the coming weeks.

Now, Unitree is tripling revenues YoY on 60% gross margin product lines, planning almost $300M of AI R&D spend, increasingly in-housing portions of manufacturing, all while pricing the cheapest humanoids on the market by far. With their upcoming highly anticipated IPO, Unitree is deservedly dominating the humanoids conversation. But historically, Unitree’s humanoid robots have a reputation for less than perfect reliability, are not viewed as useful beyond entertainment and R&D, and have a reputation for being “cheap.”

Despite this, we believe Unitree's cost structure is one of its greatest advantages over competitors. Unitree has slashed pre-tax pricing from $50K+ to $27.3K over the past 12-18 months. Even at that price, we estimate they still hit 67% gross margins on their flagship G1. With their BoM set to plummet as manufacturing scales, we've already heard pricing well under $20K in some deals.

We developed these BoMs by going through a full design examination of Unitree’s robot, speaking with manufacturers of every component, and verifying items with multiple supply chain buyers/sellers.

And, lastly, despite countless dismissive comments against the company, we argue their G1 humanoids are crossing the viability threshold of real-world deployments.

However, no one understands Unitree’s strategy, cost, and manufacturing as well as concerns of the usefulness of the robots at all. But today we are here to set the record straight. In our research we present the history of Unitree mimicking the BYD and DJI strategy, by generating their own ecosystem, spawning new markets, and then eating said markets. This strategy is in process as we speak. New markets on the horizon means that Unitree’s explosive growth should continue.

Next we examine their specific hardware strategy, and how their QDD actuator design choice led to a potentially structural advantage, and how their actuator has improved into near-deployment grade.

Lastly we argue that Unitree’s improvement and cost advantage is now breaking into the world of economic viability to displace labor. There are likely over 250 Unitrees deployed in labor settings today, and we detail how the deployment math works out. Notably, Unitree has made it this far on the back of the small hobbyist/researcher market. Should Unitree unlock viable deployments and hit critical mass, they may accelerate at unreal speed.

All of this comes at a level of scale and manufacturing that crushes the West’s cost and lead time, and Unitree itself is a standout in the competitive Chinese ecosystem. Behind the paywall we specifically discuss the new robot handmakers that are aiming to unlock more tasks and markets, as well as who will be eaten and who will benefit from the Unitree supply chain.

Unitree’s IPO is the birth of the robotics moment. They are unlocking markets, ecosystems, and are pursuing a strategy of scale that may lead down the path of other Chinese hardware juggernauts. Let’s first turn to the past to understand how Unitree could potentially work out.

The Making of A Chinese Hardware Giant

What does a fully-matured Chinese hardware giant look like in practice? There’s a great example of the carmaker BYD (Build Your Dream) today as the mature expression of Unitree’s strategy: own the most expensive and challenging component in the BoM, use this ownership to compound cost advantage that nobody can match, and create new markets while adding more value from inhousing your supply chain.

BYD initially focused on the battery cell. Batteries can consume around 30-40% of an EV’s BoM (now less, thanks to BYD). BYD was founded in 1994 to make battery cells that Japanese incumbents exited due to toxicity. BYD spent nearly a decade refining their product before entering EVs in 2011, and at first was only a niche player. When BYD launched its first battery-only EV, the e6, to the Chinese market in October 2011, annual EV sales across all of China were just 8,159, or 0.04% of new car sales. There was no EV market, but BYD helped create one.

Owning the cell at increasing automotive volumes passes through demand, creating improved supply and ecosystem, where players like Hunan Yuneng and Shenzhen Dynanonic (LFP cathodes), Inovance (motors and inverters), and Sanhua (thermal management) emerged to supply BYD’s next-generation, improved parts at a lower cost. None of these existed at a meaningful scale in 2010.

BYD has freedom to in-house manufacturing where it compounds for them. They took in house the battery cell, drives, motors, IGBTs and SiC power modules (one of the only companies in the world running an IDM model), the transmission, the chassis and outer body shells, and even the engine itself. By the late 2010s, nearly every component of an EV was being built under BYD’s roof.

This creates a positive feedback loop: owning and improving the right hardware lets BYD create a new product that opens a new market, like their 2020 Blade Battery. Pre-Blade, the battery chemistry (LFP, Lithium Iron Phosphate) was cheap, safe, but low-density, fine for vehicles that never traveled far from a charger, like forklifts that charged during operator breaks or buses that returned to depot every night. However, passenger EVs, which had to handle road trips and unpredictable home charging, saw LFP as unviable.

In 2021, their Blade Battery took a new packaging geometry, pushing battery pack space utilization per kg up 50%. Now, LFPs could remain the same size, while improving distance to a viable threshold. Tesla shifted the Model 3 and Y to an LFP battery, Ford licensed CATL’s LFP technology, and overnight, through meticulous hardware iteration, BYD created the modern affordable EV market to dominate.

Before the Blade Battery, BYD shipped 189K new EVs in 2020. After the Blade Battery in 2021, BYD shipped 600K new EVs, and by 2025, not only did BYD become the #1 EV producer, but they overtook Tesla as the #1 BEV producer (battery only), Tesla’s main product. BYD has now inhoused such a significant portion of its process (75% for the Seal), that their cost-structure is almost untouchable, like the 2023 Seagull model for ~$11,000 (newer models are under $8k in China!) BYD even owns supply chains further upstream with moves like a 2023 refining Joint Venture with Huayou Cobalt and acquiring direct lithium mining rights in Brazil’s “Lithium Valley.”

This scaling outcompeted European EVs so much that VW announced its first-ever German plant closures and Stellantis cut guidance, all citing Chinese EV pressure. Even the United States had to increase its tariffs on Chinese EVs to 100% to protect its domestic industry. Now, BYD has scaled so much they even own their own freight to ship their cheapest, and best EVs in the world.

The DJI Playbook - Niche Researcher/Hobbyist Markets Are Viable Bootstraps

DJI pioneered a playbook different from BYD, and Unitree is running that playbook today, starting with the researcher/hobbyist beachhead, and a low-quality product.

In 2013, “useful consumer drone” wasn’t a category. Parrot’s AR.Drone, the leading product, won its 2010 CES award in the Electronic Gaming Hardware bracket and shipped alongside augmented-reality dogfighting games. The drone had no camera stabilization, no GPS, and only took 640x480p pics/videos. Anyone who actually wanted a useful flying camera had two options: pay $19,995 for a Draganflyer X6, or stitch together frames, motors, flight controllers, and gimbals (stabilizers) from disparate vendors for up to $1,200 in parts, plus dozens of hours of assembly and PID (controller) tuning, usually ending in expensive crashes.

Researchers, hobbyists, and early professional camera workers were a willing market for something new. DJI’s Phantom 1 shipped January 2013 at $679, and was not a fully-fledged product at the time. It had no built-in camera, no gimbal (stabilizer), ten minutes of flight, no live video feed, but it was roughly half the cost of the build-it-yourself drone, with none of the assembly burden. A far cry from DJI’s drones today, but DJI went from $4M in revenue in 2011 to $130M in 2013, after the Phantom 1’s release. This was more than enough to kickstart DJI’s flywheel.

DJI then reaped the fruits of the Shenzhen consumer electronics ecosystem, now huge from the Smartphone boom. GPS prices went from $800 to below $14 from 2003-2013, controllers went from $2,000 to $400 from 2006-2011, and more. From DJI’s boom, there are now over 3,000 drone component suppliers for most anything you desire.

DJI chose to inhouse the most expensive and technically difficult component first: the flight controller. Third-party suppliers were still selling at $200-400 even in batches of thousands in 2014. Later on, DJI brought inhouse the gimbals, motors, and ESCs.

Like BYD, new DJI generations unlocked a new market the previous one couldn’t address. The 2013 Phantom 1 ($679, no camera, 10-minute flight, no live feed) was the bootstrap, engaging hobbyists/researchers. The Phantom 2 Vision+ in 2014 then came with a 3-axis gimbal (stabilizer) into the frame, where previously broadcast-stable aerial video had required a $2,000+ aftermarket gimbal mounted to a hand-built rig, operated by a skilled pilot.

Pre-Vision+, professional aerial photography was the domain of helicopters and Hollywood second-unit teams, but now, small businesses could perform this on their own. As such, whole new markets were unlocked for DJI, like real estate listings, wedding videos, local news, agricultural surveying. By the Phantom 4 in 2016 ($1,399, 4K camera, 28-minute flight, forward obstacle avoidance, 44 mph sport mode), the enterprise market was unlocked: surveying, inspection, first response, and more that we detailed here.

In 2016-17 DJI held roughly 70% global consumer drone share, and global drone shipments hit 6.4 million units and $1.9 billion in revenue, a market that hardly existed before. Multiple competent dronemakers were crushed. 3DR, GoPro’s Karma, and Parrot’s consumer line had all exited or were exiting the category. Chris Anderson, the 3DR CEO, estimated that DJI cut prices by as much as 70% in less than a year during the Phantom era.

For brevity, we’ll call this the “DJI Strategy” throughout the rest of this piece: own one critical component, bootstrap a willing audience, ride the ecosystem, and let each hardware generation unlock the next market. (An earlier version of this framing appeared in our first robotics paper.)

Unitree As An Early DJI

Unitree is a live DJI Strategy case study: own the bottleneck component, bootstrap a willing audience, ride and seed the ecosystem, unlock new markets generation by generation. So far, Unitree has

Let’s walk through the history, design, strategy, and the critical threshold they are crossing right now humanoid deployments.

In 2016, Wang Xingxing, a former employee at DJI, developed a low-cost quadruped robot called XDog for his master's thesis. He iterated upon the same quadruped in his new company named “Unitree.” For Unitree, their core component of choice was the actuator, the integrated joint that moves the robot’s limbs. Like BYD and their cells, DJI and their flight controller, Unitree chose the expensive actuator (50%-70% of the humanoid BoM) to improve and scale out from.

Unitree started with the academic robotics community as merely a quadruped company. Just as DJI had hobbyists paying out the nose for half-finished drones, Unitree saw university labs looking for a legged platform that didn’t cost $70-100K+. The Laikago shipped in 2018 at $45,000. The A1 followed in 2020 at $15,000; the Go1 in 2021 started at $2,700 for the Air trim and ran up to $8,500 for the Edu; the Go2 today starts between $1,600 and $2,800 depending on trim and region.

A 94-96% price decline in entry-level quadrupeds over six years pushed Unitree from academia into consumers and now even industry deployments -- with the broader AI wave lifting the hardware’s capabilities. More importantly, it gave Unitree years of real volume on the same systems for a humanoid: actuators, controls, suppliers, and production process. When Unitree released the H1 in 2024 at roughly $90K, the humanoid was less a brand-new product, but rather the direct result of their quadruped scaling curve. We’ve heard from people close to Unitree that the H1 was essentially a quadruped standing on two legs - note the bent knees and awkward walk. The H1 showed how far quadruped-era IP could be pushed into humanoids, but the following G1 changed Unitree’s world.

The $30-50K G1 - A New Possibility In 2024

In mid-2024, affordable, off-the-shelf humanoids were few and far between until Unitree came along. Agility’s Digit was just beginning to deploy a handful of robots into factories. Apptronik’s Apollo, unveiled August 2023, was pre-commercial. Figure’s initial commercial agreement with BMW, signed January 2024, was shipping in single-digit volumes. Tesla wasn’t (and as of V3, still isn’t) shipping Optimus externally at all. On the Chinese side, UBTech’s Walker, Fourier, and the early signs of AGIBot existed, but not as cheap nor at volume. No one could just “buy” a humanoid.

The G1 enabled an incredible academia sized market. Ask any researcher and they will tell you how dramatic a step-change in accessibility it was using a $30-50K, ready-to-purchase bot. This share of the research community has since bled into top-tier AI research companies via hirings, like Nvidia, Apple, and Meta, all purchasing hundreds of G1 units. Unitree has become the leading platform for humanoid AI research.

The Ecosystem Advantage

Unitree inherits both the supplier base of DJI and BYD’s past. China assembled 31.3 million vehicles in 2024, 40.9% of them new-energy (BEV or PHEV), and the aforementioned 3,000 drone component suppliers had already scaled many of the BLDC motors, drives, encoders, batteries, and manufacturing processes that general robotics can reuse. However, Unitree’s gravity is exemplified by the rise of new humanoids+quadrupeds supply chains. Every province now has several manufacturers of rightly sized/spec’d gearboxes, high-torque BLDC motors, and more. Within China, there are now ~200 humanoid companies all reaping the benefits of and contributing to this ecosystem.

All of this stemmed from Unitree’s decision to perfect the actuator. However, the first generations of their actuator didn’t work very well.

In 2024, Their Humanoids Weren’t Good

For brevity, we’ll use QDD in this piece to mean the combination of a brushless DC motor and a low-ratio planetary gearbox, typically in the single digits and up to 20:1, which can still provide adequate backdrivability, but understand there are nomenclature debates.

Let’s be clear: DJI and BYD unlocked markets when the product worked, and the H1 and original G1 were not very capable upon shipping. Whenever users pushed them toward real work, the motors often overheated. The G1 could only hold a 2kg payload, like a 2-liter bottle of soda, with arms fully outstretched for a few seconds before mandatory cooldown. The same 2-3kg with arms bent or contracted for maybe 2-3 minutes, like the diagram below.

After that, the robot typically needed about 30 minutes to return to functionality, and likely a full hour before resuming a real work task. Five minutes of work followed by the rest of the hour cooling down is not a productive robot.

The issue primarily came from Unitree’s core actuator choice: the QDD, or quasi-direct-drive, which are simpler and cheaper than typical robotics actuators. Historically, companies favored high-precision, high-power actuators that could both move the robot and support its weight. Industrial arms often use HarmonicDrives. Boston Dynamics’ early humanoids and quadrupeds used bulky hydraulic actuators. Even today, many humanoid companies default to high-gear-ratio actuators, like strainwaves from HarmonicDrive.

Those architectures work, but they are expensive, hard to manufacture, and often difficult to maintain. In 2018, the MIT Mini Cheetah popularized the QDD (quasi-direct-drive): a cheaper, simpler alternative. The open question was whether QDDs could scale and prove reliable enough for real-world robotics. Unitree believed this to be a yes.

Why The Issues Around QDD?

QDDs flip the historically common robot-joint setup. Instead of a small motor with a big gearbox, they use a beefier motor with a much smaller gearbox. If the robot wants to lift 5kg, there’s two options.

While there’s bonuses to Unitree’s QDD, like easily adapting to opposing forces (like collisions), or invoking a rapid, dynamic range, it does come with tradeoffs. Because the motor carries more of the torque burden directly, rather than relying on the gearbox to amplify it, early critics argued Unitree’s motors would draw very high current, run hot, and prove too unreliable for practical work.

Two Years Later, The QDDs Are Defying Expectations

The early critique was fair, QDDs gave Unitree a cheaper, simpler actuator, but pushed too much thermal burden onto the motor, hence the aforementioned overheating. But while most stuck with the strainwave (HarmonicDrive) route, Unitree took the chance and iterated.

Over the last few years, Unitree appears to have improved the actuator in several ways, now showing promising signs of scalability, with many other (Chinese) humanoids switching to the QDD wave and, consequently, building up the ecosystem. We will aim to cover here of the improvements Unitree has displayed, but to clarify: we have not had access to cycle test numbers, this is simply our best approximation of evaluating their hardware improvements.

Let’s begin with the usual P=I^2R, or in this case, Heat roughly scales with I^2R. So you may do two things: lower the current (I) the motor needs to draw, or lower the resistance (R) of the windings the current flows through. Most focus on Current because it’s squared, but both contribute.

One major lever to reduce wasted current is to make the motor’s torque smoother through each rotation. In plain terms: pedaling a bike with a slightly wobbly wheel takes more energy than a true one. Each rotation, you fight a small surge of resistance, and you push harder on average to hold the same speed.

A motor with a rough magnetic pull has the same problem: the rotor jerks slightly with every revolution, and the extra current you draw to overcome that jerkiness turns directly into heat. Those hiccups come from effects like cogging torque, where the rotor magnets interact with the stator teeth, and from non-ideal magnetic-field shapes that create torque ripple. Less ripple means less vibration, less wasted current, and more usable torque before overheating.

To resolve this, we can reshape or curve the magnets and slots so the pull stays smooth between rotations. Additionally, we can also skew the magnets so the stator teeth take turns engaging the field rather than all snagging at once.

Another useful solution is to pack more copper wire into the motor. A thicker, more densely-packed wire carries the same current with less resistance, known by Unitree as their “Low Copper Consumption Coil.” Halodi, now known as 1X, famously tweaked their copper fill as well by using thick, square copper wire.

Cooling is still important for a functional robot, but Unitree’s architecture is fairly conservative on this front. We’ve found passive cooling across most of the body, with active air cooling only at the main control board and hip joint, and a vapor chamber heat spreader at the knee. Unitree has iterated here too, adding active cooling around the pelvis in an October 2025 update that improved thermal headroom for subsequent G1 batches.

We suspect Unitree not focusing much on its cooling can be attributed to wanting to reduce costs and manufacturing complexity while focusing on the core problem we mentioned earlier: lowering the amount of current its motors need.

So Why QDDs In The First Place? A Quick Detour On Speed+Cost

Unitree made a bet on the QDD, taking an unproven architecture (in deployments) on the most expensive component in the robot. While difficult to work with, they are still higher efficiency (95%-98% vs 85%-90% for strainwaves), and up to 80% cheaper. Importantly, low-ratio planetary gearboxes are also common industrial components, machined with standard gear hobbing (think grinding) on widely available equipment, enabling many suppliers to exist.

Meanwhile, competitors choosing strainwave gearboxes for example, like from HarmonicDrive or LeaderDrive, face a more complicated ~13-step process. Multi-hour heat treatment of the metal grains to allow “flexing,” (shown below) precision hobbing into micron-level tolerances, etc, all lead to a process that took HarmonicDrive decades to perfect, and LeaderDrive is still considered by many to be lagging HarmonicDrive’s reliability after 20+ years.

Instead of verticalizing an entire decades-long learning curve, Unitree chose the QDD. Now, a new QDD redesign can become a sample actuator within weeks for Unitree. For context, a custom motor and gearbox subsystem may take a Western humanoid company 3+ months due to so many supply chain handoffs. Weeks of spec iteration, 6-8 weeks for motor+gearbox samples, then validation and reorders. The result is both cheaper production, as shown in our BoM, and faster iteration, as seen in the active pelvis-cooling rollout that went largely unnoticed.

From Burnout To Lightweight Tasks

Now, Unitree has iterated and improved their G1’s and actuators enough that small, but legitimate, tasks are within arm’s reach. Arms bent, the G1 sustains upwards of 10-15 minutes of operations carrying 5 kg, a roughly 2x increase in payload and a 5x increase in duration over our original numbers! With arms fully outstretched, 5kg (roughly a bowling ball) can be held for ~1 minute before hitting thermal limits. Even for humans this is considered a workout. But how close are we to Unitree being a “viable” humanoid?

Heading Towards Useful Work

Unitree is obviously very cheap, and we know the strategy they are taking, and how their hardware is improving, but does it matter? Many might point to the previous section declaring that 15 minutes carrying 5kg doesn’t matter: perhaps “dexterity is lacking,” “it doesn’t have hands,” “that’s not very long,” etc. However, we estimate that Unitree may have shipped roughly up to 250 humanoids into productive industry pilots or deployments in 2025, aside from their research/hobbyist sales. We’ve even found one company with 30 G1s deployed today, and multiple companies with 5-6 G1s deployed. Granted, these deployments are likely software limited (e.g. AI model capabilities), leading to differing economics based on what solutions are used, like 100% teleoperation (which is what we assume in our calculations).

Unitrees do not need to run perfectly, nor for extended periods of time, nor do we expect them to. Rather, it is a question of what’s sufficient for useful work. The G1’s arms are still underpowered, they don’t have enough “degrees of freedom” for perfectly human-like movements, and they still overheat on overly strenuous jobs. But this only puts a ceiling on the types of tasks the G1 can do, not a floor on whether it can do anything at all.

Looking past performative backflips, Unitrees can already perform “useful work,” non-exhaustively defined here as either.

A task performed in a business to produce some economic output, like sorting boxes, or a task performed to relieve a physical burden of a human, like folding clothes.

So what are these Unitrees actually doing? Essentially, carrying boxes/items from A to B. Currently, it tends to be lightweight materials handling, like in e-commerce tote (bin) handling with <3-5 kg payloads, or moving even empty boxes/totes.

None of these are 24-hour, fully-autonomous production lines, and most are still teleoperated. Nonetheless, let’s demonstrate that moving boxes around is becoming economically viable.

Unitrees Are Crossing The Deployment Viability Threshold

\"Thanks to Adamo for helping us understand teleoperated deployments further!\"

We’ve discussed humanoid economics in depth in our Levels of Autonomy paper, but here we perform a full calculation for Unitree. Taking Agility Robotics’ (fantastic) task as a baseline, and plugging in Unitree inputs, we find Unitrees are currently passing below the $30 per hour labor cost of a human.

No humanoid is at mass-scale production/deployments yet, all are still iterating on the technology, and these figures may change from deployment to deployment (grippers/hands, payload, throughput, etc). Instead, we will show our best estimate of what they’re capable of today and what that looks like on a specific task, but want to note: these are early-stage numbers, and if Unitree’s hardware continues its improvement trajectory, and Autonomy improves, the economics likely only improve from here.

A Lovely Task: Tote Handoff

In this particular job, Agility serves as a “bridge” between automation systems, for example, carrying totes (bins) off the AMRs to place onto conveyor belts. It’s a standard, but particular, logistics workload, where a human typically waits on automation systems to line up for the transfer, but leaving dead time in between. It’s why Agility’s 66 totes per hour (from a conference demo, likely higher in deployments) is perfectly fine.

Additionally, in Agility’s GXO deployment, the totes are 2-4kg (good for our aforementioned Unitrees). With light payloads, low-throughput, forgiving retries upon failure, and little dexterous manipulation, it’s a perfect task for the current Level of Autonomy.

Calculating Unitree’s Economic Viability

Agility currently operates using a 2:1 utilization model, two units of operating time for every one unit of charging, giving Digit a two-thirds utilization rate relative to a human.

Operators deploying Unitrees on similar 2-4 kg tote transfer work reported to us a throughput matching Digit on this task. However, Unitrees aren’t the most robust robots. A Unitree G1 typically runs a similar task for 10-15 minutes, while needing 5-10 to cool down, but still leaving a 50%-67% utilization rate. In our assumptions, we remain very conservative, assuming full teleoperation, 15% service contract (typically 5-10% in industrial scenes), a two-year useful life, zero residual value, and only two shifts. Even still, the robot proves viable today.

To get the caveats out of the way, we are not proposing Unitree as the full package. Many layers still need to be added. Agility, for example, has a great operating system to coordinate with the Warehouse Management System, a deeper functional safety approach, their own layer of autonomy, and much more. However for Unitree, even full teleoperation seems to be viable in the right business settings now.

Where This Goes

To our knowledge, this is the first stage of Unitree’s humanoids exhibiting useful work in deployments. It’s not that this task is exceptional, or a $100T TAM, but it shows Unitree may be crossing a threshold, and their trajectory is quite remarkable. Recall that DJI shipped a half-perfect drone to niche hobbyists and 32x’d their revenue, quickly catapulting themselves into drone dominance. In the beginning, the G1 would overheat from holding a box for 2-3 minutes too. The performance improvements thus far were funded by hobbyists/researchers. If Unitree unlocks even a slice of the warehouse TAM, progress can accelerate at breakneck speed.

With signs of robotic AI models gaining new capabilities, BoMs falling, and the quality of hardware improving thus far, Unitree could become the cheapest, competent humanoid platform in the market. Let’s talk about how they got to their world-leading $8,976 BoM.

China’s Manufacturing Competence

Throughout the piece so far, we’ve hinted at Unitree’s manufacturing competence and the ecosystem surrounding them. For robotics, and many other industries, working with the Chinese ecosystem is necessary to compete on a global scale when: Suppliers are mere hours away by train, samples arrive the same/next day, vertical iteration cycles can run in weeks instead of quarters, and components can run 20-40% cheaper than Western equivalents. It should be noted that strainwave gearboxes from Leaderdrive, for instance, sometimes cost a third of HarmonicDrive’s -- granted HarmonicDrive currently remains the reliability leader.

Most American robotics startups are already working with Chinese supply chains, like Sunday Robotics, Dyna, and XDOF, who all run their hardware teams out of China. Even Tesla Optimus sources from the Chinese supply chain, and will likely continue to do so.

Unitree’s Verticalization Is Remarkable - Even In China

The Chinese ecosystem is amazing, but we’ve harped on how great verticalization is, so let’s expand on this. Unitree self-develops BLDC motors, planetary gearboxes, LiDARs, and depth cameras, each typically (or even previously for Unitree) outsourced by other Chinese humanoid OEMs. Instead, Unitree’s self-produced motors can run as low as 30-40% of equivalent Western motors, and they now make some of the cheapest humanoid gearboxes in the world.

These same benefits are very apparent in the IPO filing. In Unitree’s First Round Inquiry Response with the Shanghai Stock Exchange (SSE), they explicitly state that scaling production gave them upstream bargaining power, which created a lasting cost advantage.

This translated to margins on their quadruped robots improving from 42.36% to 55.49% while costs dropped nearly in half. However, SemiAnalysis subscribers knew these quadruped margins already back in September.

This verticalization is a spectacle in Western markets, but even within China, where razor thin hardware margins require verticalization for survival. BYD and DJI from earlier showed their path to dominance via verticalization. UBTECH and AGIBot (two of the main Chinese humanoid competitors) are actively moving to bring more of the hardware under control, like gearboxes, motors, etc -- components Unitree already self-develops.

Both UBTECH and AGIBot still lean on ODM/OEM partners for manufacturing and, in some cases, final assembly, like AGIBot outsourcing European production to Minth Group in Serbia. Even then, AGIBot is licensing full tech transfer at $4M, scaling through partnerships instead of owning the manufacturing learning curve. Meanwhile, Unitree states in their S-1 they plan to bring further development inhouse, including “tooth-profile design, simulation optimization, material validation, and high-precision machining.”

However, neither Unitree nor other humanoid companies are at high-volume production. This implies that even as they scale production, Unitree may likely maintain a structural cost advantage from a first-mover standpoint. A future piece will detail the Chinese industrial ecosystem in all its glory, but for now, assume most humanoids will source from China, and even in China, Unitree is exceptional.

Conclusion

While Western manufacturers were/are still prototyping humanoids, Unitree has profitably shipped tens of thousands of quadrupeds, built up an entire humanoid market, entered into useful humanoid work, but everyone should consider: where does this stop for Unitree? BYD began with just battery cells, DJI with just controllers, and are now titans of industry.

Watching how Unitree has accelerated success through multiple robot form factors, and increasingly solidified a manufacturing advantage, they may continue to expand at this same pace and unlock markets previously unthinkable.

For now, let’s discuss robotic hands and which of Unitree’s suppliers may benefit, all now beyond the paywall.

We developed these BoMs by going through a full design examination of Unitree’s robot, speaking with manufacturers of every component, and verifying items with multiple supply chain buyers/sellers.

And, lastly, despite countless dismissive comments against the company, we argue their G1 humanoids are crossing the viability threshold of real-world deployments.

Next we examine their specific hardware strategy, and how their QDD actuator design choice led to a potentially structural advantage, and how their actuator has improved into near-deployment grade.

The Making of A Chinese Hardware Giant

The DJI Playbook - Niche Researcher/Hobbyist Markets Are Viable Bootstraps

DJI pioneered a playbook different from BYD, and Unitree is running that playbook today, starting with the researcher/hobbyist beachhead, and a low-quality product.

Unitree As An Early DJI

Unitree is a live DJI Strategy case study: own the bottleneck component, bootstrap a willing audience, ride and seed the ecosystem, unlock new markets generation by generation. So far, Unitree has

Let’s walk through the history, design, strategy, and the critical threshold they are crossing right now humanoid deployments.

The $30-50K G1 - A New Possibility In 2024

The Ecosystem Advantage

All of this stemmed from Unitree’s decision to perfect the actuator. However, the first generations of their actuator didn’t work very well.

In 2024, Their Humanoids Weren’t Good

Why The Issues Around QDD?

Two Years Later, The QDDs Are Defying Expectations

So Why QDDs In The First Place? A Quick Detour On Speed+Cost

From Burnout To Lightweight Tasks

Heading Towards Useful Work

Looking past performative backflips, Unitrees can already perform “useful work,” non-exhaustively defined here as either.

A task performed in a business to produce some economic output, like sorting boxes, or a task performed to relieve a physical burden of a human, like folding clothes.

None of these are 24-hour, fully-autonomous production lines, and most are still teleoperated. Nonetheless, let’s demonstrate that moving boxes around is becoming economically viable.

Unitrees Are Crossing The Deployment Viability Threshold

\"Thanks to Adamo for helping us understand teleoperated deployments further!\"

A Lovely Task: Tote Handoff

Calculating Unitree’s Economic Viability

Agility currently operates using a 2:1 utilization model, two units of operating time for every one unit of charging, giving Digit a two-thirds utilization rate relative to a human.

Where This Goes

China’s Manufacturing Competence

Unitree’s Verticalization Is Remarkable - Even In China

Conclusion

For now, let’s discuss robotic hands and which of Unitree’s suppliers may benefit, all now beyond the paywall.

Everyone has been talking about datacenters in space. Interviews given by Elon Musk in the past few months have spent lots of time on orbital compute:

“Five years from now, my prediction is we will launch and be operating every year more AI in space than the cumulative total on Earth... I would expect to be at least, sort of five years from now, a few hundred gigawatts per year of AI in space and rising.”
- Elon Musk on Dwarkesh Podcast, February 2026

Furthering space-based compute was also one of the stated motivations behind the merger of xAI into SpaceX (as a ‘reorganization of entities under common control’), and is a key part of SpaceX’s plans to go public, as stated in their S-1 filing on 20 May 2026.

“Our goal over time is to launch 100 gigawatts of compute to space each year. If operated continuously, the generation resources used to support 100 gigawatts of compute could generate approximately one-fifth of the annual power production in the United States, which was 4.4 thousand terawatt hours in 2025… We expect space‑based compute to massively increase AI compute scale, while also improving token economics.”
- SpaceX, S-1 Filing, May 2026

As expected, many part-time prognosticators in the Substack-verse have emerged from the woodwork to weigh in on the concept. Some articles bring up insightful points, but there are more than a few that are built upon ideas that fly in the face of science.

A few casual arguments made in favor of space datacenters include the following:

Many of these points sound like they hold merit on the surface, but a deeper analysis of each apparent advantage reveals a far more complex story.

While we think that it is possible that space datacenters could scale one day, deploying orbital compute using today’s technology currently costs several times more than deploying terrestrial compute. Achieving Space-Earth cost parity will require significant engineering work, material science breakthroughs and cost scaling progresses and will still take years to achieve. There are also important reliability and servicing obstacles to overcome - for instance - how GPU servers will recover from faults that require human intervention, effectively shielding accelerators from radiation, among many others.

When we deploy compute in space, it won’t be because of the four superficial reasons we have cherry-picked above. Rather, Space-based datacenters make sense in the world where AI demand well exceeds all of the four layers of terrestrial datacenter supply that we will introduce below. For Space datacenters to step up to this call - it is a necessary condition that major space datacenter cost items like radiators, solar arrays and launch costs decline considerably, and that a number of key operational obstacles are overcome.

The four layers of incremental power supply for terrestrial datacenters include:

A necessary condition for AI related IT equipment demand to reach levels exceeding terrestrial datacenter supply is for there to be enough chip fabrication capacity to fulfill this demand in the first place, before we even discuss datacenters! We wrote about this in great detail in our recent article on the Great AI Silicon Shortage, where we concluded that the industry has moved from a power-constrained to an accelerator-constrained regime. Available datacenter capacity and power now exceed AI compute demand, but TSMC’s N3 wafer capacity and HBM supply cannot keep pace with the pace of accelerator deployments. This means that today, and for the next few years, chip manufacturing will be the global constraint before we even worry about supply for these four layers.

The chip constraint forms a separate fifth layer of supply - Semiconductor Production, and it is a “universal” constraint on all chip deployment, whether deployed on Earth or in Space. Users of our AI Space Datacenter TCO Model can see how this constraint applies well into the future, and under what scenarios regarding chip manufacturing capacity addition that Semiconductor Production may not be the constraint.

Elon Musk is clearly well aware of this constraint, and it is the impetus behind his Terafab Initiative. The AI Space Datacenter TCO Model also includes knobs and sliders for users to tune to test out various Terafab scenarios.

Framing the Space Datacenter Debate

Our various industry models such as the Accelerator Model, the Foundry Industry Model and WFE Models illustrate the aforementioned chip tightness. Meanwhile our AI Datacenter Model forecasts accelerating incremental datacenter additions in 2027 and 2028. Thus, datacenter capacity addition will run ahead of chip constraints in the next few years until fab capacity additions accelerate to catch up. Our suite of industry models will only forecast such wafer fab and datacenter capacity additions once such plans are confirmed.

However, the world in which AI demand is so overwhelming as to exceed the already formidable datacenter capacity additions is a world with no time for half measures. As such, our AI Space Datacenter TCO Model base case departs from our industry models to reflect this world, assuming accelerating incremental datacenter capacity additions and a meaningful step up in the pace of chip fab capacity addition. It is a world where all the stops are pulled out and many obstacles from gas turbine availability to EUV tool production constraints are overcome because clear long-term AI end use ROI justifies enough capital investment to overcome them.

The below chart illustrates what this world could look like - with incremental datacenter capacity additions eventually in the hundreds of GW annually, though adding chip capacity will still be more difficult than adding datacenter capacity:

In such a base case, our model shows that the cost difference between space and terrestrial datacenters starts at more than 4x in 2026 before narrowing to parity in ~2040, with levelized costs of compute in space declining below terrestrial thereafter. By the early 2030s, space datacenters could be only ~30% more expensive than Earth-based datacenters, opening the door to the first scaled space datacenters as soon as the turn of the decade!

Space-Earth datacenter cost parity opens the door, but in our base case, there is still ample terrestrial capacity - so going into space is a matter of preference and optimization rather than necessity. But if regulatory and capacity bottlenecks starve terrestrial datacenter capacity - space becomes a necessity. This scenario is the basis for our model’s Elon Musk Scenario.

In the Elon Musk scenario, terrestrial datacenter incremental capacity addition peaks out in 2028 and remains low for decades while chip production expansion marches ahead. In this scenario, Space becomes the only alternative for scaled AI datacenter deployments.

In the Elon Musk scenario, the space datacenter TAM could easily reach high hundreds of GW of incremental capacity per year.

Our Total Cost of Ownership Framework is the foundation that allows calculation of levelized cost of compute and it powers the AI Space Datacenter TCO Model. In the rest of this introduction, we will briefly explain the TCO framework and key conclusions from our AI Space Datacenter analysis.

Analyzing Space vs Earth Datacenters using our Total Cost of Ownership (TCO) Framework

In a manner similar to our standard AI Cloud TCO model, we split out various cost categories for space and terrestrial deployments, namely IT cluster capital cost, datacenter capital cost (including launch costs), and operating cost.

Headline TCO Findings using Present Costs and Technology

A 30.5kW B300 cluster (16 GPUs across two servers) deployed in 2026 has a total program capital cost (IT cluster capital cost plus datacenter capital cost) of $4.1M for a space deployment and $1.4M for terrestrial deployments. We use B300s and contemporary mainstream GPU as references for TCO analysis for the next few years, but it is much more likely that smaller, efficient and specialized chips akin to Tesla’s FSD chips will actually be deployed.

Transformed into a levelized monthly ownership cost incorporating respective WACCs and useful lives and adding monthly operating costs, we see a total monthly cost of ownership of $100,925/month for space deployments vs $27,724/month for terrestrial deployments.

Space datacenters are more costly because of a larger upfront capital cost of deployment, with the largest driver being launch costs at $1.6M out of the total $3.1M datacenter capital cost. The cost difference is even starker when considering monthly levelized datacenter costs - because space datacenters are expected to have a useful life of only 5 years vs the 15 years for Earth-based datacenters, monthly levelized datacenter capital costs are a whopping 18x higher than for terrestrial datacenters!

GPU rental pricing is typically quoted on a $/hr/GPU basis - and so we also present the above costs on a per hour basis. Using today’s technology, total cost of ownership on a GPU-hr basis for a B300 GPU is estimated at $8.64/hr/GPU for a space-based deployment, compared to $2.37/hr/GPU for a terrestrial deployment.

In addition to total cost of ownership (TCO) in $/hr/GPU, we also look at Levelized Cost of Compute (LCOC). The difference between TCO and LCOC is that LCOC reflects the net cost of compute needed to meet a certain SLA, given expected cluster reliability. LCOC will be higher than TCO because operators will need to account for radiation availability (i.e. compute availability that is temporarily affected by solar radiation), as well as additional GPUs required (i.e. provision for redundancy given GPU failures that cannot be replaced or repaired).

For terrestrial datacenters, the radiation availability and SLA requirement result in a gross up of about 5% on top of TCO of $2.37/hr/GPU to an LCOC of $2.49/hr/GPU. For space, this means a much larger 26% gross up on top of the $8.64/hr/GPU TCO to reach an LCOC of $10.91/hr/GPU.

All of the above TCO calculations including a further more detailed space cost breakdown are available for all years from 2026 to 2050 in our SemiAnalysis AI Space Datacenter TCO Model.

When Will Space and Terrestrial Costs Reach Parity?

Terrestrial datacenters are clearly more cost effective today, but will that always be the case? When will the effect of technological innovations driving cost efficiencies in space datacenter components as well as the ~80% drop in launch costs from ~$1,400-$1,800/kg for Falcon 9 today to only ~$250/kg for Starship in the future envisioned by SpaceX put costs for space AI datacenters at parity with Earth datacenters?

To answer this question, the AI Space Datacenter TCO Model presents a Levelized Cost of Compute (LCOC) analysis, comparing projections for the cost of compute for space-based and Earth-based datacenters.

In our base case scenario, the cost difference between space and terrestrial datacenters starts at over 4x in 2026, before narrowing to parity in ~2040, with levelized costs of compute in space declining below terrestrial thereafter.

The model and this report will show that successfully scaling technology and launch costs is a necessary condition for deploying space datacenters. Once Earth-Space cost differentials narrow meaningfully, then it is the potential shortfall of terrestrial capacity that will drive actual demand rather than further cost differentials.

Another obstacle to overcome stems from chip reliability and servicing. On Earth, 3-6% of GPUs in a cluster annually suffer failures that require human intervention. To ungate space datacenters, we will need to solve this problem either through robotics, greater reliability, over provisioning or a combination of all of the above.

In what scenario would the cost differential be narrower or could we even see space datacenters come out as decisively more cost effective? The AI Space Datacenter TCO Model allows users to customize many of the underlying assumptions.

In our base case, we make the following assumptions: (a) critical technology and engineering challenges (namely radiation impact and GPU reliability) are sufficiently solved or mitigated by ~2040, (b) large cost categories (launch, radiators, and solar) also achieve sufficient cost-down scaling, and (c) a generally bullish outlook on overall AI demand and chip production ramp.

The second scenario, the “Elon Musk case”, is where, on top of the base case assumptions, we assume incremental terrestrial datacenter capacity becomes even more difficult to ramp and costly (this premise is supported by rising ground DC input costs already being seen today), which further incentivizes space-based datacenters (i.e., more cost-effective to use orbital compute). This scenario also assumes a successful ramp of an additional ~1,000k wafer starts per month (WSPM) by 2040 added by Musk’s Terafab - in itself no mean feat, though by then we already expect a considerable acceleration in fab capacity expansion. Again - both the base case and the Elon Musk case are upside cases and depart from our industry models which focus instead on confirmed capacity additions.

The base scenario sees Space-Earth cost parity, as measured by Levelized Cost of Compute (LCOC) in $ per PFLOP-hour, by ~2040, while the Elon Musk Case sees near-parity earlier - by the early 2030s.

Compared to our base case, the Elon Musk case also sees greater and earlier space datacenter demand given less terrestrial datacenter supply to fulfill overall demand (subject to AI silicon constraints).

There is a lot of ground to cover to support the above conclusion, and in the rest of this report, we will deep dive into the underlying details of this framework and the build up of all the major assumptions underlying this analysis.

Part One is a warm-up round that will debunk a few misleading arguments made regarding space datacenters. This is a great way to introduce readers to important and foundational concepts.

Part Two will focus on discussing the four layers of incremental power supply here on earth after briefly addressing a few of the casual arguments for space datacenters.

Part Three introduces the AI Space Datacenter Total Cost of Ownership (TCO) Model and explains the SemiAnalysis TCO framework that is used in the model to discuss and compare costs for terrestrial and space datacenters.

Part Four explains in detail the build up of a space datacenter with a system by system cost breakdown as well as a discussion of the science behind space datacenter design and implementation.

Taken together, this deep dive will give readers a framework for evaluating future scenarios that may make space datacenters a viable decision and evaluating the arguments for or against space datacenters.

Launching our AI Space Datacenter TCO Model

Today, we are also launching our AI Space Datacenter TCO Model into General Availability. The model provides a first-principles, system-level framework for evaluating orbital compute economics, engineering constraints, and supply-demand dynamics across both terrestrial and space-based infrastructure.

It spans from launch vehicle physics and thermal rejection limits to AI demand curves and GPU-level cost of ownership, enabling users to stress-test when and how compute may transition off Earth. Our model spans 2026 - 2050, with dynamic scenario modeling driven by user-controlled assumptions.

The model will answer the question of if and when space datacenters will become economical and will show the scenarios with respect to end demand and costs on Earth that make Space a viable deployment option.

Please reach out to sales@semianalysis.com to find out more!

Part One: Debunking the Four Casual Arguments for Space Datacenters

Deploying datacenters in space will be hard, but for different reasons than most think. Let’s first debunk the overly simplistic arguments behind these four assertions regarding orbital compute.

Argument #1: You Get 24 Hour Free Solar Energy in Space

Most orbits do not actually achieve 24-hour exposure to the sun. The ISS and majority of the Starlink constellation are in Low Earth Orbit (LEO - within 400-500km of earth) and complete ~15 orbits per day, meaning objects in LEO are receiving sunlight for ~60% of the time. This means that out of 1,361 W/m² potential solar irradiance, a space datacenter in LEO might only capture ~800 W/m² in an average 24 hours. The datacenter will also require battery energy storage sufficient to supply 100% of the IT power when the space datacenter is in eclipse, adding hardware cost and complexity.

Instead, a Sun-Synchronous Orbit (SSO) is ideal for Space datacenters. SSO is a retrograde orbit (in the opposite direction of earth’s rotation) at a high inclination of >90 degrees that tracks the Earth’s terminator and faces the sun most of the day, save for an eclipse of up to 35 minutes per day. The battery energy system required to power the datacenter during eclipse is of far lower capacity than for LEO, but it still entails integrating some power electronics as well as some engineering complexity.

Argument #2: Cooling is Free

This is perhaps the most misunderstood claim - cooling is decisively not “free”. In fact, it’s quite the opposite. Space is cold, but it is called “Space” for a reason - unlike on Earth, where the atmosphere can conduct heat away from datacenter cooling towers using convection, there is nearly nothing in space to conduct away the heat via convection, and the primary way to remove heat from a datacenter in space is through radiating the heat away. Radiation can remove heat from an object even in the vacuum of space as internal thermal energy leads to the oscillation of charged particles generating photons that remove the heat energy from the object.

Cooling is the largest structural constraint for orbital compute. The International Space Station’s radiator system can only remove 70kW of heat (a quantity about half of what is required to cool a 140kW GB300 NVL72 rack!), requiring a total area of 325 m² and a total cost of $340-$500M – more on this later.

Admittedly this system was based on 30 year-old technology with plenty of cost bloat baked in, and costs have improved considerably since then. However, it illustrates the fact that heat removal is one of the core engineering problems to solve when it comes to space datacenters.

Argument #3: You Get The Lowest Latency in Space

Low communications latency is often cited as a key benefit enjoyed by deploying compute in space. Nothing is faster than light in a vacuum. However, the truth is more complicated. A LEO compute satellite completes about 15 orbits per day and is only over a given ground station for 5-7 minutes per pass. If the satellite is over the ground station nearest to you, then the connection will be strong, but this will only happen for 5-7 minutes per day.

Outside that window, the traffic must go to another satellite in the constellation or go through multiple inter-satellite links to reach a gateway that is closer to the end user. If there is a satellite passing over the Indian Ocean that is serving a US Customer, those hops through multiple Inter Satellite Links (ISLs) can accumulate 30-80ms of one-way delay. This problem worsens when you switch to optical ground links instead of RF as there are a limited number of ground stations at any given time that do not suffer from some degree of atmospheric interference. This necessitates the deployment of many ground stations spread throughout the world. Communicating from these dispersed ground stations to the end user is yet another source of added latency.

Argument #4: No Need for Permitting in Space

Permitting is more than just a headache when building a datacenter on Earth. In the US, projects face transmission cost inflation and study-related uncertainty, on top of the 5-7 year waits to connect to the grid. As we have outlined in the latest major expansion of our Energy Model, there has been substantial growth in onsite gas and behind-the-meter (BTM) projects as a workaround to waiting in the interconnection queue, but air permitting remains a significant hurdle. Wait times for turbines are also long, with the main OEMs, such as GE Vernova and Siemens Energy, having limited capacity to serve projects planning to come online before the end of the decade.

When it comes to space, one wrinkle is that SSO is a constrained subset of LEO, rather than a separate orbital regime. LEO spans 400-2,000 km across many inclinations, with carrying-capacity estimates ranging from ~100,000 to over a 1M satellites depending on separation and collision-tolerance assumptions. SSO, by contrast, requires a specific altitude-inclination relationship (roughly 97–99° inclination at 500-1,000 km) so that the orbital plane precesses at the rate needed to maintain a fixed local solar time.

Most operational SSO traffic concentrates in a 600–800 km sweet spot. Dawn-dusk SSO, the specific subset that rides the terminator and is relevant to orbital compute, is even narrower. It represents a single local-time slot within an already constrained orbit class, and is materially smaller in usable capacity than LEO.

Other than dawn-dusk SSO, there are other “orbits” that are exposed to the sun 24/7 - for example the Sun-Earth Lagrange Point L1. But here latency is a clear deal-breaker - it takes light ~10 seconds to travel the 3 million km round trip between Earth and L1.

Part Two: Terrestrial Datacenter Constraints and the Four Layers of Incremental Power Supply on Earth

The Four-Layer Framework: How Constrained is the Terrestrial System?

In a recent appearance on the Dwarkesh Podcast, Elon Musk promoted the idea of orbital compute, arguing that Space would shortly be the most compelling place to deploy AI. Musk’s argument is not just that space-bound chips may matter eventually, but that terrestrial power, turbines, and permitting will hit a wall fast enough that space could become the cheapest place to run AI within roughly three years. There are warnings from the turbine manufacturers, and queues to connect to US grids that could last years, so you can see why he thinks this.

Musk and SpaceX clearly believe in this and have put their money where their mouth is, with corporate commitments rewarding Musk with up to 302M Class B SpaceX shares (between ~$30B and ~$60B USD assuming a $100-$200 IPO price range) for delivering 100 TW per year of non-Earth based datacenters – though we will size the analytical realism of that milestone later in this report. On top of that, the recent launch of the Terafab initiative acknowledges one other additional constraint: you can’t build a datacenter anywhere if you don’t have the chips to fill it with.

To stress-test Musk’s power argument on its own terms, it’s worth asking: even if chip supply were unconstrained, would terrestrial power suffer constraints so severe as to force AI compute into orbit?

Borrowing the Peak Oil Framework

The “Peak Oil Theory” framework is helpful in answering this question. In the 1970s, this theory that global oil production was nearing a hard geological ceiling, and that oil would soon become impossible to produce and use. Instead, new supply sources became economical as conventional supply tightened and oil prices increased, boosting the incentive to drill harder. Supply moved up the cost curve and became more infrastructure-intensive, and in more recent decades, the technology improved to access more oil supply, stabilizing prices.

That framework can be applied to datacenter power, where the sources of power supply are varied and the barriers to entry are a lot lower than the much more consolidated chip manufacturing space, which has plenty of near-monopoly producers across the supply chain.

Let’s borrow from the Peak Oil Theory as we walk through the terrestrial power stack layer by layer. As we move through the latter half of this decade, power access is set to get scarcer and trickier to execute on even when available - the end result is that power becomes more expensive. Cost escalates as we tap into more difficult to access sources of power.

All of these layers will have to become exhausted before space becomes an economically viable alternative, or even a preferred option. The fifth layer is a “universal” constraint on all chip deployment, whether deployed on Earth or in Space.

While datacenter capacity was the bottleneck in the past few years, the overriding constraint today has now moved up the stack into semiconductor manufacturing — specifically advanced-node logic capacity at TSMC, HBM production at SK Hynix, Samsung and Micron, and DRAM output across the industry.

Moving giga- or even terawatts of chip capacity into space is a non-starter if we cannot even manufacture the chips to put up there.

As chip manufacturing constraints begin to limit the pace of AI accelerator production, we will now counterintuitively have sufficient datacenter supply in 2027 to cater to chip demand. Our SemiAnalysis AI Datacenter Model models confirmed facility and land bank pipelines, and in the AI Space Datacenter Model, we take a more optimistic capacity view by including potential capacity sources that have yet to be enumerated. Adding layers one through four show that total tracked global datacenter capacity (excluding China) could rise from 89 GW in 2026 to up to 338 GW in 2030 - nearly quadrupling overall capacity over four years. However, this capacity is delivered by tapping layers two to four to allow for more datacenter capacity, and accordingly we see converted capacity, behind-the-meter (BTM) generation, and industrial production expansion kicking in.

Current projected builds and allocated capacity implies tens of gigawatts of headroom in the next few years vs planned accelerator deployment. Unfortunately, Accelerator demand will be bottlenecked by chip production capacity for the next few years, and this is a problem that Space Datacenters cannot solve.

The below table from the SemiAnalysis AI Space Datacenter TCO Model illustrates how silicon capacity holds back AI chip deployment even when more datacenter capacity can be stood up.

Notwithstanding the above, let’s step through each layer of supply and discuss the constraints involved for each layer in detail.

Layer One: Grid-Connected Supply

Grid-connected power is still the cheapest on paper, with infrastructure costs in the $12–15M/MW range. But in most US markets, the real cost is waiting in queue for power interconnection from the grid. North Virginia’s PJM interconnection timelines now run to roughly seven years in practice and that timeline is not workable for hyperscalers that can barely keep up with end demand.

Of course there is an element of overbooking interconnect capacity, but even discounting for speculative filings, the approved and under-construction pipeline is large enough that physical transmission and substation constraints in the major build markets constrain the pace of power addition for the next few years. More on this can be found in our article on the Electric Reliability Council of Texas, i.e. ERCOT’s new batch study process.

Across US ISO territories, gross positive grid reliability headroom (defined as the buffer of effective supply above peak demand plus required reserve margins, measured on SemiAnalysis's penetration-adjusted ELCC methodology) stood at roughly 18.3 GW in 2025, down from 70.2 GW in 2021. In practical terms, grid reliability headroom is a metric that is linked to how much capacity can be connected to the grid at all.

That buffer narrows to just 15.9 GW in 2026 before net headroom turns negative in 2027, reaching an aggregate deficit of approximately 40 GW by 2030. Negative headroom means the grid is oversubscribed on a planning basis - more load is projected than accredited supply can reliably serve under each ISO's reserve margin requirements.

From a global energy perspective, datacenter electricity consumption reached nearly 340 TWh in 2024 - roughly 1.1% of global power output. AI-specific demand accounts for around 0.3% of global generation today, crosses 1.7% around 2027 in our base case, and reaches just under 5% by 2030. Even in an accelerated scenario it stays around 7% by 2030, with that ceiling representing roughly 380 GW of continuous power demand.

US planned power generation capacity additions are stepping up sharply in the coming years. Raw nameplate overstates what is actually usable, so we use Effective Load Carrying Capability (ELCC) to measure how reliably each source can meet peak demand. This adjusts for intermittent solar and wind power sources which contribute far less to the grid’s firm capacity than their headline numbers suggest.

This is a picture of progress. The US grid is adding capacity, solar is coming in faster than any prior generation technology, and international buildout continues to absorb demand across EMEA and APAC.

The ~106 GW of global datacenter capacity by 2027 generation additions and grid-connected supply can keep pace with accelerator demand. Beyond that, the grid alone cannot carry the load and we are pushed into the next layer of capacity addition.

Layer Two: Converted Capacity and Powered Land

When looking beyond the grid - the first place to look is converting existing facilities with power into AI Datacenters. Cryptominer conversions are the clearest example of this second supply layer. Core Scientific, IREN, Cipher Mining, Applied Digital, and TeraWulf together point to roughly 2 GW of tracked contracted conversions by the end of 2026 and roughly 5 GW by the end of 2027.

These are sites with existing grid connections, permitted substations, and in some cases usable cooling infrastructure. Although the quality of the existing technology on-site can vary, leading these sites to be similar or even lower in cost to grid-connected supply at around $10–15M/MW. Firms like Fermi Energy and Cloverleaf are bringing large, grid-connected sites to market by taking interconnection risk themselves rather than leaving it to hyperscalers. Oracle has committed at least $3.65B to support 1.4 GW of capacity at the Related/Oracle/DTE Electric campus in Michigan. That would have sounded extreme not long ago, but looks rational now that 1 GW of AI capacity can support on the order of $12–13B of annual revenue.

In aggregate, converted sites and powered land can add 8-10 GW of near-term supply, with Cryptominer conversions supplying 8GW cumulatively by 2028. This near-term supply can act as a relief valve for a few years before the repurposable inventory is mostly absorbed, and the industry turns to the next layer: off-grid power supply.

Layer Three: Behind-the-Meter (BTM) Generation

BTM generation was once considered a last resort, but when major AI cloud contracts imply annual revenue in the range of roughly $12–13M per MW of contracted critical IT load, getting 200 MW online six months early carries an NPV of roughly $400M-$500M. So any additional capex from bringing your own generation (BYOG) and skipping the queue, can result in quick reward. The all-in BTM costs can still fall in roughly the $110–170/MWh range depending on technology, not dramatically different to grid power that can already clear around $150/MWh in major US markets.

BTM annual additions in critical IT power terms — predominantly US-based — are set to be the primary source of power for half of new AI datacenter power capacity additions by 2028 — up from fewer than 7% of capacity added in 2025. Cumulative confirmed BTM critical IT capacity is set to reach 26 GW by the end of 2030, but this number will likely be much higher still from unannounced projects, with OEMs stating the majority of their orderbooks and enquiries come from datacenter projects. Small modular reactors for BTM operations could come into play, with perhaps 1–3 GW of supply post-2030.

The system-wide DC-addressable ceiling across all BTM generation categories runs into the tens of gigawatts per year by 2027 at a cost of $15-20M/MW — spread across six or seven distinct supply chains that don’t all tighten at once.

The BTM market is growing incredibly quickly and existing OEMs and new entrants alike are flooding into the sector. We will soon be launching an expansion to the SemiAnalysis Energy Model, tracking more than 30 OEMs, forecasting quarter by quarter manufacturing capacity, installations, and availability.

Layer Four: The Production Constraint in Industrials

The headroom for more compute on Earth is there from a power and shell perspective once we pull the levers of converted capacity and BTM generation, but the next layer, Layer Four: Industrial Production, consists of supply created by mustering capital for building additional manufacturing to enable more provision of all the prior layers.

Large power transformers already have some of the longest lead times in the electrical stack, this is because building transformers depends on a small number of grain-oriented electrical steel (GOES) producers globally. Transformers are generally required in both grid-fed and BTM scenarios alike. Copper is another restriction, given how broadly it sits across transformers, turbines, cables, busbars, and cooling equipment. Copper is difficult to mine, and has seen a nearly 20% price jump over the past year. That said, the increase in prices further downstream to the transformer manufacturers, are mostly demand-signal driven, and not reflective of actual supply. And the transition to HVDC and optics in networking are increasingly being considered as antidotes to this issue.

Labor for construction and operations is getting more costly, but modularization and digitalization of datacenters has already been implemented in buildout, and can reduce on-site labor needs by more than 50%. Still, a substantial number of skilled man-hours remains, and this would compound substantially once we head into the multiple hundreds of GWs of compute territory.

Tapping into this layer would push costs per MW beyond the $20M/MW range - how far beyond will depend on how many tens or even hundreds of gigawatts of capacity but how costs escalate once we are in this fourth layer will determine whether space datacenters make economic sense.

There are a wide variety of outcomes cost-wise. Users of our new AI Space Datacenter TCO Model can adjust assumptions around base Layer Four costs and the price elasticity of supply to analyze various scenarios.

Layer Five: The Semiconductor Ceiling

In the podcast with Dwarkesh, Musk does not name the hardware that would be best suited to his stellar ambitions. But does this even matter? As we have established, ramping up production for any chip is also the biggest constraint to mass cluster buildout on Earth. Given trends in Semiconductor production tightness - we will hit a silicon capacity constraint well before the other layers.

AI-related demand is modeled to consume just under 60% of TSMC’s N3 output in 2026 and approximately 86% in 2027, nearly squeezing out remaining smartphone and CPU demand. Additional fab area must first be built before chips can be projected into orbit.

Memory faces the same problem from another angle: incremental DRAM capacity is increasingly absorbed by HBM, which consumes roughly three times the wafer capacity of commodity DRAM on a bit per wafer basis basis. Our recent report on silicon and memory shortages goes deeper into this. On the DRAM side, AI-related demand is projected to consume roughly 70% of total DRAM wafer capacity by 2027, up from 12% in 2023 — a near six-fold increase in four years.

The structural reason this ceiling is harder to move than the power stack is cleanroom physics. Adding advanced fab capacity requires building cleanrooms first, then installing tools, then qualifying processes — a sequence that is hard to meaningfully accelerate in the near term regardless of capital availability. The point where that changes looks more like 2032–2034 than 2027–2029, as we will look into further.

In our AI Space Datacenter Model base case - logic and memory capacity additions move in lockstep, though memory tends to be the binding constraint for much of our forecast horizon.

Our base case assumes that there are considerable resources mustered to add semiconductor manufacturing capacity - this is clearly not a conservative assumption and there are plenty of bottlenecks that could derail this base case. Yet - even with these loosened assumptions, the silicon constraint is binding in the long-term even though we forecast it supporting hundreds of GW of AI capacity additions in our base case.

In the Elon Musk Scenario - we keep the same robust chip manufacturing capacity expansions, and with much lower terrestrial datacenter capacity, the AI industry must shift deployments to space.

Getting even close to fulfilling 800 GW of AI demand will require the entire global EUV fleet dedicated to AI with nothing left for phones, PCs, or anything else. Some semiconductor relief could materialize in the 2032–2034 window, as TSMC’s Arizona and Japan capacity comes fully online and memory fabs respond to sustained HBM demand with dedicated capacity additions.

Musk’s Terafab Initiative

During the March 2026 launch, Elon Musk framed Terafab as a 1 terawatt per year compute factory. Tesla, SpaceX, and xAI would build it together in Austin on a $20-25B budget, starting at 100K wafer starts per month and climbing toward 1M wafer starts per month (WSPM), roughly 70% of TSMC’s entire global output today.

The plant sits on 100 million square feet of floor space, which Musk compared to 10x Giga Texas or 15 Pentagons, across thousands of acres, and draws more than 10 gigawatts. Scope covers logic, memory, mask shop, advanced packaging and testing. Compute allocation splits 80% space and 20% terrestrial, roughly 800 GW for orbital datacenters and 100-200 GW for terrestrial inference - this aligns with SpaceX’s recent S-1 disclosures to have one chip variant optimized for terrestrial edge and inference (e.g. Tesla’s Optimus robots, vehicles), and another chip variant for orbital compute. First wafers are claimed to be out as soon as next year in 2027, with mass production in 2028.

We are not one to bet against Musk. He built reusable rockets, scaled EV production to nearly 2M cars a year, and designed custom inference chips that compete with the best in the industry. Even a fraction of Terafab is a meaningful success. The numbers still matter.

Start with capacity. Our Foundry Model puts global 300mm foundry capacity at 4M+ WSPM in 2025. Terafab at the 100K entry ramps in as 2.5% of the entire world of foundry wafer starts. Terafab at 1M full scale is 24% of global foundry capacity, or 68% of TSMC alone. Nobody has added that much in a decade outside of TSMC itself, and TSMC took three decades to get there.

Our Accelerator and HBM Model puts 1GW of deployed compute at 354K wafer starts across logic, memory, and packaging, with memory over 60% of the total. Wafer value per GW runs near $3B. If “1 terawatt” means 1TW of simultaneously deployed compute, that is 354M wafer starts a year, or 21x TSMC’s entire global output. If it means cumulative installed base over 15 to 20 years, the math works but only barely. Our best guess is that “terawatt” plays the same branding role that “giga” played for Gigafactory, originally pitched as producing more batteries than the rest of the world combined.

Process IP is the deeper constraint and the reason the Samsung and TSMC deals matter most. Tesla has no manufacturing IP. The incumbents hold proprietary process technology. GAA transistor design, interconnect, lithography and etch recipes, and yield engineering refined over decades. Licensing is the only realistic path. Our view is that Terafab, if it ever reaches volume, operates as an integration fab on a licensed node the way Rapidus is attempting, not as a greenfield process developer.

The memory claim is the hardest to square. Musk expressed his opinions in the Tesla Q4 2025 call that “memory is an even bigger limiter than AI logic” without specifying what type of memory - HBM, LPDDR, or NAND. Each is a distinct process with IP concentrated in Samsung, SK Hynix, and Micron and thousands of patents. Long-term supply contracts or co-investment with an existing DRAM maker is the realistic path, not fabrication from scratch.

The SemiAnalysis AI Space Datacenter TCO Model allows users to bake Terafab capacity additions into the overall chip constraint model. Our Terafab base case doesn’t fully adopt the full extent of Elon Musk’s ambitious plans, but it still lifts the silicon constraint somewhat.

Assuming the aforementioned scenario adopted by our AI Space Datacenter TCO Model in its base case, whereby we depart from our other industry models to show accelerating incremental datacenter capacity additions and a meaningful step up in the pace of chip fab capacity addition, we see the following results.

With incremental datacenter capacity additions eventually in the hundreds of GW annually, and obstacles from gas turbine availability to EUV tool production constraints are removed, demand could be met by space or terrestrial datacenters.

Users of our AI Space Datacenter TCO Model can dynamically adjust these capacity parameters and silicon constraint assumptions to dial in scenarios as needed.

In such a scenario, the question then shifts from whether Earth or Space has more datacenter capacity, to a question of the total cost of ownership in Earth versus Space - bringing us to Part Three of this article.

Part Three: The Total Cost of Ownership Framework for Space Datacenters

In our total cost of ownership framework, we separate costs into three distinct buckets: IT Capital Costs, Datacenter Capital Costs, and Operating Costs. We begin with a line-by-line build of cluster capital requirements and then roll those upfront costs together with recurring operating expenses and an amortized cost of the datacenter to produce the below summary cost of ownership in $/hr/GPU and ultimately Levelized Cost of Compute expressed in units of $/hr per PFLOP.

Looking at a 30.5kW B300 datacenter deployed in 2026, we see that the total monthly cost of ownership is nearly four times higher for a space deployment as compared to a terrestrial deployment. Levelized Cost of Compute (LCOC) measured in $/PFLOP-hour for space is well over 4x the cost for an Earth datacenter. This difference is entirely driven by the much higher datacenter capex cost - 8x higher for space datacenters, but with a monthly cost of ownership 17x higher given the 5y useful life for space datacenters vs a 15Y useful life for Earth datacenters.

We use B300s and contemporary mainstream GPU as references for TCO analysis for the next few years, but it is much more likely that smaller, efficient and specialized chips akin to Tesla’s FSD chips will actually be deployed.

For IT Capital Cost of Ownership, space and terrestrial datacenters are largely identical. Servers, networking fabric, and the software, storage and orchestration stack are the same in orbit as on the ground with the exception of warranty and capitalized burn-in costs.

Turning to Datacenter Capital Cost of Ownership, the space and terrestrial models diverge most sharply. Terrestrial datacenters’ largest cost buckets are shell construction, chillers and cooling towers, transformers and grid interconnects, and physical security. Orbital deployments eliminate land entirely and replace each of these line items with space-specific equivalents which we list below.

When analyzing Operating Cost of Ownership, the cost profile inverts. Terrestrial datacenters pay for grid power and run a steady stream of on-site technicians for maintenance, while space datacenters do not have an incremental cost of power (they get power from solar panels) and they pay ground control operating costs which are allocated to the satellite.

IT Capital Cost of Ownership

For the first few years of space datacenter deployment, we expect that IT equipment deployed will be fundamentally similar, with a low degree of customization and similar costs. Over time, if the pace of space datacenter deployments increase, we could expect to see some limited divergence in cost, though with AI XPUs, much of the IT cost is related to the core GPU+HBM package, leaving limited room for variance due to customization.

Using our 2026 30.5kW Datacenter concept, capital costs for a B300 cluster include the server cost itself, scale-out networking, attached storage and several smaller line-item costs required to bring a cluster online. Two servers consisting of 8 B300s each carries a base server cost of $880,400 with Service, Networking, Storage, Software and other infrastructure components contributing an additional $89,956. With an additional capitalized burn-in cost of $10,526 for space deployments, and a capitalized server service cost of $15,802 for terrestrial deployments, this brings the total upfront IT cluster capex to $980,882 for space deployments and $986,158 for terrestrial deployments.

To reflect the increased risk from the relative immaturity of space datacenter deployments, we use a flat Weighted Average Cost of Capital (WACC) of 10.3% for terrestrial deployments, which corresponds to a ~7% pre-tax cost of debt, a 20% cost of equity, and a 75/25 debt-equity split. This compares to an initial WACC of 15.0% for space deployments that declines over time to reach parity at 10.3% within ~10 years once space datacenters are more mature and de-risked. Assuming a 5-year useful life for IT equipment, this translates into an IT capital cost of ownership of approximately $2.00/hr/GPU for space versus $1.81/hr/GPU for terrestrial.

Minor adaptations include tolerating radiation-induced faults such as Single Event Upsets (SEUs) or Single Event Functional Interrupts (SEFIs) through Error Correcting Code (ECC) memory, watchdog resets, and graceful restart mechanisms, an approach already demonstrated at scale by constellations such as Starlink which did not require expensive rad-hard processors.

In practice, the compute nodes themselves remain fundamentally the same as their terrestrial counterparts, but the surrounding power, thermal, and reliability systems are engineered so that standard datacenter hardware can operate reliably in the radiation, thermal, and power environment of orbit. We cover these costs in the following sections.

Datacenter Capital Cost of Ownership

While IT Capital Costs for space datacenters and Earth datacenters are similar, the cost gap is much wider when we look at Datacenter Capital Cost of Ownership. For terrestrial datacenters, there are two ways to incorporate datacenter facility costs, an opex-based approach and a capex-based approach.

In the opex based cost model, the Hyperscaler or Neocloud rents colocation capacity and cost is quoted in terms of USD per kilowatt-hour (kWh) of critical IT power per month. In the capex-based model, we assume the Hyperscaler or Neocloud builds and owns the facility outright, and we calculate a levelized cost of ownership based on the following: 5-year useful space datacenter facility life until 2032, then assuming improvements drive this to 10-year useful life after.

The AI Space Datacenter TCO Model exclusively uses the capex-based model for Earth datacenters to allow a like for like comparison as we think the most likely business model will be a vertically integrated deployment where a provider like SpaceX will own the space datacenter as well as the IT compute installed within. Further breakdowns of the useful life and capex required for self-built Earth datacenters can be found in our earlier articles on datacenter energy and cooling.

Compared to Earth datacenters, orbital deployments eliminate land entirely and replace each of the usual Earth datacenter capex line items with space-specific equivalents: satellite structure and shielding instead of buildings, radiators, cold plates and closed-loop liquid cooling systems instead of HVAC, solar panels and batteries instead of grid electrical, and the satellite bus (ADCS, propulsion, command and data handling) instead of facilities. On top of that, space carries key cost lines that simply do not exist terrestrially — launch ($/kg to orbit), one of the major cost driver of the entire model, as well as other key costs covering radiation shielding, propulsion and assembly, integration and test.

For our 30.5kW B300 datacenter deployed in 2026, overall datacenter capital costs (this excludes IT costs) at today’s technology come out at $3.1M for a space deployment versus $382K for a terrestrial deployment. Space datacenters deployed in 2026 are more costly because of a larger upfront capital cost of deployment, with the largest driver being launch costs at $1.6M out of the total ~$3.1M datacenter capital cost.

The cost difference is even starker when considering levelized datacenter costs - because space datacenters are expected to have a useful life of only 5 years (this is assumed to eventually extend to 10 years from 2032 onwards due to improvements in in-space robotics) vs the standard 15 years for Earth-based datacenters (due to buildings and facilities that outlast the GPUs themselves). When these are taken into account, datacenter capital costs of ownership are a whopping 17x higher than for terrestrial datacenters at $6.29/hr/GPU for space compared to $0.36/hr/GPU!

Within our AI Space Datacenter TCO Model, most of these line items are fully adjustable, allowing users to customize individual items based on their assumptions around cost and technology scaling for these space systems.

Operating Cost of Ownership

As mentioned above, when it comes to operating cost of ownership, the TCO profile inverts between space and terrestrial. Terrestrial datacenters run a steady stream of on-site technicians, grid power tariffs (electricity and onsite gas), and continuous hardware replacement and cooling upkeep.

The orbital model has none of these recurring costs: there is no servicing once in orbit, no ongoing power bill because solar capex is paid upfront, and failures are absorbed through roughly 20% spare redundancy rather than physical maintenance. The orbital model does have several opex lines related to ground operations cost (launch, control, communications) that do not apply to a terrestrial model.

Turning back to our Earth-based B300 cluster, we assume an electricity cost of $0.087/kWh, a power utilization rate of 80% typically achieved by serving steady training workloads and a Power Usage Effectiveness (PUE) of 1.35. PUE represents the additional power above Critical IT Power required for cooling, power delivery, and other facility-level systems. Total operating cost of ownership is dominated by power tariff and lands at $0.21/hr/GPU.

Operating costs for space datacenters are dominated by allocated operations costs, at $4,167 per month, amounting to $0.36/hr/GPU. This scales rapidly however, as fixed ground operations costs are amortized over a larger fleet of space datacenters - our modeling shows operating costs for space datacenters dropping as low $0.15/hr/GPU by the mid-2030s - one third to half of the operating cost of Earth datacenters, though it’s important to remember that IT and datacenter capital cost is still the dominant cost component for both Earth and Space.

From Total Cost of Ownership (TCO) to Levelized Cost of Compute (LCOC)

Taking our per-GPU-hour costs for IT capital costs, datacenter capital costs, and operating costs together, we see that total cost of ownership per GPU-hour a B300 datacenter is $8.64/hr/GPU for an orbital deployment compared to $2.37/hr/GPU for a terrestrial deployment.

The final adjustments required to bridge the total cost of ownership to a levelized cost of compute (LCOC) stem from radiation availability and additional GPUs required. Radiation availability refers to compute availability net of what is temporarily degraded or affected by solar radiation, i.e. occasional faults that take the affected hardware offline temporarily - we model this at 95% for space versus 100% (no effect) for terrestrial. Additional GPUs required to reach a 99% SLA refers to additional GPUs provisioned for redundancy given hardware failures that cannot be fixed through software or a soft reboot - on earth we see cold spares of up to 5%, however given that orbital chips are not able to be mechanically repaired, we assume a 20% redundancy for failures.

This gives us a Levelized Cost of Compute (LCOC) in 2026 for our B300 cluster of $10.91/hr/GPU for an orbital deployment compared to $2.49/hr/GPU for a terrestrial deployment.

However - the silicon area per GPU and thus compute throughput and IT power per GPU continues to evolve. The trend is for a greater number of individual silicon dies per GPU with more compute dies and input-output (I/O) dies finding their way into each contemporary GPU. However - the trend may be the opposite for AI Chips deployed into space datacenters. It is much more likely that smaller, efficient and specialized chips akin to Tesla’s FSD chips will actually be deployed. Larger chips will have a greater compute throughput in PFLOPs and produce more tokens/s, but ultimately both smaller efficient space AI chips and larger ground GPUs will have capabilities that scale by power.

To normalize for this and model in a way that is agnostic to the definition of a GPU or the size of an AI chip, we project PFLOPs/Watt, Capex/Watt, and Watt per silicon area. As a result, our figure of merit for measuring LCOC in the long term is LCOC per PFLOP-hour - the cost per PFLOP of compute throughput for one hour.

On a marketed FP8 dense FLOPS basis, B300s are rated at 4,500 dense FP4 TFLOPS, giving ground-based deployments an LCOC of $0.17/PFLOP-hr per marketed FP4 Dense PFLOP, and space datacenters an LCOC of $0.73/PFLOP-hr.

Similarly, our open benchmarks under InferenceX suggest that with disagg TRT, MTP using Deepseek R1, B300 token throughput under FP4 is ~5,100 Tok/s per GPU, mapping to an inference LCOC of $590 per Billion Tokens for a 2026 space datacenter vs only $135 per Billion tokens for an Earth datacenter.

As we have alluded to many times above, deploying a few 30kW space datacenters in 2026 is clearly sub-scale. For the space-ground cost parity and eventual crossover to happen within a commercially meaningful planning horizon, space LCOC needs to fall much faster than that of terrestrial datacenters.

So When Does Space Actually Get Interesting?

Space compute becomes interesting when it becomes economically viable by being at or near cost parity with terrestrial datacenters. There is still a world of engineering and cost challenges between now and then that SpaceX, the supply chain, and the space compute ecosystem will have to go through iterations of trial, error, and operationalization before cost parity is achieved.

Despite these challenges, the economic opportunity to meet exponential AI demand at a more cost-effective unit basis over the long-term is the north star for SpaceX, and something that operators, investors, and enthusiasts must not ignore.

In the next sections, we’ll work through the base case and ‘Elon Musk’ case and how cost parity evolves differently between the two cases. We will dive into an explanation of each major space datacenter cost item, deriving these costs from first principles and how they evolve over the years. We will also discuss key considerations regarding launch costs, system and subsystem capital and operating costs, compute payload costs and more. Subscribers to our AI Space Datacenter TCO Model also have full access to the above calculations and assumptions and can dynamically adjust almost all of the underlying assumptions.

Everyone has been talking about datacenters in space. Interviews given by Elon Musk in the past few months have spent lots of time on orbital compute:

A few casual arguments made in favor of space datacenters include the following:

Many of these points sound like they hold merit on the surface, but a deeper analysis of each apparent advantage reveals a far more complex story.

The four layers of incremental power supply for terrestrial datacenters include:

Framing the Space Datacenter Debate

In the Elon Musk scenario, the space datacenter TAM could easily reach high hundreds of GW of incremental capacity per year.

Analyzing Space vs Earth Datacenters using our Total Cost of Ownership (TCO) Framework

Headline TCO Findings using Present Costs and Technology

All of the above TCO calculations including a further more detailed space cost breakdown are available for all years from 2026 to 2050 in our SemiAnalysis AI Space Datacenter TCO Model.

When Will Space and Terrestrial Costs Reach Parity?

The base scenario sees Space-Earth cost parity, as measured by Levelized Cost of Compute (LCOC) in $ per PFLOP-hour, by ~2040, while the Elon Musk Case sees near-parity earlier - by the early 2030s.

Part One is a warm-up round that will debunk a few misleading arguments made regarding space datacenters. This is a great way to introduce readers to important and foundational concepts.

Launching our AI Space Datacenter TCO Model

Part One: Debunking the Four Casual Arguments for Space Datacenters

Deploying datacenters in space will be hard, but for different reasons than most think. Let’s first debunk the overly simplistic arguments behind these four assertions regarding orbital compute.

Argument #1: You Get 24 Hour Free Solar Energy in Space

Argument #2: Cooling is Free

Argument #3: You Get The Lowest Latency in Space

Argument #4: No Need for Permitting in Space

Part Two: Terrestrial Datacenter Constraints and the Four Layers of Incremental Power Supply on Earth

The Four-Layer Framework: How Constrained is the Terrestrial System?

Borrowing the Peak Oil Framework

Moving giga- or even terawatts of chip capacity into space is a non-starter if we cannot even manufacture the chips to put up there.

The below table from the SemiAnalysis AI Space Datacenter TCO Model illustrates how silicon capacity holds back AI chip deployment even when more datacenter capacity can be stood up.

Notwithstanding the above, let’s step through each layer of supply and discuss the constraints involved for each layer in detail.

Layer One: Grid-Connected Supply

Layer Two: Converted Capacity and Powered Land

Layer Three: Behind-the-Meter (BTM) Generation

Layer Four: The Production Constraint in Industrials

Layer Five: The Semiconductor Ceiling

In our AI Space Datacenter Model base case - logic and memory capacity additions move in lockstep, though memory tends to be the binding constraint for much of our forecast horizon.

In the Elon Musk Scenario - we keep the same robust chip manufacturing capacity expansions, and with much lower terrestrial datacenter capacity, the AI industry must shift deployments to space.

Musk’s Terafab Initiative

Users of our AI Space Datacenter TCO Model can dynamically adjust these capacity parameters and silicon constraint assumptions to dial in scenarios as needed.

Part Three: The Total Cost of Ownership Framework for Space Datacenters

IT Capital Cost of Ownership

Datacenter Capital Cost of Ownership

Operating Cost of Ownership

From Total Cost of Ownership (TCO) to Levelized Cost of Compute (LCOC)

This gives us a Levelized Cost of Compute (LCOC) in 2026 for our B300 cluster of $10.91/hr/GPU for an orbital deployment compared to $2.49/hr/GPU for a terrestrial deployment.

So When Does Space Actually Get Interesting?

During the 1980s and 90s, macroeconomic data could not detect the contribution of the emerging computer revolution. Famously, Robert Solow quipped “You can see the computer age everywhere, but in the productivity statistics.” And yet, despite the dot com boom and bust the Magnificent 7 now have a market cap 1.8x that of Europe. A similar issue is arising with AI where the macroeconomic data is not yet equipped to capture the value produced by AI while the headlines, public sentiment, and governments around the world are quick to capture the costs incurred in dollars, watts, gallons and jobs.

A boring 2013 methodology revision added R&D and investment in intellectual property to GDP accounting boosting total production for the 1990s by ~$3.6T. In the official accounts it was spread evenly, so the growth rate only rose marginally, but it amounted to nearly 30% of full year 2000 GDP. The magnitude of the measurement problem from AI dwarfs prior measurement issues, we call the work AI does that national accounts can’t currently see Dark Output. Even more of the new output from AI is likely to be invisible as it is clustered in the service sector where national statistics have longstanding issues with capturing productivity growth.

Incoming Fed Chairman Kevin Warsh acknowledged as much in December 2025 “If you’re looking at the data, my view is you’re backward looking. You’re going to be late. You’re not going to realize the country is able to have non-inflationary growth faster. So you’re going to have to make a bet.” With the transition of AI growth to more active capital market funding, any measures that fail to show results from AI will be scrutinized for signs of a bubble.

Dark Output

AI output will be real before it is measurable. We can capture token spend, and we can capture jobs lost. But unless AI’s output is sold at a visible price, only token spend is captured in GDP. Normally when the price of something collapses, we can see this deflation and call the results productivity. Due to well-known difficulties in the service sector (see Appendix 1), GDP will record those as declines, and prices may even show inflation. Like the dark energy that makes up our universe, Dark Output will likely only be visible in its effects on other elements of the economy and not through direct observation. One of the most visible effects is job displacement which we are now tracking on our Dark Output Monitor.

We are at risk of having an event on the scale of the Industrial Revolution where most of the new output is invisible even as businesses spend increasingly large amounts on AI services.

Types of Dark Output

Dark output is AI-enabled economic value that exists but is not visible, or is badly distorted, in GDP, prices, labor statistics, or industry accounts. We categorize this into two buckets:

In both cases, value exists despite the statistical system failing to see it. This is not a unique problem (see Appendix 1).

Substitution Dark Output

An example of substitution Dark Output is a simple legal document which in theoretical GDP should have the same inflation adjusted value to a user whether a lawyer drafts it or AI drafts it. But service sector GDP and inflation is hard to estimate (see Appendix 2), there is no ’unit’ of legal services, just lawyers’ receipts and surveys of firms for the cost of services rendered. When AI takes over the task, the receipts vanish as the cost is absorbed in tokens, and when government officials survey lawyers on the cost of services they may find that the average price has gone up, as the simplest documents are now completed by AI and not lawyers. From the perspective of GDP, the transaction has effectively vanished except for a few dollars of tokens sitting in an unrelated sector of the economy.

For Tokenomics subscribers, we track the frontier of tasks that market signals show current AI has the potential to replace. These tasks, depending on how they are performed by AI may vanish from the national accounts all together (see the Dark Output Monitor section below).

Other than housing, most services are measured in the national accounts through this sort of receipts and list prices system which backs into ‘quantity’ of tasks being done by dividing spend by price. This sort of accounting doesn’t allow for productivity gains. When the accounts record lower receipts, they will read this out as an output decline.

A basic will as seen in the figure below has fallen in price for generations as technology changed the process of creation, but because it was gradual, the induced error was less extreme. A drop from $400 to $150 in 30 years is less than 5% a year. A drop from $150 to $0.50 in a year is more than a 99% cost decrease. One introduces bias, the other vanishes from the dataset. Legal services prices were only added to the CPI in 1987, and since then the price index is up 4.6x (as of September 2024). The price index is effectively an employment cost index because there is no accounting for the increased productivity.

New Dark Output

In contrast, new Dark Output is work that did not happen before AI made it cheap enough to do. No wage bill disappears because no firm or household would have paid a human to do that work at prevailing prices. For example, when literature reviews fall from $2,000 to $2, we do not do the same number and pocket the savings, we do them before every project! Summarizing the last six months of emails on a theme in your inbox is useful. Running an academic literature review before an interview is useful. Both can create real value, but neither leaves a clean economic trace beyond the tokens, API calls, cloud spend, or subscription that made the task cheap enough to run.

There are anecdotal signs that a large fraction of current token spend is for new work that wasn’t previously paid for rather than replacing existing work. But the exact magnitude is opaque as it sits behind the anonymizing curtain of tokens. Identifying if a specific AI task is creating value and how much would likely be difficult even if you had the full conversation trace, as it is the national accounts will at best see AI revenue.

Captured AI Output

A final category of AI output is work that was previously done by humans and now is done by AI, but that can still charge the same amount as before. This captured AI will only exist where companies have genuine market power, and can protect prices in the face of declining costs of production. Consider two scenarios, first a firm that used to buy a $10,000 HR service from an outside provider now buys that HR service for $10,000 from an AI HR provider. In that case the output still is captured in national accounts and all that disappeared was the wages and workers. In the second version that $10,000 service is now done internally for $10 of tokens. In that scenario GDP has declined by $9,990 despite the same work being done.

Why Services aren’t like Goods

Manufacturing automation gave statisticians something to count. If machinists got better at making screws, the factory would report they made more screws, at lower costs, or better margins. Real GDP could rise because it was based on the quantity of output. So as the price of screws fell by 99+% over the past 6 centuries, we can count that the quantity of screws also went up on the order of 10 billion times. Real GDP correctly captures this as growth and productivity

We lack a functional vocabulary for units of services, and mental work. As useful as it would be, there is no measure of ‘mind power’ that does for AI what horsepower did for the Industrial Revolution. Horsepower gave people a way to compare machine output with animal and human labor. Tokens do not do that. A million tokens can produce junk, a useful email summary, a legal document, or a decision that changes a company’s strategy. The economic value depends on the output, not the token count.

Finger Prints of Dark Output

A common observation in AI commentary is that junior staff are being displaced from routine work first. The corollary is that average wages in exposed occupations can rise because the lower-paid workers leave the sample. The cheapest workers disappear from the data. No one got a raise, and yet wages rose.

Employment in the most AI exposed sectors of the economy is falling relative to the broader economy. Yet those same underperforming segments are showing relative wage increases.

This mismatch between employment and wages is one fingerprint of AI displacement we track in our Dark Output dashboard. This is not a direct measurement of dark output, but the sort of odd measurement artifact that dark output would create as it became more prevalent.

An initial sign of new dark output is the heavy prevalence of Token usage in segments of the economy that are not showing signs of rapid labor deterioration. In Anthropic’s Economic Index from March 2026 they show 37% of tokens are being used in computers and mathematics and yet the contribution to GDP from investment in software has not broken from its pre-AI trend and wasn’t even at an all time high.

Why We Use Market Signals, Not Benchmarks

Benchmarks answer the wrong question, and they answer it late. Expert evaluations ask whether AI can satisfy an evaluator under test conditions, often an evaluator who expects expert work. They are expensive, slow, subjective, and backward-looking because expert time is scarce. Labor augmentation and displacement does not require AI to beat the best lawyer, analyst, or engineer. It requires AI to be good enough, cheap enough, and reliable enough to aid or replace the worker who would have done the task at prevailing wages. That is why we monitor public claims companies make about their own business practices, rather than abstract claims about what another firm could do.

The Evidence Ladder

Market signals vary in strength. Tiers 1 and 2 are benchmark-driven. They suggest a model can complete a task under test conditions, we only use them to estimate the cost of AI completing the task. Tier 3 is the hype layer: a public claim that a product or company can do the work. Unfortunately it is also the human verified benchmark layer. In our view a business saying the tools are in use in production is a stronger level of evidence. A court fight where a firm successfully defends AI work is stronger than that. An insurer underwriting the risk is the strongest signal because a third party has priced the failure mode and taken on that risk. Our analysis shown in the dark output monitor treat these as an evidence ladder, not a binary yes or no.

From Exposed Labor to Dark Output

The headline $1.5T estimate is based on Tier 4 and above evidence from this ladder. It is not a claim that $1.5T of labor has disappeared. It is a claim that tasks tied to roughly $1.5T of labor cost sit inside categories where current AI has credible displacement potential. The number should be read as exposed labor, not missing output. We have not yet seen evidence of Tier 5 or Tier 6 activities, and that should read as a cautionary note on AI boosterism.

Most of the evidence we have collected to date points to AI augmentation and not AI replacement.

When AI aids or takes over a task, the output does not automatically disappear. It only vanishes from the national accounts if the prices fall or worse the task gets moved inside the purchasing firm from the outside. If the market is uncompetitive a firm could still charge the same price as when a human did the work and the value would be captured as an explosion in margins and correctly show up in the national accounts.

What the Dark Output Monitor Can and Cannot Say

Our Dark Output monitor currently shows a map of pressure, not a forecast of layoffs or dark output. It identifies where firms have incentives to move from human labor to AI labor, where importantly the cost gap is largest.

It tracks tasks, occupations, wages, evidence tiers, token costs, and possible FTE displacement. Those are labor-side and input-side measures. They are enough to show where the transition can start, and in the areas where large numbers of tokens are being used without displacing labor we get a hint of where new Dark Output is being created.

A high-exposure sector should not be read as a sector where jobs have already disappeared. It should be read as a sector where the economics of substitution are visible: the task is identifiable, the wage pool is large, the market evidence is strong, and the token cost is low enough to matter. Only time will tell whether demand is elastic enough for new output and labor augmentation to keep employment near historical levels. Or if civic, legal and government pressures are enough to prevent transition.
A useful corollary is the slow rollout of self-driving cars, where challenges in culture, insurance and market structure have been as important or more important than fundamental technological hurdles.

Where the Statistics Break

There are multiple measurement errors that could make output go dark. Different economic data sets will capture those errors in different ways. Treating all of those failures as ‘GDP missing AI’ makes the problem sound simpler than it is.

Boundary Shift

Boundary shift is work that used to be bought in the market that moves inside the firm or household. A paid research brief becomes an internal AI workflow. A contractor task becomes an employee prompt. The value may remain, but the transaction that made it visible disappears.
Price Collapse

There are no truly separate measures of quantity and quality in services. Receipts, wages, and hours worked, are captured but not quantity. As mentioned above there is no standard unit of legal services, no metric ton of literature reviews, and no barrel of consulting. If the accounts see lower receipts (because prices fell) and higher average wages (because junior staff are displaced), it will read as higher inflation and falling productivity and output.
These are genuinely hard questions, but there have unambiguously been productivity gains in the service sector from the march of technology. When those gains were relatively slow and constant it was a smaller issue than now that productivity has soared, and seems to be accelerating.

Sector Misrouting

Sector misrouting can occur when AI creates value in one sector while the transaction appears in another. The accounts end up counting the screws while missing the houses being built with them. A hospital may use AI to process paperwork faster, but if the only place AI shows up is in the revenue for an AI company or software provider, it will skew the national statistics. GDP-by-industry can make AI vendors look like the source of the value while the adopting sector looks stagnant.

New Work Invisibility

We have already discussed new work invisibility, but if there is no receipt beyond tokens, the work is only visible at the cost of tokens. Real economic work is being done, we are better prepared for a meeting because AI wrote a dossier on the person across the table for a few tokens, but that value does not show up anywhere. It can seem ludicrous to estimate what it would have cost in dollars and hours even a few years ago to accomplish many of the tasks that are facilitated now for pennies and minutes. Any reasonable measure of the macro economy must in some way account for this or the AI boom may read to the data as an AI bust.

Macro Data Is Our Best View of the Economy. Breaking It Risks Errors

Macro data is our best view of the economy. It is always imperfect, and playing catch-up with changes in the economy. But it is part of how investors decide whether a boom is real, how policy makers decide to weigh the balance of unemployment and inflation, and how firms decide whether to hire, automate, or build. If AI breaks the data links between labor, output, prices, and sectors, decisions get worse. The risk is that we keep using the same data even as they are becoming less accurate.

The economy can clearly see the costs of AI. Data centers, GPUs, electricity, water, and token spend are all visible. Knowing whether token spending represents real economic output that is durable or whether firms are playing with a shiny new tool is critical for good decisions.

Monitoring the Invisible: The Work Ahead

The Dark Output monitor shows us one measurable corner of the problem. It shines a light on labor that may be displaced, and token costs are low enough to matter. We will work to find other shadows of Dark Output and include them in the monitor.

These measurement errors cannot stay inside one dashboard. If AI creates real value that our statistics cannot see, everyone has a stake in learning how to measure it. With visible costs and revenue, but invisible output critics can dismiss AI as a bubble and not adapt. Politicians and investors need ways to classify, price, and tax new forms of output as the labor-based tax base weakens. AI has the potential to create surplus for communities to share but it also creates disruptions for workers and governments alike.

Dark output is not a reason to dismiss AI’s costs. It is a call to work to measure the other side of the ledger. Labor displacement, power demand, water use, and land use are visible now. Tokens spent is visible. The output is harder to see. Cheap screws became countable output. Cheap AI work may not. If AI is creating an event on the scale of the Industrial Revolution, we need economic data that can see more than the displacement it causes.

Appendix 1. AI and Feminist Economics

There is an economic precedent worth naming explicitly: Care Economics, within the broader Feminist Economics tradition. It helps us think seriously about goods and services that have no line item in GDP. Even granting that GDP, as currently constructed, is the right metric for evaluating AI’s economic impact, if AI generates enormous consumer surplus while displacing a great deal of formal output for a relatively small quantity of tokens, the score will still be misread. The Stiglitz Commission argued exactly this in 2009, and they were right that GDP is a poor proxy for well-being.

The Feminist Economics literature documented these issues at scale. Marilyn Waring showed in 1988 that the committees who drafted the original System of National Accounts were 91.7% male. One sentence in the founding document dismissed much of women’s economic contribution, raising children, maintaining households, caring for the elderly and sick, as “of little or no importance” to the national accounts. Duncan Ironmonger calculated that Australia’s household economy was 78% the size of its entire market economy. The UK’s Office for National Statistics put household production at 63.1% of measured GDP. The International Labour Organization estimated 16.4 billion hours of unpaid care work performed daily, worth $11 trillion a year, three times the global technology industry. By the conventions of national accounting, all of it has zero value.

This is not ancient history. It is not a resolved methodological debate. It is the same production boundary, updated but structurally unchanged, that is about to encounter AI-generated output at industrial scale. AI could push a large fraction of work out of the priced-and-produced region into just-produced, with the cost delinked from the production.

The economist Margaret Reid proposed a test in 1934 that remains the sharpest diagnostic: if work could be delegated to a paid third party, it is productive. When a family hires a housekeeper, the housekeeping enters GDP. When a family member does the same work, it does not. The act is identical. The accounting treatment depends entirely on whether money changes hands.

AI makes virtually every information task delegable. A large language model can draft a legal brief, analyze a financial statement, write a marketing plan, triage a patient complaint, generate code, or compose a research summary. In each case, the work was previously performed by a paid human and counted in GDP.

If an AI is asked to take notes on a medical consultation today, the only place that transaction can show up in the national accounts is buried in the bill from the AI company. Nowhere is the use itself reported in a way that would let disinflation or output be calculated correctly. We are using the same old GDP ruler we always have, while the production function pushes more of the economy into the no-man’s-land of Dark Output.

A candid note on what our own framework inherits from this history. Displacement Dark Output measures only paid market labor: BLS wages, BLS employment counts, O*NET work activities. It does not measure AI’s impact on unpaid care work, household production, or the informal economy. We invoke Waring and Ironmonger to establish that the production boundary is constructed and politically contested, then build a measurement system that operates entirely within that boundary. This is a deliberate choice, not an oversight. The data infrastructure for measuring market labor displacement (BLS, O*NET, employer surveys) exists and is auditable. The infrastructure for measuring household AI adoption does not, and inventing it would stack measurement uncertainty on measurement uncertainty. But the limitation is real. Dark Output reproduces a known exclusion. The 16.4 billion daily hours of unpaid care work that the International Labour Organization documented are no more visible in our framework than in the one we critique. We do not claim otherwise. Our key observation is that the problem documented by these alternative frameworks is about to get worse. Much of the displacement we measure also risks falling disproportionately on occupations with high female employment shares (administrative work is 72% female, BLS). We do not yet disaggregate by gender, but it is a logical extension.

All AI uses in the non-transactional production sphere are another form of Dark Output. When someone uses AI to do a domestic task faster, more easily, or better than before, the activity does not move from produced/priced into produced/unpriced; it just enlarges the produced/unpriced economy.

Appendix 2. AI and Feminist Economics

Since the 1990 Griliches conference, service accounting has improved, but in targeted ways. BLS expanded service producer price indexes, with PPI service coverage reaching more than 70% of the services sector by 2009 and the headline PPI system moving to Final Demand-Intermediate Demand in 2014 to include services, construction, government purchases, and exports. BEA moved to chain-type Fisher quantity and price indexes, integrated GDP-by-industry with input-output accounts, capitalized software and R&D, and improved treatment of difficult sectors like finance, insurance, and R&D.

But the core problem remains. BEA still says most detailed NIPA components are measured in dollars, not units, so real quantity is usually estimated by deflating current-dollar spending with a price index. That works tolerably when the transaction, product, and price index all still describe the same thing. It breaks down when AI moves service work into subscriptions, tokens, or internal production. The accounts can see receipts, wages, and sampled prices, but not necessarily the legal memo, literature review, HR task, or code review that still got done. It also has no unit of quality, if an AI augmented literature review is 10x more exhaustive there is no current method to capture that fact.

Dark Output

We are at risk of having an event on the scale of the Industrial Revolution where most of the new output is invisible even as businesses spend increasingly large amounts on AI services.

Types of Dark Output

Dark output is AI-enabled economic value that exists but is not visible, or is badly distorted, in GDP, prices, labor statistics, or industry accounts. We categorize this into two buckets:

In both cases, value exists despite the statistical system failing to see it. This is not a unique problem (see Appendix 1).

Substitution Dark Output

New Dark Output

Captured AI Output

Why Services aren’t like Goods

Finger Prints of Dark Output

Employment in the most AI exposed sectors of the economy is falling relative to the broader economy. Yet those same underperforming segments are showing relative wage increases.

Why We Use Market Signals, Not Benchmarks

The Evidence Ladder

From Exposed Labor to Dark Output

Most of the evidence we have collected to date points to AI augmentation and not AI replacement.

What the Dark Output Monitor Can and Cannot Say

Where the Statistics Break

Boundary Shift

Sector Misrouting

New Work Invisibility

Macro Data Is Our Best View of the Economy. Breaking It Risks Errors

Monitoring the Invisible: The Work Ahead

Appendix 1. AI and Feminist Economics

Appendix 2. AI and Feminist Economics

I’ve worked on compilers for ML for the last decade across Google, Waymo, and OpenAI. This includes CUDA support in clang, XLA:GPU, Triton, and OpenAI’s custom hardware. I’ve seen stuff. But over the past week or so I had one of the unsettling experiences of my career: In one afternoon, I spent more than $10,000 running AI agents over compiler code, finding hundreds of plausible bugs in LLVM, including many miscompiles and at least one that’s Quite Serious. This is the story of how I got here and where we might be going.

In January 2026, I decided to try to find some bugs in LLVM (the compiler behind clang, rustc, and AMD’s GPU compiler, among others), as a personal project. Codex and I collaboratively wrote a fuzzer. The basic idea is to generate a random program, run it through part of the compiler, and then check that the resulting program after compilation does the same thing as the original program (usually just by running the two programs). I spent a few weeks on it, and I found and fixed five bugs in instcombine, LLVM’s peephole optimization pass. After that, my fuzzer started taking longer to find bugs, and I lost interest.

Fast forward to mid-May 2026. I joined SemiAnalysis as a contractor, and I decided to try applying the same technique to NVIDIA’s low-level compiler, ptxas. I expected this to be less fruitful than fuzzing LLVM, for a few reasons.

I expected maybe I’d find a handful of bugs after a few weeks of work, like before.

Instead, in three days, I had 40 programs that ptxas miscompiles. (A week later, this number is up to about 80.) Although some of these test cases probably reflect the same underlying bug in the compiler, I was still kind of astonished. Many of the reproducers reduce to fairly “normal-looking” instruction sequences.

If you want to have a look at the bugs I found (really, that Codex and Claude found), they live in the FuzzX repo on GitHub.

Why was my fuzzer so much easier to write this time? As far as I can tell, it was the difference between ChatGPT 5.2 and 5.5. This time, I vibe-coded this entire fuzzer, never looking at a line of code. The LLM did the tedious job of altering the fuzzer after every bug we found so that we wouldn’t get stuck finding the same bug over and over. It also minimized each test case it generated, often spending an hour or longer doing so. It independently decided which PTX instructions to fuzz, and which sequences of instructions were “safe” (i.e. didn’t trigger undefined behavior). I just put it in a loop using /goal and went to sleep.

To be clear, it’s not surprising that fuzzing found bugs. But it was surprising that I found this many bugs so quickly, with almost no manual effort.

Naturally, my next question was whether I could also find bugs in LLVM’s AMDGPU backend. You won’t be surprised to hear that I could, at roughly the same rate as I found bugs in ptxas. At some point, my personal ChatGPT Pro account ran out of credits and I switched to SemiAnalysis’s Claude account. I didn’t notice a difference in quality between Opus 4.7 and ChatGPT 5.5, they were both great.

The story could end here. I reported the bugs I’d found to AMD and NVIDIA. As of this writing, AMD has already fixed five of them, and because their compiler is open-source, I can apply their fixes immediately. If any of these bugs were critical to me, I could even fix them myself. Hooray for open-source toolchains.

At this point my ptxas and AMDGPU fuzzers started to slow down; they were running for increasingly long times without finding any new bugs. I almost stopped there, but I had a thought. What if I just asked Claude to read through LLVM and find bugs? In other words, what if I was building a fuzzer because I hadn’t absorbed the Bitter Lesson?

I asked Claude to spin up 50 subagents at a time looking for bugs. Boy did I get them. Claude found bugs at a rate of one every four minutes. (In contrast, at this point the fuzzer was taking hours to find new bugs.)

My initial reaction was, I don’t even think this means LLVM’s AMDGPU backend is particularly buggy. Indeed, a friend suggested I do the same to the x86 backend, and it produced bugs at a rate of almost two per minute.

My bug-finding agents didn’t show any signs of slowing down before I stopped them. I don’t know how many more issues are out there. Probably a lot?

The question with automated bug-finding usually is, “Do these bugs matter?” I’ve only gone through about 20% of the x86 bugs so far. The bugs found by agents are definitely less severe on average than those found by the fuzzer; every fuzz bug is a demonstrable miscompile, whereas agents are using their judgement to decide what is and isn’t a bug, and sometimes they’re wrong.

On the other hand, one of the 30 or so agent-discovered bugs I’ve examined includes this extremely frightening case where LLVM will turn an atomic store into two non-atomic stores. This bug would have been very difficult to find via fuzzing (fuzzing atomics is hard), and would likely be both quite bad and quite hard to root-cause if it happened in production (because downgrading an atomic to a non-atomic store will be fine 99% of the time, and 1% of the time will corrupt your data).

The other question you should ask is, how much did all this cost? The vibe-coded fuzzer was relatively cheap. I was using about double the weekly quota for my $200/month ChatGPT Pro account to write the AMD and NVIDIA fuzzers at the same time. It was more expensive with pay-per-token Opus 4.7, on the order of $1000 for a few days’ work. I don’t know if it would have been cheaper with a Claude Max account, and I don’t know if the work I was doing with Claude was more token-intensive than the work I’d been doing with Codex earlier.

On the other hand, having an army of subagents read the code was, um, not cheap. I spent more than $10,000 in a few hours, and I wasn’t even using fast mode. (Thanks for the tokens, Dylan!). Even though bugs found this way were on average lower-severity than fuzz bugs, the fact that code inspection can find whole classes of bugs that would be very challenging to find via fuzzing makes it valuable to me.

Frankly, just the one atomics bug mentioned could do far more than $10,000 in damage under the right (or perhaps wrong) conditions; silent data corruption like this will break just about any production system. And even if your system had safeguards that allowed it to notice this kind of corruption, you’d still likely burn months of engineering time hunting for its source. If I were doing this again, I wouldn’t bother with fuzzing, assuming I had the budget to unleash my herd of agents.

I’m still trying to make meaning out of this experience. I think it’s bigger than simply “With enough subagents, all bugs are shallow.” Maybe the lesson is this:

A corollary is, if you don’t have the budget, you’re operating in a smaller part of the possibility space than those who do. And I expect that gap to grow substantially over the coming months.

SemiAnalysis spent an order of magnitude more on my tokens than on paying me the day I ran the agents. If the value of something is how much someone is willing to pay for it, then that was the first time in my career that I delivered less value to my employer than my AI did.

It makes me wonder how much SemiAnalysis will be willing to pay for tokens in six months, and what life will be like for people and companies who can’t or won’t pay.

One last thought. I could do the “have Claude read the code” approach with the AMDGPU and x86 LLVM backends because I have the source code. I can’t do it with ptxas because I only have a binary. But…how sure am I that, given enough tokens, Opus 5.7 (or even 4.7) couldn’t find bugs just by reading the assembly?

I expected maybe I’d find a handful of bugs after a few weeks of work, like before.

If you want to have a look at the bugs I found (really, that Codex and Claude found), they live in the FuzzX repo on GitHub.

To be clear, it’s not surprising that fuzzing found bugs. But it was surprising that I found this many bugs so quickly, with almost no manual effort.

My bug-finding agents didn’t show any signs of slowing down before I stopped them. I don’t know how many more issues are out there. Probably a lot?

I’m still trying to make meaning out of this experience. I think it’s bigger than simply “With enough subagents, all bugs are shallow.” Maybe the lesson is this:

A corollary is, if you don’t have the budget, you’re operating in a smaller part of the possibility space than those who do. And I expect that gap to grow substantially over the coming months.

It makes me wonder how much SemiAnalysis will be willing to pay for tokens in six months, and what life will be like for people and companies who can’t or won’t pay.

While other CSPs have seen declining-to-flat operating margins over the last several quarters, Amazon’s AWS margins inflected this past quarter driven primarily by customer spending growth on Claude through Bedrock. AWS’ higher share of 3P model API spend, Anthropic/Bedrock deal structure, and Anthropic’s ARR outperformance in 1Q26 all contributed to EBIT margins increasing 213bp Q/Q while other CSPs lagged. SemiAnalysis’ work in the new Tokenomics 2.0 model shows how AWS has pulled ahead of the pack and found a strong avenue to grow margins. Our model estimates quarterly revenue, profits, ROIC and compute requirements of every single business vertical of hyperscalers and AI Labs, e.g. Gemini API revenue & margins, Microsoft Copilot ARR, OpenAI ChatGPT subscriptions across plans, etc.

Although all CSPs are benefiting from increased AI and non-AI revenue, margins are a whole different story. Oracle and Coreweave both disappointed the market with lower-than-expected profits from their cloud arms. Azure is also seeing a downward trend in margins. Google Cloud has had a great upwards climb recently, but margins are inflated since they do not include training costs from DeepMind in the GCP segment. The only CSP with a true rising trend is AWS – a remarkable achievement considering their server depreciation (5yrs) is the lowest of all CSPs.

The Amazon Story & Background

Now, we see a new era for AWS where the firm combines accelerating revenue growth AND outperforming margins. Amazon brings a unique combination of the following:

Let’s dig in, starting by covering the economics and market drivers of Bedrock, Amazon’s most differentiated product. We then dive into their datacenter footprint relative to others, and then at the end of the report dive into the outlook for other CSPs.

Amazon Bedrock Deep Dive

Bedrock is an AWS service that enables customers to choose their favorite LLM among many options, benefit from AWS security and compliance and unified billing (among other things) and run AI workloads. This market, which we call “API endpoints”, has many competitors, including Microsoft Foundry and Google Gemini Enterprise Agent Platform (previously Vertex), as well as many providers focused on open-source models such as TogetherAI, Fireworks, Baseten, etc.

While some of these criteria do matter, they miss the single most important differentiator: access to frontier LLMs. As demonstrated by our Tokenomics Model, Frontier LLMs make up the vast majority of AI API industry revenue.

In the API endpoint market, this is the massive advantage that AWS, Microsoft, and Google boast over everyone else. For a long time, AWS had access to Claude, Google to both Claude and Gemini, and Microsoft to OpenAI models. Recently, AWS gained OpenAI access, while Microsoft gained Claude. No other CSP currently has the ability to sell OpenAI, Claude, and Gemini tokens.

Having access to these models is one thing, but building a substantial business around them is another. AI Inference notably has huge compute needs. To understand this, let’s dig into the economics of Bedrock, Vertex, and Foundry.

Token-as-a-Service Platform Economics

The economics of TaaS platforms are very different if they own the IP (or can freely use it, eg open source), versus if they “distribute” the IP:

Anthropic was the first AI Lab to roll this out with AWS and Google, recently followed by OpenAI with AWS. Security is of maximum importance: ideally, the CSP doesn’t have access to model weights, or a very limited one, but can still use the model and run it on its infrastructure.

For the CSP, this means more risk relative to a standard GPU IaaS deal, since revenue is not guaranteed via a 5yr IaaS take-or-pay deal. However, the margins are significantly more attractive. And Amazon with its deal structuring and execution has been the main beneficiary of its bet on Bedrock, dwarfing the size of their rival platforms.

And this is the main difference for the AMZN, MSFT, and GOOGL against ORCL and neoclouds. Their Token as a Service (TaaS) businesses are $10B+ in ARR today. Contrast this with neoclouds and Oracle at practically nothing. As we show in the Bedrock/Anthropic deal section, margins on these TaaS deals are significantly higher than being a wholesale seller of AI IaaS rental compute. The margin mix and distribution advantage these hyperscalers now possess is significantly widening the gap amongst the top hyperscalers and the rest of the field.

Vertical Integration Positions AWS for Industry-Leading Margins

Having a greater business mix towards Bedrock is a great combination with Amazon’s bet on custom silicon. In prior reports, we’ve covered Trainium2 and 3 in great depth, arguing that the chips have leading perf/TCO in certain scenarios where memory bandwidth is most valuable, such as high-batch inference and RL.

In Bedrock, the underlying hardware is abstracted from customers; they only see tokens. This means that an inference-optimized chip is a natural fit with Bedrock, with the main caveat being that porting models can be longer than with Nvidia GPUs, but it’s typically just a matter of days. This means that Trainium has a natural offtake in the form of Bedrock, and Amazon is taking full advantage of this, per the CEO:

“Our AWS Trainium chips, designed in-house for AI workloads, now power more than 50% of Amazon Bedrock token usage.” - Matt Garman, AWS CEO November 2025

Google benefits from the same advantage, but there are caveats, as we’ll discuss below.

In addition to AI Accelerators, CPUs are also increasingly relevant to training and inference of frontier LLMs. In the world of CPUs, vertical integration is an even bigger differentiator, and Amazon’s fifth generation custom chip, Graviton, is industry leading. We wrote a full deep dive on CPUs here, and we explained that Graviton4 and 5 provide a great perf/TCO advantage to Amazon. Graviton4 will be integrated to Trn3 as head node and will be used separately to handle RL workloads and agentic. Amazon now has large CPU and Graviton deals with Anthropic, OpenAI, and Meta. Upselling Graviton to Bedrock customers will also be easier for Amazon. In our Tokenomics Model, we quantify the revenue growth contribution on AWS driven by this non-AI chip business.

AWS Bedrock Mix: The Numbers

Our Tokenomics Model goes into detail within each hyperscaler AI business to show the breakout of segments such as Bedrock at AWS, GitHub CoPilot within Azure, and Gemini API vs AI IaaS at Google Cloud. We have triangulated AWS’ AI Business and the mix of Bedrock to better assist investors and enterprises in understanding these trends. For AWS, the mix of AI as a % of total revenue has increased from 2% in 1Q24 to 10% today in 1Q26. Our estimates going forward, available to Tokenomics subscribers, show this continuing to increase. In 1Q26, the mix of AI as a % of total GCP and Azure revenue was 36% and 27%, respectively. However, we believe the mix difference between IaaS andTaaS is a major contributor to margin differences despite AWS’ lower total AI mix.

As discussed above and as we dive deeper into the deal economics below, AWS’ Bedrock mix vs Azure Foundry and GCP Gemini API is important. Bedrock has steadily grown as a % of total AWS AI revenue to 37% today, up from 9% in 1Q25 when IaaS dominated the AI business. In comparison, AI IaaS is still 80%+ of the AI business mix at Azure and GCP. Per Amazon’s 1Q26 and 4Q25 earnings call, Bedrock revenue grew 170% and 60% Q/Q, respectively. SemiAnalysis believes Bedrock is a $5.5B run rate business today with the vast majority of customers (80-90%+) using Anthropic models.

Clearly, Amazon has found a winner in terms of revenue growth and mix with Bedrock and the Anthropic deal. Microsoft, in contrast, is heavily IaaS driven, and other AI efforts like CoPilot in 365 and GitHub have failed to create traction. Google has done well with Gemini API within Gemini Enterprise Agent Platform, but has not benefited from the same coding market trends that have helped drive Anthropic’s API revenue up ~13x y/y.

Bedrock/Anthropic Deal

Our understanding of Anthropic’s Bedrock deal includes a flat IaaS fee, revenue share, and performance hurdles for outperformance above certain levels of token/spend throughput.

The Tokenomics team models Anthropic’s and OpenAI’s businesses extensively by segment (Consumer, Business/Team, Enterprise) and plan (Free, Pro, Max20, API, etc) at an ARR, revenue, margin, token volume, and token pricing level to help investors and corporate strategy departments understand this flow of dollars and tokens between customers, labs, and CSPs. This work includes modeling the Anthropic and Bedrock deal, which we have previewed below to illustrate the Gross Margins of Anthropic and EBIT Margins of Bedrock at differing levels of Anthropic ARR per MW on Bedrock Compute. We illustrate that margins for Bedrock, or any Token as a Service product, have superior margins than the typical IaaS fee business. The takeaway for hyperscalers is that these TaaS arrangements are excellent margin accretive additions when structured beneficially for both the hyperscaler and lab to drive revenue higher.

We believe that Anthropic’s recent revenue outperformance (discussed below) puts Anthropic Bedrock revenue around $26M per MW in 1Q26, implying Bedrock EBIT margins of ~55% in the above analysis. For 1Q26 vs 1Q25 at AWS, we believe that reported incremental EBIT margins were primarily driven by Bedrock, with Bedrock accounting for 30% of the step-up in Gross Profit Dollars Y/Y despite accounting for only 4% of total AWS revenue. For 2Q we see Anthropic ARR per MW of Total Compute around $42M and Bedrock Revenue mix increasing to 53% of AWS AI Revenue contributing an incremental 9 points to total AWS revenue growth. We note that given Anthropic’s ARR recognition accounting on a gross basis, that margins on Bedrock are slightly negative mix vs their current blended low 60% inference gross margins.

Anthropic’s Blowout Q1

Anthropic’s 1Q26 was characterized by rapid growth which helped propel Bedrock revenue and margins higher. The company added $21B in net new ARR in the quarter to reach $30B of ARR. Most of Anthropic’s revenue is API and Enterprise based whereas OpenAI skews consumer and has a relatively heavy inference cost load from free users. This revenue explosion is driven by Claude Code, which has taken enterprises by storm. Consumers, too, have begun to flock to Claude. In a recent note to subscribers, we noted Anthropic’s increasing (and now majority) market share of net new customers vs OpenAI within card panels as well as higher average transaction values. We continue to see strong demand for Anthropic - primarily driven by enterprise API spending - through the end of the year. Our latest Tokenomics model estimates see potential for well over $100B in ARR by year.

In addition, Anthropic’s margins on inference compute have shot up. We believe inference gross margins are now in the mid 60s, up from 38% in 2025 and -94% in 2024. Per a Wall Street Journal article on 5/20/26, Anthropic is expected to be Operating Income profitable in 2Q after adjusting for stock-based compensation. Our Tokenomics model for Anthropic allocates costs to Free and Paid User Inference, Training & Research Spend, and OpEx. Our latest estimates published to clients last week were extremely aligned across revenue, cost bucket mix and dollar amounts as well as overall GAAP (unadjusted for stock-based compensation) EBIT profitability.

Hyperscaler Takeaways: Amazon Met Demand With Capacity

Key to deploying a token-as-a-service platform at much larger scale than rivals was simply having more compute capacity, given the high compute-intensity of AI inference. And as shown below, Amazon is adding significantly more capacity than rivals, with only Microsoft a close tie in 2024-26 but largely dwarfed in 2027. Our Datacenter Industry Model provides quarter-by-quarter capacity forecast by hyperscaler.

However, total capacity is not the only relevant item. We need to break it down by end-user, which is what our Datacenter Industry Model can provide quarter by quarter. Our model quantifies the capacity going to Microsoft’s internal AI efforts, which is higher than that of Amazon, reducing compute available to the broader customer base. In addition, the vast majority of Microsoft’s AI compute goes to OpenAI via long-term compute contracts, as demonstrated by a remarkably large share of Microsoft’s backlog, with OpenAI’s backlog alone being 2.5x that of the total Azure annual revenue.

To deploy more capacity than peers, Amazon has been remarkably aggressive in growing its power pipeline, and signing multibillion PPAs with IPPs like Talen, Vistra, and NiSource.

On the other hand, Microsoft had a year-long datacenter pause (see our Datacenter Freeze article), which significantly lowered their 2027 capacity forecast. In addition, as we’ve covered in our Microsoft’s AI Strategy piece, Microsoft has been remarkably slow at building large-scale AI clusters in Wisconsin, the opposite of AWS’ lighting speed to build close to 2GW in Indiana and Mississippi. The only way for Microsoft to catch up is to contract significant amounts of capacity from Neoclouds, which is much more expensive and will reduce the margin advantage. This is happening all while Amazon keeps innovating to accelerate capacity growth. The company is rolling out a new datacenter design at a very large scale, with increased modularity and prefabrication as they continue to prosecute against the AI opportunity.

The Google Pushback

The most natural pushback to our AWS outperformance thesis is that Google Cloud is also seeing increased margins, boasts the same vertical integration as AWS (if not more), and has been an even bigger outperformer, with revenue growth skyrocketing to >60% YoY in the latest quarter and margins a record high for GCP. We’ve been well ahead of the market in calling out Google Cloud’s acceleration, with our first reports coming out in August 2025 in Accelerator, Datacenter and Core Research. Our TPU Deep Dive explained in great detail why Google Cloud is outperforming.

However, we think the margin rise is an illusion and is more akin to an “EBTIT” margin, i.e. Earnings Before Training, Interest and Taxes. Alphabet is seeing increased costs due to DeepMind/Gemini training expenses, but these costs are accounted for in “Alphabet-Level Activities,” not GCP. Per Google’s latest 10-Q, Alphabet-Level Activities, “primarily reflect expenses related to our shared AI research and development” and were $5.4B in 1Q26 up from $3.0B in 1Q25. Thus to say, all the Gemini API revenue is flowing to Google Cloud at higher-than-average margins, which boosts the margin profile of GCP while $10B+ of run rate costs are bucketed elsewhere.

In addition, Google Cloud might have benefited from one-off royalty payments due to the sale of TPUs to Anthropic, through Broadcom, with Google acting as an IP vendor.

Cloud vs AI Lab vs Hardware: How Can Google Satisfy All Demand?

Google is the ultimate supply-constrained business. This is a single company which attempts to compete simultaneously with AWS on Cloud, Nvidia on hardware, Anthropic & OpenAI on models, Meta on ads, Tesla on autonomous driving, and more.

When analyzing Google’s capacity growth, we simply do not see a large enough buildout to serve all demand. Internal capacity is enough to build a successful AI Lab, but leaves little room for the Cloud business (excluding hardware) to grow to the same extent as AWS. In particular, Gemini Enterprise Agent Platform (previously Vertex) as a distribution platform for Claude (and maybe OpenAI in the future) is seeing a significantly lower capacity additions than Bedrock.

For Google, GCP appears to be a platform to upsell additional services. Meta is the best example of this: a large GPU deal that subsequently led to large-scale Gemini adoption, and then a massive TPU hardware deal.

Implications for Hyperscalers and Labs

The Amazon Story & Background

Now, we see a new era for AWS where the firm combines accelerating revenue growth AND outperforming margins. Amazon brings a unique combination of the following:

Amazon Bedrock Deep Dive

Token-as-a-Service Platform Economics

The economics of TaaS platforms are very different if they own the IP (or can freely use it, eg open source), versus if they “distribute” the IP:

Vertical Integration Positions AWS for Industry-Leading Margins

“Our AWS Trainium chips, designed in-house for AI workloads, now power more than 50% of Amazon Bedrock token usage.” - Matt Garman, AWS CEO November 2025

Google benefits from the same advantage, but there are caveats, as we’ll discuss below.

AWS Bedrock Mix: The Numbers

Bedrock/Anthropic Deal

Our understanding of Anthropic’s Bedrock deal includes a flat IaaS fee, revenue share, and performance hurdles for outperformance above certain levels of token/spend throughput.

Anthropic’s Blowout Q1

Hyperscaler Takeaways: Amazon Met Demand With Capacity

To deploy more capacity than peers, Amazon has been remarkably aggressive in growing its power pipeline, and signing multibillion PPAs with IPPs like Talen, Vistra, and NiSource.

The Google Pushback

In addition, Google Cloud might have benefited from one-off royalty payments due to the sale of TPUs to Anthropic, through Broadcom, with Google acting as an IP vendor.

Cloud vs AI Lab vs Hardware: How Can Google Satisfy All Demand?

Implications for Hyperscalers and Labs

We’d like to thank DG Matrix, Novos Power, and Aran Industries for their contributions and insights during the preparation of this deep dive.

Introduction: Welcome to the Power Chain Roller Coaster

Across every major industry conference in the first half of 2026, our research team kept walking past the same scene: a booth ten or fifteen people deep, leaning in to catch every word from another datacenter equipment messiah preaching the gospel of 800VDC. The pitch was the same every time. 800VDC is about to change the electrical infrastructure of the datacenter.

Every architectural shift looked excessive at first. Operators spent decades keeping water and leaks out of the data hall, then GPU thermal density made running coolant right up against the precious silicon unavoidable. Each shift happened anyway, because physics and the economics of compute do not negotiate. 800VDC is next, and the logic is the same. Tokens per watt are what matters.

As GPU clusters become increasingly dense, with Kyber Ultra approaching 660kW per rack, the physics start to break down. Resistive losses scale with current squared, and at these power levels copper mass and thermal envelope exceed what fits inside a rack. Moving to 800VDC eliminates conversion stages, reduces resistive losses, and cuts facility-level power consumption by ~5%. At 1GW of IT load, that is over 50MW of continuous savings, tens of millions in annual electricity costs, or new compute capacity unlocked. For all the inference-king proponents out there, 800VDC is a transition forced by physics and motivated by system economics.

We have been tracking this transition through our InferenceX and Industrials Models, which provide a bottom-up view of where efficiency gains materialize and which equipment categories absorb the disruption. The Industrials Model includes a dedicated 800VDC module, building up from individual accelerator architectures to a top-down view of 800VDC penetration, MW adoption, and market sizing for equipment like the power sidecar and Solid-State Transformers (SSTs).

This deep dive traces the transition phase by phase: from the sidecar retrofit, through faciliy-level DC distribution, to the SST endgame. For each phase, we analyze the BoM and map the changes in equipment content/MW, what survives, what gets redesigned, and what gets eliminated.

This has enabled us to successfully call out both winners, and companies inaccurately pictured as losers by the market, before anyone else. If you are wondering whether UPS systems have a place in upcoming 800VDC distribution, what is the market opportunity for SSTs, or which suppliers are leading this transition, stick with us.

Part 1 of this 800VDC Revolution series covers datacenter layout and equipment implications. Part 2 will focus on power electronics and the semiconductor revolution underneath it.

Understanding The Basics: What is 800VDC and Why It’s Inevitable

At its simplest, 800VDC in this context means distributing power at ~800 volts direct current through the data hall or row and into the rack, then stepping it down near the compute. The number 800 is not arbitrary, but a voltage high enough to materially reduce current (and therefore copper loss and thermal burden) while remaining within the broad regulatory and product-safety classification of “low-voltage DC” in many jurisdictions. For context, EU rules around the Low Voltage Directive scope reference DC equipment ratings up to 1,500 V DC (and AC up to 1,000 V).

Current datacenter electrical architectures generally rely on AC distribution at the facility level. Datacenters today use three-phase AC at 415V or 480V, and the topology relies on conventional UPS architectures before distributing 48-54V DC within the rack.

This works at today’s rack power levels, but starts to fail as rack densities in the next two years approach ~600 kW+, for several reasons:

At the end of the day, 800VDC is the physics enabler for 2,300W TDP chips and 600kW racks, and those 600kW racks are the direct consequence of the push for density, because density is what drives cost per token down. Cost per token is dictated by the size of the scale-up world you can build at full NVLink bandwidth: bigger domains mean wider Expert Parallelism (EP) / Tensor Parallelism (TP), MoE routing on NVLink rather than scale-out, and less serialization across decode. As we laid out in our Vera Rubin Deep Dive and GTC 2026 pieces, Nvidia’s design rule is to pack compute tightly enough that copper reaches everything in the rack. Reiner Pope made this point cleanly on our friend Dwarkesh’s podcast a few weeks ago, indicating that a single rack bounds the size of the expert layer you can build, because the moment an all-to-all crosses a rack boundary, it falls onto a scale-out fabric that is roughly eight times slower than NVLink.

Bigger scale-up worlds mean denser racks, denser racks mean 600kW envelopes, and 800VDC is what makes those envelopes possible.

The Four Chapters of the HVDC Transition

The move to 800VDC is a complex metamorphosis that rewrites the entire electrical architecture, introduces new safety standards, requires new regulatory frameworks, and, most importantly, forces operators to make very different strategic choices about when to walk away from their legacy AC distribution.

We frame the 800VDC transition as progressing through four distinct phases. Phases 1 and 2, starting in late 2026 / early 2027, retrofit the existing AC distribution into 800VDC at the rack level via the power rack. Phase 1 is the early-mover stage, driven by hyperscalers willing to pay up for future-proofing and efficiency gains. Phase 2 kicks in once 800VDC-native systems begin shipping at volume. Phase 3 rewrites the electrical architecture itself, taking 800VDC distribution facility-wide. Phase 4 is the end state, built around new pieces of equipment that promise to render much of today’s electrical stack obsolete.

The result is a progressive adoption curve for 800VDC. We expect total incremental capacity powered by 800VDC to reach ~39GW by 2030. Through Phases 1 and 2, all addressable capacity is served by sidecars, since the underlying facility is still AC-distributed and the conversion happens at the power rack. The mix inflects in 2029 as facility-level HVDC distribution becomes viable and the first 800VDC-native sites come online, shifting the conversion stage upstream from the rack to the SST or MV rectifier.

Before diving into how the datacenter layout changes, we encourage readers to revisit Part 1 of our datacenter anatomy series, which explains many of the core concepts behind datacenter electrical equipment.

Phase 1 (2026/2027): The White Space Retrofit

The HVDC journey begins primarily with two operators, Google and Meta. Both have been pushing their 800VDC architectures through the OCP working groups for over 18 months, most visibly with the Mt. Diablo reference design, first announced in October 2024 and published as an open specification in May 2025. Neither is being forced into the transition, but they are doing it to take a leading position in the upcoming shift and because they want to squeeze every megawatt and every point of efficiency out of their existing power chain before the rest of the market is forced to catch up.

This matters because 800VDC is not yet a hard requirement. The chip generations ramping in late 2026 and 2027, like Vera Rubin NVL72, top out at rack densities of 180-220kW. Three-phase AC can still deliver that without hitting the physical limits of conductor sizing or distribution losses. Phase 1 is therefore voluntary future-proofing, not a forced response to a hardware constraint.

This initial phase kicks off the “White Space Retrofit” era. New HVDC hardware, primarily a row-level cabinet called the HVDC power rack, layers on top of existing white space infrastructure rather than replacing it. The datacenter’s electrical backbone stays intact. Same transformers, same UPS, same switchgear, same ATS.

Power Flow Overview with HVDC Power Rack

At the facility level, Medium-voltage AC enters the grey space and is stepped down via transformer to 415V or 480V three-phase AC. That feeds into a UPS, which performs double conversion (AC-DC-AC), then outputs 415V AC. AC is then distributed through the data hall via busway. So far, this is the traditional power flow we have extensively covered in previous articles.

The change occurs when we get closer to the IT racks. Instead of feeding 415V directly into in-rack power supply units, the AC feed now terminates at a standalone 42U cabinet named the HVDC power rack deployed at the row level.

The rack receives AC from the overhead busway and outputs 800VDC through cable to adjacent IT racks. Inside, it performs three jobs: rectification of 415V AC to 800VDC, BBU modules for ride-through during outages, and optionally, capacitor shelves for transient buffering during GPU load spikes.

In a Nutshell: The Power Rack

It is worth looking in more detail into the building block that underpins Phases 1 and 2 of the 800VDC transition: the disaggregated power rack. This is a dedicated rack that consolidates AC-to-DC rectification, energy storage (BBU and/or capacitor bank), and power management into a single unit, freeing the compute rack to be entirely dedicated to GPUs, networking, and cooling. Microsoft’s Mt Diablo project originated the concept; the OCP Diablo 400 specification, co-authored by Google, Meta, and Microsoft, standardizes it.

But the sidecar concept did not emerge fully formed. It evolved through several OCP rack and power specification versions. The earlier iterations (ORv2 at 12V, ORv3 at 48V, and the HPR V1/V2 variants that pushed single-rack 48V designs up to ~190 kW with liquid-cooled busbars and upgraded 72 kW power shelves) are covered in our Datacenter Anatomy series. Here we focus on the versions directly relevant to 800VDC: the disaggregated sidecar designs where the voltage transition occurs.

ORv3 HPR V3: The Disaggregation Threshold (50V Sidecar, up to 300 kW)

HPR V3 is really where power and compute separate into distinct racks, the genesis of the “sidecar” concept. PSU and BBU shelves move into a dedicated 50VDC side power rack connected to the IT rack through horizontal busbars at the top and bottom of both. Both remain ORv3 HPR standard form factor. Power capacity tops out at 300 kW, limited by the horizontal crosslinks and the air-cooled vertical busbar inside the power rack.

The insight is putting power conversion hardware in a rack optimized for power, with appropriate cooling, safety, and serviceability, rather than cramming it into a rack optimized for compute. The V3 power rack can be serviced independently, shrinking the blast radius of power-side failures. But V3 still distributes at 50VDC, which means busbar currents remain high (6,000A at 300 kW) and the crosslinks become the bottleneck.

This persists today. Even the VR NVL72 rack, when fed by an HVDC power rack at 800VDC (Nvidia spec) or ±400VDC (OCP spec), still distributes internally over a 50V busbar. A DC-DC power shelf inside the rack steps the high-voltage DC down to 50VDC before it reaches the compute trays. At the far end, VRMs on the GPU board convert from 50V to sub-1V.

ORv3 HPR V4: HVDC Sidecar at +/-400VDC (up to 800 kW)

HPR V4 is the version that bridges the OCP HPR lineage into the HVDC era. It makes two critical changes: the voltage steps up from 50VDC to +/-400VDC (800V total), and the busbar-based crosslink is replaced with discrete power cables.

V4 effectively represents the “pre-Diablo” state of HVDC sidecar design, developed primarily by Meta’s rack and power team. It proved the concept of disaggregated HVDC power delivery but was not yet a multi-vendor, multi-hyperscaler specification.

The Diablo 400 Specification: Standardizing the HVDC Sidecar

The Diablo 400 specification (named after Mt Diablo, Microsoft’s original internal project name) formalizes and standardizes the HVDC sidecar concept that HPR V4 pioneered. Co-authored by Google, Meta, and Microsoft, Diablo 400 was released as a draft specification (v0.5.2) in May 2025, with a subsequent v0.7.0 revision following industry feedback.

The choice of 400VDC as the nominal voltage was deliberate. As Google’s engineers stated at OCP EMEA 2025: “selecting 400 VDC as the nominal voltage allows us to leverage the supply chain established by electric vehicles, for greater economies of scale, more efficient manufacturing, and improved quality and scale.” In the bipolar configuration, each individual rail sits only 400V from the grounded midpoint, keeping the system within the voltage range where mature EV-grade power electronics (650V GaN FETs, 400V-class capacitors, connectors, and fuses) can be used directly.

No One-Size-Fits-All

There is no one-size-fits-all 800VDC power rack. Yes, Diablo 400 provides a shared base specs, but the reality on the ground is fragmented. Nvidia sits entirely outside it and is developing a monopolar 800V reference design at 660kW, with air-cooled samples and production in mid-2026, and a liquid-cooled VR Ultra variant sampling in late-2026.

Even within Diablo 400, the three co-authors diverge meaningfully. Meta runs 600-800kW with 50kW HVDC output cables and 8x 200A AC input whips. Google push to 900Kw by reallocating rack space from BBU and supercap slots to PSUs, run 100kW output cables, and need 12 AC whips at the 1.1MW roofline. Amazon’s design lands at 800kW on ±400V. Microsoft co-authored the spec but we believe they are making slower progress.

Besides, an alternative sidecar topology uses an LV-input SST in place of the conventional rectifier-plus-PSU stack, like DG Matrix’s Interport Cell Series.

The cost of the power rack

The HVDC power rack is the headline new-equipment cost in the early retrofit phases. We estimate the ASP for the Power Rack to reach $400-500k per unit, roughly 10x the ~$40k ASP of standard AC power-rack equipment. On a deployed-MW basis, that lands near $500k/MW.

The Sidecar Market Opportunity and TAM sizing

In our SemiAnalysis Industrials Model, we size the 800VDC equipment TAM, specifically for the sidecar (power rack) and Solid State Transformer (SST), by applying this phase-by-phase adoption timelines to incremental datacenter capacity build and doing a chip-by-chip SKU calculation.

We expect sidecar TAM to peak at ~$11B in 2028 before declining as facility-level 800VDC takes share in Phase 3. We assume a power rack content of $0.5M/MW.

Phase 1 Summary

The white-space retrofit represents a clear cost uplift in electrical content/MW versus current architectures, because Phase 1 essentially deletes nothing. We estimate the delta at roughly +$400-500k/MW, with the HVDC power rack accounting for the large majority.

Phase 2 (2027/2028): The Turning Point Comes with 800VDC-Native Compute

Phase 1 was the start of the retrofit era. The real inflection point comes with the arrival of 800VDC-native systems. At that point, 800VDC stops being a future-proofing pilot and becomes a mandatory transition forced by physics and rack density. Operators electrifying the Kyber Rack have no AC fallback at the rack inlet, and we expect 800VDC penetration to spike sharply in this window. Because 800VDC-native silicon will land before facility-level 800VDC distribution is ready, the retrofit phase persists.

Architecturally, Phase 2 looks very similar to Phase 1. Both retrofit the white space with the HVDC power rack, both leave the grey space intact, and both rectify AC to DC in the row-level power rack. The key difference is where the voltage steps down to chip-usable levels. In Phase 1 (Oberon rack), a power shelf inside the IT rack converts 800VDC to ~50VDC before it reaches the compute trays. In Phase 2 (Kyber rack), the 800VDC bus runs directly to the compute blade, and an on-blade power module handles the final step-down to 50V.

Earlier Kyber designs shown at OCP depicted a DC-DC PSU sidecar adjacent to the compute rack, but we now believe this approach is unlikely to be adopted at scale. A standalone sidecar consumes more aggregate floor and rack space than integrating the conversion stage into the blade itself, and the power module form factor has proven feasible within the compute tray’s volume constraints.

Because most servers and trays still take roughly ~50V input, both architectures retain a high-power 800V-to-~50V DC-DC conversion stage. The difference is where that conversion happens.

Some discussions have explored delivering 800VDC directly into the compute tray and stepping it down to an intermediate bus voltage (IBV) before further conversion to point-of-load rails. While Kyber’s on-blade power module does accept 800V input, it converts to the established ~50V bus level rather than an IBV scheme. A full 800V-to-IBV-to-PoL architecture within the tray remains extremely challenging given the limited space and safety constraints involved.

What Happens With UPS and Battery Storage

Traditional central UPS systems are probably the most contested piece of infrastructure in the 800VDC transition. In the 800VDC architecture, we expect centralized Low Voltage UPS systems to progressively lose their role and eventually become obsolete. In the retrofit era, the power rack sits directly on the 800VDC bus and houses BBU modules and supercapacitors, which we cover shortly. Both are natively DC-coupled. BBUs bridge seconds-to-minutes during outages and supercapacitors absorb millisecond-scale GPU load transients. Together, they replace the centralized short-term battery storage and UPS ride-through function without the 2-3% conversion loss of an AC-DC-AC UPS pair.

As we covered in our electricals deep dive note, Google and Meta already took this aggressive approach years ago, bypassing the central monolithic UPS with “distributed UPS” architectures. In their architecture, AC power is distributed directly to the rack, the in-rack PSU handles AC-DC conversion, and rack-level Li-Ion Battery Backup Units (BBUs) provide the short-duration bridge power. This removes the central UPS’s AC-DC-AC conversion pair and improves efficiency, while also cutting in half the total battery capacity needed for the datacenter, since there is no longer a need for both an A-side and a B-side UPS.

That said, managing distributed UPS or battery backup is more operationally challenging than running a traditional central UPS. We expect operators other than vertically integrated hyperscalers like Google and Meta to keep the Low Voltage UPS in place for redundancy and load fluctuation management, at least in the medium term.

This is especially true for colocation providers, which prioritize flexibility and need to support mixed workloads: CPU racks, storage arrays, networking equipment, and older GPU racks that still run on AC. Keeping the grey-space AC infrastructure intact lets these operators deploy 800VDC for their densest AI racks while running standard AC distribution for everything else.

We expect different operators to adopt different architectural approaches to backup, and new alternatives are emerging. Medium Voltage UPS, operating at 4.16-34.5 kV directly at the grid connection point, is functionally similar to the rack-level Battery Rack but centralized at the grid interface rather than distributed across the data hall. ABB’s HiPerGuard runs at 98% efficiency and is already deployed at Applied Digital’s 400MW North Dakota AI campus. ON.energy was awarded few weeks ago a US patent that protects their MV double-conversion UPS architecture. The second alternative is facility-level BESS, which as we covered in our deep dive operates at megawatt-to-hundreds-of-megawatts scale, provides 1-4 hour duration backup, and increasingly replaces or shrinks the diesel generator.

Phase 3 (late 2028/2029): Redesigning the Electrical Architecture With a Centralized Rectifier

In Phases 1 and 2, the AC-DC conversion happens close to the rack, inside the row-level HVDC power rack. Phase 3 changes the datacenter layout itself, and 800VDC becomes the building’s electrical core. This is the true inflection point, where things start to become interesting. Let’s unpack what happens in each area of the datacenter.

What Happens in the Grey Space: Power Distribution Goes DC

In Phase 3 a dedicated upstream rectifier that sits in the grey space or outdoors converts 415V AC to 800VDC, distributing DC across the entire hall. These are mature units using silicon IGBTs or thyristors rated at 1200-1700V.

The grey space splits in two. MV transformers connecting the datacenter to the grid are unchanged. MV switchgear stays because the utility feed is still AC, and the upstream MV infrastructure (11-34 kV) is expected to grow more complex as facilities scale to gigawatt clusters. LV transformers remain, stepping MV down to 415V AC for the upstream rectifier. The 480V AC switchgear between LV transformers and PDUs has no role once 800VDC flows through the busway, and AC floor PDUs are eliminated along with it, since the DC busway feeds the battery rack directly with no AC distribution PDU in between. In summary, everything above the AC-DC conversion point stays, while everything below it, designed for AC distribution, goes.

Understanding DC Distribution: Switchboards, Busway, and Protection

In Phase 3, the AC switchboard’s function of splitting one feed into multiple protected outputs has to land somewhere. Three product categories are positioned to absorb it: (i) MW-scale rectifiers built with multiple outputs and integrated SSCB protection per output, turning the rectifier into its own distribution device; (ii) DC busway with breaker-equipped tap-off boxes that hold protection in the distribution medium, once DC-rated tap-offs with adequate arc interruption mature; and (iii) prefabricated grey-space pods that bundle rectifier, switchboard, and busway into a factory-built skid, particularly for hyperscaler procurement.

Major AC switchboard incumbents (Schneider Electric, ABB, Eaton, Vertiv) have not named discrete 800VDC switchboard products. ABB’s October 2025 Nvidia partnership covers distribution inside its “modular power block” rather than as a standalone switchboard. EPEC Solutions sells a publicly marketed 800VDC LV switchboard with high-interrupting-capacity DC breakers. We expect the discrete switchboard to retain a niche in retrofits with existing single-output rectifiers, and where operators want vendor-neutrality across the rectifier and protection layers.

Once the power is rectified, a DC busway replaces AC busway for hall-level 800VDC distribution. In traditional AC datacenters, busway systems have modular plug-in connections called tap-offs that branch power to individual racks or rows, similar to outlets on a power strip. You can add or remove these while the busway is energized. A feeder-only busway, by contrast, has no intermediate openings or tap-offs. Power enters at one end and exits at the other end or at predefined termination points.

We expect early 800VDC deployments to use feeder-only busway because, essentially, tap-offs become more complex. At 800VDC, interrupting current under load creates a sustained arc (a plasma discharge producing extreme heat) that does not self-extinguish because DC has no zero-crossing point, while AC arcs naturally extinguish 100-120 times per second as the waveform crosses zero. Besides, DC-rated tap-off units with adequate arc interruption are physically larger, making them impractical today. Delta and ABB have publicly disclosed 800VDC busway programs, and we expect other major busway vendors like Legrand and EAE to follow in 2026.

To address these challenges, multiple proven protection paradigms exist at this voltage class from adjacent industries. The likely implementation combines multiple approaches, one being new generations of circuit breakers. More specifically, following the same solid-state trend already underway with Solid State Transformers, Solid State Circuit Breakers (SSCBs) are now being adopted. SSCBs use SiC or GaN to interrupt fault current in microseconds. Because semiconductor switches can simply stop conducting with no physical contact separation, there is no arc to extinguish in the first place.

The new-generation circuit breakers are already commercialized today. ABB has the Emax 2 (1500V DC) used in solar, energy storage or marine, as well as the SACE Infinitus (solid-state, 1000V/2500A, datacenter adaptation with Nvidia announced October 2025). LS Electric has the first UL-certified DC molded case circuit breaker at 1500V, listed for datacenter applications.

Alternative Path Using LV Solid State Transformers

An emerging alternative to the centralized AC/DC rectifier is using LV SSTs. It performs the same conversion, 415V AC to 800VDC in the grey space or outdoors, but in a more compact and programmable form factor. The LV-SST sidesteps the 3,300V-class SiC supply constraint that gates MV-input SSTs, making it the earlier-to-market SST variant.

What Happens in the White Space: From the Power Rack to the Battery Rack

As you can imagine, in Phase 3 we no longer need the Power Rack doing the 800VDC conversion. Instead, we salute a new friend, the battery rack.

The battery rack shares most of the power rack’s components and functions. The main difference is that it no longer performs AC-DC rectification, because it receives 800VDC directly from the grey space. Three main components remain:

The battery rack sits generally at the same row level as the power rack it replaces, although some operators are deploying these in the adjacent grey space or in outdoor enclosures. The trade-off is simple: rectifiers go away, BBU and supercapacitor content goes up. We expect content per MW for the battery rack to reach around $200k/MW.

We covered supercapacitor chemistry and technical specifications in our deep dive on AI training load fluctuations. Part 2 of this 800VDC series will go deeper on supercapacitor economics, cell chemistry, the vendor landscape, and the practical tradeoffs of deploying them in production.

BBU modules scale up

Current modules are rated at roughly 5.5kW. With Rubin Ultra and 800VDC architectures, individual module wattage rises to 8-12kW. Infineon’s BBU roadmap, announced in March 2025, uses modular 4kW Partial Power Converter cards that parallel to 12kW per unit at up to 99.5% peak efficiency.

Delta, at GTC 2026, went further at the shelf level: its new 110kW power shelves embed 80kW of BBU capacity each, totaling 480kW across a six-shelf rack. Higher rack power demands proportionally more backup energy per rack, and higher-wattage modules deliver that energy with fewer physical modules, preserving space in the power rack.

What Happens at the Facility Level

After analyzing the complete transformation of the grey space and the white space, the facility level is the part that changes least.

Here, cooling stays on AC. Chillers, pumps, and fans still run on AC motors, requiring DC-to-AC inverters. Delta unveiled a 2.4MW In-Row CDU supporting 800VDC at GTC 2026, the first major cooling component engineered for native DC. But the full stack (chillers, compressors, pumps, building controls) remains AC-dependent, and no vendor sells an integrated DC-native cooling system.

Generator architecture is already loosening at some hyperscalers independently of 800VDC. Meta is likely bypassing generators at new sites entirely, and Microsoft’s new designs use partial generator coverage. 800VDC could accelerate that direction, as supercapacitors, BBUs, and BESS form a distributed backup hierarchy that absorbs the functions generators used to own.

Medium Voltage Rectifiers: Is There Room for Everyone?

One reasonable question to ask is why power is rectified at the LV level and not directly from MV? The answer comes from semiconductor ratings. Rectifying from 13.8kV or 34.5kV requires devices rated above 10kV, which barely exist in commercial form today. That said, the gap is closing, and Wolfspeed’s 10kV SiC MOSFET has been commercially available as bare die since March 2026.

The development of SiC MOSFETs above 10kV opens the door to a second evolution of Phase 3, where even the LV equipment drops out of the main power bus. Continuing the trend, this collapses additional conversion steps and brings new efficiency gains.

The end state of our HVDC timeline will push even further. Even though conventional rectifiers with series-stacked silicon devices can handle MV rectification, an emerging technology promises to do it in a much more efficient, compact, and faster way. That technology is our protagonist for the next chapter of our journey: Solid State Transformers.

Phase 4 (>2029): SSTs, the End-State

Finally, we get to the holy grail of DC power distribution: Solid State Transformers, or SSTs. These are a new category of power electronic devices that replace conventional iron-core transformers with high-frequency, semiconductor-based converters.

Phase 4 and its datacenter layout is very similar to Phase 3. The major change is that the SST replaces the LV AC-DC rectifier and low-voltage transformer with a single piece of equipment that converts directly from medium voltage to 800VDC. If we consider the ending of the previous section, the possibility of using a MV rectifies that rectifies directly from MV AC, the architecture is essentially identical.

In a Nutshell: Solid-State Transformers

Introduction to SSTs

An SST does the same job as the massive iron-and-copper transformers in every datacenter’s grey space: step voltage from utility-level medium voltage to a level IT equipment can use. A conventional transformer uses magnetic induction at grid frequency. An SST uses semiconductor switching stages to achieve the same conversion in a fraction of the volume.

The datacenter SST is a three-stage device. The input stage converts AC to DC, handling the dangerous medium-voltage level (13.8 to 45kV) using SiC MOSFETs rated at 3,300V or higher. The isolation stage is where the size reduction happens. A high-frequency transformer steps the voltage down while providing galvanic isolation between the utility / power-source and the datacenter. The output stage produces the final 800VDC that the distribution system needs, with no inverter required.

Pros and Cons of SSTs

SSTs’ core value proposition is energy efficiency, which translates directly to OPEX savings or unlocked compute capacity. By collapsing the medium-voltage transformer and rectifier into a single power-electronic stage, SSTs eliminate two conversion steps from the electrical chain. Vendors target up to 15% total system efficiency improvement, claiming the path rises from around 82-85% towards over 97%.

SSTs are also dramatically smaller. A conventional transformer operates at 50 or 60 Hz and needs a massive iron core. An SST switches at 20,000 Hz or higher, shrinking the core by roughly 90%. That is where Infineon’s claimed 40x weight reduction and 14x size reduction (!) come from.

In addition, SSTs are programmable. A conventional transformer steps voltage at a fixed ratio. An SST actively regulates output, adjusting under load. It also supports bidirectional power flow (pushing power to the grid during demand response, or charging a BESS). That said, SSTs with bidirectional capability and integrated BESS may trigger DER reclassification by the interconnecting utility, requiring IEEE 1547/2800 compliance.

One additional major value proposition from SSTs is input flexibility. Some SST architectures extend this flexibility into multi-port topology, where a single device aggregates several inputs (utility AC, on-site generation, DC sources) and routes power across multiple outputs in software, including bidirectionally. The case for multi-port is that it reduces stranded power between zones and lets operators orchestrate flows across the site.

Reliability

Conventional transformers last 30-40 years as passive devices. No SST vendor has published field reliability data at datacenter scale, as the longest deployment is the Hitachi-ABB PETT on Swiss Federal Railways, running since 2011. SSTs concentrate heat in semiconductor junctions and require active cooling, with DG Matrix using integrated liquid cooling and Novos Power using air cooling through proprietary insulation.

ETH Zurich’s comparative evaluation found that a line-frequency transformer paired with a SiC rectifier can match SST efficiency and functionality. Datacenter-scale SSTs depend on SiC MOSFETs at 3,300V+ for the MV input stage, still in limited production. GaN, capped at roughly 650V, serves only downstream stages converting 800VDC to rack-level voltages.

Current Efficiency State

The best public SST benchmark comes from ETH Zurich: 98% efficiency at 400kW in a 13.2kVAC-to-800VDC prototype presented at INTELEC 2025. Johann Kolar frames 98.0-98.5% as today’s state of the art for full-scale SSTs, with 99% as the next engineering target for datacenter units.

Different vendors now converge on that 98.5% ceiling: DG Matrix’s Interport platform claims up to 98.5%, Amperesand’s third-generation system claims greater than 98.5%, and Heron Power’s Heron Link targets 98.5% MV-to-rack efficiency. Novos Power reports peak efficiencies over 98%. These are encouraging, but datacenters will need 3-6MW units sustaining 99%+ efficiency under continuous load.

Two data points suggest the scale-up is underway. Chinese trade press reports that China XD Electric has deployed 2.4MW datacenter SSTs under the “East Data West Compute” program. NC State’s FREEDM Systems Center, the academic foundation which marked the origin of DG Matrix, has demonstrated 210 kHz switching at 3.3kV SiC with a 99% efficiency target for modular DC-DC SST variants.

Vendor Landscape

The vendor landscape is moving fast. DG Matrix (ABB-backed, Infineon SiC supply deal) is shipping pre-certification units and targeting UL certification by end of Q2 2026. It is the only SST included in Nvidia’s MGX reference architecture. Amperesand targets 30MW of commercial deployments in 2026. Heron Power is building a 40GW US manufacturing facility for its 4.2MW Heron Link units.

Within the SST category, products are bifurcating along LV and MV input. DG Matrix and Amperesand are pursuing both, starting with LV-input SST skids (3.2-4.8 MW) that can be deployed today alongside existing AC distribution, and following with MV-input units as 3,300V-class SiC matures. Heron Power and Novos Power are concentrating on direct MV-input units that collapse the LV transformer and rectifier into a single device. Both paths converge on 800VDC at the output, but the LV path offers a shorter time-to-deployment at the cost of retaining the upstream MV-to-LV transformer.

Novos Power claims a direct MV-to-800VDC SST with 50% smaller footprint and air cooling. On the incumbent side, Eaton acquired Resilient Power Systems in August 2025 for SST expertise. More than $320M flowed into SST startups in the twelve months ending March 2026.

Datacenter Layout Implications

The SST eliminates the LV equipment at ∼$0.55M/MW and the Phase 2 rectifier at ∼$0.20M/MW. At an estimated SST cost of ∼$1.0-1.5M/MW, we expect the first instances of SSTs to come at an upfront Capex premium over directly replaced equipment.

The rest of the electrical architecture remains the same as in Phase 3. The 480V AC auxiliary bus for cooling, lighting, and facility systems carries over unchanged. On the IT rack-side, we expect that by the time SSTs are deployed, compute trays are already 800VDC native. However, we could see deployments of SST adoption with 800V microgrid and IT racks using a DC-DC power shelve converter, which could accelerate the adoption.

On Phase 4 timings, this emerging technology is still in design phase, and we don’t expect major SST adoption at scale until early 2029. That said, we are aware that all major hyperscalers are running pilot and testing with main SST vendors, with commercial contracts already in place. As we cover in the following section, technology development itself will not be the only factor determining the adoption curve here. Regulatory framework and standards is a big one. In the SST space, no vendor has completed UL certification for datacenter SST deployment as of May 2026.

The SST Market Opportunity and TAM sizing

By 2030, we expect SST TAM to reach ~$13B, capturing the demand displaced from the sidecar layer plus the incremental MV-to-800VDC conversion. We consider a content of $1.25M/MW. A portion of this opportunity is contested by MV rectifiers, but we expect SSTs to capture the majority share.

Datacenter Layout Summary: Total Cost Barely Moves, Content Shifts, Efficiency Climbs

Electrical System Cost

Total electrical content per MW stays in a $3.6-4.8M band across four of the five architectures we model. The main headline is a content migration from grey space to white space, and the resulting change in equipment mix.

Grey space content shrinks in Phase 2 as the centralized UPS ($1.2M) exits. White space peaks in Phase 1 because the HVDC power rack arrives. By Phase 4, total content climbs to $4.0M as the SST replaces the LV transformer and rectifier.

Electrical System Efficiency

We calculate the baseline AC power path at 82.0% cumulative efficiency across seven conversion stages. The VRM (92%) and PSU (94%) are the two largest single-stage losses. The VRM stays in every architecture, but the PSU’s loss is the largest penalty the 800VDC transition can eliminate. Phase 1 barely improves to an estimated 83.7%. The UPS double-conversion loop still eats 3 percentage points, and the new power rack rectifier (97.5%) plus DC-DC stage (97.0%) only marginally outperform the old single-stage PSU.

The real jump comes in Phase 2 (86.5%) when UPS elimination cuts the chain from seven stages to five. Phase 3 pushes to 86.9% because the centralized grey-space rectifier operates at MW scale (higher efficiency than modular rack-mounted units) and 800VDC hall-level distribution eliminates AC skin effect and reactive power losses. We estimate Phase 4 to reach 87.4% as the SST replaces two stages with a single device.

At 1GW of IT load, the Phase 2 gain translates to roughly 58MW of continuous grid power savings. Phase 3 extends that to 63MW and Phase 4 to 69MW. Nvidia cites up to 5% efficiency improvement, implying roughly 50MW at 1GW. Our Phase 4 efficiency delta calculations of 5% vs baseline matches Nvidia’s reported figures.

The Other Side of the 800VDC Transition: Challenges and Limitations

So far, we’ve mapped out a promising path, but as always, various challenges will arise along the way. We now unpack four main obstacles that will determine how fast 800VDC moves from small-case pilots to broader adoption.

Challenge 1: Regulation, Safety and Grounding

Regulation

The National Electrical Code (NEC), published by NFPA on a three-year cycle, governs electrical installation in the United States. Adopted by nearly every state and municipality as binding law, it determines whether an operator can build to a standard design or must negotiate site-by-site with the local Authority Having Jurisdiction (AHJ). Full 800VDC code support targets NEC 2029. Pre-2029 deployments therefore require custom AHJ approvals and OEM-level UL certification for each site. This is workable for hyperscalers with in-house code engineering teams but could represent a meaningful barrier for colocation operators and smaller builders.

We see a useful parallel in the early days of the EV industry, where Tesla designed and approved its own internal safety frameworks because industry-wide standards had not yet arrived. Hyperscalers deploying 800VDC pre-2029 will be in a similar position.

NEC 2029 would already be fast by historical standards, considering prior DC power standardization timelines in marine, telecom, and EV. However, timelines could benefit from extreme buyer concentration with five hyperscalers and Nvidia as both demand creator and solution architect and an EV 800V component supply chain.

We think NEC 2029 will achieve partial provision, while full code maturity probably lands at NEC 2032 or 2035. Partial means the basic framework exists (voltage classification, conductor sizing, overcurrent protection) but DC-specific arc flash PPE tables, busway standards, and stored energy maintenance protocols will likely be absent.

Safety

The biggest safety risk is arc flash. IEEE 1584 does not cover DC, and NFPA 70E has no PPE table for 600-1000VDC. UL Solutions has launched a Direct Current Safety Research Consortium to build the missing hazard models, explicitly citing 800V DC datacenter architectures among the target applications.

Even beyond the code gap, daily reality may be harder. At 48V, a technician can hot-swap a server tray with minimal PPE. At 800V, many rack-adjacent tasks that were routine at 48V likely require a qualified person under NFPA 70E, with arc-rated clothing, insulated gloves rated to 1000V, and a face shield. Capacitor banks and BBU modules retain dangerous charge after power-down, and standard lockout-tagout procedures for AC do not account for stored DC energy. Multiple sources must each be verified de-energized before maintenance.

Flex, a major Nvidia manufacturing partner, has publicly advocated for in-depth hazard identification and safety training at 800VDC facilities.

Grounding

Grounding cascades into protection-device count, fault behavior, insulation monitoring, personnel safety, and vendor compatibility, which makes it on the most consequential early design choice in an 800VDC facility.

The Siemens/Nvidia paper “Protections for Data Centers Powered by Direct Current” identifies four options. A ±400V system can use high-resistance grounding (HRG), which tolerates the first ground fault and only requires fast interruption on the second, or solid grounding, which demands immediate clearing of any fault. An 800V monopolar system can float, with insulation monitoring on every branch, or run a solid-grounded return conductor.

The tradeoff is cost. HRG and floating systems need protection devices rated for the full 800VDC on both conductors plus insulation monitoring infrastructure. Solid-grounded-return cuts the protection device count but eliminates galvanic isolation between parallel converters. OCP Diablo 400 permits both ±400V bipolar and 800V monopolar, leaving the call to the operator.

The reality is that no industry consensus exists. SST and power-electronics vendors are optimizing around different grounding assumptions, which makes the choice a vendor-ecosystem commitment, not just a technical one.

Challenge 2: Cooling and Auxiliary AC Workloads

Cooling is the largest AC load in an 800VDC datacenter, and no vendor sells a DC-native cooling ecosystem. Some vendors like Delta and Danfoss are doing progress. Danfoss’s Turbocor compressors, dominant in datacenter chillers, run internally on DC at 700-813V. Danfoss also manufactures the VACON NXP variable-frequency drive, which accepts 640-1200 VDC input directly, placing 800V within its operating range. DCAirco ships 800V DC chillers for e-mobility at 4-8kW, 100-1000x too small for datacenter scale, but proof the refrigeration cycle works at this voltage.

Beyond cooling, switchgear operating mechanism, lighting, fire suppression pumps, building management sensors and security systems all run on AC. As Nvidia team presented at OCP Global Summit 2025, 800VDC reference architecture will retain an AC auxiliary bus alongside the 800VDC compute distribution for exactly this reason.

That said, the supply chain is moving. The Delta CDU noted above is the leading edge, but most auxiliary categories (lighting, fire suppression, security) lack DC variants. With datacenter industrial capex heading toward >$400 billion in 2026 and electrical equipment at 30-35%, the incentive to develop DC-native products is growing.

Challenge 3: Supply Chain Standards

Innovation in DC distribution is ahead of codification, and standards still lag across most 800VDC equipment categories. Busway is a good example of progress. UL 857, the standard governing busway systems, originally capped coverage at 600V and defined values in root-mean-square (RMS). Edition 14, published in 2025, raised the ceiling to 1000VDC, and Edition 15 in development targets 1500VDC. Outside busway, certification paths remain absent, and every installation becomes a custom engineering project where the operator must qualify the product, negotiate conductor ratings, and obtain AHJ approval on a case-by-case basis.

An OCP white paper targeting 2026 may help, and OCP working groups are coordinating with regulators and certification bodies to land initial standards by year-end 2026, but vendors are already presenting their prototypes. Delta demonstrated 800VDC air-cooled busway at OCP 2025, LS Electric exhibited DC power equipment at DistribuTECH 2026, and in almost all recent conferences the team has been present in, 800VDC-ready prototypes have been without a doubt the protagonists.

Challenge 4: Grid Interconnection and Regulatory Pressure

As we covered in our deep dive on AI training load fluctuations at gigawatt scale, datacenter load-loss events during grid disturbances have become a serious concern for grid operators. 800VDC sharpens the problem by moving grid-facing behavior into software-defined power electronics (SST control algorithms, converter current limits, DC bus capacitance…).

Grid operators now have to model and constrain those dynamics, and the regulatory bar is also rising. NERC issued a Level 3 Essential Actions Alert (its highest tier) on May 2026 covering large computational loads, with a mandatory response deadline of August 3, and has proposed a Computational Load Entity registration for datacenters consuming 1MW+ within a 20MW+ aggregate at 60kV+. ERCOT’s NOGRR282 adds voltage and frequency ride-through requirements and mandates both PSS/E and PSCAD electromagnetic transient models for all large loads.

Why 800VDC raises the burden of proof

Traditional AC datacenters have a grid-facing vocabulary that planners can model: UPS transfer thresholds, ATS timing, motor-load behavior, generator controls, and composite load models like CMPLDW. None of that captures an 800VDC facility, where the response to a grid voltage dip depends on the SST control algorithm (grid-following vs grid-forming), BESS state of charge, instantaneous GPU load profile, and interactions between multiple parallel SSTs.

800VDC also collapses layers of the power stack. In an AC facility, the utility studies the interconnection and aggregate load while the operator engineers UPS, switchgear, and rack distribution independently. In an SST-based 800VDC facility, the same converter controls determine DC bus stability, fault ride-through, current limiting, harmonic injection, and post-fault load recovery. Interconnection therefore becomes an engineering product that requires EPC capabilities bridging power electronics design, grid-level dynamic modeling, and regulatory engagement. This is leading to new entrants like Aran Industries, building AI-native EPCs to deliver PE-stampable 800VDC engineering packages.

Understanding The No-So-Basics: The Physics Behind 800VDC

Why Going Super Dense Makes Low-Voltage Distribution Break: Heat and Weight

At a fixed power level, raising voltage from 54V to 800V cuts current by ~15× and resistive losses by ~220×. That is what makes 800VDC a step-change in copper mass, thermal load, and distribution cost.

For a fixed rack power P, raising V reduces I linearly. Lower current means smaller conductors, less copper mass, and easier routing.

That drop is the energy dissipated as heat in the conductor. Substituting into the power equation yields the resistive loss equation:

Current appears squared, so the voltage-to-loss relationship is quadratic, not linear. This is the equation that makes 800VDC inevitable.

That’s a 14.8× reduction in current. Now apply the loss equation. For the same conductor resistance R, the I² ratio implies:

Resistive heating at 54 V is roughly 219 times higher than at 800 V for the same conductor. In the more commonly cited comparison using 48 V:

In practice, operators do not keep the same conductor and pocket all 219-278× of loss reduction. They shrink the copper, trading loss headroom for reductions in weight, cost, and routing space. Even after right-sizing for 800V, the efficiency gain remains transformative.

800VDC vs. ±400VDC: The Topology In Question

‘800 VDC’ may refer to two distinct electrical configurations, and the distinction matters for deployment strategy, safety engineering, and downstream semiconductor selection. ‘800 VDC’ may refer either to a single-ended 800V bus or a bipolar ±400V bus (800V pole-to-pole):

Single-ended 800V

In a single-ended 800VDC architecture, the bus is a single 800V rail referenced to return, plus protective earth. At 1MW, the bus carries 1,250A. Lower current means smaller conductors, smaller connectors, and lower I²R losses throughout the distribution path. The bus structure is also simpler to implement because it does not rely on maintaining symmetry between two rails. Power stages can be designed directly around the full bus voltage using standard high-voltage devices and conventional converter topologies. No midpoint to sense, regulate, or control.

Bipolar ±400V

The alternative splits that 800V into two symmetric 400V rails around a grounded midpoint: three power conductors (+400V, midpoint, -400V) plus protective earth. The load still sees 800V across its input, but each rail sits only 400V from ground. The central argument here is not electrical, but economic. 400V power electronics are mature because the EV industry built at scale on 400V platforms. Google stated at OCP EMEA 2025 that selecting 400VDC “allows us to leverage the supply chain established by electric vehicles”. The OCP Diablo 400 specification considers a disaggregated power rack converting 3-phase AC to ±400VDC at 100kW to 1MW per rack. The spec also includes 800VDC monopolar as a design option, leaving the door open.

There’s also a tradeoff. That third conductor must be routed, terminated, and protected at every point in the power path. Across thousands of racks, it adds meaningful copper, connector hardware, and installation labor, and it complicates hot-swap connector design where the midpoint must make and break contact in a controlled sequence to avoid transient voltage spikes.

Behind paywall we will now discuss the main winners and losers of the 800VDC revolution, and who is better positioned to be benefited from the transition.

We’d like to thank DG Matrix, Novos Power, and Aran Industries for their contributions and insights during the preparation of this deep dive.

Introduction: Welcome to the Power Chain Roller Coaster

Part 1 of this 800VDC Revolution series covers datacenter layout and equipment implications. Part 2 will focus on power electronics and the semiconductor revolution underneath it.

Understanding The Basics: What is 800VDC and Why It’s Inevitable

This works at today’s rack power levels, but starts to fail as rack densities in the next two years approach ~600 kW+, for several reasons:

Bigger scale-up worlds mean denser racks, denser racks mean 600kW envelopes, and 800VDC is what makes those envelopes possible.

The Four Chapters of the HVDC Transition

Phase 1 (2026/2027): The White Space Retrofit

Power Flow Overview with HVDC Power Rack

In a Nutshell: The Power Rack

ORv3 HPR V3: The Disaggregation Threshold (50V Sidecar, up to 300 kW)

ORv3 HPR V4: HVDC Sidecar at +/-400VDC (up to 800 kW)

The Diablo 400 Specification: Standardizing the HVDC Sidecar

No One-Size-Fits-All

Besides, an alternative sidecar topology uses an LV-input SST in place of the conventional rectifier-plus-PSU stack, like DG Matrix’s Interport Cell Series.

The cost of the power rack

The Sidecar Market Opportunity and TAM sizing

We expect sidecar TAM to peak at ~$11B in 2028 before declining as facility-level 800VDC takes share in Phase 3. We assume a power rack content of $0.5M/MW.

Phase 1 Summary

Phase 2 (2027/2028): The Turning Point Comes with 800VDC-Native Compute

Because most servers and trays still take roughly ~50V input, both architectures retain a high-power 800V-to-~50V DC-DC conversion stage. The difference is where that conversion happens.

What Happens With UPS and Battery Storage

Phase 3 (late 2028/2029): Redesigning the Electrical Architecture With a Centralized Rectifier

What Happens in the Grey Space: Power Distribution Goes DC

Understanding DC Distribution: Switchboards, Busway, and Protection

Alternative Path Using LV Solid State Transformers

What Happens in the White Space: From the Power Rack to the Battery Rack

As you can imagine, in Phase 3 we no longer need the Power Rack doing the 800VDC conversion. Instead, we salute a new friend, the battery rack.

BBU modules scale up

What Happens at the Facility Level

After analyzing the complete transformation of the grey space and the white space, the facility level is the part that changes least.

Medium Voltage Rectifiers: Is There Room for Everyone?

Phase 4 (>2029): SSTs, the End-State

In a Nutshell: Solid-State Transformers

Introduction to SSTs

Pros and Cons of SSTs

Reliability

Current Efficiency State

Vendor Landscape

Datacenter Layout Implications

The SST Market Opportunity and TAM sizing

Datacenter Layout Summary: Total Cost Barely Moves, Content Shifts, Efficiency Climbs

Electrical System Cost

Electrical System Efficiency

The Other Side of the 800VDC Transition: Challenges and Limitations

Challenge 1: Regulation, Safety and Grounding

Regulation

Safety

Flex, a major Nvidia manufacturing partner, has publicly advocated for in-depth hazard identification and safety training at 800VDC facilities.

Grounding

Challenge 2: Cooling and Auxiliary AC Workloads

Challenge 3: Supply Chain Standards

Challenge 4: Grid Interconnection and Regulatory Pressure

Why 800VDC raises the burden of proof

Understanding The No-So-Basics: The Physics Behind 800VDC

Why Going Super Dense Makes Low-Voltage Distribution Break: Heat and Weight

For a fixed rack power P, raising V reduces I linearly. Lower current means smaller conductors, less copper mass, and easier routing.

That drop is the energy dissipated as heat in the conductor. Substituting into the power equation yields the resistive loss equation:

Current appears squared, so the voltage-to-loss relationship is quadratic, not linear. This is the equation that makes 800VDC inevitable.

That’s a 14.8× reduction in current. Now apply the loss equation. For the same conductor resistance R, the I² ratio implies:

Resistive heating at 54 V is roughly 219 times higher than at 800 V for the same conductor. In the more commonly cited comparison using 48 V:

800VDC vs. ±400VDC: The Topology In Question

Single-ended 800V

Bipolar ±400V

Behind paywall we will now discuss the main winners and losers of the 800VDC revolution, and who is better positioned to be benefited from the transition.

Every advanced chip on earth is designed using Electronic Design Automation (EDA) software from three companies. Synopsys, Cadence, and Siemens EDA bridge the gap between what a chip needs to do and what a foundry can manufacture, translating billions of transistors into manufacturable silicon.

The Big-3 hold over 85% combined market share (Ansys now part of Synopsys), and the industry has posted positive revenue growth every single year for over a decade. Synopsys generated $8B in CY2025 (including Ansys), Cadence $5.30B, and Siemens EDA an estimated $2.2-2.5B, putting Big-3 combined revenue at around $16B across EDA tools, semiconductor IP, emulation hardware, and simulation software. The broader EDA+IP industry totals $18B when including smaller vendors and Chinese EDA companies.

EDA grows at 13% CAGR while semiconductor R&D grows at 7%. That 6-point spread widened after 2018 as hyperscaler AI chip programs, emulation hardware economics, and advanced node verification costs created EDA demand that outgrew the traditional R&D base.

EDA software tools represent roughly 9-12% of total semiconductor R&D spending, depending on how both the numerator and denominator are defined. When including semiconductor IP licensing revenue from EDA vendors (Synopsys IP at $1.7B, Cadence IP at $0.7B+), EDA vendor revenue as a share of semiconductor R&D rises to 12-15%.

Synopsys CEO Sassine Ghazi noted in early 2025 that semiconductor R&D intensity is rising from roughly 6% of industry sales toward 9%, driven by AI workload complexity. EDA vendors benefit twice from this shift. The R&D budget they sell into is growing as semiconductor companies spend more on design, and their share of that budget is expanding through verification intensity, AI tool premiums, and node transition pricing.

Part 1 of our EDA Primer explained the journey from RTL to signoff. Part 2 follows the businesses behind the tools that make that journey possible. Part 3 will examine how AI is beginning to reshape the entire chip design stack.

Reduce time to market. A chip designed in 18 months instead of 24 captures 6 months of protected revenue. For a $200M product, that’s $100M+ in value, because EDA automates placement, routing, and verification tasks that would take human engineers 10-100x longer.

Optimize PPA (performance, power, area). Every chip design is a three-way trade-off between how fast it runs, how much power it consumes, and how much silicon area it occupies. EDA tools run thousands of automated iterations to find the optimal balance across these dimensions for a given process node. A 5% improvement in area means 5% more chips per wafer and millions in manufacturing savings at scale. A 10% reduction in power determines whether a mobile SoC fits within its thermal envelope. PPA optimization is the core technical value proposition of EDA.

Manage complexity that exceeds human capacity. A modern flagship chip contains 50 to 200 billion transistors, and more in multi-die packages. At 3nm, foundries impose 25,000+ design rules, each representing a manufacturing constraint that must be satisfied simultaneously. The number of process-voltage-temperature corners requiring signoff has grown from 5-7 at 28nm to 20-30+ at 3nm. Manual design stopped being possible at 65nm, and automated optimization is the only path to functional silicon at leading-edge nodes.

Prevent silicon failure. A single respin at leading-edge nodes costs $50-100M and delays the product 6-12 months. Proving correctness before committing to a $40M mask set is the highest-ROI activity in the design cycle.

Who Buys EDA Tools

Fabless chip designers (NVIDIA, Qualcomm, AMD, Broadcom, MediaTek) are the largest traditional segment, spending $80-150K per engineer annually on tools, IP, and verification. These companies design chips but own no fabs, making EDA their core technical infrastructure.

Systems companies now account for 45% of EDA demand according to Cadence. This is the fastest-growing and most consequential category. Hyperscalers (Google, Amazon, Microsoft, Meta) each run multiple custom silicon programs with full EDA tool stacks at advanced nodes. Apple employs 8,000+ chip designers across the M-series, A-series, and modem programs. Tesla designs its own FSD and Dojo chips. Automotive OEMs and Tier-1s (Continental, Bosch, Denso) are entering chip design for the first time. These companies arrived as EDA customers within the last decade, and their spend is incremental to the traditional semiconductor R&D base.

IDMs (Intel, TI, Analog Devices, Infineon, STMicroelectronics) spend less per engineer ($40-80K) but run larger teams across both design and manufacturing. They negotiate enterprise-wide agreements covering thousands of seats and develop some internal IP, reducing external licensing costs.

Memory companies (Samsung, SK Hynix, Micron, Kioxia) use specialized tools for DRAM, NAND, and HBM design. HBM verification now approaches logic-chip complexity as stacking and interposer routing requirements grow with each generation.

Foundries (TSMC, Samsung Foundry, Intel Foundry, GlobalFoundries, Rapidus) are both customers and partners. They co-develop PDKs with EDA vendors 24 months before production and specify which tools their customers must use for tape-out, effectively mandating specific signoff software for the entire ecosystem.

Turnkey ASIC design houses (Broadcom ASIC Group, Marvell Custom Silicon, Alchip, GUC) are among the largest per-customer EDA spenders. They hold EDA licenses on behalf of hyperscaler clients and run multiple concurrent tape-outs at advanced nodes. Broadcom’s ASIC group alone is estimated to spend $200-500M annually on all-in EDA tool, IP licensing, and emulation hardware.

IP companies (ARM, Rambus, Alphawave) license EDA tools to design IP blocks that ship inside other companies’ chips. Their per-engineer spend is lower because they design once and license repeatedly.

What Drives EDA Revenue Growth

Node transitions. Each new process node adds design rules, verification corners, and tool requirements. 3nm tools cost 3-5x more than 28nm tools, and customers pay because they have no alternative path to leading-edge silicon.

Verification intensity. Proving chips work before manufacturing consumes 60-70% of design time and grows 15%+ annually. Hardware emulation alone is a $1.5B+ market. Every new protocol (PCIe Gen6, HBM4, UCIe) adds verification surface area that compounds on existing workloads.

AI accelerator proliferation. Hyperscaler custom silicon created $15B-$20B in new chip design activity that barely existed five years ago. Google TPU, Amazon Trainium, Microsoft Maia, Meta MTIA, each requires a full EDA tool stack at advanced nodes, incremental to traditional R&D budgets.

Pricing power from lock-in. 95%+ customer retention combined with 3-7% annual contractual escalators means EDA vendors grow revenue from existing customers every year without adding seats. $10M ELAs signed in 2020 renew at $12-14M in 2025 without adding engineers.

The divergence started in 2018. Before that, EDA revenue tracked fab R&D spend 1:1. Hyperscaler AI chip development, emulation hardware economics, and advanced node verification costs all grew faster than design complexity, pulling EDA revenue above R&D trendlines. With Synopsys’s $35B Ansys acquisition, the addressable market expands to $31 billion ($18B EDA+IP + $10B simulation + $3B systems software), meaning the oligopoly just absorbed its only adjacent market.

Synopsys and Cadence revenue (2012-2025). Synopsys: $1.76B to $7.05B (~11% CAGR). Cadence: $1.15B to $5.30B (~12% CAGR). Thirteen years of unbroken growth through every cycle.

What EDA Tools Actually Do: RTL to Silicon in 12-24 Months

EDA tools transform abstract hardware descriptions into manufacturable silicon through a sequential pipeline. Engineers write RTL code (Verilog or VHDL), which synthesis tools (Synopsys Design Compiler, 84-85% share) map onto foundry-optimized standard cells. Place and Route (Synopsys Fusion Compiler or Cadence Innovus) positions gates and routes billions of wires through dozens of iterations over 2-3 months.

Signoff analysis (Synopsys PrimeTime 90%+ share, StarRC, Redhawk) validates timing, parasitics, and power integrity across all PVT corners. Physical verification (Siemens Calibre, 85%+ share) checks DRC against foundry rules and LVS to confirm layout matches the circuit. Foundries mandate these signoff and verification tools for tape-out, as detailed in the competitive moat section. Tape-out delivers GDSII files to the foundry.

Chip design pipeline from RTL to tape-out. Each stage feeds the next; changing one tool re-runs all downstream steps. 12-24 months for 7nm/5nm/3nm.

Verification is where the majority of design time and budget goes, as described in the growth drivers section above. Functional simulation (Synopsys VCS 45-50% share, Cadence Xcelium 40-45%) runs billions of test vectors. Hardware emulation (Cadence Palladium 55-60% share, Synopsys ZeBu 35-40%) maps designs onto physical hardware for full-SoC validation, and a flagship AI chip requires 6-12 months of continuous emulation. The sequential dependency matters more than any individual tool’s merits. Change your synthesis tool and you must re-run place-and-route, signoff, and physical verification. The flow itself is the lock-in.

Design time breakdown. Verification: 65% (8-15 months). Implementation: 30% (4-7 months). Physical verification: 5%. A 7nm chip requires 10-50X more verification compute than a 28nm chip of equivalent gate count.

The EDA Market: Sizing, Share, and Structure

The remaining 10-15% is fragmented across dozens of vendors, with Ansys pre-Synopsys), Keysight ($1.5B, partially overlapping), and Zuken ($500M, PCB/IC packaging) as the largest independents. No vendor outside the Big-3 holds more than 5% in any core EDA category.

Renesas acquired Altium ($5.9B, 2024) to use Altium’s PCB design software for promoting its component portfolio and BoM optimization. Altium generates $280M in annual revenue from PCB design, placing it among the larger independent EDA players in that specific category.

These shares have been roughly stable for a decade. The only category with meaningful movement is Place & Route, where Cadence Innovus gained 10-15pp against Synopsys ICC2 (IC Compiler II, Synopsys’s flagship place-and-route tool) between 2015-2020, then stabilized as Synopsys launched Fusion Compiler. Everything else is locked.

SNPS+CDNS combined market share trending upward as complexity drives consolidation toward the two largest vendors.

How EDA Licensing Actually Works: Seats, Tokens, Hardware, and the Renewal Machine

EDA pricing is opaque by design. Vendors don’t publish price lists, and every deal is negotiated individually.

Model 1: Seat-Based Licenses (Traditional)

One license equals one engineer running one tool at a time, and seat-based pricing is still used for small customers and specific tools.

Seat-based pricing scales linearly with headcount, which is simple but limits vendor upside to headcount growth alone.

Model 2: Token/Capacity-Based Licenses (Modern)

Tokens decouple licensing from individual seats. A customer buys a pool of compute capacity, any engineer can use any tool drawing from the shared pool, and peak usage gets throttled or billed at overage rates.

Token licensing is the growth model for EDA vendors, and four dynamics explain why.

The shift from seats to tokens is the most important pricing dynamic in EDA. Synopsys stated at its 2024 Investor Day that AI-enhanced tool renewals generate ~20% revenue uplift over baseline contract values. That uplift comes from token consumption growth while headcount stayed flat.

Model 3: Enterprise License Agreements (ELAs)

For the top 50-100 customers, the actual unit of sale is the ELA, a multi-year contract bundling broad portfolio access.

These licensing structures are reconstructed from vendor disclosures, customer interviews, and quarterly earnings call commentary, as neither Synopsys nor Cadence publishes pricing details.

ARM uses a similar model with its Flexible Access program, offering customers all-you-can-evaluate access to the full ARM IP portfolio for an annual fee, with per-chip royalties only triggered at production. This model has been adopted by 70%+ of ARM’s new license agreements since 2019.

Hardware Licensing: Emulation Is a Different Business

Emulation hardware (Cadence Palladium, Synopsys ZeBu) follows capital equipment economics, with physical systems that have depreciation schedules, installation teams, and cooling requirements.

Once a customer installs $50M of Palladium systems, four forces lock them in for the life of the hardware. Testbenches written to Palladium APIs run millions of lines. Engineers specialize in Palladium-specific debug workflows. The 5-7 year depreciation schedule creates a financial commitment. And $3-5M annual software/maintenance fees per system reinforce the vendor relationship. Every Palladium system pulls $2-3M in annual software licensing on top of the hardware investment.

Geographic Pricing Differences

What Happens When Customers Merge: The EDA Licensing Windfall (and Risk)

Scenario 1: Same primary vendor (e.g., both use Synopsys)

The combined entity has two ELAs that get consolidated at renewal. The larger company negotiates better per-seat pricing through volume discounts, and total spend usually declines 10-20% from the sum of the two standalone agreements. This outcome is bad for the EDA vendor in the short term.

Scenario 2: Different primary vendors (e.g., acquirer uses Synopsys, target uses Cadence)

The acquirer standardizes on its preferred platform, the target’s engineers get retrained, and the losing vendor’s contract gets run off over 2-3 years because teams can’t switch mid-project. The winning vendor gains seats, the losing vendor loses them, and total spend stays roughly flat.

Scenario 3: The transition creates evaluation opportunity

When AMD acquired Xilinx ($49B, 2022), the combined entity had overlapping EDA agreements and the merger forced rationalization. Both Synopsys and Cadence competed aggressively for the combined contract, and the result was that the winning vendor got a larger deal at compressed margins from competitive pricing to win the consolidation.

Recent examples:

The net effect of semiconductor consolidation on EDA revenue is slightly negative, since fewer independent customers means fewer separate ELAs. But the surviving entities are larger, design more complex chips, and spend more per engineer. Historically, the complexity growth has more than offset the consolidation discount.

What Drives Revenue Growth Beyond Adding Seats

EDA revenue grows at 12-15% CAGR while global semiconductor design headcount grows at 3-5%. The delta comes from six sources.

This breakdown explains a critical point. EDA vendors are selling genuinely new capabilities at each node transition - multi-patterning aware routing at 7nm, backside power delivery at 2nm, 3D-IC integration at advanced packaging nodes. Customers get new functionality and also pay more for it. The pricing is justified at the tool level, but the monopoly dynamics determine how much of the value the vendor captures versus the customer.

Do Customers Pay for Updates?

Under the old perpetual model, customers paid 15-20% annual maintenance for updates, and they could skip updates and coast on old versions (many did during downturns). Under the current time-based model, updates are included in the annual fee with no separate charge. Customers always run the latest version, and stopping payment means losing access entirely. This is why the perpetual-to-TBL transition was so important for vendors, because it eliminated the “maintenance holiday” that customers used during downturns.

Both Synopsys and Cadence now generate 70-83% of revenue from time-based/subscription arrangements, with the remainder from upfront hardware deliveries, IP milestones, and perpetual licenses. The upfront share has actually grown in recent years as emulation hardware sales expanded. The transition from perpetual to time-based took a decade (roughly 2005-2015) and permanently improved business quality.

The Renewal Machine

EDA revenue is a self-reinforcing renewal engine, and the renewal math is straightforward.

A customer who signed a $10M/year ELA (Enterprise License Agreement) in 2020 renews at $12-14M in 2025, driven by contractual escalators, AI premiums, and verification expansion. A customer renewing a $10M ELA in 2025 pays $12-14M for the same headcount but upgraded tools, AI features, and expanded verification capacity. Management frames it as value creation while procurement teams see annual inflation, and both are correct.

EDA pricing power index. Contractual escalators, AI premiums, and verification expansion compound on a captive base.

Competitive Pricing Dynamics

Most EDA competitive evaluations serve as negotiating leverage rather than genuine switching attempts. The typical pattern is that a customer announces an evaluation, the incumbent responds with a 15-25% discount offer, and the customer accepts without completing the evaluation. Sales teams at both Synopsys and Cadence have learned to distinguish real evaluations, where the customer allocates a dedicated engineering team and provides actual design data, from pricing negotiations disguised as technical assessments.

Retention Rates by Tool Category

Synopsys: The $35B Platform Bet

“Fusion Compiler is the embodiment of what happens when you break down the walls between synthesis, place and route, and signoff. That unified data model is what gives us structural advantage. Competitors can bolt tools together, but they can’t replicate a unified architecture.” - Sassine Ghazi, President & CEO, Synopsys Investor Day 2024

Synopsys’s strategy is platform maximalism, owning every tool in the design flow, cross-selling IP, and expanding into adjacent simulation domains. The $35B Ansys acquisition (completed July 2025) extends this logic from chip design into system-level simulation covering thermal, structural, electromagnetic, and CFD (computational fluid dynamics, for cooling analysis) analysis.

Modern chips don’t exist in isolation. A 700W datacenter GPU must dissipate heat through complex cooling, and an automotive SoC must meet EMC (electromagnetic compatibility) requirements on a vibrating engine block. Traditional EDA stops at the package boundary. The Synopsys-Ansys combination creates a device-to-system simulation stack covering TCAD (Technology Computer-Aided Design, for device physics simulation) for device physics, EDA for chip design, Ansys for package thermal, system EMC, CFD, and structural stress. No competitor offers this breadth.

The synergy math (from the 2024 Investor Day) projects $400M run-rate cost synergies by year 3 and $400M run-rate revenue synergies by year 4, with long-term revenue synergies of $1B+ annually. At the Morgan Stanley TMT Conference in March 2026, Ghazi indicated synergy realization is tracking ahead of the original plan. Combined company targets include non-GAAP operating margins in the mid-40s%, unlevered free cash flow margins in the mid-30s%, and high-teens EPS growth. Ansys adds end-market diversification at 31% semiconductor/high-tech, 22% aerospace, and 18% automotive.

Risks include integration complexity (different customers, sales motions, cultures), leverage (~3.9x at close, targeting <2x within two years), valuation ($35B is 12x revenue), and management distraction from core EDA competition.

The Margin Staircase: From 14% to 37.3% (FY2006-FY2024)

The pattern is consistent across two decades. Major acquisitions (Magma, Coverity, Black Duck) temporarily compress margins by 100-200bp, followed by systematic recovery. Management told analysts in 2011 that “If we see the opportunity to grow the top line a little bit more, we want to do that. If we see that, for whatever reason, the top-line growth is more difficult, we will immediately revert to a higher pressure on the operating margin.”

23pp of margin expansion came from four structural factors: (1) shift from perpetual to time-based licenses, (2) verification/IP mix shift toward higher-margin products, (3) AI tools commanding 15-25% premiums with minimal incremental cost, (4) platform cross-selling reducing customer acquisition costs.

The CEO Transition: Founder to Operator

Aart de Geus (CEO 1986-2023, now Executive Chair) handed the company to Sassine Ghazi (CEO from January 2024), and the tone shift is subtle but material.

Two major moves in Ghazi’s first year demonstrate the pivot. The Software Integrity Group was divested for $2.1B (“compelling investment opportunities in design automation and Design IP with much higher expected growth and return profiles”), and Ansys was acquired for $35B. The “tale of two markets” framework, distinguishing AI infrastructure customers from traditional semis, is distinctly Ghazi’s analytical approach, signaling operational rigor over visionary expansionism.

Synopsys acquisition timeline. The buy-over-build strategy accelerated with Ansys ($35B) in 2024.

The Backlog Fortress

$11.3B in backlog provides extraordinary forward revenue visibility for a software company.

Synopsys vs Cadence backlog. $11.4B and $7.8B respectively (FY2025 year-end), providing 1.5-1.6 years of forward revenue visibility.

100% Advanced Node Market Share, Verified

The historical arc from quarterly earnings transcripts shows steady accumulation over more than a decade.

The full design start data supporting these claims is presented in the competitive moat section below. The absence of post-2019 design start disclosures doesn’t signal share loss. The data became an antitrust liability, and the consistent revenue growth since then supports the same conclusion by different means.

Synopsys Revenue by Segment. IP revenue grew from $200M (2011) to $1.91B (2024), a 9.5x increase in 13 years. The IP business now represents 31% of FY2024 revenue pre-Ansys, dropping to ~25% of the larger FY2025 base.

Synopsys revenue by segment. IP revenue grew from $200M (2011) to $1.91B (2024), IP share fell from 31% (FY2024) to ~25% (FY2025) as Ansys revenue diluted the denominator.

Near-Term Headwinds: The FY2026 Transition Year

Underneath the Ansys-inflated headline numbers, Synopsys’s organic business is decelerating. The Ansys merger masked a slowdown in the standalone EDA+IP business in FY25, with organic revenue growing only ~3% YoY ex-Ansys compared to 15% reported. In FY26, the same dynamic persists, with organic growth of roughly 7-8% ex-Ansys versus 36% reported. The gap between reported and organic growth rates is the widest it has ever been.

The primary drag is IP. Design IP revenue declined quarter-over-quarter for 3 of the 4 quarters in FY25, breaking from a historic 13% CAGR (FY20-FY24). Two specific gaps drove the decline. First, Synopsys lacked foundry-ready IP for Intel 18A during the critical ramp window, missing early design wins that went to Cadence. Second, the company had a coverage gap in HPC IP titles that it expects to fill in 2HFY26.

Management guided IP growth as “muted” (low-single-digit %) in FY26 with sequential improvement, far below the company’s mid-teens long-term IP target. The processor IP solutions business is being divested to GlobalFoundries, sharpening focus on interconnect and foundation IP but creating a near-term revenue air pocket. Design IP adjusted operating margin dropped to 16.2% in 4QCY25, well below the 30%+ margins IP generated at scale.

China compounded the organic weakness. Excluding Ansys, China revenue declined 22% in FY25 as export restrictions tightened and local EDA companies captured share at mature nodes. Management acknowledged directly that “the companies we cannot sell to are looking for alternatives, and these alternatives are typically local EDA or IP companies.” China exposure dropped from 16% of revenue in FY24 to 12% in FY25, and management expects further deceleration below corporate average growth in FY26.

Core EDA (ex-IP, ex-Ansys) grew 8% in FY25, below the company’s long-term double-digit targets. Management guided core EDA for only 9% growth in FY26. Hardware (ZeBu/HAPS) posted a record year driven by AI silicon demand, but this business remains well behind Cadence’s Palladium in both market share and revenue scale. Cadence also continues gaining traction at Intel, historically a Synopsys stronghold.

Synopsys is attempting to restructure its IP business model from flat NRE-plus-usage charges to a model that includes royalties, responding to hyperscaler customers who increasingly demand custom IP. This model transition creates near-term revenue headwinds as legacy contracts roll off and royalty streams take time to build. The first joint Synopsys-Ansys physics solutions are expected in 1H26, which could drive pricing upside, but the integration risk remains real. The key catalysts to watch through FY26 are IP revenue stabilization by mid-year, evidence of the royalty-based IP model gaining traction with hyperscaler customers, and organic EDA growth re-accelerating toward the double-digit target.

Intel Customer Concentration: A Two-Decade Dependency

Intel has been Synopsys’s largest customer for over two decades. At peak concentration in FY2017, Intel represented 17.9% of total revenue ($363M on a much smaller revenue base). By FY2024, Intel’s share had moderated to 12.6% ($772M), reflecting Synopsys’s revenue diversification more than any reduction in Intel spending. In FY2025, for the first time in Synopsys’s history, no single customer exceeded 10% of revenue, though this milestone owes more to Ansys dilution expanding the denominator than to Intel spending declining.

Intel remains the single largest account. Cadence is gaining traction at Intel, a historically weak position for Cadence relative to its strength at TSMC. Intel Foundry’s restructuring and leadership turnover have created evaluation opportunities across the EDA stack that did not previously exist. Every foundry transition opens a window for competitive re-evaluation, and Intel’s current transformation is the largest such window in a decade.

Synopsys Intel customer concentration over 20 years. Intel peaked at 17.9% of revenue in FY2017 and declined to below 10% in FY2025 as Ansys expanded the revenue base.

Cadence: From Near-Death to the Highest Margins

The IP segment is the growth engine. SDA growth includes the BETA CAE acquisition ($1.24B, Q2 2024), which brought structural analysis to top 10 global automakers and F1 teams. Hardware backlog entering 2026 is at record levels.

2026 Outlook: Implied Upside Across the Board

Record backlog of $7.8B (+11% q/q) covers 67% of FY26 revenue before a single new booking. Hardware is derisked with H2-weighted delivery schedules locked in. China is guided at 12-13% of revenue with H2 prudence baked in, the same conservative framing that proved overly cautious in FY25 when China grew above guidance. The Hexagon Design & Engineering acquisition (~$150M annualized revenue) is excluded from the $5.9-6.0B guide entirely. Incremental margins are guided at 51%, well below the 59% Cadence actually delivered in FY25. Every assumption embedded in guidance tilts conservative, creating multiple paths to upside.

System companies now account for 45% of EDA demand at Cadence, up from 40% two years ago. A marquee hyperscaler adopted the Cadence digital full flow for its first full COT (Customer-Owned Tooling) AI chip tape-out, a milestone that validates Cadence’s digital competitiveness at the most demanding customers. Cadence added 25 new digital full flow logos in 2025, extending the trajectory from 10 wins per year in 2014 to consistent double-digit annual additions. The IP portfolio hit critical mass with HBM4, 224G SerDes, and LPDDR6 as key titles. IP revenue grew nearly 25% in 2025, now in its third consecutive year of strong growth. The multi-foundry dynamic across TSMC, Samsung, Intel, and Rapidus is a structural tailwind that Cadence is better positioned to capture than Synopsys, which has acknowledged FY26 IP will be a muted year.

Hardware posted another record year with 30+ new customers and substantially higher repeat demand from AI and hyperscaler programs. Seven of the top 10 hardware customers were Dynamic Duo (emulation + prototyping) customers, embedding Cadence across the verification workflow. Management stated they are taking share in all major product segments. CFO John Wall laid out the agentic AI monetization framework in three tiers. Subscriptions serve as the anchor revenue base, usage-based pricing captures AI-driven compute intensity, and a virtual engineer tier prices agents as additive headcount equivalent. Full monetization takes two contract renewal cycles, making this an FY27-28 revenue story, but the architecture is already in place.

The Hexagon Design & Engineering acquisition (~$200M annualized revenue) closed in February 2026, adding physical AI and automotive simulation capabilities that compete directly with Synopsys-Ansys in the systems simulation space. Cadence also expanded its TSMC collaboration for N2 and A16 process flows, deepened its partnership with Broadcom on agentic AI workflows, and formalized a new partnership with Rapidus, the Japanese government-backed foundry. Each foundry relationship adds IP porting revenue, tool certification fees, and long-term design ecosystem stickiness.

Cadence revenue by segment. SDA grew from $500M (2019) to $1.5B+ (2024), driven by Palladium hardware and BETA CAE.

The Near-Death Experience That Defines Everything

Under CEO Mike Fister (2004-2008), Cadence pursued aggressive adjacencies and attempted a hostile bid for Mentor Graphics, and the result was catastrophic.

Revenue fell 36% in a single year, producing a GAAP loss of $6.57 per share and a $200M goodwill writedown. Earnings transcripts from 2008Q3 through 2011Q1 are missing from public records, a gap that speaks volumes about the severity of the crisis.

The Lip-Bu Tan Turnaround (2009-2024)

Lip-Bu Tan became CEO in January 2009 at the absolute trough. In 2014 he summed it up on the earnings call. “From 2009 to 2013, revenue grew 71%. Non-GAAP operating margin expanded from near zero to 24%, and operating cash flow grew from just $26 million to $368 million.”

53pp of margin expansion in 15 years, powered by Tan’s operating rule that “50% of incremental revenue drops through to operating income.” Cadence achieved that target for 7+ consecutive years.

Virtuoso: The Tool No One Can Kill

Cadence owns analog design through Virtuoso, a tool with no credible competitor because analog methodology evolved inside Virtuoso over four decades. The tool embeds tribal knowledge about matching, noise, and linearity that can’t be replicated by writing better algorithms, because it requires decades of customer feedback layered into the product.

450+ customers, and no major customer defection on record. The absence of competitive losses over decades of public earnings disclosures confirms what the market share numbers suggest.

Palladium: The 10-Year Hardware Lead

200 repeat customers annually, with a custom ASIC architecture that creates a development moat requiring a competitor roughly a decade to cross.

The Digital Gap That Keeps Closing

From 10 wins in 2014 to 36 wins in a single year by 2024. Cadence never directly counters Synopsys’s “95% advanced node” claim, and instead cites TSMC Partner of the Year awards while letting the revenue numbers carry the argument.

Cerebrus AI: 1,000+ Tapeouts in 8 Quarters

5.6x increase in under 2 years with 100% penetration of top 10 digital customers. Cadence focuses on proliferation first and pricing capture later through ACV growth.

Named customer results tell the technical story. MediaTek achieved 5% die area reduction and 6%+ power reduction. Renesas saw 75% improvement in total negative slack on advanced-node CPU. Samsung SARC got 4x productivity boost, and Samsung India (SSIR) achieved 8-11% PPA improvement. IBM is deploying Cadence AI-enabled digital implementation. The JedAI (Joint Enterprise Data and AI) Platform underneath Cerebrus aggregates waveforms, coverage reports, timing analyses, and physical layouts into a unified training data repository, creating a compounding data moat that makes Cadence’s AI tools improve with each deployment.

Cerebrus AI tapeout trajectory. 180 to 1,000+ in 8 quarters, with 100% penetration of top 10 digital customers.

CEO Transition and the Three-Horizon Strategy

Anirudh Devgan became CEO in December 2021 and reframed Cadence as a “computational software company” with three expansion horizons.

Horizon 1 (present - 3 years): Datacenter AI. Core EDA, IP, and emulation for AI accelerator designs, already the largest revenue driver.

Horizon 2 (3 - 7 years): Automotive and “Physical AI.” BETA CAE ($1.24B, Q2 2024) brought structural analysis serving top 10 global automakers and F1 teams. MSC Software ($3.25B, September 2025) added mechanical simulation. Combined with Cadence’s existing CFD capability (NUMECA, acquired 2021) and Pointwise (mesh generation, 2021), Cadence now has a full multi-physics stack for automotive. The acquisition sequence was deliberate, with small bets first (NUMECA $189M, Pointwise $31M) followed by scale when proven (BETA CAE $1.24B, MSC $3.25B).

Horizon 3 (5 - 10+ years): Life Sciences. OpenEye Scientific ($500M, September 2022) provides computational molecular modeling used by 19 of the top 20 global pharmaceutical companies. The thesis is that the same algorithms that optimize transistor placement can optimize molecular docking, with TAM estimated at $2B growing at ~15% CAGR. This is Devgan’s most non-consensus bet, and if it works, Cadence transcends the EDA category entirely.

When Lip-Bu Tan explored returning to the company in 2024, the board confirmed Devgan, and Lip-Bu departed to become CEO at Intel. The transition was unusually public, signaling governance strength and conviction in Devgan’s strategy.

Cadence acquisition timeline. NUMECA, Pointwise, OpenEye, BETA CAE, MSC Software - systematic expansion beyond core EDA.

The Margin Reversal. Cadence: from -11% (2009) to 42.5% (2024). Synopsys: from 14% (2006) to 37% (2024). Cadence is now the more profitable company despite being smaller. The turnaround story is one of the great comebacks in enterprise software.

The Blocking Position: Siemens EDA

Calibre physical verification is the industry standard. TSMC specifies “Calibre-clean” DRC/LVS for tape-out, Samsung and Intel do the same, and this single tool ensures Siemens EDA’s permanent relevance in the market regardless of what happens in other tool categories.

How Mentor Became #3: The Release 8.0 Catastrophe

Mentor Graphics was the largest EDA vendor in the late 1980s with $380M revenue in 1989 and $44.8M net income. Then management attempted a complete ground-up rewrite of the entire software suite (”Release 8.0”), and it spiraled out of control. The project missed deadlines by years, and Cadence surged past Mentor in software revenue during the chaos. Mentor reported its first quarterly loss in April 1991, followed by a $61.6M annual loss and 15% workforce cuts. When Release 8.0 finally shipped in 1992, it was slow and riddled with bugs.

This is the canonical cautionary tale in EDA, and it explains three enduring realities. First, it explains why Mentor fell from #1 to #3 and never recovered. Second, it explains why all three Big-3 vendors acquire rather than build from scratch (Synopsys-Ansys, Cadence-BETA CAE, Siemens-Altair). Third, it explains why no startup can replicate an EDA platform by rewriting from zero. The codebase complexity defeats clean-sheet approaches every time.

CEO Wally Rhines (1993-2017) rebuilt Mentor through M&A instead, assembling Calibre, PCB tools, embedded software, and automotive electronics into a coherent portfolio. Activist pressure from Carl Icahn (2011) and Elliott Management (2016) eventually pushed Mentor toward a sale, and Siemens acquired it for $4.5B in 2017 before rebranding it as Siemens EDA in 2021.

The Siemens ownership is double-edged.

Advantages include cross-subsidization from an industrial conglomerate, bundling EDA with Teamcenter PLM and Opcenter MES, and the fact that Siemens supplies automotive OEMs while Siemens EDA serves the chip designers who supply those OEMs.

Disadvantages are real. EDA is <5% of Siemens revenue, there is no independent stock for acquisitions, reporting is opaque because it’s buried in Digital Industries Software, and investment competes against automation, healthcare, and energy priorities for parent company capital.

The Simulation Arms Race: Siemens Acquires Altair

All three Big-3 vendors simultaneously acquired simulation/CAE companies in 2024-2025 in a move-for-move escalation.

The EDA-CAE boundary is permanently dissolving, and Siemens’ Altair deal is the third leg of a three-way race to own the “silicon to systems” simulation stack.

Siemens AI Products: Catching Up

At DAC 2025, Siemens launched three AI product families: Aprisa AI for digital implementation, Calibre Vision AI for DRC violation clustering (cutting debug time in half), and Solido AI for custom/analog design. These target Siemens’ 85%+ Calibre installed base and represent the company’s first serious AI-driven push into domains that Synopsys and Cadence have led. A partnership with NVIDIA uses NIM microservices for EDA-specific AI inference.

PAVE360: The System-Level Digital Twin Play

Siemens’ differentiated response to chip-level competition is PAVE360, which provides full vehicle simulation, software-hardware co-verification, and production vehicle integration testing. Partnerships with AMD, Elektrobit, KPIT, TIER IV, and Qt create an end-to-end workflow from system requirements to production validation.

The TAM for system-level automotive verification sits adjacent to EDA ($800M-1.2B opportunity by 2030) but targets different buyers, specifically vehicle integration teams and Tier-1 suppliers. Siemens can capture this adjacent opportunity without directly competing for chip-level sockets against Synopsys and Cadence.

Q1 FY2026 Update (February 2026): EDA Outgrowing the Portfolio

Siemens’s Digital Industries software business grew 11% in Q1 FY2026, with EDA and simulation specifically driving healthy double-digit growth within that segment. The Altair integration is progressing well, and the NVIDIA partnership for chip design software continues to expand. PLM (excluding simulation) grew 7%, meaning EDA and simulation are outgrowing the broader Siemens software portfolio by a meaningful margin. This bifurcation matters because it signals that semiconductor design complexity and automotive simulation demand are pulling Siemens EDA faster than the industrial PLM base, reinforcing the strategic rationale behind both the Altair and Mentor acquisitions.

R&D Intensity: Synopsys at 34%, Cadence at 30%, Siemens EDA estimated at 25-28%. The spending gap explains the technology gap, but Siemens’s lower intensity reflects Calibre’s entrenched position requiring less R&D to defend.

In the full report, subscribers get access to competitive dynamics analysis (Cadence pulling ahead of Synopsys organically in 2026), the six-layer lock-in architecture with franchise tool and PDK moat evidence, design cost breakdowns for NVIDIA Blackwell and Apple Silicon, China vendor financials and export control forensics from earnings transcripts, our proprietary R-squared lock-in intensity matrix ranking 20+ fabless companies by EDA dependency, AI disruption risk assessment with CEO quotes from February 2026 earnings calls, and catalysts. All data is available in our interactive EDA Dashboard, updated quarterly.

Who Buys EDA Tools

What Drives EDA Revenue Growth

Synopsys and Cadence revenue (2012-2025). Synopsys: $1.76B to $7.05B (~11% CAGR). Cadence: $1.15B to $5.30B (~12% CAGR). Thirteen years of unbroken growth through every cycle.

What EDA Tools Actually Do: RTL to Silicon in 12-24 Months

Chip design pipeline from RTL to tape-out. Each stage feeds the next; changing one tool re-runs all downstream steps. 12-24 months for 7nm/5nm/3nm.

The EDA Market: Sizing, Share, and Structure

SNPS+CDNS combined market share trending upward as complexity drives consolidation toward the two largest vendors.

How EDA Licensing Actually Works: Seats, Tokens, Hardware, and the Renewal Machine

EDA pricing is opaque by design. Vendors don’t publish price lists, and every deal is negotiated individually.

Model 1: Seat-Based Licenses (Traditional)

One license equals one engineer running one tool at a time, and seat-based pricing is still used for small customers and specific tools.

Seat-based pricing scales linearly with headcount, which is simple but limits vendor upside to headcount growth alone.

Model 2: Token/Capacity-Based Licenses (Modern)

Token licensing is the growth model for EDA vendors, and four dynamics explain why.

Model 3: Enterprise License Agreements (ELAs)

For the top 50-100 customers, the actual unit of sale is the ELA, a multi-year contract bundling broad portfolio access.

These licensing structures are reconstructed from vendor disclosures, customer interviews, and quarterly earnings call commentary, as neither Synopsys nor Cadence publishes pricing details.

Hardware Licensing: Emulation Is a Different Business

Emulation hardware (Cadence Palladium, Synopsys ZeBu) follows capital equipment economics, with physical systems that have depreciation schedules, installation teams, and cooling requirements.

Geographic Pricing Differences

What Happens When Customers Merge: The EDA Licensing Windfall (and Risk)

Scenario 1: Same primary vendor (e.g., both use Synopsys)

Scenario 2: Different primary vendors (e.g., acquirer uses Synopsys, target uses Cadence)

Scenario 3: The transition creates evaluation opportunity

Recent examples:

What Drives Revenue Growth Beyond Adding Seats

EDA revenue grows at 12-15% CAGR while global semiconductor design headcount grows at 3-5%. The delta comes from six sources.

Do Customers Pay for Updates?

The Renewal Machine

EDA revenue is a self-reinforcing renewal engine, and the renewal math is straightforward.

EDA pricing power index. Contractual escalators, AI premiums, and verification expansion compound on a captive base.

Competitive Pricing Dynamics

Retention Rates by Tool Category

Synopsys: The $35B Platform Bet

The Margin Staircase: From 14% to 37.3% (FY2006-FY2024)

The CEO Transition: Founder to Operator

Aart de Geus (CEO 1986-2023, now Executive Chair) handed the company to Sassine Ghazi (CEO from January 2024), and the tone shift is subtle but material.

Synopsys acquisition timeline. The buy-over-build strategy accelerated with Ansys ($35B) in 2024.

The Backlog Fortress

$11.3B in backlog provides extraordinary forward revenue visibility for a software company.

Synopsys vs Cadence backlog. $11.4B and $7.8B respectively (FY2025 year-end), providing 1.5-1.6 years of forward revenue visibility.

100% Advanced Node Market Share, Verified

The historical arc from quarterly earnings transcripts shows steady accumulation over more than a decade.

Synopsys revenue by segment. IP revenue grew from $200M (2011) to $1.91B (2024), IP share fell from 31% (FY2024) to ~25% (FY2025) as Ansys revenue diluted the denominator.

Near-Term Headwinds: The FY2026 Transition Year

Intel Customer Concentration: A Two-Decade Dependency

Synopsys Intel customer concentration over 20 years. Intel peaked at 17.9% of revenue in FY2017 and declined to below 10% in FY2025 as Ansys expanded the revenue base.

Cadence: From Near-Death to the Highest Margins

2026 Outlook: Implied Upside Across the Board

Cadence revenue by segment. SDA grew from $500M (2019) to $1.5B+ (2024), driven by Palladium hardware and BETA CAE.

The Near-Death Experience That Defines Everything

Under CEO Mike Fister (2004-2008), Cadence pursued aggressive adjacencies and attempted a hostile bid for Mentor Graphics, and the result was catastrophic.

The Lip-Bu Tan Turnaround (2009-2024)

Virtuoso: The Tool No One Can Kill

450+ customers, and no major customer defection on record. The absence of competitive losses over decades of public earnings disclosures confirms what the market share numbers suggest.

Palladium: The 10-Year Hardware Lead

200 repeat customers annually, with a custom ASIC architecture that creates a development moat requiring a competitor roughly a decade to cross.

The Digital Gap That Keeps Closing

Cerebrus AI: 1,000+ Tapeouts in 8 Quarters

5.6x increase in under 2 years with 100% penetration of top 10 digital customers. Cadence focuses on proliferation first and pricing capture later through ACV growth.

Cerebrus AI tapeout trajectory. 180 to 1,000+ in 8 quarters, with 100% penetration of top 10 digital customers.

CEO Transition and the Three-Horizon Strategy

Anirudh Devgan became CEO in December 2021 and reframed Cadence as a “computational software company” with three expansion horizons.

Horizon 1 (present - 3 years): Datacenter AI. Core EDA, IP, and emulation for AI accelerator designs, already the largest revenue driver.

Cadence acquisition timeline. NUMECA, Pointwise, OpenEye, BETA CAE, MSC Software - systematic expansion beyond core EDA.

The Blocking Position: Siemens EDA

How Mentor Became #3: The Release 8.0 Catastrophe

The Siemens ownership is double-edged.

The Simulation Arms Race: Siemens Acquires Altair

All three Big-3 vendors simultaneously acquired simulation/CAE companies in 2024-2025 in a move-for-move escalation.

The EDA-CAE boundary is permanently dissolving, and Siemens’ Altair deal is the third leg of a three-way race to own the “silicon to systems” simulation stack.

Siemens AI Products: Catching Up

PAVE360: The System-Level Digital Twin Play

Q1 FY2026 Update (February 2026): EDA Outgrowing the Portfolio

One of the other things that has changed is Cerebras’s fortunes. With the arrival of fast tokens on the mainstage and a 750MW compute deal with OpenAI notched, Cerebras is feeling ready for the scrutiny of public markets. Up until just 6 months ago, we felt that the Wafer Scale Engine, despite its bold innovations, had some technical weaknesses that were too hard to cover up. Thus, the continued popularity of HBM-based accelerators such as GPU and TPU. The strengths of Cerebras (namely: speed), have been overlooked for years in favor of total throughput. But now, with frontier labs releasing fast, priority, standard and batch tiers of the same model weights, the world has revealed their preference for fast tokens with their wallets. This brings Cerebras’s strengths to the fore and is the key reason why OpenAI is willing to fork over tens of billions of dollars for Cerebras compute.

Today, on the verge of Cerebras’s IPO, and because we love the wafer, we are shipping an article that is as long as 4 normal articles. Inside, we will dive deep on:

For paid subscribers we also show the economics of the huge OAI Inference deal that has changed the company’s fortunes and share our insights on how far along Cerebras is in becoming a neocloud (i.e. securing the 750MW they need by 2028 for OpenAI). Furthermore, we will talk about Cerebras’ future plans of hybrid bonding an wafer scale optical transceiver onto their WSE compute engine, which they claim they are pursuing strictly for the love the game as it is not needed for LLM inference, but is needed for HPC boomer workloads. The HPC customers whom NVIDIA has effectively abandoned after reducing FP64 native hardware on their GPUs to basically nothing.

The Need for Speed

While SemiAnalysis has historically been an SRAM machine hater, all this changed when Nvidia licensiquihired Groq in December 2025. Clearly Jensen saw at least $20B of value, and he was proven right just a couple months later when we hit the Claude Code Inflection Point. Now, the wafer is here to stay.

Many (including Andrej Karpathy) previously believed that raw intelligence/capabilities mattered far more than speed, but our revealed preferences ended up proving that there are times when the opposite is true. Past a certain threshold of intelligence, developers prefer faster tokens to smarter tokens. And in a world where AI is involved in almost every aspect of your workflow, the speed at which tokens are generated can be the bottleneck to “flow state”, i.e. how much productive work is completed.

Opus 4.6 fast mode famously charges 6x the price for 2.5x the interactivity (though its now under 2x faster, see chart below). In April, 80% of our AI spend (which peaked at $10M annualized) was on Opus 4.6 fast. When Opus 4.7 came out, many of our engineers refused to switch over because it didn’t include fast mode. Notably, this is the first time we’ve ever decided to forgo frontier intelligence in exchange for faster tokens (and at a significant price premium too!).

As an aside, Opus 4.6 fast has become an increasingly worse deal as of late. Standard Opus 4.6 interactivity in Claude Code is consistently around 40 tps (tokens per second). Opus 4.6 fast used to deliver > 100 tps, fulfilling the 2.5 faster guarantee. But it recently degraded to ~70 tps (only 1.75x faster). We recently worked with our friends at OpenRouter to gather this data on the two operating modes of Claude Opus.

We believe Opus 4.6 Fast is Anthropic’s highest margin SKU and a big reason for their explosion in ARR this year. However, we’ll see if this remains true given the slower speeds, delayed 4.7 support, and upcoming Mythos release. For in-depth details on OpenAI/Anthropic revenue broken down by model, see our Tokenomics Model.

The Throughput-Interactivity Frontier

To fully explain the architectural decisions Cerebras has made with their wafer scale chip, we first need to revisit inference fundamentals.

As Jensen repeatedly emphasized during this year’s GTC, throughput (tokens/sec/gpu) vs interactivity (tokens/sec/user) is the fundamental trade-off for inference. In our original InferenceX writeup, we described it as a bus vs a Ferrari: you can choose to serve lots of users slowly, a single user quickly, or anything in between.

Of course, users are also willing to pay more money for higher interactivity, so it’s currently unclear exactly which spot along the pareto frontier maximizes overall revenue and profitability of inference for a given model provider. In reality, providers are currently deploying multiple options in an attempt to capture the entire market. Fast mode, priority mode, batch pricing, and specific model architectures are all experiments from OpenAI and Anthropic to find the optimal combination for their user base.

Manipulating batch size (or “concurrency”, the number of users you serve simultaneously) is the primary way to move along the curve given the same hardware. This is the beauty of InferenceX. Whereas most other public inference benchmark only considers a single workload at a single interactivity level, InferenceX builds the entire pareto frontier across 3 different input/output sequence length combos for all the top open-source models. This allows you to make charts like the following, which shows that GB300 NVL72 achieves 20x more throughput than H100s at low interactivity (40 tps) and 100x more throughput at high interactivity (120 tps).

Alternatively, you can move along the frontier by changing the underlying hardware. This is the promise of SRAM machines like Cerebras and Groq. Their extremely high memory bandwidth allows them to increase throughput at high interactivity, and in the extreme case, achieve interactivity levels that are simply impossible for HBM-based accelerators. Cerebras offers speeds in the thousands of tokens per second, which is literally off the chart compared to the accelerators we benchmark in InferenceMax

In a world where people are willing to pay more for faster tokens, SRAM machines look quite attractive as they let you both (a) serve more users concurrently at premium speed (pushing the frontier “up”) and (b) serve some users at even faster, more expensive speeds (extending the frontier to the right).

The Wafer-Scale Engine

Cerebras’s fundamental bet has been to go beyond the reticle limit for a single piece of silicon. Instead of splitting a wafer into multiple chips, the goal is to make the entire wafer a chip. This clever scaling was to address a whole host of problems incurred by the slowdown of Moore’s law and the hard constraint of silicon being no larger than 858mm²; the size of a single reticle pattern in mask-based lithography. This single wafer-sized chip is called their Wafer Scale Engine (WSE).

The WSE is a 12 x 7 grid of 84 identical steppings/die on a whole wafer that forms one piece of silicon. Each wafer or chip has a large pool of very fast SRAM. 50% of silicon area is dedicated to SRAM cells with the remaining 50% consisting of compute cores. The key innovation is having both the silicon and memory on one piece of silicon instead of interconnecting multiple different chips together. This saves power, latency, and cost of moving data off-silicon or off-package.

“Traditional” GPUs and XPUs need advanced packaging and networking to achieve greater levels of aggregate compute and memory, which incurs costs in terms of power, speed and more networking equipment. While not a like-for-like comparison, Cerebras compares its on-wafer dataflow speeds to Nvidia’s off-package scale-up bandwidth based on the assumption that data can stay on the WSE whereas GPU data needs to move off-package.

Cerebras is on its third-generation product, WSE-3, which is fabricated on TSMC’s N5 node. One WSE-3 has 44GB of SRAM across a wafer or “single chip.” This is a lot of SRAM. A typical large processor has on-chip SRAM in the 100s of megabytes. Even the Groq SRAM machine is only 500MB for each LPU3. SRAM is very fast, so it can deliver 21PB/s of bandwidth, thousands of times more than what HBM offers. Again, this is significantly more than the very high bandwidth Groq LPU due to the WSE having several more banks of SRAM and with the bandwidth of individual banks aggregated together.

While Cerebras markets a lot of FLOPs for the WSE-3: 125 PFLOPs of FP16 compute, this is a sparse number, not a dense number. This is taking a page out of the Jensen Math playbook but taking it further. Unlike Nvidia, Cerebras doesn’t actually state dense FLOPs in public WSE marketing materials. However, Cerebras assumes 8:1 unstructured sparsity in its sparse number, so dense FLOPS is actually 1/8^th or 15.6 PFLOPS of FP16 compute throughput. We call this “Feldman’s Formula.” For the CS-2/WSE-2 a 10:1 ratio was assumed – as we see below, the sparse and dense spec is an order of magnitude different. While WSE-3 still wins on absolute compute throughput relative to other chips, compute per silicon area is not that impressive, especially today. This is likely down to each core being much smaller than a GPU’s functional array size, which is necessary for the purposes of yield harvesting, which we describe below.

The last part is off-wafer networking, which stands as the weakest part of the WSE. In total there is only 150GB/s of bandwidth, a fraction of GPU/XPU competitors who place huge importance on network to scale capability. We will talk more about the implications of low I/O as well as the structural difficulty in adding more I/O.

In summary, the WSE is a very big chip with a lot of SRAM, a decent amount of compute but not that much relative to silicon area, and almost zero network. We will now talk about the implications of this.

SRAM Machines

Where the WSE is clearly very strong is SRAM capacity. Like Groq’s LPU, the WSE is in the class of accelerator we call “SRAM machines,” where more silicon area is dedicated to super-fast SRAM, which is used as the primary memory where model weights and KV Cache are stored. In contrast, mainstream GPUs and ASICs such as TPU and Trainium use HBM to store model weights and KV Cache. They still have SRAM, just less of it. In general, trading HBM for SRAM means much higher bandwidth, lower latency and faster token output, but at the cost of capacity and therefore total throughput per {chip, watt, $}. SRAM is also just a lot more expensive per bit. Here is a chart from our recent article on NVIDIA + Groq’s use of SRAM comparing the technologies:

Even though the WSE-3’s 44GB of SRAM is a huge amount of SRAM relative to any other chip, it is not much more capacity than the 36GB provided by a single stack of HBM3E 12-Hi. With the norm trending towards 8 stacks per accelerator, this is 288GB for a single GPU or TPU package (e.g. the current generation Blackwell Ultra), which is 6.5x more than the SRAM capacity of a WSE.

Some readers may have noticed that DRAM has been in demand , and a lot of it is because AI system designers are trying to pack in as much capacity as they can. More memory in a system allows model providers to:

3. support larger context windows, i.e. larger sequence lengths per request (more KV Cache)

Inference providers make a business out of using all the above, which is why memory capacity per GPU is increasing. Not only that, but usable memory is not limited to a single package, since a workload can be sharded over multiple chips and aggregate memory can be pooled together within a scale up fabric. That’s why networking is such a key competitive battleground for all the AI hardware companies. That is, all of them except for Cerebras who have accepted the trade-off of little network and are working around it. So, with on-wafer memory capacity limited, the escape hatch of networking more wafers together is also much narrower for Cerebras. The lack of network bandwidth, while not fatal, is certainly a handicap in the WSE-3 design preventing Cerebras from launching their business to the stratosphere.

With that said, Cerebras is now on the path to being a healthy and rapidly growing business, with its OAI deal being a game-changer: until 2028 Cerebras will need to ship an order of magnitude more servers than they have since inception. The demand surge is already visible in TSMC’s wafer loadings, which step up materially each quarter through the year to meet OpenAI’s deployment requirements. We expect Cerebras revenue to inflect sharply in the coming years, with OpenAI as the primary growth driver.

Cerebras’s Technology

To reach this point, Cerebras has had to solve many technical problems from silicon to system to software. To their credit, there is a lot of proprietary hardware technology here, especially when compared to the innovations (or lack of) that a lot of other accelerator startups bring to the table. The wafer is a bold bet and not easy for incumbents and competitors to replicate.

1. Cross-die wiring and routing. Cerebras uses the scribe lines as wiring for the on-wafer data fabric that connects all the dies together. In a typical wafer, these are keep out zones where the wafer is diced to singulate individual dies.

2. Redundancy and failure routing. For the purpose of having an acceptable level of yield, the ability to route through defective cores is critical. Defects are inevitable especially for near reticle-sized units. Typically, dense processors that are near reticle sized have sort yields of well below 50%. For the sake of redundancy, there are a total of 970,000 cores on the WSE, of which 900,000 are enabled. Each core is deliberately made much smaller for the sake of better yield harvesting. However, this is not simple and there is a significant additional cost required. One of the interesting things done is that each batch of wafers will have a custom mask set for the upper metal layers. This is for the purposes of having different wiring for each batch to route around all the defective tiles. The cost of additional masks is a material increase in cost on top of the nominal TSMC wafer cost. Why is this for every batch of wafers? This comes down to intra-batch process variation being lower than across different batches. Read here to learn more about semiconductor manufacturing process variation. The net result of this is that wafer-level yield ends up being high. Nearly 100% of the TSMC wafer output is good enough to be assembled into a production server.

3. Power delivery and cooling. One of the major challenges that Cerebras has solved is getting over 20KW of power into one wafer, and it will be even more next gen. This much power necessitated the need for a custom power delivery solution from Vicor. This power will of course be turned into heat that needs to be removed, which requires specialized cooling. The power delivery and cooling sub-assembly in each CS server is called the “engine block.” This is another key component which, like the WSE silicon itself, is uniquely architected for Cerebras.

Despite these commendable technical achievements, the WSE architecture runs into a few technical limits that constrain their technical roadmap and ability to serve tokens.

Thermal Design and Cooling

Cooling 25 kW in a single 46,225 mm² wafer is the central thermal problem in CS-3 design, which translates into roughly 50 W/cm² averaged across the die, before accounting for hotspots. Air cooling was rejected because a 3DVC vapor chamber heat spreader (like we see in HGX H100 servers), scaled to span the 21.5 cm die, exceeds its wick’s capillary limit and dries out before working fluid can return to the evaporator. The CS-3 uses a custom liquid-cooled stack that presents architecture, flow rates, and rack-level plumbing different from Nvidia’s more recognizable direct-to-chip single-phase deployments.

The thermal solution is 100% custom and co-designed with the wafer. The silicon and the PCB underneath it expands at different rates as they heat up, and across a 21.5x21.5cm wafer that mismatch is large enough to crack a conventional package. The cold plate, the connector that bridges wafer to PCB, and the assembly tooling all had to be built from scratch. Cerebras calls its system the “engine block”, a four-layer sandwich including the cold plate, wafer, compliant connector, PCB, with the cooling manifold mated to the back of the cold plate. We will go over the system architecture in more detail in the next section.

Heat rejection runs through the cold plate. Coolant flows through micro-fin channels machined into the back of a copper plate. The wafer-facing side of the plate is polished and held against the silicon under preload, letting the two-slide relative to each other as they expand at different rates while maintaining contact to spread heat.

We find another architectural challenge at the rack-to-CDU interface. The OCP/Nvidia reference design for GB200 NVL72 sizes facility-side flow at ~1.5 LPM/kW. That constant is the one the majority of today’s CDU fleet is sized against. The WSE-3 runs at around ~100 LPM at 25kW, roughly 4 LPM/kW, or ~3x the NVL72 reference. That delta forces operators to use larger pumps, larger pipes, oversized CDUs, and quick-disconnects rated for higher flow. We believe that CS-4 should bring rack-level flow back toward 1.5–1.7 LPM/kW, which, if delivered, would converge Cerebras onto standardized infrastructure.

One of Cerebras’s main cooling partners is LiquidStack, which Trane Technologies acquired in March 2026. LiquidStack and Cerebras initially started working on two-phase solutions, and they have jointly developed L2L single-phase CDUs sized to the CS-3’s flow and pressure envelope.

Inlet temperature is a final axis where Cerebras diverges from other chips. Cerebras’s Oklahoma facility runs a 6,000-ton chiller plant producing 5°C (42°F) chilled water, which is then warmed across a heat exchanger to ~21°C (~70°F) before reaching the engine block. NVL72, by contrast, is specified up to 45°C (113°F) inlet temperature, which lets operators run free cooling for larger portions of the year. The CS-3’s wafer-level heat flux requires the colder envelope, and the cost is a chiller-heavy facility.

The CS-3 Architecture and BOM

Let’s take a step back from liquid cooling and zoom out to the Cerebras CS-3 system.

Each CS-3 includes the following: one WSE-3 engine block, peripheral compute and I/O modules, two mechanical pumps, 12 3.3kW power supply units, and a liquid-to-air or liquid-to-liquid cooling system.

Zooming into the WSE-3 engine block, the WSE-3 engine takes in 25kW of power alone. Power delivery and cooling of the WSE-3 wafer is extremely customized and innovated. The power is fed into the WSE-3 engine block via the blind mated power connectors from the 12 3.3kW power supply units. The PSU delivers power at 50V to 12 PDB boards that stack on top of each other horizontally. Each PDB board matches to a row of 7 Vicor power bricks, which matches to a row of 7 blocks on the WSE-3 wafer. With 12 PDB, that is 84 power bricks and 84 blocks on the WSE-3 wafer. Then, 12V power will be delivered to Vicor’s power delivery module which is on the PCB with the WSE-3 wafer on the other side, and the Vicor brick will convert the power to 1V before sending it to the wafer. The WSE-3 is socketed onto the customized PCB via an elastomer socket.

At the top of the WSE-3 engine block sits the I/O FPGA module connected to the WSE-3 PCB via board-to-board connectors. These FPGAs essentially serve as NICs that take in the Cerebras proprietary I/O off the wafer and converts it to Ethernet for scale out as well as PCIe. Customized cold plates are attached to the WSE-3 engine, the Vicor power delivery module, the CPUs, and the I/O FPGAs. The cooling loops connect to the manifold on the right side of the WSE-3 engine block. The manifolds have 6 couplings, in which 4 goes to the pump and 2 goes to the liquid-to-air or liquid-to-liquid heat removal system.

In addition, each CS server has a separate ‘KVSS’ node. This is a dual socket AMD CPU node with 6TB of DDR5 RDIMM which is used for KVCache offload. We estimated the BoM cost of the CS-3 system and the KVSS CPU node to be $350k USD per rack before the memory price hike that started in Q4 last year. Accounting for the latest memory price hike, we have raised the estimate of the BoM of the CS-3 system and the KVSS CPU node to $450k USD per rack.

This is very high especially relative to silicon content. While nominally the accelerator silicon, usually the most expensive part of the server, is one TSMC N5 wafer that costs around $20k, there are a lot of additional costs. The requirement for masking for each wafer substantially adds to the costs. The other major BOM item is the power delivery modules from Vicor. This is a custom VRM that needs to deliver 25kW to a wafer and uses VPD. The bespoke nature of this also means a high cost, and we believe VICR’s content in each WSE is not too far from TSMC’s content. The same goes for the customized cooling solution. Assembly and packaging are also performed in-house by Cerebras rather than at a contract manufacturer. There are also some peripheral components like 12x 100GbE Xilinx FPGAs that effectively act as NICs converting the Cerebras’s own I/O into Ethernet for external comms.

Where the Wafer Wins

To understand the extremely high memory bandwidth of Cerebras in context, one must put on the hat of a performance engineer working on LLM inference. To performance engineers, a chip is a tool. Whether you are using 10,000 LPUs, 72 GPUs, or 1 wafer to get the job done, what matters is the “arithmetic intensity” of the chip – how many FLOPs the chip can perform for every byte it transfers to/from memory (FLOPs/byte). Below is a table of chip specs to put the WSE-3 in context. Note that these are theoretical maximum numbers.

On a relative basis, the performance of AI applications depends on the performance of individual kernels (i.e. software that runs on the device, not the host CPU) on these chips. The canonical example of a kernel used in AI is GEMMs (general matrix multiplication). GEMMs can have different shapes, dictated by the shapes of the matrices being multiplied. Certain shapes running on specific hardware can be memory bound (i.e. performance is limited by the available bandwidth), or compute bound (i.e. performance is limited by the available FLOPs).

It is striking to see the FLOPs of a WSE-3 compared like-for-like with NVIDIA GPUs. In terms of dense FP16 or INT8 FLOPS (the actual FLOPs that developers using a Cerebras WSE use), an entire WSE-3 is only capable of 15.625 PFLOPS. Compared to NVIDIA GPUs running native FP4, B300 comes in at 13.5 PFLOPS (or 15 for GB300) and the Rubin GPU has 35PFLOPS. Of course, the astute reader will note that FP4 FLOPs and FP16 FLOPs are not always comparable, but with most production inference today shifting to FP4, it’s the best real-world comparison. Astute readers should also note the impact of Cerebras product marketing. Cerebras marketing materials, as well as their S1, claim much higher PFLOPs per wafer than our table shows. Thanks to the “Feldman Formula”, they use a factor of 8x (claiming 8:1 unstructured sparsity) to get there. An even bigger sparsity factor than the hallmark 2:1 rule of Jensen Math!

To compare Cerebras to alternatives, it is not useful to compare directly, chip-to-chip (or wafer-to-chip). We illustrate a more useful comparison below, with round numbers, to demonstrate where the wafer fits in.

It is most instructive to compare a single wafer’s worth of cost and performance to around $1M worth of hardware on both HBM and SRAM. Namely: 2x NVIDIA HGX systems (16 GPUs), 4x NVL72 sleds (16 GPUs), or around 50x Groq LP30s. So, we will progressively add more rooflines to the plot in the following charts.

Finally, this chart demonstrates how this analysis can be extended to the system level (albeit in a naive way), comparing the roofline of a single Wafer’s SRAM to DGX systems and a GB300 NVL72 rack. One has to assume zero network overhead and add many racks of GB300 NVL72 just to be able to realize the same FLOPs as Cerebras on kernels with equivalent arithmetic intensity.

To finish with a complete understanding of which AI workloads are a good fit for Cerebras, we can just look at common GEMM shapes. GEMMs generally use “mnk” notation, meaning that the input matrices have size “m” and “n” respectively, with a contracting dimension of “k”.

We can calculate the Arithmetic Intensity of a given GEMM using the following formula:

And finally, here is how those kernels would theoretically perform on different chips. Just trace from bottom to top on one of the vertical lines that represent the arithmetic intensity of a given kernel to see the (theoretical) performance that a given chip will be able to realize on that GEMM shape (measured in TFLOPs).

At a high level, it is clear that Cerebras has very unique performance characteristics, with an optimal arithmetic intensity of 0.74 on the WSE-3’s SRAM and FP16 or INT8 FLOPs. With HBM-based GPUs going the other direction over time, i.e. an arithmetic intensity increasing to over 1000, there is a clear difference between the GEMM shape (or more generally, which kernels) will make the most effective use of Cerebras hardware.

For the reader to get a sense of what the realized FLOPs looks like for a given decode kernel, just imagine a decode kernel with (m=batch=1) and arithmetic intensity of (AI=2). This is the leftmost vertical bar on the previous chart. As you trace your finger from bottom to top on that line you will cross many chips before you reach Cerebras: all NVIDIA GPUs and Groq LPUs will only be able to realize dozens or hundreds of TFLOPs in an absolute max, theoretical case. Meanwhile, the Cerebras wafer can (again, theoretically) realize its full 15.625 PFLOPs. This is the key point of the wafer. Absolutely massive amounts of memory bandwidth from the 44GB of SRAM on the wafer mean that decode kernels can realize equally massive amounts of performance.

Going back to our job as a performance engineer, this means that decode kernels with low arithmetic intensity have a much higher theoretical limit in terms of the amount of FLOPs that can be realized. The SRAM bandwidth can keep up with the compute, while the HBM of a GPU running the same kernel leaves Blackwell SM100 FP4 Tensor Cores starving. And as a result, the types of models and workloads that will be designed to run on the Cerebras WSE-3 in the future, such as GPT-5.3-Codex-Spark (with an architecture that also goes by the name of gptoss-120b), will be developed with the performance characteristics of the wafer in mind.

The Wafer Taketh and the Wafer Giveth

The WSE has several clear weaknesses that we have mentioned. It has a lot of SRAM, but given SRAM is inherently not dense on a per-watt or per-dollar basis, HBM-based GPUs and XPUs offer far more memory capacity per watt or dollar. This HBM is currently used to serve larger models with longer context length, as well as more batching of users to drive throughput. Networking more wafers together to overcome the lack of memory per wafer is also constrained by the lack of off-wafer bandwidth. Absent a heroic technical achievement (hybrid bonded optical transceiver wafer anyone?), both these issues are an intentional part of the Cerebras architecture and make it hard for Cerebras to economically serve large models or even medium size models with long context lengths, that are representative of today’s agentic workloads.

To illustrate this point, we have made an interactive calculator available at tokenomics.info/cerebras. This is a taste of the kind of research that our Tokenomics subscribers get.

As shown above, when adjusting the average request size, number of concurrent requests supported, model size, and quantization for weights and KV Cache, the total number of WSEs required to run inference varies significantly. This, of course, leads to different performance characteristics on inference or decode, and $/Mtok cost conclusions.

A notable assumption in this calculator is our 96.3k average request size. While Cerebras chooses to build their inference product for their customers around an assumption of 64k avg request size, we believe this is an artifact of running models with limited context windows of 128k. In other words, confirmation bias in action.

To get an understanding of exactly what real-world traffic patterns are, we built a proxy that collects fully anonymous traces from popular agentic coding harnesses such as Claude Code, Codex, Cursor, and OpenCode. This is part of an ongoing effort to collect production agentic traces for offline replay on InferenceX.

A relatively large sample size of ~432k requests (about 80B tokens) leads us to believe that a typical P50 ISL is ~96.3k tokens, not 64k or fewer. We also deduce that the P90 or P95 requests can be exponentially more valuable than the initial requests and still critical to support. In total, almost 50% of our requests are over 128k, which is the maximum context window that Cerebras currently supports on public endpoints. Many sessions we see have an initial context length of over 100k tokens due to tool use context, system prompts, and things like skills and various other forms of primer context.

Moreover, the industry is trending towards larger context windows ad infinitum -- 128k context will certainly not be acceptable for long, especially with the prevalence of agentic workloads. The obvious conclusion of this analysis is that to run the latest open-source models with full context windows for real-world traffic patterns, Cerebras needs to deploy a lot of wafers.

Just for the DeepSeek v4 example above, with 24 CS-3 a CS-3 customer could get 5 GB300 racks. Each rack has 20TB of HBM which is easily able to absorb the model weights leaving over 19TB for KVCache. That is a lot of KVCache to serve more users and to support long sequence length, and there are 5 of these racks also. While we’ve shown the speed gap in favour of Cerebras, this is how the throughput gap is well in favour of HBM-based GPUs.

SRAM Scaling is Dead

Arguably, Cerebras is the company most exposed to the death of SRAM scaling, with Cerebras’s key draw being SRAM and 50% of wafer area dedicated to SRAM. It’s already showing up on their roadmap. WSE-1 on TSMC 16nm shipped with 18 GB of SRAM; WSE-2 on 7nm jumped to 40 GB, a decent 2.2x gen-on-gen. WSE-3 on 5nm advanced to just 44 GB. That’s a 10% increase across a full node transition, while logic transistor count grew ~50%.

As we look to the future, this only gets worse. We can see that beyond 5nm (what the WSE-3 is currently fabbed on), SRAM scaling basically stops dead. The most common flavour of 3nm, N3E, has zero shrink relative to N5, and this continues to be the case for N2 and beyond. Now, the only way for Cerebras to increase SRAM capacity is by increasing wafer area dedicated to SRAM, sacrificing compute area. It’s a strict tradeoff when the chip is wafer scale. This is why the next generation CS-4 system will use the same N5 based WSE-3, but with higher power to sustain higher clock speeds and compute but stuck at the same SRAM capacity.

By comparison, this isn’t as critical for Groq as they are able to scale in the Z direction: using hybrid bonding to add additional SRAM tiles to vastly expand SRAM per package, which is on the roadmap for the Nvidia Groq LP40.

The logical path would be for Cerebras to do the same: wafer-on-wafer bond another wafer to expand SRAM and or compute per system. This is something that Cerebras is seriously exploring, having shown their concept of a DRAM wafer hybrid bonded onto the WSE to add more fast memory capacity. However, the timeline and technical feasibility of this is a concern for us given the litany of thermo-mechanical and bond-wave challenges. Yes, wafer-on-wafer bonding is an established process, but not where the whole wafer is stitched together as a whole chip. Cerebras has overcome these sorts of challenges in the past and will need to continue to innovate.

The Island Problem - bandwidth is geometry

Despite the SRAM scaling issue, WSE still delivers an overwhelming amount of more compute and SRAM per single piece of silicon compared to other chips. Now comes the biggest tradeoff: the network. As mentioned earlier, each WSE has just 1.2 Tb/s (150GB/s) of off-package bandwidth. This is low compared to the average accelerator, and especially low relative to the amount of compute that the WSE has. No, this is not because the Cerebras architects have missed the importance of I/O for AI compute and overlooked adding more SerDes, this is just an inevitable tradeoff that comes with a wafer-scale chip.

By comparison, each Groq LP30 that NVIDIA will produce includes 96 lanes of 112G SerDes. That’s a 9.6 Tb/s pipe in and out of a much smaller chip. It is clearly well prepared for the PDD + AFD inference setup that Jensen debuted at GTC this year.

So why the bandwidth tradeoff? At the current 150 GB/s (1.2 Tb/s) of off-wafer bandwidth, that’s just 0.17 GB/s per mm of edge, so Nvidia’s off-chip I/O is 130x denser!

Cerebras’s lack of shoreline density comes down to the wafer scale architecture and reticle stepping problem. The WSE is patterned one reticle field at a time, tiling the same reticle pattern across the wafer in an 84-die array (12 columns × 7 rows on WSE-3). For the cross-scribe-line interconnect to work, every reticle exposure has to be identical, with the same logic, the same memory, the same routing, in the same positions. That’s what allows the on-wafer 2D mesh fabric to extend uniformly across die boundaries: every die’s east edge connects to its neighbor’s west edge with matching pin assignments.

This uniformity requirement is non-negotiable, and it has a punishing implication for IO. You cannot dedicate one reticle to PHYs while the other 83 reticles do compute. Every reticle has to be the same reticle. So, if you want more SerDes lanes on the wafer edge, you have to spend reticle area on SerDes in every reticle, not just the perimeter ones. Most of those PHYs will be in the middle of the wafer where they cannot reach the outside world, doing nothing. You pay a full silicon cost for IO that’s stranded inside the wafer.

An alternative, putting PHYs only in perimeter reticles, would require a non-uniform stepping pattern, which is unfeasible from a process point of view. It would require swapping out reticles on a partially patterned wafer which would introduce untenable process risk and complexity, especially given all these reticles need to be stitched together which breaks the cross-scribe-line interconnect that makes wafer-scale work in the first place (what we called the “scale-up network” earlier).

Even if Cerebras accepted stranded silicon and burned area on PHYs everywhere, they would hit a third constraint: on-wafer dataflow blocking. During inference, the on-chip 2D mesh fabric carries the activations, weights, and gradients between cores (again, why we called it the scale-up network). Every PHY block placed inside a reticle is a hole in the mesh, a region where compute and routing cannot exist. PHYs are large (high-speed SerDes are typically 1–3 mm² each at 5nm, including the analog circuitry that doesn’t scale with logic), and their analog circuitry is hostile to neighboring digital logic due to power and EMI concerns, demanding guard regions. Putting PHYs in the middle of the wafer means the 2D mesh fabric has to be routed around that area, increasing latency between reticles and reducing total bandwidth. Too much of this excess routing would defeat the purpose of going wafer-scale, since the whole point is fast and low-power dataflow across tiles.

In summary, the uniform tiling that makes wafer-scale possible (one reticle pattern, one mesh fabric) is what makes adding IO bandwidth hard. Cerebras must be looking for ways around this limitation.

A lot of the issues we just described come from the realities of moving data in the electrical realm, which are circumvented with optical I/O. The solution that Cerebras is working on (again proof that Cerebras recognizes the problem) is a photonic interconnect wafer hybrid bonded onto the WSE. As with the additional DRAM wafer to solve the memory constraint, the bandwidth constraint is also being addressed with another wafer.

Cerebras claims that for LLM inference they don’t need any more bandwidth and is only aggressively pursuing hybrid bonding wafer scale photonic I/O to help their HPC boomers. The HPC customers whom NVIDIA has effectively abandoned after reducing FP64 native hardware on their GPUs to basically nothing. This is great that Cerebras is aggressively reinvesting completely back into moonshot R&D instead of doing buybacks. Buybacks is not an good idea for companies that are lots of r&d things to reinvest into, for example, AMD did ~$221million dollars of buybacks last quarter yet internally multiple AMD internal teams continue to lack development interconnected GPU clusters.

This allows data to move in/out of the wafer up through the z-axis, rather than having it go through the edges. The photonics partner developing this photonic wafer is Ranovus. This reintroduces the issues of WoW hybrid bonding for wafer scale silicon. Optical components are thermally sensitive (cannot be too hot or too cool) and it will be sandwiched directly against a wafer that runs hot. Lastly, there is the practical difficulty of fibers needing to be perfectly coupled off to the wafer. This is still being figured out at the optical engine level for conventional CPO, let alone for something wafer scale.

With all this in mind, let’s look at how the architecture shapes inference workloads

Pipeline Parellelism is Forced

One of the key concerns that we have already highlighted with using Cerebras in any inference deployment is just how big models have gotten. Both in terms of total parameter count (e.g. DeepSeek V4 is 1.6T total parameters), and in terms of KV Cache (256k context is the norm, with DeepSeek V4 debuting 1M context).

The combination of limited single wafer SRAM capacity of 44GB in the WSE-3 and low IO bandwidth results in challenges effectively serving models of these sizes.

Each CS-3 has just 12x100GbE of IO bandwidth -- roughly 150 GB/s for the entire wafer. This is one sixth of the scale-up bandwidth for Blackwell with NVLink5 at 900 GB/s per GPU, and an order of magnitude below the bandwidth of HBM.

This bandwidth constraint is what makes it difficult for Cerebras to serve larger parameter models. Any large tensors to be used must be resident on the wafer; streaming on/off the wafer is impossible with such a small amount of I/O. Similarly, any sharding strategy that requires high-bandwidth collectives at each layer is categorically ruled out.

The only real option is pipeline parallelism, which slices the model layer-wise across wafers and only transfers activations between stages, relying on the fact that activations are small relative to weights. This reduces network requirements and keeps the capacity-demanding components (the weights, and to some extent the KV cache) stationary instead of moving on or off the wafer. For instance, Cerebras shards Llama3 70B across 4x WSE-3, transferring only the activations between each wafer and staying well within the available 1.2Tbps I/O.

As you increase the number of wafers used to host the model, there are several factors to wrestle with to increase scale. First, the pipeline bubble: to keep N pipeline stages busy, you need at least N in-flight microbatches. A 4-stage config needs ~4 microbatches in flight; a 16-stage config needs ~16. Second, each in-flight microbatch carries its own KV cache, and on Cerebras that KV cache must live in the same 44GB of on-wafer SRAM that’s already mostly consumed by weights. Even if there is enough capacity in the SRAM with the heavily compressed KVs of recent models such as DeepSeek V4, the time to transfer the KV cache on or off the wafer is still quite large. Additionally, scaling the model size scales the number of wafers needed to hold the weights and therefore increases the number of times the latency of wafer->wafer activation transfer adds to the decode time.

In summary, the way the wafer is being used in production today basically goes against the entire ethos of the wafer. The whole point of the wafer is to run really fast at small batch sizes!

Running the Numbers

Let’s take a look at some napkin math with a few open-source model architectures to better understand how different models map to Cerebras’s SRAM footprint. Below are some rough ballpark numbers showing the footprint of several models.

And now some rough numbers considering the WSE-3 specs. We make some assumptions here, including that the transfers will use the full 12x100Gbps.

Here we define the minimum number of wafers to store the model weights by sharding strictly along layer boundaries, but we don’t include the space to store KV caches. In practice, more wafers may be used to give more space for KV caches. Activation transfer times are not included because activations are so small that their transfer will be bound by the propagation time across the I/O path.

It is clear from the table that recent KV cache compression techniques such as those published by DeepSeek might significantly alleviate issues Cerebras has with long-context serving. However, the problem of slow I/O does not completely disappear. Firstly, KV transfer times on- and off-chip are still quite large at several milliseconds, both impacting TTFT and making it more difficult to achieve high utilization due to issues of batching, pipelining, and latency-hiding related to KV cache storage and transfer. Secondly, the fixed I/O latency of activation transfer must be paid in proportion to the number of wafers used to host a model instance. This is a fixed cost in the TPOT that scales linearly with the number of wafers used to host the model.

The key takeaway is that Cerebras, while fast, pays a large latency cost to move data on and off the wafer, and therefore their cost-to-performance ratio (or perf per Joule) will depend on how much of that latency they can hide or minimize. A clue about the difficulty of this in practice may be reflected in Model offerings on Cerebras Inference Cloud. The largest production model is GPT-OSS, which is only 120B total parameters. There are larger preview models, but even those top out at 355B (GLM 4.7). For reference, Sonnet and Opus are 1T and 5T parameters respectively, per Elon. Notably, the formerly popular Llama 70B and 405B models were also deprecated, potentially due to the economics of serving them.

It’s also worth emphasizing that two of the most popular frontier open-source models of 2025, DeepSeek V3 and Kimi K2, have never been offered on the public Cerebras Cloud. This is despite the large KV cache size reduction in DeepSeek V3 due to the use of Multi-head Latent Attention (MLA), which would leave it with better serving economics than Llama 3 405B.

With that said, our analysis above shows that the even newer DeepSeek V4 Pro can have a similar deployment shape to Llama 405B (which they have already served on Cerebras cloud), with significantly smaller KV cache sizes. For that reason, with modern KV cache compression techniques and enough concurrency, Cerebras might indeed look attractive even for large 1T+ models.

The Cerebras OpenAI Deal

OpenAI plays a huge role in Cerebras’s future. It is simultaneously the company’s secured lender, its largest warrant holder, and the source of essentially all of its $24.6B backlog. OpenAI’s financial stake in Cerebras means Cerebras’s fortunes are tied to a single counterparty through three interlocking mechanisms that all move in the same direction. If the relationship succeeds, the loan is repaid through capacity delivery rather than cash (with the 6% accrued interest waived on capacity-repaid portions), the warrant vests and aligns incentives, and revenue scales into the billions. On a fully diluted basis, OpenAI could hold as much as 12% of Cerebras shares (not including any new issuances and offerings).

While Cerebras had been largely left out of the neocloud boom, OpenAI’s February release of GPT-5.3-Codex-Spark (a model using the gptoss-120B architecture that was distilled from the real 5.3 Codex) is turning things around. Spark runs on Cerebras at up to 2,000 tok/sec/user and led to the announcement of a long-term deal between the two companies, driving their IPO prospects (and the value of sama’s stake) ever higher.

Cerebras’s chips are only economically capable of serving relatively small models today, or at least based on what’s available to the public. GPT-5.3-Codex-Spark, for example, is NOT at all the same thing as the full GPT-5.3-Codex; it’s gpt-oss-120b fine-tuned on GPT-5.3-codex traces. In other words, it’s a distilled model that’s over 10x smaller.

While GPT-5.3-Codex-Spark is really fast, its tokens likely aren’t worth $10B today. For OpenAI to run any model above 1T total params with a 1M context window for modern agentic workload patterns, they will need to accept significant tradeoffs on cost (and recoup it by selling those tokens at a significant premium), and we expect the realized performance to be below 1000 tok/sec interactivity. On the other hand, algorithmic improvements will certainly make small models smarter. We’re probably less than a year away from GPT 5.5-level intelligence in a 120B form factor.

As mentioned earlier, many of our engineers were willing to forgo the frontier level intelligence of Opus 4.7 in exchange for faster tokens from Opus 4.6 fast. With GPT-5.5, OpenAI finally has an Opus 4.5 level model. Will people be willing to pay for really fast GPT-5.5- quality tokens a year from now even after the true bleeding edge frontier has moved far beyond it? For the first time ever, we think the answer may be yes. While the first 750MW is locked, there is much more upside for Cerebras if OAI chooses to take the full 2GW or even more. This is all dependent on the quality of the model they can fit on Cerebras hardware.

Behind the paywall, we will go through just how the OAI deal’s profitability Cerebras and the major execution risk - how far along is Cerebras in securing the DC capacity.

Today, on the verge of Cerebras’s IPO, and because we love the wafer, we are shipping an article that is as long as 4 normal articles. Inside, we will dive deep on:

The Need for Speed

The Throughput-Interactivity Frontier

To fully explain the architectural decisions Cerebras has made with their wafer scale chip, we first need to revisit inference fundamentals.

The Wafer-Scale Engine

SRAM Machines

3. support larger context windows, i.e. larger sequence lengths per request (more KV Cache)

Cerebras’s Technology

Despite these commendable technical achievements, the WSE architecture runs into a few technical limits that constrain their technical roadmap and ability to serve tokens.

Thermal Design and Cooling

The CS-3 Architecture and BOM

Let’s take a step back from liquid cooling and zoom out to the Cerebras CS-3 system.

Where the Wafer Wins

We can calculate the Arithmetic Intensity of a given GEMM using the following formula:

The Wafer Taketh and the Wafer Giveth

To illustrate this point, we have made an interactive calculator available at tokenomics.info/cerebras. This is a taste of the kind of research that our Tokenomics subscribers get.

SRAM Scaling is Dead

The Island Problem - bandwidth is geometry

So why the bandwidth tradeoff? At the current 150 GB/s (1.2 Tb/s) of off-wafer bandwidth, that’s just 0.17 GB/s per mm of edge, so Nvidia’s off-chip I/O is 130x denser!

In summary, the uniform tiling that makes wafer-scale possible (one reticle pattern, one mesh fabric) is what makes adding IO bandwidth hard. Cerebras must be looking for ways around this limitation.

With all this in mind, let’s look at how the architecture shapes inference workloads

Pipeline Parellelism is Forced

The combination of limited single wafer SRAM capacity of 44GB in the WSE-3 and low IO bandwidth results in challenges effectively serving models of these sizes.

In summary, the way the wafer is being used in production today basically goes against the entire ethos of the wafer. The whole point of the wafer is to run really fast at small batch sizes!

Running the Numbers

And now some rough numbers considering the WSE-3 specs. We make some assumptions here, including that the transfers will use the full 12x100Gbps.

The Cerebras OpenAI Deal

Behind the paywall, we will go through just how the OAI deal’s profitability Cerebras and the major execution risk - how far along is Cerebras in securing the DC capacity.

AI demand has been driving the explosion in compute over the past few years, resulting in chip designs getting ever more complex, with silicon area and power per package seeing continued growth as designs push for even greater performance. With each successive generation, new process nodes with more design rules and restrictions further increase chip design costs.

At the same time, the rush to bring compute into the market as quickly as possible has put design teams under immense pressure to compress timelines and speed up validation cycles from years to months. If you’re not fast, you will get lapped up and beaten by your competitors. Even a 3 month delay means billions of dollars.

All this is happening while the engineering talent base is shrinking. Lucrative salaries and flexible working arrangements have enticed most students into the Software and Information Systems tracks, resulting in a dwindling number of Electrical Engineer graduates that could enter the chip design workforce. Siemens presented the engineer-hours demanded of these numerous complex AI accelerator designs that far outstrip the engineering talent coming into the workforce.

One-third of the current U.S. semiconductor workforce is over 55. The pipeline of new graduates is nowhere close to filling that gap. Even Apple is actively funding education programs to encourage uptake in engineering. While their New Silicon Initiative has contributed to increasing interest and number of EE graduates, it barely moves the needle compared to the explosion in manpower requirements as transistor count grows at a Moore’s Law pace.

With this trifecta of increasing chip complexity, compressed design timelines and a shortage of engineers, a massive bottleneck has formed at the design stage. The latest AMD MI455X packs 320 billion transistors across 12 logic dies on 2nm and 3nm processes with advanced Hybrid Bonding 3D die stacking, HBM4 memory integration and high speed 224G SerDes. Designing something at this scale is not a matter of hiring more engineers or buying more verification servers. It tests a company’s tooling, methodology, and human capital organization as to whether the design succeeds or fails.

After spending hundreds of millions of dollars on a new SoC design, there is no guarantee the chip will work. Multiple steppings are usually required that need new mask sets, with A0 rarely going into production. When a single advanced mask set costs tens of millions of dollars, every respin is a gut punch to the balance sheet. Furthermore, it adds months to the schedule for high volume production start.

As designs get more complex, testing is becoming more important to ensure all modules within a chip are interoperable and locally sound. Verification, the process of proving a design does exactly what it should before committing it to silicon, now consumes up to 70% of total project effort, depending on the design. Verification engineers are the fastest-growing job category in chip development, and the industry still cannot hire them fast enough.

While chip complexity grows at roughly 50% per year, driven by new nodes and larger SoCs, design productivity improves only about 20% each year. This design productivity gap means every new generation of silicon demands exponentially more engineering effort, more compute, and more sophisticated automation.

The semiconductor industry’s ability to keep building more powerful chips depends not on physics or lithography alone, but on EDA (Electronic Design Automation) software. These tools effectively translate human intent into manufacturable silicon. Without EDA, no chip designed after the mid-1980s would exist.

This primer is your guide to EDA in the semiconductor industry. In this first part, we will walk the entire journey from RTL (Register Transfer Level) code, the high-level hardware description language that engineers actually write, all the way to manufactured, packaged silicon. We will name the tools, explain the tradeoffs, and show why EDA is one of the most consequential and underappreciated sectors in technology.

In part 2, our EDA Market Primer dives deep into the business of EDA, profiling the major companies (Synopsys, Cadence, Siemens) and their revenue and business models. We provide comprehensive market analysis and monitoring the Chinese EDA effort, as well as IP licensing and outsourcing to design partners and the transition to Customer Owned Tooling (COT) with hyperscaler ASIC designs.

Part 3 then assesses how AI is disrupting the EDA industry, covering the full gamut from startups and engineer dashboards to agentic chip design flows from NVIDIA and the big three. The concept of using AI accelerators to create superhuman designs that go into future AI accelerators is the most exciting development that our industry has seen in decades. Stay tuned as we cover the incoming revolution in chip design.

A Brief History: From X-Acto Knives to the Big Three

In the 1960s and 1970s, designing an integrated circuit meant drawing it by hand. Engineers sketched layouts on graph paper, and technicians transferred those sketches onto sheets of Rubylith — a red cellophane film laminated onto clear Mylar. Using X-Acto knives and light tables, they cut away sections of the film to define each layer of the chip. The finished masters were then photo-reduced up to 100 times to create production photomasks. A single slip of the blade could ruin weeks of work. This was the standard design process up to and including the Intel 8080 with its Rubylith pictured above.

The first step toward automation came in 1971, when Calma shipped its Graphic Design System (GDS) to Intel, allowing engineers to digitize and edit layouts on minicomputers. In 1978, Calma released GDS II, whose stream file format became the de facto standard for exchanging mask data. Remarkably, GDS II remains the dominant interchange format today, nearly five decades later, alongside its modern successor OASIS.

The EDA industry as we know it was born in 1981, when three companies launched within months of each other: Daisy Systems, Mentor Graphics, and Valid Logic Systems. Known collectively as “DMV,” they introduced computer-aided engineering to the front end of the design flow, schematic capture, simulation, and logic verification, running on dedicated workstations. By the late 1980s, all three had migrated to standard Unix workstations from Apollo and Sun Microsystems, establishing the software-centric business model that defines EDA today.

The Big Three Emerge

The modern EDA landscape is dominated by three companies. Synopsys, founded in 1986 by Aart de Geus and colleagues from General Electric’s research group, introduced Design Compiler in 1987, the first commercial logic synthesis tool. Logic synthesis automated the translation of high-level hardware descriptions into optimized gate-level netlists, a breakthrough that enabled the leap from thousands of hand-placed transistors to the billions we design today. Cadence Design Systems formed in 1988 through the merger of SDA Systems and ECAD, quickly becoming the leading provider of IC layout and place-and-route tools. And Mentor Graphics, one of the original DMV trio, was acquired by Siemens in 2017 for $4.5 billion, rebranding as Siemens EDA in 2021 and bringing deep verification and physical design expertise into the Siemens Digital Industries portfolio.

Compared to the early Rubylith days, logic synthesis not only speed up design, it fundamentally changed what was possible. By abstracting away manual gate placement, it unlocked a multi million-fold increase in design complexity to form today’s multi-billion-transistor SoCs.

The Chip Design Waterfall

Building a chip is a multi-year relay race with thirteen legs. Miss a handoff and the whole schedule slips, by months, or even by quarters. The diagram below lays out the full flow from a blank whiteboard to volume production. This article will go through the stages where EDA tools are used in the design flow.

This is a simplified “waterfall” view. In practice, many of these stages overlap heavily and iterate. Architecture bugs found during verification force RTL changes; timing failures in physical design send engineers back to re-optimize critical paths. A modern SoC program manages dozens of these feedback loops simultaneously, which is exactly why EDA tooling exists, no human team could track it all by hand.

1. Planning

The first stage to any chip is to decide on what role the chip serves. Each design department usually specializes in a given family of chips, be it CPUs and accelerators to the more mundane system controllers and embedded sensors. The product requirements and high level specifications are defined with respect to the current generation of products in the market, along with competitive analysis of others in the target market.

Strawman concepts are proposed that evolve rapidly as Program Managers work within the insertion schedules of various IP blocks from the design teams that may be ready for integration. Learnings from Post-Mortems of previous projects are factored in, forming a knowledge base to work from on what works and what is too ambitious for a given timeframe.

The key high-level metrics here are PPACt: Performance and Power consumption, usually given as a percentage improvement over the prior generation and where it might sit in the competitive landscape, the area that such a design takes up in silicon on a given process node, which translates to Cost. Time to Market is the final metric that determines whether the product is viable both from a design time and product competitiveness standpoint. In a fast growing market where performance doubles every few years, being 1 year late could spell the end of a project’s success.

These feasibility studies will then need to be greenlit by management before project kickoff begins in earnest. Each company has with work within their R&D budget with finite engineering resources. Scheduling resource allocation with ongoing projects in the roadmap requires strict completion deadlines so engineers can be released to start working on the next project. Communicating early with suppliers to project the wafer, memory and packaging demands for each design is now increasingly important to secure capacity.

2. Architecture Layout

Closely tied to planning, the architectural layout is done alongside design space exploration. A high-level floorplan diagram sets the initial area bounding boxes for each logic and I/O block design team to work within. Each functional block is broken down into smaller elements that are easier to design and can be repeated multiple times across the design. These area budgets may increase over the design cycle based on features that may be added later that take more area. For example, a feature update in an Instruction Set Architecture (ISA) with additional computing elements to support new instructions. On the AI accelerator side, this equates to adding dataflow accelerators and doubling Matrix Multiplication engine widths.

Block diagrams are drawn up with relations and Network on Chip (NoC) bandwidth requirements decided for each functional block, with memory bus widths and SRAM area budgeted based on cache hierarchy and early simulations of performance vs memory pressure. These simulations, known as Design Space Exploration, have traditionally been done with targeted Design of Experiments that simulate the performance impacts and interactions between each functional block, varying unit sizes, widths and bandwidths to find the lowest hanging fruit to maximize performance gains.

Going forward, this step has increasingly been accelerated with AI, as the task is easily verifiable with assignable reward functions for PPA in a multi-dimensional input space. First party AI-driven exploration tools such as Synopsys’ DSO.ai have followed the many internal efforts by the fabless design houses to leverage AI to accelerate pathfinding and planning decisions. An in-depth analysis on this will be featured in Part 3 of this EDA series.

3. RTL Design

With the architecture specified, engineers must then describe exactly what the chip does. This is done at the level of registers, data paths and combinational logic, which will later be translated into transistor implementations. This description is called RTL (Register Transfer Level) code, and it is where the design’s behavior is defined in a language that both humans and synthesis tools can read. Most of the engineering hours in the chip design flow is spent writing and verifying the RTL code. Below we look at the aspects to RTL design.

Signal Timing

In the real world, transistors don’t switch instantaneously. There is a propagation delay where it takes some time for an input change to produce a stable output. This delay has two components: the gate delay (how fast the transistors themselves switch) and the wire delay (how long the electrical signal takes to travel along the metal interconnect to the next gate). At advanced process nodes, wire delay ends up dominating gate delay as transistors switch faster while datapaths lengthen with complex designs.

Digital chips use a clock signal to synchronize all operations. Two timing constraints govern correctness. Setup time requires that input data be stable for a minimum period before the clock edge arrives. Hold time requires that data remain stable for a minimum period after the clock edge. The clock period (the inverse of frequency) must be long enough to accommodate the slowest signal path in the entire design. This worst-case path is called the critical path. If your critical path takes 0.2 nanoseconds and you want a 5 GHz clock (0.2 ns period), you are right at the edge, with no margin for process variability. This is why timing optimization consumes enormous effort in chip design, with many trade-offs in performance and complexity.

State Elements

Combinational logic computes outputs from inputs, but it needs to be combined with memory to build useful functions such as a counter, a processor pipeline stage, or a protocol engine. These memory registers are implemented as flip-flops. A flip-flop captures and holds one bit of data on each clock edge, acting as a tiny one-bit memory. Multiple flip-flops are chained together with combinational logic to form a Finite State Machine (FSM). This circuit steps through a defined sequence of states, one clock cycle at a time. This is sequential logic, which forms the base for chips to compute. Thus, RTL is an abstraction that describes how data moves between registers and combinational logic on each clock cycle.

Writing the RTL

RTL is written in a hardware description language (HDL). The dominant choice today is SystemVerilog, an extension of the original Verilog language that adds features for both design and verification. VHDL, the older alternative, still appears in aerospace and legacy applications. A designer writing RTL specifies what happens on every clock edge, where data moves between registers, arithmetic operations execute, and state machines transition. Synthesis tools (covered in the next section) then convert this description into actual gates and transistors.

Once written, RTL passes through linting, a static analysis that catches coding mistakes, race conditions, and syntax errors. This is done as a quick code review without requiring simulation. VC SpyGlass from Synopsys is the industry-standard linting tool, flagging seemingly subtle issues that could cause intermittent silicon failures. This is essentially the chip design equivalent of a compiler’s warning flags, just with far costlier consequences.

IP Integration

In most modern SoC (System on Chip) designs, only about 20-30% of the RTL is truly custom logic designed in-house. It is easier to reuse previous designs for non-critical components, with the rest comprised of licensed IP blocks. These are pre-designed, pre-verified modules purchased from third-party vendors. ARM provides processor cores, GPU and other IP. Synopsys DesignWare supplies USB, PCIe, DDR memory controllers, and hundreds of other interface blocks. Broadcom’s excellent high speed IO can be used if they are handling the rest of your chip design. Meanwhile, smaller IP vendors sell everything from GPIO interfaces to cryptographic accelerators.

IP licensing is the result of economics. Designing a custom PCIe Gen 6 controller from scratch would require spinning up a dedicated team of I/O design and verification engineers working to prove compliance with PCI-SIG’s specification. Licensing one costs a fraction of that and comes pre-verified against the spec. However, the IP integration itself can be challenging, something we will cover for our subscribers below.

4. RTL Verification

The RTL code then goes through the verification process, crucial to iron out any bugs or design errors within. This is done through simulation, which runs the design in software, applying stimulus, and checking the outputs. Three commercial simulators dominate the market, in order of ubiquity:

Most large chip companies license at least two of these. Running a full regression suite with tens of thousands of test cases on a complex SoC can consume thousands of CPU core-hours per run. Dedicated on-prem verification servers are usually insufficient these days, with cloud-based simulation on AWS and Azure shoring up short-term demand as teams try to burst capacity during crunch periods before tapeout. The amount of data this generates is also staggering, with multiple Petabytes of disk space required to house just a single chip’s entire definition and test items.

As mentioned above, you will usually find more Verification engineers than any other single role in a chip design house. With chips getting more complex, even more things need to be verified with one another, placing huge demands on the verification staff. We will dive into what this means for chip design in reality for our subscribers below.

The Verification flow takes two paths: Standard DV testing on one end, and Formal Verification with proofs on the other.

UVM Testbench

RTL simulation is structured in UVM (Universal Verification Methodology). This is an industry standard SystemVerilog library and method for building reusable testbenches. Before UVM was standardized by Accellera in 2011, every team rolled their own testbench architecture. UVM brought the industry together by defining a common set of components:

The testbench is used for constrained random verification. Instead of hand-writing every test case for directed testing, the engineer defines constraints such as legal address ranges, valid packet formats and protocol rules. The tool then randomly generates millions of input combinations within those bounds. This approach finds corner-case bugs that might not be caught with directed testing. These constrained random regression tests are very resource intensive due to the large sample range, but is usually more effective and fault detection compared to writing directed tests.

Formal Verification

Formal verification takes a fundamentally different approach from simulation. Instead of applying specific inputs and checking outputs, formal tools use mathematical proof engines such as SAT solvers and model checkers to exhaustively prove that a design property holds for all possible inputs and all possible sequences of states. If the property can be violated, the tool produces a concrete counterexample showing exactly how. FV is done with properties, usually SystemVerilog Assertions (SVA) that define the expected behavior.

The leading tools are JasperGold (Cadence) and VC Formal (Synopsys). Formal verification shines for protocol compliance (e.g. the handshake signal is never asserted for more than 3 cycles), control logic correctness, and security properties (e.g. this register is reserved for software with elevated privileges). However, FV’s limitation is scalability. Formal engines hit capacity limits on datapath-heavy designs with wide buses. In practice, formal and simulation are complementary. FV proves critical properties exhaustively on targeted blocks, while simulation covers the full chip at statistical confidence.

5. RTL Freeze

To know when verification is complete, engineers look at several coverage metrics, a quantitative measure of what each testbench has exercised. There are two categories:

Coverage closure is the final step in the verification process. While 90% of test cases coplete quickly, ironing out the remaining 10% with functional coverage takes serious effort, sometimes requiring weeks to write targeted tests while adding or modifying constraints and exclusions in other tests. The more specific and complex a test case is, the more esoteric the knowledge is in whether a design might be susceptible. Design houses tap on their vast history of leanings from previous designs to help inform and prioritize the most important tests.

When all coverage goals are met and no open bugs remain at the target severity level, the project’s RTL is frozen. This formal milestone, known as RTL Freeze, signals that no more functional changes to the RTL are permitted. From this point forward, any modification must go through a formal process known as Engineering Change Order (ECO), requiring re-verification and equivalence checking. ECOs may be required in the design process at a late stage to fix a bug or tweak timings that were not caught earlier. RTL Freeze ensures that the next step, Physical Design, has a concrete base to work from, separating front-end design from back-end physical implementation.

While verification is often overlooked as the unglamorous side of chip design, it is critical to the development of new architectures. Designing a chip is easy. Knowing your design works with all possible scenarios is hard.

6. Firmware / Software Development

With the chip development process already taking years, software teams cannot afford to wait for silicon to arrive before beginning to write software for it. An operating system, firmware stack, and driver suite need to be substantially ready before the first chip comes back from the fab. To write software concurrently with hardware development, engineers rely on Pre-Silicon Hardware Emulation. The chip’s RTL design is mapped onto large arrays of FPGAs that execute the chips functions at 50MHz speeds. Programmable logic elements in the FPGAs are routed to roughly match the logical configuration of each design, enabling these emulators to run 1000x faster than pure software RTL simulation on a CPU.

The two dominant platforms are ZeBu from Synopsys and Palladium by Cadence. Synopsys’s latest ZeBu-200 clusters can emulate up to 23 billion gates and delivers up to 2x the runtime performance of its predecessor. Cadence’s Palladium Z3 can scale to design with up to 48 billion gates and is 1.5x faster than the Z2 generation. These systems allow firmware teams to boot Linux, test firmware, and conduct software validation months before silicon arrives.

7. Physical Design

Up to this point, the chip exists only in high-level RTL descriptions. Before Physical Design can take place, a crucial translation step must be done.

Logic Synthesis

Logic synthesis transforms RTL code into a gate-level netlist, a connectivity map of logic gates drawn from a foundry’s standard cell library. These synthesis tools parse the RTL code and determines the right combination of logic gates, connected in a certain order, to carry out the functions described by RTL. We will explain what these logic gates are below.

On top of this, the synthesizer optimizes the netlist and works within the limits set by the design. It balances timing (can the gates in this circuit complete the operation within 4 clock cycles?), area (how many gates can I squeeze into the area set out by the architectural description?), and power (how many watts worth of power loss from the dynamic and static leakage of these logic gates?). These conflicting goals are somewhat mitigated by techniques such as minimizing repeated logic, sharing logic gates across multiple functions and retiming functions to reduce the load on critical paths. The tool explores thousands of alternative implementations to find the best tradeoff between these demands.

The original and still dominant tool is Design Compiler from Synopsys, which established the entire category with multiple versions such as NXT and Ultra with greater integration and features Design Vision, a graphical interface for engineers to evaluate the synthesis flow. Cadence offers Genus as their synthesizer. Synopsys now pushes Fusion Compiler, which unifies synthesis with place and route in a unified flow to allow cross-probing between RTL, timing and layout. We cover these Unified EDA flows in more detail below.

Equivalence Checking

Once RTL has been synthesized into a gate-level netlist, you have to check that the synthesis tool did not introduce any bugs. To do this, the design is proven mathematically with Equivalence Checking, a formal technique that verifies two representations of a design (RTL and gate netlist) are functionally identical, input for input, output for output. Formality (Synopsys) and Conformal LEC (Cadence) are the standard tools.

Equivalence checking is run at every major translation step, not just after synthesis. Later gate-to-gate transformations such as clock tree insertion, scan chain stitching, routing optimization, and after every ECO. With each transformation comes a potential vector to introduce errors. Equivalence checking is thus the safety net that catches errors introduced by the tools themselves.

Logic Gates

The synthesizer selects from a range of logic gates in a standard cell library, with each logic gate carrying out a Boolean function, translating a given set of binary inputs into an output. The permutations between inputs and outputs are listed in Truth Tables as shown above. The seven basic logic gates are INV and NAND as shown above, AND, OR, NOR, XOR and XNOR. Transistors laid out in standard cells then carry out these operations in the real world, with the output signal voltage pulled up to Vdd for “1” or pulled down to Vss for “0”.

Standard Cell Libraries

The above gates are not designed from scratch. Instead, layouts are pulled from a standard cell library, a catalog of pre-designed, pre-characterized logic building blocks provided by the foundry or a third-party IP vendor that obeys all design rules. Each cell has a fixed height and a variable width, tiling neatly in horizontal rows across the die like bricks in a wall. This standardized, ordered layout with two sites for transistors (usually PMOS above NMOS) and standard pitches for power delivery is crucial to enable automated place and route.

Apart from the basic logic gates above, more complex standard cells are offered with advanced functionality. These combine logical operations into a higher level function that will be used repeatedly throughout a design. Those who work closely with the foundry may also design their own custom standard cells for these complex operations. Examples include (in increasing complexity), AOI, OAI, Registers, Multiplexers, D-Flip Flops and Full Adders. Special purpose cells are also featured throughout the design that deal with functions beyond logical computation. Examples include IO pad cells, Tap cells, Filler and Decap cells, Tie-high and Tie-low cells, power and clock gate cells and level shifters to carry signals over multiple voltage domains. SRAM Macro compilers are also included in each node that generates Static RAM blocks which are the densest memory cells in a logic process.

A modern standard cell library for an advanced node like TSMC N2 contains tens of thousands of individual cells. Each logic gate will have multiple layout options to choose from depending on wire routing and signal pin accessibility. Each gate will also come in several drive strength options, where higher drive strength cells are used when its output needs to drive multiple input signals. These are used selectively as higher drive current transistors have greater leakage power. Modern TSMC process nodes have upwards of 6 Threshold Voltage options to enable fine tuning of signal reliability and power. One the problems with Intel’s initial 18A was that there were only 3 VT options vs 6 at TSMC meaning even if Intel performs better at a given point, the design has a harder time being on the pareto optimal curve across all VTs. Intel’s 18AP fixes this.

Synthesis tools pick the right size for each cell based on timing and power constraints. With millions of permutations to choose from, EDA tools in both Synthesis and Physical Design are mandatory to navigate this optimization problem.

With Moore’s law slowing, newer nodes with novel area scaling techniques have exacerbated the layout complexity further. TSMC’s N3 FinFlex and N2 NanoFlex mix standard cells of different cell heights together, adding another layer of design constraints. Multiple library options are also offered for each node, such as HD high density and HP High Performance with different standard cell dimensions and power characteristics. Designers can mix and match different libraries for each region of a chip to extract the best PPA from the new process. Apple for example uses TSMC’s 3-2 FinFlex for its high performance CPU cores, with the rest of the die adopting the much denser and lower power 2-1 FinFlex library.

The standard cell library is the foundry’s main commercial interface with chip designers. It encodes not just logic functions but the foundry’s manufacturing capabilities, design rules, and process characteristics into a form that EDA tools can consume. When a fabless company “ports” a design to a new foundry, migrating the standard cell library is the first and most impactful step, and the one that triggers the most re-work across the entire tool flow.

Process Corners

Physical Design tools must also account for the effects of the real world, namely on variability in manufacturing. A cell’s speed and power consumption may shift significantly depending on variations in Process, Voltage and Temperature (PVT). Libraries are therefore characterized across a range of PVT corners as shown below.

TT (typical-typical) is the nominal case. FF (fast-fast) means both NMOS and PMOS transistors came out faster than nominal. SS (slow-slow) means both came out slower. FS and SF capture skew between NMOS and PMOS. These cases are particularly tricky because they affect circuit balance in ways that symmetric corners do not.

Voltage is varied around the nominal supply. For example, 0.75 V nominal with corners at 0.675 V (-10%) and 0.825 V (+10%). Lower voltage means slower switching and at lower power. Temperature ranges depend on the application. Consumer chips might cover 0°C to 105°C, while automotive-grade libraries must be characterized from -40°C to 125°C. Designs must meet timing at the slow corner and stay within its power budget at the leaky corner.

Metal Interconnect

Standard cells define the active transistor layer at the bottom of the metal interconnect. The lowest metals, M0 and M1, are used within cells to connect transistors to pins. These are thin, high-resistance wires, fine for short distances inside a cell but not for routing signals across the die. Each alternating metal layer has lines that are perpendicular in orientation, with odd layers for north-south routing and even layers for east-west routing.

Semi-global metals (M3 through M5) handle block-level routing, connecting standard cells to each other within a functional unit. These are progressively thicker and wider, reducing resistance for longer runs. The thickest top metals above are reserved for the power distribution network and global clock trees. The total number of metal layers depends on the complexity of the chip, from 10 layers in the low-cost mobile SoC to 19 layers in a high performance AI processor. Backside Power Delivery schemes such as Intel’s 18A above unlock another dimension for routing tools to utilize by segregating power and signal lines, reducing parasitic capacitance and signal interference.

Process Design Kit

All of the parameters above are detailed and provided by the foundry in a Process Design Kit (PDK). This dense package of files, models, and rules translates a foundry’s manufacturing process into terms that EDA tools can understand.

Each PDK comes with the following information, each having their own industry standard file formats:

GDS (GDSII Stream) layout parameters may also be provided, giving an example of the full, exact physical layout for every transistor and metal layer, including diffusion regions, polysilicon gates, contacts, and vias. This file is what ultimately goes to the foundry for tapeout.

A standard cell as seen in a GDS viewer. Each color represents a different fabrication layer. Polysilicon gate (red), metal interconnect (blue), and vertical contacts/vias between layers (blue squares).

PDK Versions

As the process is being developed, customers begin to design test chips and learn about the process characteristics. As such, PDKs go through many revisions on its way to version 1.0 for production tapeouts. The major PDK milestone releases are:

Between each major version, the foundry issues point releases that tighten design rules, narrows model uncertainty and adds DFM (Design for Manufacturability) hotspot rules discovered from real fabrication data.

PDK Licensing and Access

Foundry PDKs are universally NDA-protected, and access is controlled through a rigid tiered system that reflects both business relationships and IP security.

Even then, Tier 1 customers never see the foundry’s physical recipe. Chip designers see electrical characteristics, with no need to know the exact doping profiles, process temperatures, or etch chemistries. The PDK is an abstraction layer that makes the fabless model possible, protecting IP between foundry and customer.

PDK Timeline: Intel 18A

Open Source PDK

As foundry PDKs are closely guarded secrets, few will ever have the access to learn and use them. To date, only one production PDK has been released as fully open source. In 2020, Google partnered with SkyWater Technology to release the SKY130 PDK, a complete, manufacturable design kit for SkyWater’s 130nm process. SKY130, originally developed by Cypress Semiconductor, includes SPICE models, DRC/LVS decks, standard cell libraries, and IO cells. Everything needed to go from RTL to fabricated silicon, published on GitHub under an Apache 2.0 license.

While the technology is over 20 years old, it has been crucial for education and open source research as well as training open source models in Physical Design. The open-source EDA ecosystem includes OpenROAD for place-and-route and OpenLane for automated flow orchestration. Google has funded several multi-project wafer (MPW) shuttle runs, manufacturing community designed chips for free, enabling students to complete the entire RTL-to-GDSII flow without a single commercial license. Unfortunately Google pulled funding for these areas. Check out https://wafer.space.

Physical Design Tools and Functions

The two dominant physical design platforms are IC Compiler II (Synopsys) and Innovus (Cadence). Together, they handle virtually every advanced-node chip taped out today. The choice between them often comes down to which tool the team has the most expertise with. Decades of institutional knowledge built over dozens of projects make switching difficult and expensive. These tools take all of the parameters and considerations above along with the synthesized netlist and foundry PDK to generate a complete GDSII layout for tapeout. Engineers use these EDA tools for:

Physical design is done over multiple optimization loops. After each major step, the tools run timing optimization, inserting buffers on long or high-fanout nets, resizing gates (swapping a standard cell for a larger or smaller variant), applying intentional skew (shifting clock arrival times to help critical paths), remapping logic (restructuring gate-level topology), and hold fixing (adding delay cells to prevent data from arriving too early). The tools iterate through these techniques to arrive at the best PPA.

8. Signoff

Much like RTL design and verification is locked in with RTL Freeze, Physical designs undergo Physical Verification (PV) before Signoff locks the GDSII in place. Signoff proves design will actually work in both function and manufacturability, across every operating condition and power scenario. Synopsys offers IC Validator, Cadence offers Pegasus, and Siemens has Calibre. All three have deep foundry entrenchment that are indispensable for the fabless design houses in properly utilizing a given foundry process. The main components to clear before Signoff are:

Occasionally, ECO flows may have to be implemented to clear Signoff. A functional ECO fixes logic bugs by repurposing pre-placed spare cells, only requiring new masks in the lowest metal layer. A timing ECO, guided by PrimeTime, resizes or remaps cells to close timing, and may require a full mask set change. Each type trades flexibility for cost and turnaround time, the mechanics of which will be explained in the Steppings section below.

Unified Flows

Traditionally, each stage of the chip design flow operated as a silo. The RTL team wrote code and handed it to the synthesis team. Synthesis produced a netlist and handed it to place and route. They generated a layout and handed it to signoff. Each tool ran independently, creating a coordination nightmare when issues are found downstream that needed a fix from layout or even at RTL. These inevitable bugs would throw programs off by many weeks as these verticals are brought together to implement fixes. To address this, EDA vendors have unified the physical design process so every step is factored in concurrently rather than sequentially.

Synopsys Fusion Compiler was the first major answer to this problem. Built on a single data model, it unifies synthesis, place and route, and timing analysis into one engine. Cadence responded with iSpatial, which embeds the Innovus placement and optimization engines directly into the Genus synthesis tool. Unified flows bring the “Shift Left” philosophy. Have signoff-quality analysis earlier in the design process to avoid late surprises.

9. Tapeout

Once every signoff check passes, the design is exported as a GDSII or OASIS file, the industry standard formats describing the geometric shapes on every layer. This file is sent to the foundry, marking the milestone known as the tapeout, a term coined from the old days where GDSII files with sent on reels of magnetic tape. With this, the first mask set is designed. OPC (Optical Proximity Correction) algorithms modify the mask patterns with SRAFs (Sub-Resolution Assist Features) to compensate for optical distortion during lithography. Then mask making begins, followed by wafer fabrication.

10. Fabrication & Packaging

The fabrication process, from tapeout to first silicon arriving, typically takes 8-12 weeks. However, these first wafers can be expedited by purchasing Hot Lots, with higher priority wafer shuttles cutting down cycle times by several weeks. This allows post-silicon bring up engineers earlier access to begin debugging the new chip.

Before debugging can happen, silicon needs to be packaged to protect the fragile die and break out the tiny I/O bumps on the silicon surface to a reliable socketable package. On modern advanced processors, Heterogeneous Integration (chiplets) and Advanced Packaging come into play here. Multiple dies per package, 3D die stacking and 2.5D interposers such as TSMC’s CoWoS (Chip-on-Wafer-on-Substrate) technology are used to scale performance beyond the 26 x 33 mm reticle limit, the maximum die size that each lithography machine can expose.

11. Post-Silicon Validation & Bringup

Automatic Test Equipment (ATE) from the likes of Teradyne and Advantest assesses every single chip coming off the production line, applying thousands of test vectors generated earlier by ATPG tools. JTAG debug interfaces within the chip provide direct access for post-silicon debugging of errata, when behavior deviates from expectations. Initial bringup features multiple rounds of testing, as more bugs are found and workarounds are implemented by updating the original firmware that was developed on FPGA emulators. In production runs, this step is known as Final Test (FT), where functionality is verified with tests that return a simple pass/fail result.

To access every single wire coming out of the package, large breakout boards known as Probe Cards are used. Bringup engineers use them to isolate each signal pin from the package, hooking up oscilloscopes to the terminals to measure signal integrity and strength that meets design requirements.

EDA vendors also have a hand in this space, with Synopsys TestMAX family of post-silicon testing tools providing comprehensive coverage from ATPG during the design phase, yield diagnosis, DFT structures that test patterns run on, and software to run and manage chip testing and results logging.

Burn-in Test

The next stage in the qualification process is for thermal reliability with High Temperature Operating Life (HTOL) testing. This is an accelerated burn-in where chips are tested beyond the typical operating temperature, weeding out any defective chips that may not survive the rated operating environment and thermal cycles over the expected lifetime of the chip. Removing these “infant mortality” class of chips greatly reduces the chance of customers receiving units that may fail early on in the Bathtub Curve of reliability.

Test durations depend on the reliability requirements of each design, though on average range from 72 to 168 hours. Reliability critical applications such as aerospace and automotive may undergo up to 1000 hours of stress testing. Lower end, cost sensitive consumer devices may only have some chips from each production batch randomly selected for extended testing. Testing requirements are standardized by JEDEC, with JESD47 covering HTOL and JESD22 for package reliability under high humidity and temperature cycling.

Steppings

The first chip that comes back from the fab is known as A0 silicon. The ideal case is when A0 meets performance, power and reliability targets with no tweaks required for production silicon. However, sometimes the first chip doesn’t boot at all, and the fundamental flaw must be discovered. As such, most design teams budget additional development time to do multiple steppings, meaning the chip design is updated with new GDSII files sent to the fab for mask making and fabrication.

From Intel’s example above, multiple rounds of Engineering Sample (ES) versions are made, with a final Qualification Sample that is validated before production wafers are run. Steppings come with many codenames such as B1 or even E5 as was the case for the long delayed Intel Sapphire Rapids. Here’s a guide to what the stepping number means:

This circuit editing is done with advanced Focused Ion Beam (FIB) tools to add jumper wires or alter transistor interconnect patterns while keeping the chip fully functional so the fix can be tested. Designs can accommodate this process by having buffer or spare cells and metal lines that leave more wiggle room for fixing routing faults. As this process can be time consuming, it is mostly used to physically validate the layout changes that are planned to go into the next stepping.

Binning

The last step is speed binning, where each chip is tested at progressively higher frequencies, and those that pass at the highest clocks are sold as premium parts. This performance spread is the result of natural variability in manufacturing. While all the chips at this stage are fully functional, some may require slightly higher voltage than others to achieve the target operating frequency. Binning also encompasses yield harvesting, where some cores or sub-components in the chip may be defective. These are then fused off and sold at a discount with lower performance.

Binning has allowed Intel to create its product mix, the famous Core i5, i7 and i9 branding for CPUs, while NVIDIA GPUs almost never have all SMs enabled due to yield harvesting.

12. System Integration

Validated chips are then placed in reference boards and connected to devices such as storage and networking, with drivers, BIOS, and OS support qualified with System Level Testing (SLT). The chips are run through various stress testing benchmarks, and typical software applications are run to ensure end-user functionality.

Additionally, these boards are bundled with ES silicon and are sampled with partners and developer to receive early customer feedback. These Reference Validation Platforms (RVP) loop the customer into the development cycle. Major applications developers can start optimizing their software to take advantage of the new chips features and ensure day 1 software support upon the chip’s availability.

13. Production

Once the production ready stepping is qualified with satisfactory yield, the design can finally begin ramping up production and deployed with customers. However, the work does not stop. Failure analysis of defective silicon returned from customers can help iron out the last kinks in the design with minor revisions to the manufacturing flow. Design houses continue to work with TSMC as part of the Continuous Improvement Process (CIP) flows to help improve chip yields. FA engineers use tools such as Synopsys Avalon to map a given defect to the corresponding circuit schematic to identify the affected gates and wires in the design.

EDA in Foundry

Apart from the tools used by the fabless design houses, EDA also features extensively in the Foundry as they simulate and design the next generation of process nodes. On top of that, tight collaboration between foundry and fabless companies in the form of DTCO and STCO are used to squeeze even more PPA from the foundry process.

Technology CAD (TCAD) is the physics simulation layer that lets engineers design new transistor architectures entirely in software before spending tens of millions of dollars on silicon experiments. The dominant tool suite is Synopsys Sentaurus, which has two core engines working in sequence. Sentaurus Process simulates every physical fabrication step. Ion implantation, oxidation, thin-film deposition, plasma etching, lithography patterning and predicts the resulting 3D structure of the device down to nanometer-scale dopant profiles. Feed in a process recipe and it simulates the output device geometry. Sentaurus Device then takes that 3D structure and simulates its electrical behavior, with I-V curves, capacitance, leakage current and breakdown voltage. Together, they let process engineers iterate on a complex transistor designs, crucial in the Gate-All Around generation and beyond to CFET and novel materials such as Ruthenium, featured in our IEDM 2025 coverage.

Synopsys Mystic then takes the simulated transistor and extracts compact model parameters (industry-standard SPICE models like BSIM-CMG) from the output. The device simulations are used to form the earliest PDK 0.1 for circuit designers to get started months before any real silicon exists.

Zooming into the atomic frontier, Synopsys QuantumATK is used in materials research, simulating at the scale of individual atoms. It uses density functional theory (DFT) and non-equilibrium Green’s function (NEGF) methods to model quantum transport and electron tunnelling to for material interface engineering. This is particularly useful in Work Function Metallization schemes that precisely control the transistor threshold voltage.

Design Feedback Loop

Silicon measurement data such as transistor characteristics, IR drop and yield maps of chips made on the current process node feed directly back into the foundry’s next-generation PDK development and the design team’s planning for future node updates. Best Known Methods (BKMs) are refined over the lifetime of the node to help customers improve yields and lower costs for the foundry. This forms a productive feedback loop where customers help define the next generation of foundry processes.

DTCO: Blending Process and Design

For decades, chip manufacturing followed a strict handoff: process engineers developed the transistor technology, characterized it, then “threw it over the wall” to chip designers. The designers worked with whatever they got. Design-Technology Co-Optimization (DTCO) demolishes this wall. It evaluates process options using chip-level PPA (Power, Performance, Area) metrics from the very first day of process development.

The DTCO flow chains the entire tool stack into a feedback loop: Sentaurus TCAD (device physics) → Mystic (PDK) → SiliconSmart/HSPICE (cell characterization) → IC Compiler II/StarRC/PrimeTime (chip-level PPA evaluation), where the results feed back to the process engineers. Process knobs are tweaked to push design rules beyond the conservative cookie cutter standard cells. not in isolation, but by its impact on a real chip design. The biggest companies such as Apple, NVIDIA and AMD have dedicated foundry departments that work closely with TSMC to engineer their own custom cell libraries to push PPA by as much as 15% over standard libraries. Routability improvements translate directly to lower delay, higher performance, lower power and even reduced area from higher area utilization in congested routing zones.

New transistor schemes such as TSMC’s FinFLEX and NanoFlex make DTCO almost mandatory to effectively utilize alternating rows of transistors with significantly different performance and leakage characteristics. Backside power designs with Intel 18A and TSMC A16 expose another dimension of signal and power routing, potentially unlocking new standard cell routing scheme that route around both sides of the transistor layer.

STCO: Co-Optimizing the Entire System

System-Technology Co-Optimization (STCO) extends the DTCO concept up one more level: from chip and process co-design to system and package co-design. It addresses chiplet partitioning decisions, packaging technology selection, cross-chiplet bandiwidth and latency trade-offs, multi-die thermal management and power integrity across the entire package. As single-die scaling hits economic and physical limits, STCO is how the industry keeps delivering generational performance gains.

Intel’s Ponte Vecchio GPU is the poster child for a chip that required extensive STCO. It integrates 47 active dies manufactured across five different process nodes, stitched together using both EMIB (Embedded Multi-die Interconnect Bridge, a 2.5D silicon bridge) and Foveros (Intel’s 3D face-to-face die stacking). However, the chip faced several design challenges and was delayed for years, with final performance coming nowhere near original targets. With proper STCO and sound design practices, complex designs can now come out on schedule and on target, which is what AMD plans to do with their MI455X GPU releasing later in 2026.

Below, we discuss the reality of chip design in the real world, and the challenges engineers face from increased design complexity while pushing for compressed design cycles. We also take a look at the hardware that EDA tools run on in simulation and emulation.

The Ugly Reality of Chip Design in the Real World

A Brief History: From X-Acto Knives to the Big Three

The Big Three Emerge

The Chip Design Waterfall

1. Planning

2. Architecture Layout

3. RTL Design

Signal Timing

State Elements

Writing the RTL

IP Integration

4. RTL Verification

The Verification flow takes two paths: Standard DV testing on one end, and Formal Verification with proofs on the other.

UVM Testbench

Formal Verification

5. RTL Freeze

To know when verification is complete, engineers look at several coverage metrics, a quantitative measure of what each testbench has exercised. There are two categories:

6. Firmware / Software Development

7. Physical Design

Up to this point, the chip exists only in high-level RTL descriptions. Before Physical Design can take place, a crucial translation step must be done.

Logic Synthesis

Equivalence Checking

Logic Gates

Standard Cell Libraries

Process Corners

Metal Interconnect

Process Design Kit

Each PDK comes with the following information, each having their own industry standard file formats:

PDK Versions

PDK Licensing and Access

Foundry PDKs are universally NDA-protected, and access is controlled through a rigid tiered system that reflects both business relationships and IP security.

PDK Timeline: Intel 18A

Open Source PDK

Physical Design Tools and Functions

8. Signoff

Unified Flows

9. Tapeout

10. Fabrication & Packaging

11. Post-Silicon Validation & Bringup

Burn-in Test

Steppings

Binning

Binning has allowed Intel to create its product mix, the famous Core i5, i7 and i9 branding for CPUs, while NVIDIA GPUs almost never have all SMs enabled due to yield harvesting.

12. System Integration

13. Production

EDA in Foundry

Design Feedback Loop

DTCO: Blending Process and Design

STCO: Co-Optimizing the Entire System

The Ugly Reality of Chip Design in the Real World

A day in AI now feels like a year in any other industry. Model releases, software breakthroughs, and hardware improvements are compressing multi-year cycles for any other industry into weeks. Over just the past few months, agentic AI has crossed a real inflection point, driving a step-change in the value of tokens while software and hardware improvements have sharply reduced the cost of generating them.

This flood of demand is driven by end users enjoying a huge return on investment (ROI) from consuming tokens, and this demand growth is arguably only in its early innings. This year Anthropic’s ARR has exploded from $9B to over $44B today, their gross margins on their inference infrastructure have increased from 38% to over 70% over the same period.

This rapid pace of AI adoption has created value across the stack, but the unique phenomenon is that the AI labs are capturing all the value now, from almost none last year.

End users are enjoying a productivity bonanza, tasks that used to take tens of person-hours costing thousands of dollars can now be accomplished in minutes with a just a few dollars’ worth of tokens. This huge surge in revenue and margins is because the value of tokens being created is dramatically improving businesses. For example, SemiAnalysis has reached as high as $10.95 million dollar annual spend rate on Anthropic Claude tokens, but the value we derive allows us to outcompete all our competitors and gain market share.

New chips such as Blackwells can generate 30x more tokens per second while running frontier workloads today vs Hoppers a year ago, and ASICs such as TPUv7 and Trainium 3 show similar improvements. Inference providers such as Fireworks, Baseten, Fal, margins are widening while their revenue trends are in hyper growth.

Even parts of the hardware stacks have repriced, with memory prices having gone up 6x in the past year. Neocloud GPU rental pricing is surging as well, up with 1-year H100 rental contract prices up 40% from the bottom in October 2025.

There are two firms in the industry with incredible pricing power that haven’t moved much though. TSMC and Nvidia have not reacted to the recent boom in value generation of AI models.

In this article we will explore where value from AI is accruing - from end users to inference providers, Neoclouds as well as hardware providers. We will unveil how TSMC and Nvidia are now venting vast value into every vertical of the ecosystem.

Finally - we introduce a new framework: the “One Chart to Rule Them All” that explores GPU Rental Economics and analyzes whom among the end users, the Neoclouds/Hyperscalers and the AI System suppliers are capturing the most value in the AI ecosystem.

AI Value Profit Pools

From 2023-2025, all the value in AI was captured by the infrastructure layer. Nvidia had their first blockbuster earnings call in May 2023 and jumped 25% after hours, officially marking the start of the AI trade. In 2024, Vistra and GE Vernova were two of the top performing stocks in the S&P 500 (+265% and +146% respectively) as everyone realized power was becoming the key bottleneck. In 2025, memory stole the show, with SanDisk, Western Digital, Seagate, and Micron all posting 200%+ gains on the year. These are all sweeping generalizations of course and many other infra names have significantly outperformed thanks to increased AI capex. Those interested in all the granular details should subscribe to our institutional products.

During this same period, gross margins for all the model creators and inference providers were famously bad. For most, the actual utility of AI still only amounted to slightly better Google search locked behind a chat interface and Studio Ghibli style selfies. Skeptics loudly proclaimed that there was simply no way AI could ever deliver on the trillions of planned capex.

Agentic AI Has Changed the Game

The world changed in December 2025, when Agentic AI began to really work. SemiAnalysis has written and talked extensively about our Claude Code usage, but it is important to emphasize that agentic AI is no longer limited to just coding. Our analysts are using agents every day to convert excel models into dashboards, create charts for all our notes, build financial models and analyze company earnings, and much more. These are all tasks that either 1) we simply wouldn’t have been able to do before or 2) would’ve previously taken our junior analysts many hours, taking them away from far more value added tasks.

The table below shows a handful of real examples from our own workflows, comparing token spend against what the equivalent human labor would have cost:

Annualized token spend at SemiAnalysis is already ~30% of employee compensation and we’re consuming just under 5B tokens per month per employee (over 5x more than Meta!). This is power law distributed though, so there are team members running over 100B tokens a month. It’s obvious that this is still just the beginning, and that all white-collar enterprises will soon embrace agentic AI.

Within the past few months, the value of each token has clearly increased. We estimate that the true blended price per million tokens for running Opus 4.7 on agentic tasks at $0.99 despite the sticker price being $5/$25 per MTok. Agentic workloads have extremely high input-to-output ratios (our Claude Code usage has a ratio of about 300:1) and high cache hit rates (90%+). Because cached input tokens only cost $0.50/MTok, most of the tokens end up in the cheapest tier. We walk through the full methodology here.

When framed this way, it’s no wonder why Anthropic ARR has exploded from $9B to potentially $44B+ YTD.

Tokens Are Getting Cheaper to Produce

At the same time, the cost of producing each token has plummeted. This is the largest driver of value accretion to inference providers, and it is a key reason for the sharp increase in margins at large AI Labs.

Cost of production for token has fallen sharply because increases in accelerator pricing generation-over-generation have been more than offset by much higher throughput (tokens/sec/gpu). Average blended price per million tokens has fallen dramatically over the past few months, agentic workloads are inherently multi-turn with longer input/output ratios and higher cache hit rates,but inference margins have gone up from < 40% to > 70% in the same time frame. For in-depth estimates on true blended price per million tokens, token production volumes, and gross margins for all the major models from OpenAI, Anthropic, and more, see our Tokenomics model.

InferenceX remains the best benchmark for tracking real-world inference performance over time for open source models given both hardware and software improvements.

The following chart shows throughput vs interactivity for B300s running DeepSeek R1 on 8k input tokens to generate 1k output tokens. The top line reflects token throughput with wideEP + disagg + MTP, the middle reflects wideEP + disagg and he lowest line is without any of the three software optimizations. The gap is startling with the same B300 able to yield~1k, ~8k, and ~14k tokens/sec/gpu on the same hardware. One can 14x throughput with software improvements alone.

If you factor in hardware improvements as well, then the difference is even more pronounced. The most optimized GB300 NVL72 configuration achieves ~17x higher throughput than the most optimized H100 configuration in FP8. If we switch to FP4, which Hopper doesn’t natively support, the difference jumps to 32x. Remember that the total cost of ownership per GPU is only ~70% higher for GB300 vs H100.

Model Provider Margins Will Continue to Increase

Many were surprised when Anthropic released Opus 4.5 at a price of $5 per million input tokens and $25 per million output tokens in late November 2025. Previous Opus models such as 4 and 4.1 (released May 2025 and August 2025 respectively) were priced 3x higher at $15/$75.

However, we think Anthropic’s margins have actually increased on Opus tokens despite the lower ASP thanks to software improvements across Trainium and Nvidia GPUs as well as replacing Hoppers with Blackwells.

Anthropic’s margin expansion so far has come from cost reductions; they can generate the same tokens for cheaper. Despite the Opus price cut, their ASP/token has also actually gone up because most of the volume shifted from Sonnet to Opus.

Even if XPU providers start dramatically raising prices to better capture their share of the throughput improvements, Anthropic still has another lever to pull to further expand margins: they can continue shifting volume to more expensive SKUs.

As mentioned earlier, the gap between the price of a frontier-level token vs the economic value of the work that can be produced by said token is the largest it’s ever been. Anthropic can either re-up the price of the base Opus family or introduce new products. We already saw the latter with Opus fast being priced 6x higher than regular Opus, and Mythos being announced at $25/$125 (5x regular Opus pricing). Both these SKUs are higher margin than regular Opus, yet the most AI-pilled businesses are still more than happy to pay the increased prices because the productivity gains outweigh the cost. If Anthropic let us pay $150/$750 for Mythos fast, we would.

The age of low gross margins for frontier model providers is over. Real agentic AI has permanently increased the market-clearing price per token, and there’s no going back.

Why Model Provider Profits Won’t Get Competed Away

The most obvious argument for why the labs won’t be able to capture higher margins despite increased utility per token is competition. However, we don’t think this is how things will play out for two reasons.

First, it’s become clear that the frontier model maintains pricing power. Regardless of what the benchmarks may say, open-source models are still noticeably worse than their closed source counterparts for real knowledge work, and there’s no reason to believe the gap will close any time soon. Kimi K2.6 ($0.95/$4) exerts very little downward pressure on Opus pricing.

Second, compute constraints means that no single frontier lab will be able to serve the entire market. Anthropic is already beginning to alienate large swathes of the market today by locking Claude Code behind a $100+/month subscription and blocking third party harnesses like OpenClaw. Token demand will far outstrip supply for the foreseeable future, which means any lab capable of providing true frontier quality will be able to charge based on the economic value delivered by the token rather than competing away each other’s margins.

Agentic AI Hits the Market, but TSMC and Nvidia Haven’t Flinched

Despite the repeated emphasis on agentic AI during Jensen’s most recent GTC keynote, Nvidia and TSMC still have not fully internalized how transformative the past few months have been for token economics. We already saw Nvidia underestimate Blackwell’s performance-per-dollar improvements based on Jensen’s reaction to InferenceX, and it now appears they have also underestimated how quickly frontier tokens would appreciate in value.

Nvidia is still operating within a framework shaped by prior assumptions, where the willingness to pay per unit of compute declines over time. That assumption no longer holds. The market has shifted materially, driven by the explosion of agentic workloads and a sharp increase in token consumption per workflow. Demand is no longer linear. It is compounding.

Demand, however, continues to accelerate. Anthropic’s ARR has reportedly reached $44B+, up from $30B in our last update, while open-weight models such as GLM and Kimi are expanding the addressable compute base. Capital raises across AI labs and neoclouds are translating directly into incremental GPU deployments.

At the same time, compute supply remains structurally constrained. Upstream bottlenecks in memory and leading-edge wafers continue to limit availability, with N3 utilization expected to exceed 100% in the second half of 2026 and DRAM fabs already running above 90% utilization. There is no meaningful relief in sight.

TSMC could raise prices materially, but they haven’t. This is a stategic error on their part. If not increasing prices, they could at least demand larger prepayments.

The current dynamics within the compute market suggest that if current trends continue, the value generated by the overwhelming token end demand will continue accruing to AI Labs, Hyperscalers, Inference Providers, Neoclouds and Memory Vendors.

AI labs are capturing a disproportionate share of the value being created, driven by strong end demand, rising token monetization, and increasingly favorable unit economics. At the same time, Nvidia’s pricing framework has not fully adjusted to reflect this shift, even as its hardware remains the critical bottleneck enabling that value creation. Despite rising token monetization and increasingly favorable unit economics, Nvidia compute is still the bedrock for enabling that value creation.

Demand for Nvidia systems remains extremely strong across all tiers, with buyers willing to lock in long-term contracts and accept higher pricing to secure capacity. Even with alternative hardware options, Nvidia retains a clear advantage in ecosystem maturity, software stack, and deployment reliability. For many workloads, especially at the frontier, substitutes are not yet fully interchangeable.

Rubin, set to launch in 2H26, sits at the center of these dynamics. It delivers a step-function improvement in performance but also embeds a much larger memory subsystem at a time when memory is the tightest constraint in the supply chain. DRAM pricing has already moved sharply higher and is likely to remain elevated, making memory the primary driver of system cost.

In this context, Nvidia has room to increase pricing, particularly for systems like Rubin that deliver step-function performance gains. The incremental value created at the system level far exceeds the incremental cost, especially when viewed through $/FLOP or end workload economics.

This creates a clear disconnect. The market has structurally shifted, with demand scaling faster and more persistently than supply can respond. Yet Nvidia’s pricing framework remains anchored to prior assumptions, rather than adjusting to reflect the increased value its systems now deliver.

Put simply, even if Nvidia raises server pricing and infrastructure providers increase compute pricing, demand would remain intact. Buyers are optimizing for access to compute, end users are optimizing for access to as much tokens as possible, and both are securing capacity at all costs - marginal cost optimization is not their primary concern today.

SOCAMM Pricing: Nvidia’s Next Margin Lever

The next question after whether Nvidia can raise prices is where within the system it is most effective to do so.

At the system level, memory is the most natural point of control. Rubin-class systems embed significantly more memory into an already constrained supply chain, and unlike compute, memory can be more cleanly segmented and continuously repriced.

This is because memory on VR NVL72 is a socketed LPDDR-based memory solution called SOCAMM (System-On-Chip Attached Memory Module). SOCAMM is designed for Nvidia’s rack-scale systems, enabling higher capacity, modularity, power efficiency, and independent pricing of memory alongside compute.

This makes SOCAMM one of the most important variables in understanding Nvidia’s pricing strategy. Two factors ultimately determine system-level pricing outcomes: the cost Nvidia secures for SOCAMM, and the markup applied when reselling that memory to customers. Developing a precise view of Nvidia’s pricing and BoM for its rack-scale systems is not an easy task, given the complexity of its “extreme co-design” approach and intricate supply chain dynamics.

This is why SemiAnalysis provides an industry-leading breakdown through our VR NVL72 BoM and Power Budget Model. Furthermore, there are two swing factor at play when determining memory pricing to end customers:

As of today, our Memory Model implies SOCAMM contract pricing paid by Nvidia at ~$8/GB in 1Q26, a sharp step-up from 4Q25 to 1Q26. This jump was driven by the broader LPDDR5X pricing surge in 1Q and overall memory supply tightness. We anchor this estimate based on two points:

Industry chatter suggests that Nvidia has secured substantial SOCAMM volume for both its GB300 NVL 72 and VR NVL72 systems, under a long-term agreement (LTA) format, which we outlined in our institutional note for Memory Model earlier. As the only scaled SOCAMM customer today, and arguably the most critical buyer in memory, Nvidia likely benefits from preferential access and pricing, and we believe Nvidia’s past track record speaks for itself when it comes to its ability to leverage the supply chain.

That said, broader DRAM pricing dynamics should still inevitably flow through. Further price hike in mobile LPDDR5X pricing in coming quarters should still be a critical pricing reference for SOCAMM, and SOCAMM should reprice accordingly given limited LPDDR5 allocation volume. We believe exit ’26 pricing for SOCAMM could exceed $13/GB, which is roughly in line with mobile DRAM pricing expected by the end of this year; accordingly, we view ~$10/GB as a reasonable assumption for Nvidia’s SOCAMM cost.

One key question some may raise is: On what basis should customers accept further price increases and margin expansion from Nvidia, and what rationale can Nvidia credibly use to justify such a position? We think it is reasonable for Nvidia to charge 60% margin on SOCAMM for three reasons:

Capex Per Watt Trends from GB300 to VR NVL72

For GB300, DRAM was bundled into the board and marked up at ~75% gross margin, making the margin charged on the memory on the board consistent with what is implicitly priced for the Blackwell systems.

For Rubin, we initially assumed the same dynamic, with the understanding that Nvidia would target an overall system Gross Margin in the mid-70s. As such, our initial Bill of Material (BoM) modeling applied a consistent margin throughout the entire Strata board leaving SOCAMM margin at the same mid 70s margin.

However, because SOCAMM2 is a socketed module in Rubin whereas GB300 uses an ordinary LPDDR5X module that is soldered onto the board, memory can be disaggregated and quoted separately from the base system. This allows Nvidia to explicitly price memory as its own line item, rather than embedding it within board-level pricing. Importantly, this also introduces an additional value for Nvidia to adjust margin on the SOCAMM2 while keeping margin on the board the same. Even if Nvidia initially absorbs some of the memory cost inflation, it retains the ability to offset this by charging higher system-level margins to customers.

Hence, we would have expected overall capex per watt to rise as we transition from GB300 to VR NVL72. Yet to the contrary – current pricing only works out to a slight creep up in capex per watt from $37.4/W for GB300 to $38.1/W for VRNVL72. This is despite chip TDP almost doubling from GB300 to VR NVL72 (1400W to 2300W), and a material increase in FLOPs.

This is unusual relative to broader trends in server capex per watt. Across AMD, Nvidia, and custom ASICs, capex per watt typically increases generation over generation as improvements in performance per watt allow vendors to capture more value at the system level. Thus, it is puzzling to us that $/GW appears to remain largely stagnant from GB300 to VR NVL72. This is even more unusual given the step up in performance/W from GB300 to VR NVL72 is more than double.

Nvidia also has the opporunity to price discriminate on memory more than they do on the GPU because memory isn’t an anti-trust concern whereas the GPU is.

Networking as a Vector for Price Discrimination

Today, Nvidia does not heavily differentiate on GPU pricing across customers within a given strata. Core components are generally sold at similar prices across the ecosystem, be it to Hyperscalers, Neoclouds, Emerging Neoclouds, sovereigns or enterprises. This creates a relatively uniform pricing structure for the core GPU and memory components, even in a market where willingness to pay varies significantly.

While GPU pricing is relatively uniform, Nvidia traditionally price discriminates on networking equipment, offering Neoclouds and other marginal cloud players a price point that is at significant premium to hyperscalers. We conducted our own survey with GPU cloud providers and found that, for instance, the SN5610 could be priced 2x more for a Neocloud as opposed to for hyperscalers. Hyperscalers clearly have stronger bargaining power but this is not because they are buying more switches and transceivers from Nvidia.

Neoclouds lack the scale and networking expertise to customize and cost-optimize their networking clusters and so they ultimately prefer Nvidia’s turnkey solutions. Hyperscalers work directly with OEMs and ODMs and have the networking engineering bench to deploy more cost effective solutions that may not be turnkey deployments and thus are a heavier lift to deploy properly.

The disparity in networking costs between a Neocloud and hyperscaler becomes less significant, however, on a total cluster capital cost basis. For two comparable clusters, a 94% increase in networking cost for a Neocloud versus a hyperscaler translates to only to a 10% increase in all-in capital cost for a full rack-scale server. This excludes other variables such as power, utilities and operations, which will further erode cost differences attributable to networking equipment pricing disparity.

Though this is a great case study demonstrating how Nvidia is pricing its solutions to value, the current price gap between Neoclouds and Hyperscalers is meaningful, leaving limited room for Nvidia to pull this lever further.

Nvidia as the Central Bank of AI

One explanation behind Nvidia restraint’s in pricing thus far might be a combination of regulatory and strategic reticence.

Nvidia’s position in the AI compute stack is already under increasing antitrust scrutiny, given its dominance across GPUs, interconnect, and software. In this environment, aggressively repricing systems to fully capture the value delivered risks drawing further attention, particularly if it results in outsized margin expansion while downstream AI labs are also generating significant profits. Holding pricing closer to prior frameworks can help avoid signaling excessive pricing power in a supply-constrained market.

This behavior is not without precedent. TSMC has historically taken a similar approach. Even while operating at full utilization and acting as the bottleneck for advanced-node supply, TSMC has generally avoided fully pricing to scarcity. Instead, it has prioritized long-term relationships and ecosystem stability over extracting maximum short-term margins, in part to avoid regulatory and customer backlash.

Nvidia appears to be following a comparable path. Rather than fully repricing Rubin systems to reflect both the increase in performance and the structural shift in memory costs, it is maintaining a more measured pricing approach. This balances margin expansion against regulatory risk, ecosystem dynamics, and the need to avoid accelerating customer diversification toward alternative compute platforms.

We made a similar point in our Nvidia as the central bank note. Nvidia is actively supporting the development of the broader ecosystem, ensuring long-term demand expansion rather than maximizing near-term extraction. Today, frontier labs benefit from Nvidia’s software-driven efficiency gains, but these improvements are not fully monetized at the hardware level. As a result, incremental value continues to accrue downstream despite Nvidia being the primary enabler. By taking the oxygen out of the room – Nvidia aims to ensure it remains the main protagonist in the AI era for the foreseeable future.

Yet - with Compute demand well over compute supply, why should those in control of the scarce resources not capture higher pricing and enjoy greater profitability?

TSMC, The Fairest and Most Just Company In The World

We’ve said that TSMC’s N3 capacity is even tighter. All major accelerator roadmaps have now converged on the N3 process node for this year and next year. Nvidia, Broadcom, Annapurna, MeidaTek and AMD are all fighting for more N3 wafer allocation from TSMC so that they can ship more compute to their customers. While N3 capacity is arguably the tightest constraint in the system, pricing remains relatively stable.

TSMC strategically looks to protect profitability through downcycles. The flipside is that this policy also blunts upside during upcycles. Regardless, TSMC is certainly leaving value on the table with all their major fabless customers enjoying very high gross margins that could be transferred over instead to TSMC.

However, TSMC could very much take a more aggressive posture on pricing, and customers would not only accept it but we would argue some would even welcome it. Nvidia would love to pay more for wafers if it means shutting out their competition who have less ability to pay up. After all, Jensen himself said in 2024 that TSMC should charge more for wafers and he meant it for this reason.

TSMC can also take a position of longer term agreements with guaranteed capacity commitments and prepayments in liu of major price increases. This is the more likely path.

Its greatest strength in this environment is procurement. Nvidia has secured disproportionate access to constrained upstream supply, particularly TSMC wafers, allowing it to serve demand that others cannot.

AI compute buyers such as Anthropic are therefore forced into Nvidia’s ecosystem, as alternative capacity from TPU and Trainium remains limited by the same upstream bottlenecks. Despite this structural advantage, Nvidia is not fully reflecting it in pricing.

For now, pricing for Nvidia remains anchored to cost-based frameworks. But this is unlikely to hold. As the return on investment for inference providers becomes clearer and more widely accepted, the focus will shift even more towards pricing to value. This reduces the scrutiny on pricing and gives GPU infrastructure providers room to move from cost-based to value-based pricing. Once that transition occurs, it creates space for Nvidia to move pricing higher and capture more of the value delivered at the system level – the manifestation of the pie growing.

Triangulating VR NVL72 Rental Pricing: Cost-Based vs Value-Based Approaches

The Cost-based pricing approach starts with the premise that GPU deployments will only occur if these projects they meet a minimum return threshold for Neoclouds. If returns fall below this level, capacity will not be deployed until pricing adjusts to meet that hurdle.

Therefore, the rental price charged under a cost-based framework is the price that earns the Neocloud a project IRR above the minimum hurdle rate for deployment. Most projects today tend to earn a return of mid-to high teens IRR. An illustrative GB300 deployment today will likely have a project IRR of 15.6% over a 5-year period with a 15% prepay.

Neoclouds will aim for a similar IRR when deploying VR NVL72, which in turn determines what debut GPU rental prices for Vera Rubin might look like. With our all-in server cost for VR NVL72, a rental price of at least USD 4.92 per Hour per GPU rental price is required for a 5-year project with a 15% prepay to achieve the same project IRR hurdle of 15.6% that most GB300 projects use.

Our second framework looks at value-based pricing. We anchor on the $/FLOP implied by existing SKUs and ask what that would translate to for Rubin. This represents the theoretical maximum a renter of compute would be willing to pay to remain indifferent between Rubin and current generation GPUs - and therefore serves as the ceiling for GPU rental pricing. Here - we look to the trend in improvement of rental prices per PFLOP.

For training workloads, we anchor on GB300 pricing by comparing rental cost per PFLOP on a marketed FP8 dense basis. Using a current 5-year GB300 rental price of ~$0.70 per PFLOP, we derive a VR NVL72 ceiling price of approximately $12.25 per GPU hour at parity.

The VR NVL72 price per TCO stands out in that, unlike for the GB300 and prior cards, there is an extremely large gap in Value-based and Cost-based pricing. If we are conservative and select a point slightly below the trend line - for instance a rental price of $0.55 per PFLOP - this would corresponds to $9.63/hr/GPU, nearly double the minimum rental price of $4.92/hr/GPU needed to cross Neoclouds’ return hurdle.

One Chart to Rule Them All

The cost-based approach forms the floor for GPU rental pricing – below this rental price, Neoclouds will not greenlight new GPU projects. The value-based approach forms the theoretical ceiling for GPU rental pricing – no customer would pay higher on a $/FLOP basis to rent a newer generation GPU.

We combine both constraints as well as a pricing curve that illustrates the returns to the Neocloud for given GPU rental prices to create one pricing chart. This “One Chart To Rule Them All” also acts as a framework to understand competitive dynamics and pricing power.

At the start of the article – we posed the question: Who are the benefits of strong AI demand accruing to?

This question can be answered by plotting observed GPU Rental Prices charged by Neoclouds and the IRRs earned by these projects. Sliding up and to the right along the orange curve in the chart below represents stronger Neocloud bargaining power: The Neocloud is able to charge higher GPU Rentals and earn well above their IRR hurdle rate.

If Nvidia increases pricing for VR NVL72, the pricing curve shifts up and to the left. This is because a higher rental price is needed to offset this higher system cost while still earning the same IRR from the Neocloud perspective. This shift represents stronger bargaining power enjoyed by system suppliers like Nvidia.

The top left corner, where the blue maximum rental price/PFLOP and the beige Neocloud Project IRR minimum hurdle intersect, represents the maximum theoretical AI Cluster pricing. If the system is priced any higher, Neoclouds and end users would be better off just purchasing or renting GB300s. The larger the gap between the current pricing curve and the top left corner, the more room there is for AI Cluster providers like Nvidia to increase system pricing.

At today’s VR NVL72 system pricing, Neoclouds can charge $4.90/hr/GPU for a 5-year contract while still earning the same 15% IRR as they do on their GB300 projects. For customers – Rental Price per PFLOP works out to $0.28/PFLOP, a 60% drop in cost per PFLOP vs the GB300 NVL72, an improvement in cost that is well below trend.

This suggests that there is meaningful room for Nvidia to increase server prices. A ~40% increase in server pricing would deliver below trend cost improvements in price per FLOP, while still leaving Neoclouds enough room to lift prices even higher so they can earn higher IRRs. Even if Neoclouds adjust pricing higher to slide along the grey curve, for instance charging $8.00/hr/GPU and earning a 38% IRR, corresponding to a cost of $0.46/PFLOP, which is still an improvement that is below trend.

It is important to point out that this analysis has mainly focused on Rental Price/FLOP - but improvements in inference performance per TCO have been accelerating at an even brisker pace. Though we have yet to benchmark a VR NVL72 system at InferenceX, it is highly likely that there is an even sharper pace in cost decreases when it comes to dollars per token delivered by VR NVL72, meaning there could be even more headroom for Nvidia to capture more value from the overall ecosystem.

VR NVL72 vs GB300 Performance per TCO

This rapid pace of AI adoption has created value across the stack, but the unique phenomenon is that the AI labs are capturing all the value now, from almost none last year.

There are two firms in the industry with incredible pricing power that haven’t moved much though. TSMC and Nvidia have not reacted to the recent boom in value generation of AI models.

AI Value Profit Pools

Agentic AI Has Changed the Game

The table below shows a handful of real examples from our own workflows, comparing token spend against what the equivalent human labor would have cost:

When framed this way, it’s no wonder why Anthropic ARR has exploded from $9B to potentially $44B+ YTD.

Tokens Are Getting Cheaper to Produce

InferenceX remains the best benchmark for tracking real-world inference performance over time for open source models given both hardware and software improvements.

Model Provider Margins Will Continue to Increase

The age of low gross margins for frontier model providers is over. Real agentic AI has permanently increased the market-clearing price per token, and there’s no going back.

Why Model Provider Profits Won’t Get Competed Away

Agentic AI Hits the Market, but TSMC and Nvidia Haven’t Flinched

TSMC could raise prices materially, but they haven’t. This is a stategic error on their part. If not increasing prices, they could at least demand larger prepayments.

SOCAMM Pricing: Nvidia’s Next Margin Lever

The next question after whether Nvidia can raise prices is where within the system it is most effective to do so.

Capex Per Watt Trends from GB300 to VR NVL72

Nvidia also has the opporunity to price discriminate on memory more than they do on the GPU because memory isn’t an anti-trust concern whereas the GPU is.

Networking as a Vector for Price Discrimination

Nvidia as the Central Bank of AI

One explanation behind Nvidia restraint’s in pricing thus far might be a combination of regulatory and strategic reticence.

Yet - with Compute demand well over compute supply, why should those in control of the scarce resources not capture higher pricing and enjoy greater profitability?

TSMC, The Fairest and Most Just Company In The World

TSMC can also take a position of longer term agreements with guaranteed capacity commitments and prepayments in liu of major price increases. This is the more likely path.

Triangulating VR NVL72 Rental Pricing: Cost-Based vs Value-Based Approaches

One Chart to Rule Them All

At the start of the article – we posed the question: Who are the benefits of strong AI demand accruing to?

VR NVL72 vs GB300 Performance per TCO

Since we called out the Claude Code inflection point on February 5th, we have seen a flurry of model releases. Opus, Mythos, Codex, Gemini, DeepSeek, Kimi, Qwen, GLM, MiniMax, Composer, Muse Spark, and more. Today we will break down all of these major model releases, explain when you can vs can’t trust the benchmarks, and give our predictions for the future of the agentic coding market.

First we have to highlight GPT-5.5 from OpenAI. In our view, GPT-5.5 is now materially better at some tasks than all other models. We believe that GPT-5.5 has arrived at the frontier. This is a huge change from November when Opus 4.5 was released. At that time, and for the 6 months since, OpenAI’s coding model was not world class in most metrics, leading to Opus being our daily driver. GPT-5.5 is now integrated in our daily work.

Meet the Models

There’s been at least one major lab releasing a new checkpoint purpose-built for coding every week for the past 3 months. GLM-5.1, Qwen3.6-Plus, Kimi K2.6, Composer 2, and Gemini 3.1 Pro all emphasize “agentic coding,” “long-horizon tasks,” or similar capabilities in their headlines. February was a particularly busy month.

New checkpoints are cool, but entirely new pre-trains are what really get the people going. Heading into April, the San Francisco rumor mill was ablaze with talk about Capybara and Spud. These are codenames for Anthropic and OpenAI’s newest pre-trains. With the release of GPT-5.5 yesterday, we now have something concrete to discuss.

GPT 5.5

GPT-5.5 is the first public release based on “Spud”. As OpenAI’s first new pre-train since the failed GPT-4.5, expectations are obviously high. And despite both NVIDIA and OpenAI claiming with precise language that the model was “trained” on a 100k GB200 NVL72 cluster, this “training” is post-training (RL) only. Pre-training is still on Hopper.

OpenAI’s flagship model has historically been cheaper than Anthropic’s, but at $5 per million input tokens and $30 per million output tokens, GPT-5.5’s API price will be 2x more expensive than GPT-5.4 and slightly more expensive than Opus 4.7. The API went live this morning after a brief ChatGPT/Codex-only window due to safety concerns. We’ve been testing the model via Codex and API during an alpha testing period and describe that experience later in this article.

Like all their other models, OpenAI will also be offering a priority tier for GPT-5.5 priced at 2.5x the standard rate. Figuring out how to charge users more money for faster tokens is becoming increasingly important, and it’s worth clarifying that priority is totally different from fast mode. Fast mode just makes some vague guarantees like “2.5x faster for 6x the price,” whereas priority makes more conservative, concrete SLAs (e.g. > 50 tokens/sec > 99% of the time). Both Anthropic and OpenAI offer fast mode and priority tiers, but we think Opus 4.6 Fast is the only SKU that’s gained real traction.

Separately, OpenAI also offers GPT-5.3-Codex-Spark, but that’s a totally different model built to run on Cerebras. Specifically, it is a distilled version of GPT-5.3. There’s a big difference between offering faster tokens via running smaller batch sizes, changing the reasoning depth, and routing requests to a priority queue without changing the underlying model (priority and fast mode) vs running a dumber, smaller model (codex spark).

Also released is GPT-5.5 Pro, which is only available via ChatGPT and API. It’s meant for scientific research or long range reasoning tasks instead of everyday agentic work. GPT-5.5 Pro earned SOTA scores on BrowseComp and FrontierMath, and is priced at the same $30/180 as GPT-5.4 Pro. We expect to see more announcements about GPT-5.5 Pro making scientific discoveries soon.

Both the standard and pro models offer different levels of reasoning: xhigh, high, medium, low, and non-reasoning, which is a tradeoff between cost vs capability. As has been clear since the release of strawberry/o1, higher reasoning levels lead to better outputs but require more tokens and users have to wait longer for a response.

Relatedly, OpenAI advertised in their model card that GPT-5.5 scores higher on benchmarks than 5.4 while simultaneously using less tokens. In other words, it’s more “token efficient.” This is an extremely important concept to understand, and we believe it will become a major talking point this year. As we explained and quantified to Tokenomics model subscribers last week, cost per task, not cost per token, is the true north star metric that determines model pricing. Mythos may be 5x more expensive than Opus on a per token basis, but much of that price increase is nullified because Mythos can solve the same problem using fewer tokens. It may also be a faster end to end response.

Opus 4.7

This all comes a short week after Anthropic’s release of Claude Opus 4.7, a drop-in replacement for Claude Opus 4.6. Opus has been the daily driver for most of SemiAnalysis, and Opus 4.7 is a small improvement. With improved scores on many benchmarks and predictably good vibes, but not a step change, 4.7 has been reluctantly adopted by our team members. Why? Fast mode does not exist yet. For the first time, we have found that many of our engineers are willing to sacrifice a bit of quality (but not too much) for faster speed, claiming that the 2.5x faster for 6x the price tradeoff lets them hit “flow state”.

In practice, the noticeable changes moving from Opus 4.6 → Opus 4.7 have been from features/functionality rather than raw performance. In general, these models have gotten so good that most day-to-day tasks are accomplished successfully, with our engineers’ criticisms of a code edit or PR being more about style, approach, architectural decisions, and token efficiency (i.e. speed) rather than success on functional tests. It is increasingly rare for any of these coding models to go haywire and botch a commit completely.

On model behavior changes, the biggest thing we have noticed in our testing is how 4.7 is using fewer tool calls by default, and using reasoning more. The jury is still out on the benefits here, but in general we don’t like it. Anthropic suggests raising the reasoning effort from high to xhigh or max to increase tool usage. And it seems that our users are doing exactly this in order to let the model bring in enough context to successfully complete a complex task or form a complete multi-step plan. Not exactly the token efficiency tradeoff claimed in the announcement.

Notably, many people have been accusing Anthropic of intentionally degrading the 4.6 model on the lead up to the 4.7 release. Anthropic has categorically denied these claims, but multiple engineers at SemiAnalysis independently said that over the last few weeks the changes in 4.6 performance have made them “feel a little schizo”. And of course, they were right.

On April 23, a week after the Opus 4.7 release, Anthropic posted a postmortem detailing three bugs that they found in March/April. All three were present for weeks, and affected basically all users of Claude Code. One of the bugs is trivial, two are interesting, and all are real. When the harness is part of the product, the model gets blamed.

Notably the three timelines are March 4 to April 7, March 26 to April 10, and April 16 to April 20. This is weeks and weeks of bugs going unnoticed. Bugs that were introduced by Claude, and likely root-caused by Claude. Live by the sword, die by the sword.

DeepSeek V4

The long awaited DeepSeek v4 drop is here. DeepSeek took the world by storm last year with its R1 release and since then there have been legitimate questions in the AI community about whether open source models will commoditize intelligence. For those keeping score at home, DeepSeek crashed the market so hard that CEOs were scrambling to explain Jevons paradox. This seems to have played out quite clearly in the 16 months since, with the Great GPU Shortage now upon us.

V4 is an improvement over V3, but it didn’t crash the market today. That said, the achievements of DeepSeek should not be discounted. They open-sourced the weights, a detailed technical report, and updated libraries such as DeepEP, DeepGEMM, and FlashMLA that are widely used by labs around the world. Ironically, DeepSeek is helping American open source AI stay alive.

This release includes two models: DeepSeek-V4-Pro and DeepSeek-V4-Flash. The former is 1.6T total / 49B active, and the latter is 284B total / 13B active. Pro is a step up from V3, which was 671B total / 37B active, while Flash is a step down. We believe that both these architectures are still meaningfully behind their closed-source counterparts on the frontier in terms of both total and active parameter counts. We detail more about how we model the architectures of leading closed source frontier models in our Tokenomics model.

The core advancement of V4 over V3 is a move from a 128k context window to 1M context. As a result, all of the main technical advancements are focused on long context performance. These include:

And result in the following claim: “In the one-million-token context setting, DeepSeek-V4-Pro requires only 27% of single-token inference FLOPs and 10% of KV cache compared with DeepSeek-V3.2.” That’s a 90% reduction in KV Cache, way more impactful than Google’s TurboQuant paper last month! NAND Flash investors, watch out.

On benchmarks, DeepSeek did not feel that standard benchmarks were good at capturing real-world task capability, so they introduced their own set of agentic benchmarks to measure how V4 compared against other SOTA models: Chinese writing, retrieval augmented search, a suite of white-collar tasks with long horizons, and coding. V4 Pro was able to compete with top models on all these tasks but lags behind in key areas. For instance, on especially difficult Chinese writing tasks, Claude Opus 4.7 still beats DeepSeek V4 Pro. Claude mogs Chinese models in it’s own language.

Unfortunately, using public announcements on model performance benchmarks as a proxy for real world performance is unreliable. Conflicting incentives cause these labs to publish certain benchmarks and not others. Like this example, where DeepSeek takes a shot at the Kimi and GLM APIs:

This is the reason why the SemiAnalysis Tokenomics Dashboard tracks all major model performance claims, pricing, release dates, usage disclosures in an unbiased manner. We also do our own hands-on testing of all the major models. Below is an example of our tracking of meaningful benchmark performance across the major model releases. We will explain later why benchmarks are bad.

DeepSeek also open sourced a Mega-Kernel inside of DeepGEMM that supports both NVIDIA GPUs and Huawei Ascend NPUs. NPU support is claimed, but only the code for SM90 (Hopper) and SM100 (Blackwell) GPUs is released publicly. It is likely a goal to run a meaningful portion of the future inference traffic on Ascends. It is notable however that the parameter size fits just inside the memory domain of an 8x H20 HGX at FP4.

Of course, the key contribution of DeepSeek V4 is that it is open source. Thanks to an all nighter, our InferenceX team, collaborating with 10x engineers from vLLM/Inferact and NVIDIA, have published day-zero support on our H200 cluster. Support for Blackwell and AMD GPUs using vLLM, SGLang and TRT-LLM with Dynamo is a work in progress.

Interestingly, day-zero support on H200 at FP81 performance of this model hits ~150 tok/sec throughput per GPU at 20 tok/sec interactivity on 8k in 1k out. For reference, V3 hits ~1.3k to 2.3k tok/sec of throughput per GPU at 20 tok/sec interactivity on 8k in 1k out. This is a new model and we expect meaningful optimization in the coming weeks. Watch inferencex.com for real time improvements.

Overall, DeepSeek is an exceptional engineering release, and is right behind the SOTA frontier. It will be the lowest cost alternative to closed source models, but it’s capabilities are not at the leading edge. SemiAnalysis’s workflows likely will not be cannibalized by DeepSeek.

VIBEZ: Our Impressions of GPT-5.5 vs Opus 4.7

SemiAnalysis is famous (infamous?) for shilling Claude, and we’ve been testing GPT-5.5 as part of an alpha program with OpenAI the past few weeks.

We think GPT-5.5 is a significant improvement within Codex specifically. Previously, ~all our engineers used Claude exclusively, and use of ChatGPT models for coding was restricted to wrappers like Cursor. Now, most of our engineers switch between Codex and Claude models depending on the task and IDE preference. Here are some quotes:

However, it’s not all positive for OpenAI. Some of our other engineers complained that Codex is still worse at inferring your true intent than Claude Code. Humans naturally give terse and not particularly well thought out instructions when prompting coding agents, and Codex often listens too literally.

Relatedly, another engineer commented that GPT-5.5 feels too conservative when it comes to actually making code changes. Yes, this improves token efficiency, but it comes at the cost of correctness. A similar tradeoff happened from 4.6 → 4.7 as we described previously. Seeing the words “narrow fix” in the output is now a signal to double check the model’s work.

Here’s a concrete example that illustrates our overall impression on the strengths and weaknesses of Codex vs Claude Code well. We asked both Opus 4.6 and GPT-5.5 to make a new dashboard for our accelerator model and gave it the current tokenomics dashboard as an example. As our institutional subscribers know, this dashboard includes a homepage that links to all the different tabs.

If we specifically asked Codex to copy the homepage in the prompt, we’re sure it would’ve done so, but it was unable to infer this intent itself.

With that said, the actual data Codex included in the dashboard was much more accurate than Claude (though to be clear neither was perfect on the first pass). This implies stronger reasoning about the data structures and relationships with a relatively complex excel file on the part of Codex. Meanwhile, many of Claude’s numbers were straight up hallucinated and it made mistakes like including Nvidia GPUs in TPU charts. This tracks with our overall impression that Codex is “smarter” and better at doing complex reasoning to solve harder, more narrowly scoped tasks, whereas Claude is better for more open ended, greenfield problems.

It’s for these reasons that some of our engineers have settled on the following workflow:

Importantly, before the GPT-5.5 release, ~all of SemiAnalysis used Claude Code for both of these steps. Our use of ChatGPT models had become restricted to Deep Research on the webapp and wrappers like Cursor Bugbot.

Critically, features in the plugins/CLIs are holding Codex back. Many of our engineers prefer fast mode with 1M context, use remote control/sandbox plugins to take sessions from laptop to phone and back, and upload images/screenshots during a conversation. All of this is possible with the Claude Code CLI, VSCode Plugin, web app and mobile app. But none of it is currently possible with the Codex CLI, VSCode Plugin, desktop app, web app and mobile app.

Even if GPT-5.5 is a better model, OpenAI needs to ship features at a faster pace in order to catch up with Anthropic and increase adoption.

Benchmarks are bad but we need to keep using them anyways

The one thing that is always prominently featured in every new model announcement is a table comparing performance on various benchmarks.

It’s very tempting to be able to point to a small set of numbers in order to prove the “objective” superiority of your new model release, but many within the AI community have long lamented that benchmarks are no longer a useful proxy for real-world utility. We tend to agree with this point of view. There’s a big difference between claiming to measure a model’s coding/finance/reasoning abilities vs actually doing so in any meaningful capacity.

That being said, we expect all the labs to continue highlighting improved benchmark performance for all future model releases, and the following section will help you separate the signal from the noise.

Anatomy of a benchmark

Really understanding the first two is how you determine if a benchmark is any good or not. To illustrate, we’ll walk through some famous benchmarks below in rough chronological order. This will also give you a sense of how benchmarks have changed over time.

MMLU and multiple choice/simple answer benchmarks

Released by academic researchers in 2020, Measuring Massive Multitask Language Understanding (MMLU) is a set of 15,908 multiple choice questions covering 57 subjects. These questions were manually collected by university students from online sources like standardized tests and college exams/problem sets. All of them have exactly 4 choices and are publicly available, but they range in difficulty from “elementary” to “advanced professional”.

MMLU has a minimal harness that essentially just formats the question into a prompt. Tools like web search are not included. The multiple choice format is crucial because it makes grading trivial–just check if the model outputted the right letter.

MMLU was effectively solved (aka “saturated”) by GPT-4 in March 2023 when it scored 86.4%. In practice, the true max score for benchmarks is usually lower than 100% because some of the tasks are ambiguous, poorly worded, or just straight up incorrect. This paper estimates that 6.49% of MMLU questions contain errors for example.

As each of these became saturated, their creators made harder versions (e.g. MMLU-Pro, MMMU-Pro, GPQA-Diamond). Tactics include filtering out easy questions from the previous version, using an LLM to up the choices from 4 to 10, paying your contractors to make harder questions, etc.

The most relevant simple answer benchmark today is Humanity’s Last Exam (HLE). Released by Scale AI in January 2025, they sourced 1000+ experts from around the world to create 2500 questions on everything from algebraic geometry to classical ballet. 80% of the questions require an exact-match short answer and 20% are multiple choice. For the harness, you can choose to run the model with or without tools (e.g. web search and code execution).

These questions obviously aren’t representative of real-world LLM usage and are also riddled with issues. For example, one study found that 30% of HLE chemistry/biology questions had answers that directly conflicted with peer-reviewed literature.

However, the labs still absolutely hillclimb all of these benchmarks during the RL stage of training. Google, for example, had a 9 figure budget in 2025 specifically for HLE style STEM questions, which they paid to data vendors like Mercor, Surge, and Handshake. It’s no coincidence that Gemini 3 Pro was a step-change improvement on the benchmark.

The generous explanation for why labs care about things like HLE is that the knowledge gained from solving esoteric multiple choice questions will transfer to other use cases. The cynical explanation is that corporate VPs want to be able to point to a single number to prove that they’re doing their job, and Scale was good at marketing HLE to win sufficient mind share.

SWE-bench and coding benchmarks

Coding is the most important AI capability, and SWE-bench (released in 2023) was the first big coding benchmark.

The tasks were automatically scraped from 12 Python repos, including django, scikit-learn, and seaborn. They used the following 3 step filtering process:

In other words, the GitHub issue is the task, and the PR that resolved said issue is proof that the task is possible. The eval is all the old tests in the repo plus the new tests included in the PR. The AI is successful if none of the old tests break (pass-to-pass) AND all the new tests pass (fail-to-pass). Importantly, the model is not allowed to see any of the new tests while attempting the task. For the harness, the model is able to inspect the codebase, but it can’t actually run any code.

It is worth emphasizing that there was no human verification at any step in the task creation process. GitHub issues are often ambiguous and poorly specified. Furthermore, the tests devs include in their PRs are typically noncomprehensive and scoped to particular implementation details. This causes two big issues

In short, many of the SWE-bench tasks were straight up broken. For example, one task required the AI to perfectly match a 19 word error message in the eval despite not mentioning it at all in the problem description.

OpenAI attempted to solve these issues by releasing SWE-bench verified in August 2024. They hired 93 python devs to manually review all the task descriptions and evals for ambiguity/unfairness. After filtering out all the problematic ones, the original 2294 problems were reduced to 500 “verified” tasks. OpenAI also added a bash tool to the harness so the AI could execute code–making the benchmark more agentic–and improved infra reliability by packaging each task as a Docker container.

In February 2026, OpenAI announced that they would no longer report results on SWE-bench verified for two reasons:

Instead, they recommended model makers report SWE-bench pro results instead. SWE-bench pro is another Scale creation. The main difference (besides making the tasks harder) is that they used public repos with less permissive licenses and private repos to avoid contamination. They also hired contractors to write evals and problem descriptions for the commits, instead of relying purely on GitHub issues and preexisting PRs. These are all good steps, but they definitely don’t fully solve either problem identified with SWE-bench verified. As you’ve probably already figured out by now, no benchmark is perfect.

SWE-bench pro and verified are both still commonly reported in model release cards today. Other popular coding benchmarks include

GDPval and non-coding agentic benchmarks

Agentic AI extends far beyond coding today, and so too do agentic benchmarks. The most famous example is GDPval by OpenAI. Released in September 2025, it aims to measure AI’s ability to complete real economically valuable tasks across 44 different jobs, from financial analysts to nurse practitioners.

To create the tasks, OpenAI hired expert contractors from each job–e.g. an ex-BofA banker for finance tasks–and asked them to provide 3 things per task:

The harness is another step up from coding benchmarks. Agents are given access to apps like LibreOffice (Microsoft Office clone) and CAD software, along with the standard web search and code execution tools. Although GDPval isn’t quite this advanced, newer agentic benchmarks also include fake calendars, emails, Slack messages, Google Drives, etc. that the AI needs to navigate in order to successfully complete the task.

Finally, to evaluate each task, OpenAI used additional expert contractors to compare the AI outputs to the human-provided solutions. They also created an AI grader that uses the rubric to rank solutions, but conceded it’s still not as reliable as the human experts. For this reason, they still use human experts for their official results, despite it being way slower and more expensive.

However, “LLM-as-a-judge” for evals is a popular technique used by other agentic benchmarks for tasks that aren’t objectively verifiable. GDPval-aa, for example, is just the public GDPval tasks but with an LLM judge.

In theory, rubrics allow you to measure important qualitative traits like style, but they have obvious limitations. For example, it’s hard to guarantee quality when your rubrics are either written by contractors or AI generated. Using an LLM to evaluate quality is also inherently suspect, especially when there’s no human in the loop making the final decision.

Another big limitation with GDPval is the clearly defined, unnaturally specified prompts. Real world tasks typically have an element of ambiguity that’s completely missing from this benchmark. Human jobs also involve iteration based on feedback, whereas GDPval is strictly single-turn.

That being said, GDPval is certainly closer to actual knowledge work than something like HLE. Other popular agentic benchmarks include:

Some sneaky benchmark reporting by OpenAI

Hopefully the previous sections have convinced you that benchmarks are often widely unrepresentative of the capability they claim to be measuring. However, they also definitely aren’t totally useless, and a 10%+ improvement on SWE-bench verified after everyone else thought the benchmark was already saturated (which is what Mythos did) still means something.

Looking at the benchmarks companies choose NOT to report can also be telling. For example, OpenAI barely included any benchmarks in their GPT-5.4 announcement and didn’t compare it to any Anthropic models. We think this is because it would’ve gotten brutally mogged by Opus 4.6—which came out a month earlier. This matches our overall vibe of the model. Until yesterday, OpenAI’s models were worse than Anthropic’s for ~all agentic tasks.

With GPT-5.5, they’re finally back on the frontier, which is why Claude and Gemini were re-included in the benchmark table.

However, there’s still one benchmark that’s suspiciously missing. Coding is the most important model capability and OpenAI literally wrote a blog post in February arguing for SWE-bench Pro to become the industry’s new de facto benchmark. So why did they use this random “Expert-SWE” benchmark instead?

Scrolling down all the way to the very bottom of the blog post reveals the answer:

GPT-5.5 got mogged by Opus 4.7 (much less Mythos which scored 77.8%). This supports our qualitative impression of the three models. GPT-5.5 is better than Opus 4.7 at some coding tasks but is not decisively better across the board. Mythos is presumably a true step up compared to both of them, but Anthropic hasn’t given us access yet :(

Why you shouldn’t use the same harness for an apple-to-apples comparison

As part of our alpha-testing, we also ran a number of benchmarks on GPT-5.5 vs 5.4 vs Opus 4.6. Here are the results:

Our numbers are generally lower than OpenAI’s and Anthropic’s for 2 reasons:

You could argue that our benchmark numbers are better than OpenAI/Anthropic because we use the same harness for a more apple-to-apples comparison. But the harness is clearly part of the product at this point. What people actually care about is how good is Codex vs Claude Code, not GPT-5.5 vs Opus 4.7.

Returning back to the importance of token efficiency, it’s worth emphasizing that the harness has a huge impact on the ultimate cost per task. Prompt caching, input/output ratio, and tool use patterns are all largely determined by the harness. SemiAnalysis is currently collecting millions of dollars worth of agentic AI traces in order to better understand how different harnesses (e.g. Claude Code vs Codex vs Cursor vs OpenCode) change cost per task. Preliminary analysis shows that Codex is likely more token efficient than Claude Code, with an average input/output ratio of 80:1 vs 100:1. Yes, a higher input/output ratio means a lower price per Mtok, but Codex still ends up being cheaper because it consumes less input tokens overall. Those interested in the full results should subscribe to the Tokenomics model.

Who Wins in the Agentic Coding Wars?

So what does this all mean for the future of coding agents? Behind the paywall, we’ll give our predictions for this soon to be multi-trillion dollar industry.

Meet the Models

GPT 5.5

Opus 4.7

DeepSeek V4

The core advancement of V4 over V3 is a move from a 128k context window to 1M context. As a result, all of the main technical advancements are focused on long context performance. These include:

VIBEZ: Our Impressions of GPT-5.5 vs Opus 4.7

SemiAnalysis is famous (infamous?) for shilling Claude, and we’ve been testing GPT-5.5 as part of an alpha program with OpenAI the past few weeks.

If we specifically asked Codex to copy the homepage in the prompt, we’re sure it would’ve done so, but it was unable to infer this intent itself.

It’s for these reasons that some of our engineers have settled on the following workflow:

Even if GPT-5.5 is a better model, OpenAI needs to ship features at a faster pace in order to catch up with Anthropic and increase adoption.

Benchmarks are bad but we need to keep using them anyways

The one thing that is always prominently featured in every new model announcement is a table comparing performance on various benchmarks.

Anatomy of a benchmark

MMLU and multiple choice/simple answer benchmarks

SWE-bench and coding benchmarks

Coding is the most important AI capability, and SWE-bench (released in 2023) was the first big coding benchmark.

The tasks were automatically scraped from 12 Python repos, including django, scikit-learn, and seaborn. They used the following 3 step filtering process:

In February 2026, OpenAI announced that they would no longer report results on SWE-bench verified for two reasons:

SWE-bench pro and verified are both still commonly reported in model release cards today. Other popular coding benchmarks include

GDPval and non-coding agentic benchmarks

To create the tasks, OpenAI hired expert contractors from each job–e.g. an ex-BofA banker for finance tasks–and asked them to provide 3 things per task:

That being said, GDPval is certainly closer to actual knowledge work than something like HLE. Other popular agentic benchmarks include:

Some sneaky benchmark reporting by OpenAI

With GPT-5.5, they’re finally back on the frontier, which is why Claude and Gemini were re-included in the benchmark table.

Scrolling down all the way to the very bottom of the blog post reveals the answer:

Why you shouldn’t use the same harness for an apple-to-apples comparison

As part of our alpha-testing, we also ran a number of benchmarks on GPT-5.5 vs 5.4 vs Opus 4.6. Here are the results:

Our numbers are generally lower than OpenAI’s and Anthropic’s for 2 reasons:

Who Wins in the Agentic Coding Wars?

So what does this all mean for the future of coding agents? Behind the paywall, we’ll give our predictions for this soon to be multi-trillion dollar industry.

Introduction: Rethinking the Total Cost of a GPU Cluster

Modern GPUs are unbelievably expensive. A single Blackwell GPU costs more than the average car, and uses more energy than a single family home. It is now common for unicorn startups to have thousands of these GPUs working for them, day and night. Many foundation model companies now spend an order of magnitude more money on GPUs than they do on employees. We know multiple companies spending over 80% of their initial funding on GPUs. Startup founders now have four important categories of spending to consider when building a financial plan for their company:

Traditionally, when deciding where to get a cluster to solve that first category, companies evaluate neoclouds on a cost-per-hour basis, focusing on the most expensive line item: the GPUs. However, focusing solely on the price per GPU-hour a provider offers can be misleading. In practice, two cloud offerings with identical pricing per GPU-hour can have very different TCO, once you account for everything that goes into training a model or building inference endpoints. Factors such as downtime, setup time, debugging time, and required performance tuning of networking and storage can dramatically impact how much useful work users can do per dollar spent. Additional costs for non-GPU expenses such as CPU compute, networking, storage, orchestration software, and support can also be hidden and not considered. In other words, what appears to be a cheaper cluster can in many cases end up being more expensive.

The central premise of SemiAnalysis ClusterMAX™ research is that cluster quality varies significantly across GPU cloud providers, and that these differences have a meaningful impact on end user experience, productivity and as a result, TCO. Many of these factors are not captured in hardware specs, reference architectures, or one-time performance benchmarks. Differences in reliability, networking behavior, storage performance, and support affect the only metric that matters: time-to-research-objective.

In this article, we introduce a methodology for calculating the TCO of GPU clusters that goes beyond raw price per GPU-hour. We define a framework that incorporates direct costs such as compute, storage, networking, and support, as well as indirect costs related to reliability, debugging, and setup. Using this framework, we compare three classes of ClusterMAX rated providers: a gold tier neocloud provider, a silver-tier hyperscaler, and a silver-tier neocloud. We apply this methodology to three representative cluster configurations, covering Large LLM Pretrain, Multimodal RL Research, and Inference Endpoints.

In order to conduct this comparison we use our GPU Cluster TCO Calculator and our Goodput Calculator, which we release for free on our ClusterMAX website. Anyone reading this can plug in their own values for custom scenarios and see the results. We explain the formulae behind this calculator later in this article and introduce our Grand Unifying Theory of Goodput.

These calculators are supported by input data from our GPU Rental Pricing data series, hands-on experience testing 80+ neoclouds, and interviews with over 150 end-user customers of neoclouds which were conducted during the research effort for ClusterMAX 1.0, ClusterMAX 2.0, and continue to this day for ClusterMAX 3.0.

Our findings demonstrate why providers in the ClusterMAX gold-tier command a pricing premium, (or win deals at equal price). Specifically, we find that when we hold GPU pricing constant, the TCO of a gold-tier provider is lower than a silver-tier provider by roughly 5-15% across a representative set of large training workloads, but the difference is reduced to near zero when considering fault tolerant workloads like single node inference clusters. In other words, we put real dollar values behind the intuition that users have built when understanding the benefits of fault tolerance.

Definitions and Key Terms

To evaluate GPU cloud offerings on equal footing, we break down the TCO of a GPU cluster as follows.

1. GPUs [$/GPU-hr]: The headline rental price for a GPU cluster. This starts from the provider’s list price, then factors in any discounts due to term length commitments or volume, planned use of spot/preemptible instances, and the orchestration premium. Orchestration premium refers to pricing increases beyond basic instances, for example if using Kubernetes or Slurm through SageMaker Hyperpod Slurm in AWS, customers typically pay a premium for the SageMaker instance type vs the standard EC2 instance type even though the underlying GPU machine is the same. We account for such discounts and premiums to derive an accurate per GPU-hour cost. Critically, the data used in this report is informed by two of our institutional products: our AI Cloud TCO Model and GPU Rental Pricing Data Series. Our default pricing for GPUs is a historical snapshot from August, 2025. We described how things have changed since then in a recent article. Please contact sales@semianalysis.com for access to our GPU pricing data series.

2. Storage [$/GB-mo]: The cost of storing data. This includes high-performance “hot” storage (e.g. NVMe-based parallel file systems), lower-tier “warm” or object storage for less frequently accessed data, and “cold” archival storage. We also include any data access costs: for instance, API call costs on object storage or data egress charges if data leaves the cloud. These can be substantial during training when moving around large datasets and model checkpoints, and during inference when considering storing logs and metrics (now including image, video, and audio data). Based on customer surveys, we adjust our assumptions across different cluster scenarios from a low point of 2TB/GPU to a high point of 25TB/GPU. We also track the public pricing (standardized to per GB, per month) across various providers and release this data for free as a dropdown menu in the Cluster TCO Calculator. Notably, storage performance can vary massively even between different offerings even from the same provider. For example, AWS FSx for Lustre has 4 different throughput tiers (ranging from 125 MB/s/TiB to 1,000 MB/s/TiB) and charges about 3x more for 4x more throughput at list price. We allow for a consideration of this difference during inputs (e.g. for job init time) in goodput calculations discussed later.

3. Networking [$/hr or $/GB-mo]: The cost of frontend/N-S networking features. Networking services include public IPs, firewalls/security groups, load balancers, data egress, and data transfer. For example, transferring training data or model weights out of AWS or between AWS regions can incur significant fees. For the backend/E-W network, we make a simplifying assumption that all clusters eventually perform at a similar level with a high bandwidth interconnect (i.e. InfiniBand, RoCE, EFA, etc.) after setup. As a result the cost differences are considered later in Setup Expense and Debugging Expense.

4. Control Plane [$/hr]: The cost of managing the cluster. In terms of the orchestration software control plane, nodes for login, code development, and job submission. Extra CPU-based nodes for data processing and environments for RL rollouts can be considered here too.

5. Support [% uplift]: The cost of support. For example, on AWS, this is an extra charge on the entire cloud bill, with three different options that range anywhere from an initial 10% to a final 3% of the bill as the monthly spend graduates to higher tiers. Of course, different tiers of support mean better response in the event of an outage or performance issue.

6. Goodput Expense [% uplift]: The first item that is not showing up on a monthly bill and is an implicit cost associated with using lower-tier providers. We use this percentage to build in an additional cost of downtime on the cluster in the form of more rental time required, or less useful work being completed. In practice, the actual amount of downtime, or number of job interruptions depends on the provider, the individual datacenter, hardware, and workload. Inputs used to calculate this expense include the total number of interruptions/failures, time to identify the failure, and the time to repair/replace a node. The impact of a single failure/interruption also depends on the cluster design, e.g. the blast radius of the failures, training initialization time, average job size, checkpoint frequency and/or use of fault tolerant software frameworks. The inputs to this piece of the calculator is also an opportunity for users to price in the risk of a bad SLA from a risky provider, on a total % basis. For example, a 95% cluster uptime SLA commitment from the provider allows for 5% downtime with no response and not credits. Since this input is so complicated we have an entire second tab with multiple scenarios covered. More on this later.

7. Setup Expense [$/hr]: The cost of having engineers setup the cluster, and tune performance. For example, on AWS, POC’s are not free, and users report that tuning NCCL + EFA parameters in order to reach the same level of performance as InfiniBand or RoCE networks can take weeks to months of effort by multiple engineers. Since in many cases this requires an entire cluster to be dedicated to this work, the additional line items of expense includes both engineering hours and the cluster time spent on performance tuning.

8. Debugging Expense [$/hr]: The cost of having engineers debug the cluster over time, i.e. the cost of engineering headaches. For example, on AWS, users report that debugging NCCL + EFA issues involves 4 or 5 layers of indirection from their pytorch code, through the driver stack and into the NIC/switch firmware/hardware recipe. In other words, these line items of expense include the engineering time spent on an ongoing basis, and the cluster time spent on failed jobs.

Our Proposed TCO Formula for GPU Clusters

The following formula is used to calculate the Total Cost of a GPU Cluster on a monthly basis:

Note: setup is amortized over the contract term (3mo to 3yr). in other words, spending time setting up a cluster you will use for 3 years is not a big deal. Spending weeks setting up a cluster you will use for 3 months is.

Next, we define G_chkpt-hot, G_chkpt-cold, and G_tolerant, i.e. the different ways to calculate goodput expense.

The Grand Unifying Theory Of Goodput

In the context of training, goodput is defined as the amount of useful work users can perform on their cluster. Goodput plays on the term throughput to mean that not all throughput is “good”. Lots of training throughput can be “bad” if a GPU fell of the bus, NCCL is stalling, or there is an OOM hiding around the corner during the next checkpoint save.

These issues are much more pronounced at scale. As we demonstrate below, larger jobs on larger clusters are much more impacted by individual failures or interruptions. If 80% of your cluster is running one job, and that job has to restart (a process that can take 10-15 minutes depending on storage, networking, CPUs, caching setup, etc.) this is costing you all of those 10-15 minutes of cluster time for job initialization time, plus all the wasted compute you did from the last checkpoint to the time of the failure/interruption/crash.

As we explained in ClusterMAX 2.0, cluster-level MTBF also plays a role here. Since all GPUs eventually fail, the bigger your job, the less time you have to do useful work (goodput) in between failures.

Here we use a convenient table to illustrate the concept. As node failures get more common (moving down the y-axis of the chart) and cluster size gets bigger (moving to the right across the x-axis of the chart), the time between failures (MTBF) gets smaller and smaller.

We summarize all of this in our TCO Calculator as “Goodput Expense”, where the following formulae are used to calculate Goodput Expense under three scenarios:

Where…

G_chkpt-cold = goodput expense when jobs restart from a checkpoint via a spare node that is “cold” (typically, provider managed). In other words, the jobs wait until a repair/replace happens. This is the worst case scenario, since these kinds of repairs typically take hours or days.

G_chkpt-hot = goodput expense when jobs restart from a checkpoint via a spare node that is “hot” (typically, customer managed but can also be from top-tier providers). In other words, the jobs (depending on defined priorities) can restart immediately on idle nodes (customer managed), pre-empt lower-priority jobs (also customer managed), or restart on a node that gets brought into the cluster from a spare pool (provider managed). Of course, a provider-managed spare pool also depends on some capacity guarantee from the customer (i.e. if one of your machines fail and you report it for repair/replacement, there needs to be spares available). Top-tier providers that are experienced running multi-tenant clusters at 4k+ GPU scale tell us that they will leave anywhere from 2-6% of their nodes in this spare pool to be used for hot-swaps.

G_tolerant = goodput expense when jobs are “fault tolerant”, i.e. they can keep running in the event of a hardware issue. This scenario is well understood for single-node inference, where a framework such as llm-d or ome or kserve will just have the load balancer stop sending traffic to the failed node and resend any failed requests to the healthy nodes. The scenario is less well understood in training.

Notably, from the user’s perspective, there are two very different approaches at the software level that we have observed on training clusters. The first is checkpoint restart (still the most common option for small and medium-scale clusters), and the second is fault tolerant training frameworks. In both cases, the inputs to the calculations depend on the approach of recovering from idle nodes vs pre-emption vs relying on the provider, and how long repair/replace flows actually take.

In the scenario of a fault-tolerant training framework, we consider three options, which we describe in more detail below:

TorchFT

TorchFT is the open source standard for fault tolerant training frameworks. The framework easily integrates with existing torchtitan code, and allows for training jobs on large clusters to continue running in the event of a hardware failure. No need for checkpoints (or really, you can checkpoint less frequently). However, the blast radius is the entire replica group.

Since TorchFT’s blast radius is the entire replica group (i.e. an FSDP shard within an HSDP job), when any GPU or node within a group fails, the whole group’s torchrun process crashes. This means that all GPUs in that group are idle until recovery completes. As a result, with FSDP shard=16 a single GPU failure takes out all 16 GPUs. With shard=32, it takes out 32 GPUs, etc.

Specific to FSDP, the relevant failure domain is the communication group, not just the raw cluster size. Because parameters are all-gathered before computation and gradients are reduce-scattered in backward, a single failed or hung rank can stall the entire participating group. In practice, HSDP makes this more explicit: blast radius becomes a topology decision at the replica-group level rather than a property of the whole cluster.

This has a tradeoff. When a replica group dies, you lose that whole group’s GPUs until the node is replaced, a surviving group serializes its full model + optimizer state via state_dict(), serves it over HTTP to the recovering group, calls load_state_dict(), syncs its step counter, and rejoins the quorum. This whole process is orchestrated by the TorchFT lighthouse server, which you must install on the cluster.

Not every large-scale failure looks like a dead GPU or dead node. A meaningful share of incidents first appear as stuck collectives or watchdog timeouts, which are just symptoms. From a TCO perspective, that means goodput loss includes not only repair or replacement time, but also the time required to detect, attribute, and unwind a hung collective across the participating ranks.

Checkpointing itself can be part of the failure tax. On FSDP2, converting a DTensor state dict back to a full tensor for saving issues an all-gather across ranks Checkpoint frequency is a reliability parameter and a communication and failure-surface parameter.

However, this fault tolerance comes at a performance cost. Since TorchFT requires the use of GLOO vs NCCL for comms across replica groups, there is a per-iteration overhead for an allreduce through the CPU via frontend TCP instead of the backend RDMA network. In initial testing we saw a performance difference of over 10% on comparable HSDP jobs. As a result, when considering goodput expense, we allow for this performance difference to be considered in a “Network overhead (%)” line item if the user chooses to run TorchFT.

Fault tolerance can affect training semantics, not just recovery latency. The number of healthy participants, and therefore effective batch, could change from step to step as replica groups dropped in and rejoined. When comparing TorchFT to checkpoint-restart or live-migration approaches, some methods preserve forward progress by accepting temporary degraded participation, which may affect optimizer dynamics and throughput accounting.

AWS SageMaker HyperPod Checkpointless Training

AWS introduced checkpointless training for their SageMaker Hyperpod EKS clusters in December 2025. This is a kubernetes-only, and NeMo megatron-only solution to the same fault tolerance problem described earlier. Amazon developed this technology internally for training their Nova models and has proven it at 1k+ GPU scale.

The core of checkpointless training is the concept of model redundancy. In other words, the model and optimizer states are contained to the replica group, and then synced cross-replica group (though AWS calls them node groups). Similar to TorchFT, the presence of this cross-group sync allows for recovery of failed nodes and groups without interrupting the running job. Blast radius is proportional to the size of the group relative to the full job size. At runtime, each GPU maintains redundant copies of its model shards on peer GPUs, meaning when a failure occurs the recovering process loads state via RDMA over EFA. This process is managed by CheckpointManager and is a relatively simple code change as long as you’re scheduling your jobs on via the SageMaker HyperPod Training Operator.

There is a clear tradeoff for memory overhead here. To quote AWS docs: “The high-precision master model weights/gradients and optimizer states will be affected. Adding one redundant model replica increases device memory usage by roughly the equivalent of one DCP checkpoint size.” In other words, to run with this approach to fault tolerance you will introduce GPU memory pressure (proportional to the size of your replica groups relative to total job size) and OOMs. The result is running at reduced batch size or different parallelism strategies, which relative to a checkpoint restart job generally means a performance impact. As a result, when considering goodput expense, we allow for this performance difference to be considered in a “Memory overhead (%)” line item if the user chooses to run with checkpointless training.

Notably, checkpointless training is integrated with AWS node lifecycle management and deep health checks, which means it is quick to swap in pre-warmed hot spares (i.e. idle nodes in the cluster) for replacement. AWS claims recovery times of 1min 45 seconds for checkpointless training, vs 15 mins for checkpoint restart. Our hands on testing confirms this recovery time for a simple megatron training job on a 4-node H200 cluster. We also tested deep health checks and saw simulated hardware failures identified in under 2 minutes, and health nodes replaced in the cluster in under 20 mins.

TorchPass

By direct comparison to the previous two frameworks, torchpass is the only licensed software product, and the only option that maintains the same training performance as jobs without fault tolerance. In other words, the code changes are minimal, there is no performance overhead. The cost comes in the form of idle nodes in the cluster or time spent pre-empting lower priority jobs.

TorchPass is implemented at the scheduler level via plugin. In the case of our hands on testing this was an 8-node GKE cluster running a torchtitan job via PyTorchJob (KubeFlow) and the native kubernetes scheduler. We primarily tested the “planned migration” case, which is applicable for interruptions such as upgrades or maintenance on nodes in the cluster, and various Xids related to ECCs, GPU falling off the bus, power failures, link flaps, etc. In these cases, TorchPass supports a simple “just-in-time” checkpoint concept via get_state() that allows for the failing node to transfer state via RDMA to an idle spare. Notably, this sort of soft failure scenario is the most common type of failure in large training clusters where nodes slowly degrade over time but are still functional.

The results are clear when compared to checkpoint restart and TorchFT with performance overhead. Recovery times are similar for planned migrations, and the job performance is similar.

In addition, “unplanned migration” or hard failures (i.e. GPU, memory, network or other hardware failure, sudden reboot, kernel panic, etc.) are also possible to support via a similar approach to what was described in the TorchFT or Checkpointless Training sections above. In other words, the same “just-in-time” checkpoint approach can migrate state from a healthy worker in another replica group to the idle node joining the cluster to replace the failred node.

The TorchPass orchestrator is installed at the cluster level, and interacts with a Manager class that is integrated into an existing training script. It is relatively simple to figure out with only a few lines of code being added to existing training scripts.

Overall, there are many fault tolerant frameworks to choose from, and as clusters scale in size it because necessary to contend with these reliability challenges, users can’t only rely on their provider to handle every failure. A training codebase at 1k+ GPU scale really needs to be designed to work with the realities of the cluster it runs on.

Overview of Three Cloud Providers Being Assessed

Now, to demonstrate how to use the calculator, we use three representative providers.

These are not direct comparisons, but rather an amalgamation of the average experience using the providers in the given tier. Roughly speaking:

Gold-tier

Typically, we see Gold-tier providers have more aggressive discounts than hyperscalers, approaching the 25^th percentile of our GPU Pricing Data for large clusters and long-term contracts. In our testing, storage performance is strong, with volume discounts available for file and object storage. On networking, we have seen expected InfiniBand or RoCE performance out of the box with little to no setup time. The orchestration software (slurm or kubernetes is generally configured properly, and is also easy to setup and use. In general, POC’s are free. All support is 24x7, with good response times and talented engineers available directly without opening tickets. This support experience is included in the price without extra hidden charges. Finally, monitoring dashboards are setup with the cluster, health checks are configured by default, and a hot-spare pool of nodes is available for quick replacements in the event of a failure, effectively guaranteeing that when hardware fails there will be a quick replacement.

Hyperscaler

We typically see hyperscalers provide volume discounts that range from the 50^th to the 75^th percentile of our GPU Pricing Data. Large enterprises with MSA’s should expect pricing above the 75^th percentile due to additional hidden costs. Out of the box, we have seen poor storage performance be the default, with limited discounts available and extra charges required to improve performance. Networking performance also tends to require lots of setup time and debugging issues over time, leading to lots of cluster time being lost to unproductive work. In addition, POC’s are generally not free, which increases the setup time expense. Hyperscalers also charge a premium for support, with different tiers that each have different response times and monthly charges. These charges generally graduate down from 10% to 3% of the monthly bill as the monthly spend increases. Finally, monitoring dashboards are not usually setup and accessible by default, and can have extra charges associated. However, health checks are generally easy to configure, the datacenters are well run (reducing the total amount of failures and interruptions that occur) and both a hot spare pool of nodes and capacity guarantees exist.

Silver-tier

The typical silver-tier neocloud is a representative amalgamation of features offered by different companies. Typically, we see silver neocloud pricing at the 50^th percentile on the high end, and below the 25^th percentile on the low end. Buyers should be wary of aggressive discounts, which are a signal of low quality. Storage performance depends on the provider’s experience with VAST or Weka, and some volume discounts are available. InfiniBand or RoCEv2 performance is generally good out-of-the-box, though time can be lost for setup and debugging of orchestration software like Slurm or Kubernetes, and POC’s not always free. Support is typically included, but 24x7 response time coupled with a direct-to-engineer model is unusual. Critically, monitoring dashboards and health checks are not usually configured by default. Hot spare pools can be available, but there are generally no capacity guarantees. We see cold spares being the typical approach as silver-tier providers tend to rely on their serve OEM to handle repairs for them.

Applying the TCO and Goodput Formulae

In order to explore a broad range of typical customer requirements, we calculate the TCO for Gold-tier, Hyperscaler, and Silver-tier in three different scenarios: Large LLM Pretrain, Multimodal RL Research, and Inference Endpoints.

Below we walk through these scenarios using screenshots from our GPU Cluster TCO Calculator and our Goodput Calculator. We release both of these calculators for free on our ClusterMAX website so that users can plug in their own values for custom scenarios and see the results with relevant inputs.

Next we will walk through three representative scenarios using the calculators. Just click “Load” to bring in one of these three scenarios and follow along.

Scenario 1: Large LLM Pretrain

In this scenario, we assume that basically the entire cluster is built to run a single large pretraining job (around 80% at 4096 of 5184 GPUs). For the rest of the cluster specs, we assume a medium storage ratio of 2TB/GPU with 500TiB of hot-tier storage and 10PiB of cold-tier storage. We also assume that there is a long setup time where users tune the cluster and try to reach best performance on communications bound workloads, which leads to more time spent on EFA than InfiniBand and Spectrum-X, for example.

Overall, the discrepancy in price when comparing Gold-tier, Hyperscaler, and Silver-tier on a 3 year term is 1x, 1.10x, and 1.15x respectively for this scenario.

For the Hyperscaler, this 10% price discrepancy is primarily attributed to the additional cost of support, and setup (EFA performance tuning). We assume this amount of setup time since pretraining jobs of this size are typically collective-bound on the scale-out network, requiring aggressive EFA tuning to reach expected performance.

For the Silver-tier neocloud, the 15% price discrepancy is primarily attributed to the additional cost of goodput loss/downtime, setup (engineering time spent on health checks, performance tuning), and storage.

Now to explain the goodput calculation in more detail, in this scenario we assume that the customer has implemented three different approaches to fault tolerance depending on the provider. In the first case, on the gold-tier we assume TorchPass (or equivalent custom code) is used. On the hyperscaler, we assume HyperPod Checkpointless training. On the silver-tier, we assume a checkpoint restart for hot spare idle nodes.

The different inputs are displayed in the table below, with the key differences being that the Silver-tier provider is assumed to have a 60% worse MTBF (i.e. more total interruptions/failures), a longer time to identify a failure (assumed 1hr vs 15 mins), a longer time to repair a failed node (again assumed 1hr vs 15 mins), and a longer job init time (15 mins vs 10 mins due to worse hot-tier storage performance).

TorchPass comes with an extra expense of 32 idle GPUs (4 nodes spare, i.e. 0.62% of the cluster idle) and Checkpointless Training is assumed to have a 5% performance impact due to memory overhead.

The results are a significant difference between the three providers: 6.14%, 10.53%, and 20.91% respectively.

Scenario 2: Multimodal RL Research

Notably in the previous scenario, we kept the price per GPU-hour equal across providers. This is not typically the case. In this scenario, we assume small jobs, but no fault tolerance. We assume a large amount of storage, i.e. a high TB/GPU ratio. We also assume not much setup/debugging time is necessary to achieve equal performance across providers as the workload is primarily compute or memory bandwidth bound, not collective bound.

Overall, the discrepancy in price when comparing Gold-tier, Hyperscaler, and Silver-tier on a 3 year term is 1x, 1.61x, and 1.15x respectively for this scenario. For the Hyperscaler, this 61% price discrepancy is primarily attributed to the additional cost of the GPUs and orchestration software, storage, and setup time. For the Silver-tier neocloud, the 15% price discrepancy is primarily attributed to the additional cost of storage, with a small amount being goodput and debugging time.

Now to explain the goodput calculation in more detail, in this scenario we assume that the customer has not implemented fault tolerance in their code. In the event of a hardware interruption, the running job will wait in a queue to restart from the latest checkpoint using provider-managed hot or cold spare machines. Thus, in the event of a failure, whether a provider has hot spares available impacts the entire job. Specifically, we assume that the average job size is 64 GPUs of the total 2,048 (around 3% of the cluster), async checkpointing is configured at a 1hr interval, and job initialization time is 10-15 minutes depending on the provider’s storage performance.

As discussed in Scenario 1, we provide example inputs for the goodput calculator based on our hands-on testing experience and customer interviews. Specifically, we assume an equal GPU-level MTBF for Gold-tier and hyperscaler of around 25,000 GPU-hr, while the example silver-tier provider is assumed to have a GPU-level MTBF of 15,000 GPU-hr. We assume that both Gold-tier and hyperscaler identify failures in 15 minutes, while the silver tier provider takes 1 hour. We also assume that on Gold-tier and hyperscaler the time to replace the failed node is 15 minutes, while on the silver-tier neocloud it is 1 hour.

The results are a small difference between the three providers: 0.23% to 0.96%, demonstrating that in a scenario with many small jobs, differences in cluster reliability is much less impactful on goodput, and therefore on cluster TCO.

Scenario 3: Inference Endpoints

Notably in the previous two scenarios, we assumed that customers have not always implemented fault tolerance in their code. For this scenario, we assume that the customer is using a modern inference framework with load balancing and autoscaling built in, so whether a provider has hot spares available will not impact how long a failed request waits before being retried on another endpoint. It will only impact the length of time a node is down in a cold-swap scenario. There is also no checkpointing and initialization time due to job restarts/cold starts in this scenario. We assume that the average job is small relative to the size of the cluster, taking just single node (8 GPUs) from a cluster of 512 GPUs (1.5%).

Overall, the discrepancy in price when comparing Gold-tier, Hyperscaler, and Silver-tier on a 3 year term varies almost exclusively based on the GPU pricing in this scenario, i.e. less than 1% for equal GPU pricing between Gold-tier and Silver-tier neoclouds. For the Hyperscaler, this 59% price discrepancy can be attributed to the additional cost of the GPUs and orchestration software, storage, and support.

To explain the goodput calculation in more detail, we assume that the customer has implemented fault tolerance based on the use of a modern LLM serving framework such as llm-d or SGLang OME. In the event of a hardware interruption or cluster autoscaling up/down, the requests in flight are retried on the load balancer. Thus, in the event of a failure, whether a provider has hot spares available impacts only the uptime of the failed node, and the job keeps running. There is basically no initialization time or cold-starts once the new node has re-joined the cluster. Just load the model into GPU memory and go.

Specifically, we assume that the average job size is 8 GPUs of the total 512 (1.5% of the cluster). Notably, in scenarios with WideEP, Disaggregated Prefill/Decode, and fault-tolerant training, this job size (and resulting blast radius) would be much larger. As discussed in Scenario 1 and 2, we provide example inputs for the goodput calculator based on our hands-on testing experience and customer interviews. Specifically, we assume an equal GPU-level MTBF for Gold-tier and hyperscaler of around 25,000 GPU-hr, while the example silver-tier provider is assumed to have a GPU-level MTBF of 15,000 GPU-hr. We assume that both Gold-tier and hyperscaler identify failures in 15 minutes, while the silver tier provider takes 1 hour. We also assume that on Gold-tier and hyperscaler the time to replace the failed node is 15 minutes, however in this case we assume 8 hours for the silver-tier neoclouds, illustrating why certain workloads can tolerate this downtime even when the provider has no hot spares available, and accommodates an entire repair/replace workflow on the physical hardware.

Notably, this doesn’t make much of a difference on the total Goodput Expense. Only around 0.5% of the Cluster TCO is impacted by all these extra failures and extra downtime for the Silver-tier provider. This is a real example of why inference providers can find unused capacity from lower tier providers all around the world and use it effectively to serve single-node inference workloads for happy customers.

Conclusions, Limitations of this Work, and Comments on Future Research Directions

This article’s intention was to provide real-world data to backup up intuition that both users and providers have built on the importance of running reliable, performant, and easy-to-use clusters. In other words: even in scenarios where pricing per GPU-hour is equal, there are always hidden costs across Storage, Network, Control Plane, Support, Goodput, Setup, and Debugging expenses. We demonstrate that in three real-world scenarios, Hyperscalers can be over 10% more expensive on a TCO-adjusted basis vs Gold-tier providers, even when holding GPU-hr pricing equal. And we demonstrate that silver-tier neoclouds can be over 15% more expensive when holding GPU-hr pricing equal.

Going forward, we intend to apply this methodology to all ClusterMAX rated providers, specifically during our upcoming ClusterMAX 3.0 testing this summer. We also intend to collect real-world data on MTBF. To that end, if you are a customer of a neocloud that tracks failure data manually or through an automated system, dmesg logs, or are willing to contribute data in an anonymous, aggregated manner, please reach out! Even intuition on past experience mentioning failures/day or failures/week and cluster size is helpful. We would love to hear from you. Again: clustermax@semianalysis.com.

Beyond the scope of ClusterMAX, we continue to work with users running large clusters to compare performance differences between neoclouds. Specifically, the NVIDIA DGXC benchmarking repo and related NCP/DGXC certification process reveals that even providers following NVIDIA’s reference architecture can experience performance differences on different workloads. This is especially true when comparing the interconnect network for collective-bound operations, which itself is becoming more and more common due to the adoption of wide EP, PDD, and other parallelism techniques that take advantage of massive interconnect bandwidth on both the scale up and scale out domains.

On pricing, all inputs for this analysis and defaults displayed on the calculator are to be considered as a point-in-time analysis based on historical pricing data from August, 2025. We track neocloud pricing of all major GPUs globally for different cluster sizes and commitment terms in our GPU Pricing Data. And as we described in a recent article, those prices are going up. We continue to update this data series over time for our subscribers. This is done on a daily basis for spot instance pricing, and a monthly basis for cluster pricing.

The functionality of fault tolerant frameworks needs to improve. Today, TorchFT is the only open source option and is not widely adopted for training. Meanwhile all three options we explored in this article (TorchFT, Hyperpod Checkpointless, and TorchPass) come with tradeoffs on communications overhead, memory overhead, handling hard failures, and cost of idle nodes. This leaves fault tolerant training as a secret sauce available to frontier labs or those willing to pay for a software license. Meanwhile, fault tolerant inference is the standard for single 8-way systems, while it is being actively built into PDD and WideEP frameworks such as NVIDIA Dynamo, including at the KV Cache offloading level with frameworks such as LMCache and Mooncake.

ClusterMAX 2.1 Update

This update adds a small set of new providers to the ClusterMAX rating system. This is not a full re-test of all providers. We are actively conducting ClusterMAX 3.0 testing with a focus on the latest and greatest: B300 and GB300 with XDR/800Gb networking.

Core42

Core42 is a division of G42 with a massive presence in the UAE and a growing presence in the US. With the backing of MGX, and the sister company Khazna Datacenters, all of whom are intimately involved with Stargate UAE, the group means business. Back on the US side, Core42 is also making moves. They have established small sites in San Jose, Grenoble, and 70MW of MI300X in Buffalo (via Terawulf). During our testing we were provided with both slurm and kubernetes clusters from that site, using AMD MI300X GPUs, and crucially some Broadcom Thor-II NICs. This was the first cluster we’d gotten with Thor-II during clustermax testing, and it was a battle. Every single container image we had previously tested on AMD clusters, and nearly every AMD base image they publish to rocm repos such as vllm, sglang, torchtitan, and MoRI are all built with AMDs own Pollara NICs. This meant downloading tarballs from Broadcom’s driver search website, scp’ing the files over to the cluster nodes, and rebuilding containers from scratch. A headache to say the least. Notably, the Core42 engineering team was ready to help the entire way, from troubleshooting these driver recipe issues to debugging slurm user errors on our side it was a really strong showing of hands-on, proactive technical support. If Core42 launches some modern GPUs in the US or starts relaxing the compliance restrictions they have in place that prevent us from testing in the UAE sites (or anyone from outside UAE renting GPUs at those sites) we expect Core42 to quickly rise into the silver tier and beyond.

BitDeer

We conducted some initial testing with BitDeer at their Malaysia site using 2 nodes of GB200 NVL72. We were limited on time and could not get the IMEX domain configured correctly to confirm the NVLink was setup for intranode communication on the NVL72 domain. We did run some training jobs and figure out the console successfully. With many more GPUs coming online this year, we are excited to see more from BitDeer in terms of orchestration software, monitoring, reliability and support for the big clusters they have announced they’re building.

FPT Smart Cloud

We got the chance to test FPT Smart Cloud back in December. FPT is based in Vietnam and at the time had H100 and H200 available. They use Soperator from Nebius for orchestration, and the cluster was well configured. We noticed some poor performance on the VAST Storage. The monitoring experience was quite strong, some of the best custom DCGM dashboards we have seen with Loki used for logging and analysis. Unfortunately, FPT is help back from the silver tier due to some serious security issues. Our testing showed that PKeys and SAKey were not configured correctly, allowing us to see every other endpoint on the network (i.e. every other customer).

Radiant/Ori

Radiant was announced recently after Brookfield acquired Ori, a Saudi Aramco backed neocloud with H100s and H200s in London and Dallas. When we tested with Ori on two occasions in the fall, we saw some quick progress but not enough to get to silver. Ori fell victim to the exact same issues as FPT, with PKey and SAKey not configured correctly. In addition, during our first round of testing, we were unable to run nccl-tests at full bandwidth on kubernetes due to an issue with the NetworkOperator picking up NICs that were intended to be for the frontend but were named/configured incorrectly to be used for NCCL. Finally, DCGMI health watches are not enabled by default, and there is no automated background health check program. Our testing of a simple hardware failure simulation showed that the system did not trigger any automated alerting or node replacement over an 18-hour window. The team is targeting Q2 2026 for the release of monitoring dashboards, and seems well on their way to having the funding they need to build Blackwell clusters with comprehensive slurm, kubernetes, monitoring and reliability features customers expect.

Others

We add coverage for Tatra Supercompute (Slovakia), QumulusAI (Texas, Oklahoma), Boostrun (Seattle, Texas, North Carolina), Moonlite (reseller/operator) Vessl (marketplace from Korea), SK Telecom (also Korea), and BytePlus (division of ByteDance) as significant players worth tracking as they bring capacity online this year.

Behind the paywall we will now dig into how Goodput Expense is impacting real companies. Specifically, we will walk through and analyze the margin of some inference endpoint providers, conducting a sensitivity analysis that considers Goodput’s impact on these margins.

Inference Endpoint Margins Sensitivity

Introduction: Rethinking the Total Cost of a GPU Cluster

Definitions and Key Terms

To evaluate GPU cloud offerings on equal footing, we break down the TCO of a GPU cluster as follows.

Our Proposed TCO Formula for GPU Clusters

The following formula is used to calculate the Total Cost of a GPU Cluster on a monthly basis:

Next, we define G_chkpt-hot, G_chkpt-cold, and G_tolerant, i.e. the different ways to calculate goodput expense.

The Grand Unifying Theory Of Goodput

We summarize all of this in our TCO Calculator as “Goodput Expense”, where the following formulae are used to calculate Goodput Expense under three scenarios:

In the scenario of a fault-tolerant training framework, we consider three options, which we describe in more detail below:

TorchFT

AWS SageMaker HyperPod Checkpointless Training

TorchPass

The results are clear when compared to checkpoint restart and TorchFT with performance overhead. Recovery times are similar for planned migrations, and the job performance is similar.

Overview of Three Cloud Providers Being Assessed

Now, to demonstrate how to use the calculator, we use three representative providers.

These are not direct comparisons, but rather an amalgamation of the average experience using the providers in the given tier. Roughly speaking:

Gold-tier

Hyperscaler

Silver-tier

Applying the TCO and Goodput Formulae

Next we will walk through three representative scenarios using the calculators. Just click “Load” to bring in one of these three scenarios and follow along.

Scenario 1: Large LLM Pretrain

Overall, the discrepancy in price when comparing Gold-tier, Hyperscaler, and Silver-tier on a 3 year term is 1x, 1.10x, and 1.15x respectively for this scenario.

TorchPass comes with an extra expense of 32 idle GPUs (4 nodes spare, i.e. 0.62% of the cluster idle) and Checkpointless Training is assumed to have a 5% performance impact due to memory overhead.

The results are a significant difference between the three providers: 6.14%, 10.53%, and 20.91% respectively.

Scenario 2: Multimodal RL Research

Scenario 3: Inference Endpoints

Conclusions, Limitations of this Work, and Comments on Future Research Directions

ClusterMAX 2.1 Update

Core42

BitDeer

FPT Smart Cloud

Radiant/Ori

Others

Inference Endpoint Margins Sensitivity

There are three major semiconductor conferences each year, IEDM, VLSI and finally ISSCC. We have covered the former two in great detail over the past few years. Today, we finally complete the trinity with our roundup on ISSCC 2026.

Compared to IEDM and VLSI, ISSCC has a much bigger focus on integration and circuits. Almost every paper comes with some form of circuit diagram, together with clear measurements and data.

In past years, ISSCC findings have been hit or miss when it comes to industry impact. This year was different, a significant number of papers and presentations were directly relevant to market trends. Topics covered range from the latest advancements in HBM4, LPDDR6, GDDR7, and NAND, to co-packaged optics, advanced die-to-die interfaces, and advanced processors from the likes of MediaTek, AMD, Nvidia, and Microsoft.

In this roundup, we will cover major categories such as Memory, Optical Networking, High-Speed Electrical Interconnects, Processors.

Memory

One key theme that caught our attention at this year’s ISSCC was memory, including Samsung HBM4, Samsung and SK Hynix LPDDR6, and SK Hynix GDDR7. Other than DRAM, logic-based SRAM and MRAM also piqued our interest.

Samsung HBM4 - Paper 15.6

Samsung was the only one among the top three memory vendors to present a technical paper on HBM4. Before ISSCC, we noted in our Accelerator & HBM model that Samsung had made great improvements in their HBM4 generation over their HBM3E. The data presented at ISSCC confirmed our analysis, with Samsung posting best-in-class performance - we have also detailed this development months ago, in a model update note.

The technical details presented at ISSCC, combined with industry chatter we have gathered, clearly demonstrate that Samsung’s HBM4 is competitive with its peers. Notably, it can meet the pin speed required for Rubin while staying below 1V. While Samsung still lags SK Hynix in terms of reliability and stability, the company has made meaningful progress in closing the gap on the technology front and could challenge SK Hynix’s dominance in HBM. Their 1c-based HBM4 paired with an SF4 logic base die appears to deliver stronger performance in pin speed.

Samsung demonstrated a 36 GB, 12-high HBM4 stack featuring 2048 IO pins and 3.3 TB/s of bandwidth, built using 6th-generation 10nm-class (1c) DRAM core dies paired with an SF4 logic base die.

The most obvious architectural change from HBM3E to HBM4 is the process technology split between the core DRAM dies and the base die. HBM4 uses the DRAM process node only for the core die while the base die is manufactured with an advanced logic node unlike previous generations of HBM that used the same process for both.

The key architectural challenge arises as AI workloads demand higher bandwidth and faster data rates from HBM. By moving the base die to the SF4 logic process, Samsung enables higher operating speeds and lower power consumption. The operating voltage (VDDQ) fell 32%, from 1.1V in HBM3E to 0.75V in HBM4. A logic-based base die provides higher transistor density, smaller device dimensions, and better area efficiency due to smaller transistors and larger metal-layer stack availability as compared to a base die fabricated on a DRAM process. This helps Samsung’s HBM4 achieve — and significantly surpass — JEDEC’s HBM4 standard that we explain more at the end of this section.

Combined with adaptive body-bias (ABB) control, which mitigates process variation across stacked core dies, the doubled TSV counts further improve timing margin. Together, Samsung’s paper claimed that the ABB and the 4× higher TSV count allow their HBM4 to achieve operating speeds up to 13 Gb/s per pin.

The improvement brought by the SF4 base die and 1c DRAM core dies comes with a trade-off. Samsung’s choice of SF4 for the logic base die comes at a higher cost compared with competing approaches even though Samsung Foundry can offer discounts for their internal base die usage. SK Hynix is adopting TSMC’s N12 logic process for their HBM4 base die, while Micron relies on their internal CMOS base-die technology, both of which are lower-cost options than the near leading-edge SF4 node, even considering vertical integration cost advantages.

The 1c front-end manufacturing process has proved challenging for Samsung throughout 2025, especially given that the company skipped the 1b node and moved directly from 1a-based HBM3E to the 1c generation. Front-end yields for the 1c node were only around 50% last year, although they have been gradually improving over time. The lower yield poses a risk for their HBM4 margins.

Historically, Samsung’s HBM has earned lower margins than those of their top competitor, SK Hynix, a dynamic that we model across all vendors comprehensively in our Memory Model. We have detailed wafer volumes, yields, density, COGS, and more for each vendors HBM, DDR, and LPDDR across various nodes.

Samsung’s strategy appears to be an aggressive adoption of a more advanced node for the base die to achieve superior performance and outpace their competitors, particularly as HBM requirements from leading customers such as NVIDIA continue to become more demanding.

Another key issue in HBM to address is tCCDR, the minimum interval required between consecutive READ commands issued across different stack IDs (SID). For AI workloads that rely heavily on parallel memory access across many channels, tCCDR directly impacts achievable memory throughput.

In a stacked DRAM architecture, multiple core dies are vertically integrated on top of a base die. This naturally introduces small delay differences across the stack, driven by factors such as process variation between the core dies and the base die, TSV propagation differences, and local channel variation.

The increased stack heights and channel counts, from 16 to 32, compound this challenge. As the channel counts and stack heights increase, the variation between the dies accumulates, causing larger timing mismatches across channels and dies that impact the achievable tCCDR and overall HBM performance.

To address this issue, Samsung introduces a “per-channel TSV RDQS timing auto-calibration scheme.” After power-up, the system measures delay variation across channels using a replica RDQS path that mirrors the timing behavior of the real signal path. A time-to-digital converter (TDC) quantizes the timing differences, which are then compensated for using delay compensation circuits (DCDL) for each channel.

This calibration accounts for both global delay variation between stacked core dies and local per-channel variation, aligning timing across the stack. By compensating for these mismatches, Samsung significantly improves the effective timing margin and increases the maximum achievable data rate while maintaining the required tCCDR constraints. This scheme alone increased data rates from 7.8 Gb/s to 9.4 Gb/s.

Some of our readers who are well versed in memory technology may be asking: How is there enough die area to accommodate the significant increase in TSV counts? This is where the 1c node becomes important. Compared with the previous 1a node, 1c further shrinks the DRAM cell area, freeing up die space that can be used to integrate the larger number of TSVs required for HBM4.

Another key innovation enabled by the logic base die is Samsung’s Programmable Memory Built-In Self-Test (PMBIST) architecture. PMBIST allows the base die to generate fully programmable memory test patterns while supporting the complete JEDEC row and column command set, meaning the test engine can issue the same commands that a real system would generate and can do so at any clock edge and at full interface speed. In practical terms, this allows engineers to replicate complex real-world memory access patterns and stress the HBM interface under realistic operating conditions, which is difficult with traditional fixed-pattern test engines.

This approach represents a notable departure from HBM3E. As discussed earlier, the HBM3E base die is fabricated using a DRAM process, which imposed strict power and area constraints on the MBIST (Memory Built-In Self-Test) engine and limited testing to a small set of predefined patterns given the natural power and area disadvantage of DRAM against logic. By moving the base die to Samsung Foundry’s SF4 logic process, Samsung enables a fully programmable testing framework capable of running complex test algorithms and flexible access sequences.

This enables much more robust debugging and better yield learning for HBM. Engineers can create targeted stress patterns to validate critical timing parameters such as tCCDR and tCCDS, identify corner-case failures earlier in manufacturing, and accelerate characterization during chip-on-wafer (CoW) and system-in-package (SiP) testing. Put simply, PMBIST improves test coverage, debug efficiency, and ultimately production yield as HBM stacks grow more complex and operate at higher speeds.

Samsung also demonstrated strong pin speed results — their HBM4 is able to hit 11 Gb/s at sub-1V core voltage (VDDC), and up to 13 Gb/s at higher voltages. We have yet to see Samsung’s peers demonstrate comparable performance albeit they do have better reliability and stability.

Samsung’s implementation significantly exceeds the baseline specification of the official JEDEC HBM4 standard (JESD270-4), which specifies a maximum data rate of 6.4 Gb/s per pin and about 2 TB/s of bandwidth. Samsung demonstrates more than 2× the JEDEC-standard pin speed, reaching 13 Gb/s per pin and delivering 3.3 TB/s of bandwidth. Even at VDDC/VDDQ of 1.05V and 0.75V, the device can sustain a data rate of 11.8 Gb/s.

Samsung LPDDR6 - Paper 15.8

Both Samsung and SK Hynix showed off their LPDDR6 chips. We will discuss Samsung’s chips first and turn to SK Hynix’s later.

Samsung presented their LPDDR6 architecture and detailed the power-saving techniques used.

LPDDR6 adopts a 2 sub-channel per die architecture, with 16 banks in each sub-channel. It also features two modes: a normal mode and an efficiency mode. In the efficiency mode, the secondary sub-channel is powered down, and the primary sub-channel controls all 32 banks. However, there is a latency penalty for accessing data in the secondary sub-channel.

The dual sub-channel architecture also means that there is twice the amount of peripheral circuitry, such as command decoders, serialization and control. From the die shots provided by both Samsung and SK Hynix, the penalty is about 5% of the total die area, leading to a reduction in total bits per wafer.

Unlike GDDR7, which uses PAM3 signaling, LPDDR6 will continue to use NRZ. However, it does not use standard NRZ as the eye would not have sufficient margin. It uses wide NRZ with 12 data (DQ) pins per sub-channel and a burst length of 24 per operation.

For those of you doing the math, 12×24 is 288, not a power of two. The remaining 32 bits are split into 2 use cases, 16 for metadata like ECC, and 16 for Data Bus Inversion (DBI).

DBI is a power-saving and signal integrity mechanism. Before a burst is sent out, the controller checks if more than half the bits would switch state compared to the previous burst. If so, the controller inverts all the bits and sets a DBI flag, so that the receiver knows to invert them to get the actual data. This limits the maximum number of simultaneous switching outputs to half the bus width, reducing power consumption and supply noise.

To calculate the effective bandwidth, you must account for these metadata and DBI bits like so: Bandwidth = Data Rate × Width (24 b) × Data (32 b) / Packet (36 b).
For 12.8 Gb/s, you get 34.1 GB/s, and for 14.4 Gb/s, you get 38.4 GB/s.

LPDDR6 has two constant power domains, VDD2C at 0.875V and VDD2D at 1.0V. By carefully choosing which peripheral logic is using which power domain, read power has been reduced by 27% and write power reduced by 22%.

LPDDR is primarily used at low data rates of 3.2 Gb/s and below when idling. Samsung focused heavily on saving power at these lower data rates through careful use of the voltage domains, reducing both standby and read/write power consumption.

By using a redistribution layer (RDL), Samsung can locate related circuits closer together physically. This shortens critical delay paths and reduces their sensitivity to voltage and temperature variation. At the high frequencies of LPDDR6, tighter timing and reduced variation are essential.

Samsung’s LPDDR6 can reach a data rate of 12.8 Gb/s at 0.97V, and up to 14.4 Gb/s at 1.025V. Each 16 Gb die is 44.5 mm², with a density of 0.360 Gb/mm² on an unknown 10nm-class process. This is considerably lower than the density of LPDDR5X on 1b at 0.447 Gb/mm² and only slightly higher than the density of LPDDR5X on 1a at 0.341 Gb/mm². While the area penalty from the dual sub-channel architecture does partially contribute, there seem to be other problems with the LPDDR6 as well. The memory density described leads us to believe that this prototype LPDDR6 chip was manufactured on their 1b process.

Samsung SF2 LPDDR6 PHY - Paper 37.3

Samsung also unveiled PHYs on the logic die interface with LPDDR6. The PHYs are fabricated on their new SF2 process and support up to 14.4 Gb/s. The PHYs take up 2.32 mm of shoreline and 0.695 mm² of area, with bandwidth densities of 16.6 Gb/s/mm and 55.3 Gb/s/mm² respectively.

The PHYs also support the efficiency mode implemented by the LPDDR6 chips, which can reduce read power by 39% and write power by 29%.

The PHYs can augment the efficiency mode by gating the high-speed clock path for the inactive secondary sub-channel. With clock-gating, the power reduction reaches almost 50% for reading and writing, and idle power is reduced by 41%.

SK Hynix 1c LPDDR6 - Paper 15.7

SK Hynix unveiled their first 1c DRAM products, both in LPDDR6 and in GDDR7 packages. Their LPDDR6 can operate at a data rate of up to 14.4 Gb/s, 35% faster than the fastest LPDDR5X, and at lower power.

Although SK Hynix did not state the area or density of the LPDDR6 chip, we estimate the bit density will reach 0.59 Gb/mm², based on the relative density increase of their GDDR7.

In their shmoo plot, SK Hynix showed that they can reach a data rate of 14.4 Gb/s at 1.025V, the same as Samsung. However, they can only reach 10.9 Gb/s at 0.95V as compared to Samsung’s 12.8 Gb/s at 0.97V. This indicates that SK Hynix may have worse power efficiency at lower pin speeds when compared to Samsung, having to run at higher voltages to maintain reliability.

Like Samsung’s LPDDR6, SK Hynix’s LPDDR6 also features two modes, a normal mode and an efficiency mode. The efficiency mode runs at 12.8 Gb/s over a single sub-channel, with 12.7% and 18.9% lower standby and operational current draw respectively compared to normal mode.

SK Hynix 1c GDDR7 - Paper 15.9

While the LPDDR6 is a generational leap with new memory technology, SK Hynix’s GDDR7 on their 1c process shows an even greater improvement, clocking up to 48 Gb/s at 1.2V/1.2V. Even at only 1.05V/0.9V, it can clock up to 30.3 Gb/s, higher than the 30 Gb/s memory in the RTX 5080.

The bit density achieved is 0.412 Gb/mm², compared to 0.309 Gb/mm² on Samsung’s 1b process, and 0.192 Gb/mm² on Samsung’s older 1z process.

GDDR7 has lower bit density than LPDDR5X, usually around 70% of the latter. Although it has much higher data rates, this comes at a cost, both in terms of power and area.

GDDR7’s lower density is a result of the significantly higher periphery area for high access speeds. The actual memory arrays thus make up a smaller percentage of die area. This more complex logic control circuit is required for the PAM3 and QDR (4 symbols per clock cycle) signaling used in GDDR7.

GDDR7 is mainly used in gaming GPU applications that require high memory bandwidth at lower cost and capacity compared to HBM. NVIDIA had announced the Rubin CPX large-context AI processor in 2025 with 128GB of GDDR7, but this has all but vanished from the 2026 roadmaps as NVIDIA focuses on rolling out their Groq LPX solutions instead.

Samsung 4F² COP DRAM - Paper 15.10

The architecture for 4F² cells is the same as SK Hynix’s, with vertical channel transistors (VCT), and capacitors above the drain.

The vertical architecture presented by Samsung is essentially the same as that used by SK Hynix, with a cell wafer hybrid bonded on top of a peripheral wafer. With this architecture, it is possible to use a DRAM node for the cell wafer while using a more advanced logic node for the periphery.

Samsung notes that hybrid bonding for COP has already been used for NAND. This is true for other NAND manufacturers, but Samsung has not brought hybrid bonding for NAND into high volume production and is still years away from doing so.

Moreover, the number of inter-wafer connections for DRAM is an order of magnitude higher than for NAND and requires much tighter pitches. To reduce the number of inter-wafer interconnections, Samsung has employed two novel approaches.

First, they have reorganized the sub-wordline drivers (SWD) from 128 per cell block to 16 groups of 8. This reduces the number of signals required for the SWD by 75%.

Next, they split the column select into an even and an odd path. This requires twice the multiplexers (MUX) but halves the column select line (CSL) count to 32 per data pin.

With hybrid bonding, the core circuitry, that is, bitline sense amplifiers (BLSA) and SWD can be placed under the cell array. The goal is for the core circuits to occupy the same area as the cell array to increase overall density.

Samsung adopted the “sandwich” structure, which allows them to maximize the area efficiency of the core circuitry, and reduce the edge region area, which is not under any cells.

The area used by the core circuitry was reduced from 17.0% down to only 2.7%, a significant improvement, directly translating to overall die size reductions.

In traditional DRAM, increasing the number of cells per bitline would result in a significant increase in chip area, while for VCT DRAM, the increase is almost negligible as the core circuitry is all below the cells.

Samsung did not provide any density figures for this chip, only stating that it was a 16 Gb chip on a 10nm DRAM process.

Samsung noted that the VCT DRAM suffers from the floating-body effect, increasing leakage and reducing retention time. Mitigating this effect remains a key challenge for 4F² adoption.

Despite these challenges, we still expect 4F² hybrid bonded DRAM to arrive in the latter part of the decade as early as the generation after 1d. Our memory model tracks the timing and ramp of each node in detail. The current memory pricing landscape largely incentivizes the ramp and introduction of new nodes with higher bit densities to improve bit output per fab. On the otherhand, performance/$ of memory is highly sought after more than capacity for many use cases.

SanDisk/Kioxia BiCS10 NAND - Paper 15.1

SanDisk and Kioxia demonstrated their BiCS10 NAND, with 332 layers and 3 decks. This is the highest reported NAND bit density, at 37.6 Gb/mm², dethroning the previous champion, SK Hynix’s 321L V9.

Despite using a similar architecture with 6 planes, 3 decks, and a similar number of layers, SK Hynix falls behind, with 30% lower bit density. In a QLC configuration, BiCS10 has a bit density of 37.6 Gb/mm², while SK Hynix’s V9 has a bit density of only 28.8 Gb/mm². While in a TLC configuration, the densities are 29 and 21 Gb/mm² respectively, another example of SK Hynix’s trailing position.

Moreover, BiCS10 features a 6-plane configuration, increasing IO bandwidth by 50%. There are two ways to implement a 6-plane configuration, 1×6 and 2×3. SK Hynix chose to use a 2×3 configuration, while SanDisk and Kioxia have decided to use a 1×6 configuration.

A 1x6 configuration has fewer ground pads and reduces area by 2.1%. However, the lower number of ground pads and vertical power tracks constrains power distribution.

By using the CBA (Cell Bonded Array) architecture, SanDisk and Kioxia are able to customize the CMOS wafer process. By adding another top-metal layer in parallel to the existing one, they created a stronger power grid and overcame their power distribution constraints.

Stacking more dies is essential to increasing storage density. However, in multi-die architectures, the idle current from unselected dies is approaching the active current of the selected die. SanDisk implemented a gating system to fully shut down the data path of the unselected dies, reducing the idle current by two orders of magnitude.

MediaTek xBIT Logic-based Bitcell - Paper 15.2

SRAM scaling is dead. Despite logic area decreasing by 40% from N5 to N2, 8-transistor high-current SRAM bitcells have only decreased in area by 18%. 6-transistor high-current (6T-HC) bitcells are even worse, only decreasing by 2%. Assist circuitry has scaled more, but it's not free lunch.

It is well known that N3E’s high-density bitcell is a regression from N3B’s, falling back to N5’s density. In this paper, MediaTek shed some light on the high-current bitcell. N3E’s high-current bitcell increased in area by 1-2% over N5. The density decreased from ~39.0 Mib/mm² to ~38.5 Mib/mm². Do note these figures do not account for assist circuitry overhead.

In modern logic nodes, 6T bitcells have 4 NMOS and 2 PMOS transistors, while 8T bitcells have 6 and 2 respectively. The unequal number of NMOS and PMOS transistors requires specialized rules and makes layout more inefficient. MediaTek’s novel bitcell is a 10-transistor cell, named the xBIT, with 4 NMOS and 6 PMOS transistors or vice versa. The two variants of the bitcell can be arranged together into a rectangular block, with 20 transistors, storing 2 bits.

When compared to the PDK’s standard 8T bitcells, the xBIT achieved 22% to 63% higher density, with the largest gains at lower wordline widths. Power has also improved greatly, with the average read/write power reduced by over 30%, and leakage reduced by 29% at 0.5V. At 0.9V, performance was similar to an 8T bitcell and at 0.5V, although it was 16% slower than the 8T bitcell, it was fast enough not to be the bottleneck in a processor, and the voltage range was large enough for voltage-frequency scaling.

MediaTek also showed a shmoo plot of the xBIT cell, going from 100 MHz at 0.35V, up to 4GHz at 0.95V.

We will be doing a deep dive into SRAM and its scaling factors in an upcoming newsletter article.

TSMC N16 MRAM - Paper 15.4

TSMC presented an updated STT-MRAM on their N16 node, building on their previous work from ISSCC 2023. TSMC is positioning the MRAM as an embedded non-volatile memory (eNVM), for use in automotive, industrial, and edge applications, which do not need the most advanced technology but instead reliability.

The MRAM features dual-port access so reads and writes can occur simultaneously — critical for over-the-air (OTA) updates in automotive, where the system cannot halt reads while firmware is being written.

It features interleaved reads across modules with independent clocks, raising throughput to 51.2 Gb/s at 200 MHz. On silicon, the 84 Mb macro achieves 7.5ns read access time at 0.8V across -40 °C to 150 °C.

The architecture is modular — configurable in 16 Mb, 8 Mb and 2 Mb modules that compose into macros from 8 Mb to 128 Mb. By combining large 16 Mb modules with a few smaller 2 Mb and 8 Mb modules, the capacity can be fine tuned to the needs of any design. For example, 5× 16 Mb modules and 2× 2 Mb modules form an 84 Mb macro.

As stated earlier, reliability is where embedded MRAM lives or dies. After 1 million endurance cycles at -40 °C, the hard error rate stays well below 0.01 ppm — well within ECC correction range. Read disturb at 150 °C is below 10⁻²² ppm at typical read voltages, effectively negligible. The 168 Mb test chip passes reflow and supports 20-year retention at 150 °C, meeting stringent automotive requirements.

Compared to the old MRAM on the same N16 node, the bitcell has shrunk 25% from 0.033 µm² to 0.0249 µm², and macro density increases to 16.0 Mb/mm² iso-capacity. The read speed drops from 6 ns to 5.5 ns iso-capacity, and the dual-port access and interleaved reads are entirely new.

While Samsung Foundry also published work on 8LPP eMRAM this year, TSMC’s is far more promising. It targets the needed features, has great performance, and is on a cheaper N16 node.

TSMC is already planning the next-generation “Flash-Plus” variant with a 25% smaller bitcell and 100× higher endurance.

Optical Networking

Several papers from major optics vendors tackled optical interconnects that will carry data between next-generation AI accelerators both within and between datacenters.

Nvidia DWDM - Paper 23.1

The choice of optical signaling format will affect the go-to-market timeline for scale-up co-packaged optics (CPO). Nvidia is ramping up production of COUPE optical engines that support 200G per lane PAM4 for scale-out switching in the near-term.

However, at ISSCC, Nvidia proposed using 32 Gb/s per lambda, with 8 wavelengths multiplexed using DWDM. A 9^th wavelength is used for clock forwarding at half rate — that is 16 Gb/s.

Clock forwarding means that SerDes can be made somewhat simpler by removing the Clock and Data Recovery (CDR) circuit as well as other circuitry, improving energy and chip shoreline efficiency.

Earlier in March, just before OFC 2026, the formation of the Optical Compute Interconnect MSA (OCI MSA) was announced, and it will focus on a 200 Gb/s Bi-directional link, with each of transmit and receive formed using 4 lambdas of 50G NRZ, which will be sent bi-directionally across the same fiber. Did I hear anyone mention OCS?

Interestingly, the OCI MSA did not make use of an additional wavelength for clock forwarding, and it appears that reserving all wavelengths for actual data transmission is the priority.

Much of Nvidia’s published research on scale-up CPO has centered on DWDM, though today’s CPO optical engines are oriented around 200G PAM4 DR Optics, which makes more sense for scale-out networking. The OCI MSA centering around DWDM for scale-up optics resolves this apparent contradiction, as it is now clear that Nvidia and others will center around the use of DWDM for scale-up and DR Optics for scale-out.

The OCI MSA also illustrates different implementations, an On-Board Optics (OBO), a version of CPO that is integrated via the substrate on the ASIC package, and a version where the optical engine is integrated directly on the interposer. The implementation illustrated in the middle figure (b) will be the most common one used for scale-up and scale-out CPO for the next few years, but it still requires some form of serialized links that can pass through the ASIC substrate and will still require some form of SerDes on both sides. For example, UCIe-S could be used as a protocol for such transmission.

The “Final Boss” when it comes to implementing CPO will be when the optical engine can be integrated onto the interposer itself, connecting to the ASIC using a parallelized die to die (D2D) connection as depicted in (c) above. This could considerably improve shoreline bandwidth density, enable much higher radix and improve energy efficiency. This implementation thus unlocks benefits of CPO in ways that the other implementations cannot, but achieving it is still a few years away and requires further improvements in advanced packaging technology.

Marvell Coherent-Lite Transceiver - Paper 23.2

Marvell presented an 800G transceiver for coherent-lite applications. Traditional transceivers have a limit on how far they can reach, less than 10 kilometers. Coherent transceivers support much further reach, but they are complex, consume more power, and are more expensive. Marvell’s Coherent-lite transceiver targets a middle ground with respect to power consumption, cost and range, which is perfect for large datacenter campuses with links spanning at most tens of kilometers.

Coherent transceivers primarily use C-band wavelengths for their low attenuation. However, the long-haul links in which coherent transmission is used typically have a very high dispersion, needing heavy DSP processing. The long range of traditional Coherent optics can often be overkill for datacenter campuses with buildings that are only tens of kilometers apart.

Coherent-Lite transceivers instead use O-band wavelengths, which have near-zero dispersion over the relatively short distances on a datacenter campus. This enables minimal DSP processing, saving power and reducing latency.

The Coherent-lite transceiver is a DSP-based pluggable module consisting of two 400G channels. Each 400G channel runs a dual polarization QAM and consists of two parallel modulation streams, X and Y.

The key to this demonstration is highlighting other methods of scaling channel bandwidth that are optimized for the campus application.

Higher-order modulation coupled with the dual polarization using X and Y axes delivers 400G channel bandwidth. There are 8 bits per channel for a total of 32 constellation points as demonstrated above. These 8 bits times the 62.5 GBd signal rate equals ~400G of total bandwidth.

This modulation scheme is not entirely new to the industry, but it is now being brought into the datacenter campus setting for use in those shorter links.

Marvell’s approach significantly reduces power to only 3.72 pJ/b excluding silicon photonics, half of other full-fledged coherent transceivers. Their measurements were taken over a fiber length of 40km, with a latency of less than 300 ns.

Broadcom 6.4T Optical Engine - Paper 23.4

Broadcom showcased progress on their 6.4T MZM optical engine (OE) consisting of 64 lanes of ~100G using PAM4 modulation. The optical engines were tested in a Tomahawk 5 51.2T CPO system. One CPO package consists of eight 6.4T OEs, each with a PIC and an EIC, on TSMC’s N7 process.

While Nvidia uses COUPE, Broadcom uses a Fan-Out Wafer-Level Packaging approach for this OE. Broadcom will be switching to COUPE in the future, but older generation products like this OE still use other supply chain partners. Here are the promising results from their demonstration:

High-Speed Electrical Interconnects

As multi-die designs become the norm, die-to-die interconnects are a critical bottleneck. Major foundries and chip designers presented approaches to pushing bandwidth density and energy efficiency on both organic substrates and with advanced packaging.

Intel UCIe-S - Paper 8.1

Intel presented their UCIe-S compatible die-to-die (D2D) interface. It can reach up to 48 Gb/s/lane over 16 lanes with UCIe-S and up to 56 Gb/s/lane with a custom protocol. It works on a standard organic package for a distance of up to 30mm. Interestingly, it was manufactured on Intel’s 22nm process.

At VLSI 2025, Cadence demonstrated their own UCIe-S die-to-die interconnect on N3E. Despite the node disadvantage, Intel has managed to best Cadence’s interconnect in data rate, channel length and shoreline bandwidth, only losing out in energy efficiency.

The interconnect presented by Intel is likely to be a prototype of what will be used on their Diamond Rapids Xeon CPUs. Efficiency should be much better when designed on the Intel 3 process compared to this 22nm test chip and could serve to replace the advanced packaging approaches like EMIB on Granite Rapids. As we have covered in our article on the Landscape of Datacenter CPUs, Diamond Rapids consists of two IMH dies, and 4 CBB dies. With the long traces between each CBB die to both IMH dies, we believe this link is a viable candidate to connect the dies over standard package substrate, negating the need for EMIB.

TSMC Active LSI - Paper 8.2

TSMC’s advanced packaging division presented their Active Local Silicon Interconnect (aLSI) solution. As opposed to standard CoWoS-L or EMIB, aLSI improves signal integrity and reduces the complexity of PHYs and SerDes on the top dies.

The device that TSMC showed used a 32 Gb/s UCIe-like transceiver. Due to aLSI improving signal integrity, the area of the transceivers could be decreased, and the bump pitch could also be reduced from 45 µm to 38.8 µm. The combination of a tighter pitch and a switch to a Manhattan grid allowed them to reduce the PHY depth from 1043 µm to 850 µm, saving space that designers can reallocate to compute, memory, or IO, or use to shrink the die. The transceiver is only UCIe-like and not true UCIe, as UCIe mandates a hexagonal bump map rather than the Manhattan grid used here.

As designers eke out every bit of die space for next-generation AI accelerators, the switch to aLSI is inevitable.

The ‘active’ part of aLSI comes from replacing the passive long-reach metal channel in the bridge die with active transistors forming an Edge-Triggered Transceiver (ETT) circuit to maintain signal integrity over longer reaches. This also reduces the signal driving requirements of the top die’s Transmit/Receive ports. ETT circuits within the aLSI only add an additional 0.07pJ/b to the energy cost, minimizing thermal concerns from adding active circuits in stacked dies. By moving the signal conditioning circuits to the bridge die, PHY area on the top die TX/RX can be reduced by using smaller pre-drivers and clock buffers and remove the need for signal amplification on the receive end.

The ETT integrates a driver, an AC-coupling capacitor (Cac), an amplifier with both negative and positive feedback, and an output stage. Running the signal through the Cac introduces peaks in the signal transition edges, which is then picked up by the dual-loop amplifier, hence the edge-triggered nomenclature. The amplifier leverages both positive and negative feedback loops to stabilize the voltage level. In this design, Cac is set to be 180 fF for a 1.7 mm channel length, with 2kΩ resistance on die A and 3kΩ on die B respectively.

These aLSI bridges can also integrate embedded deep trench capacitors (eDTC) along the front-end to improve power delivery to the PHY and D2D controllers. Instead of compromising the power grid by having a bridge die in the way, aLSI with eDTC improves both the power and signal routing along the D2D interfaces.

Only 388 µm of shoreline is required for 64 TX and 64 RX data lanes, coming out to a total area of 0.330 mm². Only the top 2 metal layers are required for routing the signals. The remaining metal layers can be used for the front-end circuitry.

TSMC explained how the Active LSI can be tested at multiple stages. The first is Known Good Die (KGD) with just the LSI for die validation. Next is Known Good Stack (KGS) with the SoCs connected by the LSI for stack functionality. Lastly is Known Good Package (KGP) with the full assembly to comprehensively verify functionality, performance and reliability.

They showed shmoo plots at the KGD and KGP stages, both showing the interconnect hitting 32 Gb/s at 0.75V and 38.4 Gb/s at 0.95V.

The package reveals two SoC dies and two IO dies. Interestingly, the test vehicle appears to match AMD’s MI450 GPU’s design, with 2 base dies connected to each other, 12 HBM4 stacks and 2 IO dies with Active LSI. Instead of each individual HBM4 stack having its own Active LSI, two HBM4 stacks share one.

As for the power, the total is only 0.36 pJ/b at 0.75V, with only 0.07 pJ/b being used by the ETT in the Active LSI. Below is a comparison with other D2D solutions.

Microsoft D2D Interconnect - Paper 8.3

Microsoft also detailed their die-to-die (D2D) interconnect. Their test vehicle includes two dies and two pairs of D2D nodes for interconnection. A full mock-up of the power delivery network and routing were included to mimic clock gating and crosstalk.

The interconnect on their test die occupied 532 µm of shoreline and had a depth of 1350 µm. The test vehicle was fabricated on TSMC’s N3P node, and the interconnect was tested at two data rates, 20 Gb/s at 0.65V, and 24 Gb/s at 0.75V.

Microsoft reported two power consumption figures, one with both analog and digital system power, and one with only analog power. The latter is what most die-to-die interconnects report. At 24 Gb/s, the system power is 0.33 pJ/b and the analog power is 0.226 pJ/b, while at 20 Gb/s, the system power is 0.25 pJ/b and the analog power is 0.17 pJ/b. The power at idle state is 0.05 pJ/b.

Microsoft also compared their interconnect to the same prior research as TSMC did for their Active LSI.

As we explained in a previous article, Microsoft’s Cobalt 200 CPU features two compute chiplets connected by a custom high-bandwidth interconnect. We believe that this presentation details that exact interconnect.

Processors

From small mobile CPUs to large AI accelerators, ISSCC featured first-time architectural breakdowns from MediaTek, Intel, AMD, Rebellions, and Microsoft. Many even included die shots.

MediaTek Dimensity 9500 - Paper 10.2

Every year, MediaTek presents a different aspect of their flagship mobile CPUs. This year is no different, and the focus of this year’s mobile CPU presentation was on boosting and thermal management.

TSMC offers two different Contacted Gate Pitch (CGP) options for N3E and N3P, 48nm and 54nm. Across most chips, the narrower CGP of 48nm has been used, as it leads to smaller cell sizes and a greater die shrink. However, it also faces problems with leakage, routing and manufacturing due to the smaller critical dimensions.

MediaTek used the larger 54nm CGP on their Dimensity 9500’s C1 Ultra high-performance cores, for the improved power efficiency. This allowed them to reach higher performance with less thermal penalty, with 4.6% more performance at iso-leakage or 3% less power at iso-performance.

The rest of MediaTek’s paper focuses on dynamic performance optimization by utilizing unused aging budget and reducing thermal overshoots. In total, they were able to increase boost clocks from 4.21 GHz to 4.4 GHz. If these optimizations sound interesting to you, we recommend you check out the paper, 10.2 A Dynamic Performance Augmentation in a 3nm-Plus Mobile CPU.

Intel 18A-on-Intel 3 Hybrid Bonding - Paper 10.6

Intel disclosed their first hybrid bonded chip, the M3DProc. It consists of an Intel 3 bottom die, and an 18A top die. Each die contains 56 mesh tiles, cores and DNN accelerator tiles, respectively. The two dies are bonded together with Foveros Direct, hybrid bonding at a 9μm pitch.

The mesh tiles are arranged in a 14×4×2 3D mesh, with SRAM being shared across both dies.

Intel found that the 3D mesh reduces latency and increases throughput by almost 40%. They also tested the energy efficiency of transferring data, with 2D being within the 56 mesh tiles of the bottom die, and 3D being 28 adjacent mesh tiles across both dies. The results show that the Hybrid Bonding Interconnect (HBI) had a negligible impact on efficiency.

Each tile has 552 pads, with just under half used for data, and just under a quarter used for power.

In terms of packaging, M3DProc is similar to Clearwater Forest (CWF). CWF has Intel 3 base dies, connected to 18A compute dies via 9μm Foveros Direct.

The M3DProc achieves 875 GB/s 3D bandwidth, while each CWF compute die only achieves 210 GB/s. This chip’s 3D NoC has a significantly higher bandwidth density. CWF uses Foveros Direct to disaggregate the CPU core cluster’s L2 cache from the base L3 with 6 clusters per top die at 35GB/s each for 210GB/s per top die. M3DProc’s 875GB/s 3D bandwidth is aggregated over 56 vertical tile connections, or 15.6GB/s each over a far smaller area.

AMD MI355X - Paper 2.1

AMD presented on their MI355X GPUs. In conference presentations, AMD usually rehashes prior announcements while only introducing one or two new pieces of information. This paper was much better in that regard, explaining how the MI355X XCD and IOD were improved as compared to the MI300X.

AMD detailed how they doubled the matrix throughput per CU while keeping the total area the same and the number of CUs largely similar. First, of course, was the move from N5 to N3P; this provided the bulk of the transistor density improvement. The additional two metal layers provided by N3P allowed for improved routing and thus, higher cell utilization. AMD designed their own standard cells, as they have with N5 before, to optimize the node for their HPC use case.

They also used denser placement algorithms, similar to how the Zen 4c cores used in EPYC Bergamo CPUs are much smaller than the Zen 4 cores used in EPYC Genoa CPUs.

There are two approaches when performing the same calculations with many different data formats like FP16, FP8, MXFP4, etc. The first is using shared hardware, where every format goes through the same circuits. However, this comes with a power cost as there is little optimization for each format. The second option is each data format using an entirely different set of circuits for calculations. However, this takes up a lot of additional space. Of course, the optimal approach is somewhere in the middle. This optimization was an important focus for AMD.

As the next node with improved transistors, N3P provides performance improvements by itself. Nevertheless, AMD was able to improve frequency iso-power by 5% before process node improvements. They also designed multiple flip-flop variants with varying power and performance characteristics deployed across different areas of the chip depending on usage and architectural requirements.

MI300X featured 4 IO Dies. MI355X cuts that down to two. By doing this, AMD saves area on die-to-die interconnects. A larger monolithic die improves latency and reduces SerDes and translations. Moreover, the efficiency of the HBM was also improved by increasing the interconnect width. The saved power could be reallocated to the compute dies to increase performance.

As a large die with many routing options between any two areas on the chip, AMD had to do a lot of work to optimize the wires and interconnects. Through custom engineering of the wires, AMD was able to reduce the interconnect power consumption by ~20%.

Rebellions Rebel100 - Paper 2.2

Rebellions is a South Korean startup working on AI accelerators. At ISSCC, they published the first architectural breakdown of their new accelerator, the Rebel100. Unlike other accelerators, which are generally manufactured at TSMC, Rebellions chose Samsung Foundry’s SF4X node. With Nvidia, AMD, Broadcom and others hogging most of TSMC’s capacity, this allows them more flexibility.

At Hot Chips 2025, Rebellions demonstrated the chip running Llama 3.3 70B. The specs have remained the same between Hot Chips and ISSCC. One key point to note is the use of Samsung’s I-CubeS interposer technology. While the Hot Chips slide mentions the usage of TSMC’s CoWoS-S, we have clarified that this was a mistake on the slide, and that it has always been I-CubeS.

We have recently mentioned that CoWoS-S capacity constraints have been easing. That said, Samsung may have offered steep discounts to bundle I-CubeS advanced packaging with their front-end process — sparing the startup from having to find and validate a separate advanced packaging supplier. Samsung may have also conditioned the availability of their HBM on using I-CubeS.

I-CubeS has not seen adoption among any of the leading AI accelerators, and this could be Samsung’s attempt to break into the market. There are only 5 confirmed users of I-CubeS: eSilicon, Baidu, Nvidia, Rebellions and Preferred Networks.

The first is a networking ASIC by eSilicon on Samsung’s 14LPP with HBM2. Baidu’s Kunlun1 accelerator is similar, using Samsung’s 14LPP process and 2 HBM2 stacks. When the CoWoS-S capacity was very tight back in 2023, Nvidia outsourced a small amount of H200 production to I-CubeS. Then, there is the Rebel100 and lastly, a planned accelerator from Preferred Networks on the SF2 process.

The Rebel100 uses 4 compute dies and 4 HBM3E stacks. Each die has 3 UCIe-A interfaces. However, only two are used on each die, clocked at 16 Gb/s.

Rebellions claims that the design is reconfigurable at the package level, where additional IO or memory chiplets can be added to integrate with Ethernet for scale-up. This is where the remaining UCIe-A interface would be used.

Rebellions stated that the IO chiplets would be taped out by 1Q2026. There was no provided timeline for the memory chiplets.

They have also integrated silicon capacitors beside each HBM3E stack to improve power quality for HBM3E and critical control blocks.

Microsoft Maia 200 - Paper 17.4

Microsoft detailed their Maia 200 AI accelerator. The paper was less of a research paper and more of a white paper, with only a single image, a spec sheet comparing it to the Maia 100. Makes sense given a lot of Maia 200 claims are dubious such as the flops/mm^2 and flops/w.

While the Maia 100 was designed in a pre-GPT era, Maia 200 was designed for the current age of models and specifically inference. Earlier this year, Maia 200 nodes were made generally available on Azure.

Maia 200 is the last holdout of reticle-scale monolithic designs. Every major HBM-equipped training and inference accelerator has moved on to multi-chip designs with 2, 4, or even 8 compute dies per package. Every single mm² of the die has been hyper-optimized for one purpose. Unlike with an Nvidia or AMD GPU, there is no legacy hardware for media or vector operations. Microsoft has pushed the reticle-scale monolithic approach to its limit on TSMC’s N3P process, packing in over 10 PFLOPs of FP4 compute, 6 HBM3E stacks, and 28× 400 Gb/s full-duplex D2D links.

On the package-level, Maia 200 is very standard, mimicking the H100. A CoWoS-S interposer, with 1 main die, and 6 HBM3E stacks.

The long sides of the chip are covered in 3 HBM3E PHYs each, while the short sides each have 14 of the 28 lanes of 400 Gb/s D2D links. In the center, there is 272 MB of SRAM, with 80 MB of TSRAM (L1) and 192 MB of CSRAM (L2).

The Maia 200 has two different kinds of links, fixed links between other chips in the same node, and switched links between a chip and a switch. 21 links are configured as fixed links, 7 to each other chip, while the remaining 7 links are configured as switched links to one of four in-rack switches.

We will be publishing a deep dive into the Maia 200, its microarchitecture and network topology, for institutional subscribers.

Other Highlight

Samsung SF2 Temperature Sensor - Paper 21.5

Samsung presented a compact temperature sensor on SF2, replacing the traditional bipolar junction transistor (BJT) approach with a BEOL metal resistor. This may not be as flashy as the next generation of memory or processors, but it is essential to making chips work.

The metal resistor offers 518× higher sheet resistance than an equivalent routing metal, requiring roughly 1% of the area for the same resistance. As it sits in the upper metal layers, it leaves plenty of room for any circuitry underneath and eliminates FEOL area overhead. Although it has a low resolution, the benefits more than make up for it.

The sensor uses a fully stacked capacitor-resistor-circuit structure with a total area of just 625 μm². As a characterized PDK element, its behavior is modeled by the foundry and validated. It is more suitable for mass production, where process variation must be tightly controlled. Even on a single chip, thousands of these sensors may be used near hotspots.

As mentioned earlier, the metal resistors have a lower temperature coefficient of resistance (TCR), just 0.2× that of routing metal — which limits sensing resolution. Samsung compensates for this by increasing the base resistance. However, this slows sensing time as the RC time constant grows. To address this, Samsung uses a time-offset compression technique: a low-resistance (0.1R) fast-charge path rapidly charges the RC filter, then the circuit switches to the full resistance for the temperature-sensitive portion of the waveform.

For the time-to-digital conversion (TDC), they replaced the large linear delay generator used in prior work with a compact ring oscillator-based (RO) TDC, cutting delay generator area by 99.1%. The RO also doubles as the system clock, with phase-interleaved counting preventing non-monotonicity.

The new temperature sensor has an accuracy figure of merit (FoM) of 0.017 nJ·%², improving upon prior work on Samsung 5LPE, TSMC N3E and Intel 4 (JSSC 2025). Prior temperature sensors could only optimize for one of these: area or speed. The sensor on N3E was small, at 900 μm² but took 1 ms, while the sensor on Samsung 5LPE was fast, at 12 μs but huge, at 6356 μm².

Compared to IEDM and VLSI, ISSCC has a much bigger focus on integration and circuits. Almost every paper comes with some form of circuit diagram, together with clear measurements and data.

In this roundup, we will cover major categories such as Memory, Optical Networking, High-Speed Electrical Interconnects, Processors.

Memory

Samsung HBM4 - Paper 15.6

Samsung demonstrated a 36 GB, 12-high HBM4 stack featuring 2048 IO pins and 3.3 TB/s of bandwidth, built using 6th-generation 10nm-class (1c) DRAM core dies paired with an SF4 logic base die.

Samsung LPDDR6 - Paper 15.8

Both Samsung and SK Hynix showed off their LPDDR6 chips. We will discuss Samsung’s chips first and turn to SK Hynix’s later.

Samsung presented their LPDDR6 architecture and detailed the power-saving techniques used.

For those of you doing the math, 12×24 is 288, not a power of two. The remaining 32 bits are split into 2 use cases, 16 for metadata like ECC, and 16 for Data Bus Inversion (DBI).

Samsung SF2 LPDDR6 PHY - Paper 37.3

The PHYs also support the efficiency mode implemented by the LPDDR6 chips, which can reduce read power by 39% and write power by 29%.

SK Hynix 1c LPDDR6 - Paper 15.7

Although SK Hynix did not state the area or density of the LPDDR6 chip, we estimate the bit density will reach 0.59 Gb/mm², based on the relative density increase of their GDDR7.

SK Hynix 1c GDDR7 - Paper 15.9

The bit density achieved is 0.412 Gb/mm², compared to 0.309 Gb/mm² on Samsung’s 1b process, and 0.192 Gb/mm² on Samsung’s older 1z process.

GDDR7 has lower bit density than LPDDR5X, usually around 70% of the latter. Although it has much higher data rates, this comes at a cost, both in terms of power and area.

Samsung 4F² COP DRAM - Paper 15.10

The architecture for 4F² cells is the same as SK Hynix’s, with vertical channel transistors (VCT), and capacitors above the drain.

First, they have reorganized the sub-wordline drivers (SWD) from 128 per cell block to 16 groups of 8. This reduces the number of signals required for the SWD by 75%.

Next, they split the column select into an even and an odd path. This requires twice the multiplexers (MUX) but halves the column select line (CSL) count to 32 per data pin.

Samsung adopted the “sandwich” structure, which allows them to maximize the area efficiency of the core circuitry, and reduce the edge region area, which is not under any cells.

The area used by the core circuitry was reduced from 17.0% down to only 2.7%, a significant improvement, directly translating to overall die size reductions.

Samsung did not provide any density figures for this chip, only stating that it was a 16 Gb chip on a 10nm DRAM process.

Samsung noted that the VCT DRAM suffers from the floating-body effect, increasing leakage and reducing retention time. Mitigating this effect remains a key challenge for 4F² adoption.

SanDisk/Kioxia BiCS10 NAND - Paper 15.1

SanDisk and Kioxia demonstrated their BiCS10 NAND, with 332 layers and 3 decks. This is the highest reported NAND bit density, at 37.6 Gb/mm², dethroning the previous champion, SK Hynix’s 321L V9.

A 1x6 configuration has fewer ground pads and reduces area by 2.1%. However, the lower number of ground pads and vertical power tracks constrains power distribution.

MediaTek xBIT Logic-based Bitcell - Paper 15.2

MediaTek also showed a shmoo plot of the xBIT cell, going from 100 MHz at 0.35V, up to 4GHz at 0.95V.

We will be doing a deep dive into SRAM and its scaling factors in an upcoming newsletter article.

TSMC N16 MRAM - Paper 15.4

While Samsung Foundry also published work on 8LPP eMRAM this year, TSMC’s is far more promising. It targets the needed features, has great performance, and is on a cheaper N16 node.

TSMC is already planning the next-generation “Flash-Plus” variant with a 25% smaller bitcell and 100× higher endurance.

Optical Networking

Several papers from major optics vendors tackled optical interconnects that will carry data between next-generation AI accelerators both within and between datacenters.

Nvidia DWDM - Paper 23.1

However, at ISSCC, Nvidia proposed using 32 Gb/s per lambda, with 8 wavelengths multiplexed using DWDM. A 9^th wavelength is used for clock forwarding at half rate — that is 16 Gb/s.

Clock forwarding means that SerDes can be made somewhat simpler by removing the Clock and Data Recovery (CDR) circuit as well as other circuitry, improving energy and chip shoreline efficiency.

Interestingly, the OCI MSA did not make use of an additional wavelength for clock forwarding, and it appears that reserving all wavelengths for actual data transmission is the priority.

Marvell Coherent-Lite Transceiver - Paper 23.2

The Coherent-lite transceiver is a DSP-based pluggable module consisting of two 400G channels. Each 400G channel runs a dual polarization QAM and consists of two parallel modulation streams, X and Y.

The key to this demonstration is highlighting other methods of scaling channel bandwidth that are optimized for the campus application.

This modulation scheme is not entirely new to the industry, but it is now being brought into the datacenter campus setting for use in those shorter links.

Broadcom 6.4T Optical Engine - Paper 23.4

High-Speed Electrical Interconnects

Intel UCIe-S - Paper 8.1

TSMC Active LSI - Paper 8.2

As designers eke out every bit of die space for next-generation AI accelerators, the switch to aLSI is inevitable.

They showed shmoo plots at the KGD and KGP stages, both showing the interconnect hitting 32 Gb/s at 0.75V and 38.4 Gb/s at 0.95V.

As for the power, the total is only 0.36 pJ/b at 0.75V, with only 0.07 pJ/b being used by the ETT in the Active LSI. Below is a comparison with other D2D solutions.

Microsoft D2D Interconnect - Paper 8.3

Microsoft also compared their interconnect to the same prior research as TSMC did for their Active LSI.

Processors

From small mobile CPUs to large AI accelerators, ISSCC featured first-time architectural breakdowns from MediaTek, Intel, AMD, Rebellions, and Microsoft. Many even included die shots.

MediaTek Dimensity 9500 - Paper 10.2

Intel 18A-on-Intel 3 Hybrid Bonding - Paper 10.6

The mesh tiles are arranged in a 14×4×2 3D mesh, with SRAM being shared across both dies.

Each tile has 552 pads, with just under half used for data, and just under a quarter used for power.

In terms of packaging, M3DProc is similar to Clearwater Forest (CWF). CWF has Intel 3 base dies, connected to 18A compute dies via 9μm Foveros Direct.

AMD MI355X - Paper 2.1

They also used denser placement algorithms, similar to how the Zen 4c cores used in EPYC Bergamo CPUs are much smaller than the Zen 4 cores used in EPYC Genoa CPUs.

Rebellions Rebel100 - Paper 2.2

The Rebel100 uses 4 compute dies and 4 HBM3E stacks. Each die has 3 UCIe-A interfaces. However, only two are used on each die, clocked at 16 Gb/s.

Rebellions stated that the IO chiplets would be taped out by 1Q2026. There was no provided timeline for the memory chiplets.

They have also integrated silicon capacitors beside each HBM3E stack to improve power quality for HBM3E and critical control blocks.

Microsoft Maia 200 - Paper 17.4

On the package-level, Maia 200 is very standard, mimicking the H100. A CoWoS-S interposer, with 1 main die, and 6 HBM3E stacks.

We will be publishing a deep dive into the Maia 200, its microarchitecture and network topology, for institutional subscribers.

Other Highlight

Samsung SF2 Temperature Sensor - Paper 21.5

Anthropic’s Claude 4.6 Opus and Claude Code have soared in demand. Anthropic’s ARR has more than tripled in just a single quarter from $9B at the end of last year to over $30 today. Open models such as GLM and Kimi K2.5 caused open model use cases to soar. Capital raises by firms like Anthropic, OpenAI, and various Neolabs also demand GPUs.

This inflection point means that demand has spiked and there’s been a run on GPUs at the hyperscalers and Neoclouds.

This new source of demand has spiked pricing for products and services across the supply chain, from DRAM and NAND memory to fiber optic cables, datacenter colocation and gas turbines.

GPU Rental Pricing is the latest of many compute related products and services to see a dramatic tightness in supply and resulting jump in pricing. H100 1-year GPU rental contract pricing has shot up almost 40% from a low of $1.70/hr/GPU in October 2025 to $2.35/hr/GPU by March 2026.

On-Demand GPU rental capacity is sold out across all GPU types – those that have locked up on-demand instances are not willing to relinquish this capacity back into the pool despite recent price hikes. Trying to find GPU compute in early 2026 has been like trying to book airplane tickets on the last flight out, high prices, and almost no availability. That’s the PC analogy, but the more accurate analogy is that trying to rent a cluster is actually like trying to buy drugs.

At SemiAnalysis, we have been deeply involved in tracking trends and topics including GPU rental pricing within the Neocloud and Hyperscale ecosystem thanks to our work on ClusterMAX, InferenceX and AI Cloud Total Cost of Ownership.

We also spend a good amount of time helping AI Labs connect with Neoclouds and find GPU rentals in the market and have been actively discussing GPU rental price trends with nearly everyone in the ecosystem.

Since 2023, for our clients, we have maintained GPU rental price indices tracking pricing for most major GPU types (H100, H200, B200, B300, GB200, GB300, MI300, MI325, MI355) across all key rental terms, from on-demand and 1 month all the way to 5 years. Our index is constructed using survey data polling many Neoclouds and buyers of compute which is also validated by transaction data as well as by negotiations and transactions we ourselves are facilitating.

Today, we are making our H100 1 year GPU rental index publicly available to provide additional data and insights to the community. We update the index monthly and will be providing commentary on the latest trends and anecdotes via our social media pages on X and LinkedIn. Access to the full-term structure and rental pricing across all other major GPU types is available to institutional subscribers of our AI Cloud TCO model.

This report will discuss the latest trends, anecdotes and data points regarding the GPU rental market, explain how we analyze the broader GPU Rental market structure and touch on what the future may hold for rental pricing.

Surge Pricing Comes to the GPU Rental Market

The chart illustrating the 1y H100 rental price hardly does the trend justice – anecdotes from our first-hand experience in trying to procure compute and feedback from others in the market paints an even starker picture.

Demand is strong across many very heterogeneous use cases and there is no one-size fits all approach. There are plenty of inference workloads like large mixture of experts (MoE) inference that run best on the latest large world-size systems like the GB300 NVL72, while training workloads can have the best price performance on H100s, keeping demand high even for older cards.

Customers are fighting to pay $14/hr/GPU for p6-b200 spot instances in AWS, some Neocloud Giants no longer sell single nodes, H100s are getting renewed at the exact same rate they were signed at 2-3 years ago and some H100 contracts are being renewed for 4 years though 2028. Hunting for even 8 nodes (64 GPUs) of H100s or H200s is not easy – half the providers we asked were completely sold out, and most providers will simply respond they have no capacity of Hopper GPUs coming off contract at all.

We have even heard of renters of compute subdividing their clusters and subletting the compute just like an apartment during the Monaco Grand Prix. Coming soon – the rise of Neocloud slumlords?

Blackwell availability is very tight too. We are hearing lead times for new Blackwell deployments now extending into June-July thanks to strong demand for open-weight models as well as the ongoing surge in inference demand, and most of these clusters are now getting taken up. Indeed, market-wide, all capacity coming online until August to September 2026 has already been booked!

GPU Rental Prices – The Comeback Kid

But how did the market come to this point? Only six months ago, most market observers were skeptical on GPU terminal value and assumed an inexorably steep fall in GPU rental rates over time. Financial analysts chastised any Neocloud or Hyperscaler that used a 6-year depreciation period for its GPU compute assets. Let’s quickly recap the story so far before we discuss how trends could evolve in the future.

Before late 2025, the prevailing expectation across the ecosystem was that Hopper (i.e. H100 and H200) rental prices would drop considerably as Blackwell deployments ramped given the latter’s much lower cost of compute. Instead, the opposite happened in late 2025: demand for H100s was holding firm, and in many cases, strengthening. The rapid adoption of open-weight models and accelerating inference demand at that time was the first sign of the insatiable wave of compute demand coming to market.

January was the next inflection point for compute when memory pricing, across both DRAM and NAND pricing, went from rising aggressively for several quarters, to going completely parabolic, with LPDDR5 and DDR5 contract prices tracking toward ~4x and ~5x year-on-year increases respectively in 1Q26 based on our Memory Model.

To manage margin risk stemming from this rapid hike in component costs, OEMs began repricing AI servers at levels that significantly exceeded the underlying increase in component costs. This complicated the cluster capital investment processes as higher server acquisition costs compressed prospective project returns, forcing some operators to slow-roll or abandon deployments. In effect, supply that would have otherwise come online was being withheld, tightening the rental market further.

Amid the server procurement disarray cause by this AI Server Pricing Apocalypse, GPU rental demand was clearly accelerating, with most remaining spare capacity taken up completely during January and February. By March – it became increasingly impossible to find any H100s, H200s or B200 rental capacity for any term. Rental pricing broke above $2/hr/GPU for a 1y contract by late January, and then shot up 15-20% by mid-to-late February vs end January and is set to rise another 15-20% month-on-month by the end of March.

A major driver of demand early this year arose from native media generation - Seedance and Nano Banana are driving massive increases in token throughput as users generate and refine images and video at scale. But the most visible driver of demand is the emergence of multi-agent workloads executing multi-step workflows, operating at high concurrency and iterating continuously, leading to parabolic growth in token and compute consumption.

Look no further than the trends regarding Claude Code that we have already called out in many articles. SemiAnalysis as a company has, over the past 7 days, consumed billions of tokens costing around ~$5/M tok on average, but the return on time saved and expansion of workflows and capabilities far exceeds that cost. SemiAnalysis now deploys a suite of AI tools across workflows beyond simple search and summarization – notably dashboarding, automated scraping, large-scale data wrangling and agentic financial modelling.

We are tracking the overwhelming demand through proxies like Claude Commits Daily. At the current trajectory, we believe that Claude Code will be 20%+ of all daily commits by the end of 2026. While you blinked, AI consumed all of software development. Institutional clients who are interested in the data set can reach out to our API documentation team. A spoiler alert: this number of commits is meaningfully higher than when we published.

It seems that almost everyone in our circle is an avid user of Claude Code – yet we recognize that our circle of contacts are immersed in everything AI and Semiconductors, and thus they are just the tip of the spear. For many Fortune 500 companies and to the broader world, Claude Code and the Agentic world are but a quirky side story that may come up on their Facebook feeds or their favorite NPR podcasts. They are completely unaware of the tidal wave of productivity and upheaval that the agentic world is about to unleash upon the world.

As those in the broader economy that recognize the amazing return on investment from using AI Tools join us in surfing this tidal wave, there will continue to be a step shift higher in token consumption over time. The debate on the true return of using AI is now a settled question – the use of AI tools can deliver value an order of magnitude greater than the cost of using the tools. The shift up and to the right in the demand curve for tokens is providing a powerful and relatively inelastic (for now) force driving up GPU rental pricing.

Put simply – if the return on investment from using AI tools is 5-10x, then there is clearly a long way to go in GPU rental pricing before prices rise enough to curtail demand. It would not surprise us for the increases in rental pricing to exert further upward pressure on server and component costs.

Introducing the SemiAnalysis 1Y H100 contract price index

Today, we are making our SemiAnalysis H100 1-year rental contract price index freely available to create greater awareness and transparency around trends in the GPU rental market.

Our index is constructed from direct survey data across a pool of 100+ market participants including Neocloud providers, buyers and sellers of compute that is captured every month to determine a representative range (25^th to 75^th percentile) for GPU rental contracts. We validate these pricing levels with transaction data as well as by arranging a few transactions ourselves, connecting buyers and sellers of compute within our network.

Since 2023, we have tracked the contract market price across 3m to 5y tenors for the H100, H200, B200, B300, GB200, GB300, and we have selected data available for the AMD complex as well (MI300, MI325, MI355).

The SemiAnalysis H100 1-year contract price index is additive to currently available GPU indices for a few different reasons:

Alongside the public release of our H100 1Y contract price tracker, we have launched the SemiAnalysis Tokenomics Dashboard for our institutional Tokenomics model subscribers – a tool for tracking and understanding the frontier AI model landscape. The dashboard lets users build custom benchmarking comparisons across coding, reasoning, math and agentic evals, compare API pricing across leading models and providers, and view key disclosures from major AI labs including token volumes, revenue, valuations and customer counts.

GPU Rental Market Structure Today

Before late 2025, GPU rental pricing was more competitive as operators had much greater GPU inventory while end demand was only starting to accelerate meaningfully. Pricing exercises were competitive, with multiple Neoclouds able to offer very competitive pricing. These operators prioritized utilization and ensuring they did not miss out on sweating fixed compute assets before the next GPU refresh cycle potentially puts pressure on pricing for incumbent GPU servers.

GPU rental providers’ strategy has pivoted 180 degrees since then. Neoclouds and Hyperscalers are now in the driver’s seat – they can now negotiate for more favorable terms such as higher prepay, better pricing, longer contract lengths and can even pick and choose the contract start and end dates to match their inventory availability. Time is also now on the Neoclouds’ side – they can plan deployments on their own time, harnessing the increasing price climate to build the best book of clients for a given cluster over time.

The GPU rental market structure is best understood by dividing it into three primary market segments, each of which tends to cater to different types of customers:

Short-term Rental: On-Demand, Spot, Less than 3-Month Contracts

Short-term rentals represent the very front end of the rental term structure and in many cases represents residual capacity, though many such as Runpod and Lambda very successfully focus on providing considerable capacity of flexible on-demand or spot capacity. On-demand pricing functions tends to function very differently than the rest of the contract GPU rental market. Providers usually set a fixed price for on-demand capacity and will only very infrequently adjust prices.

Providers will adjust pricing on a one-off basis in response to utilization levels – if utilization is too low, they will drop pricing to attract demand. If utilization is maxed out – then prices will be hiked as the provider determines that utilization will remain high even at higher pricing levels. This is why a time series of Neocloud posted on-demand pricing will be flat for a long period of time before gapping up or down. With the on-demand market, utilization is the best high frequency indicator of demand, and not price.

Mid-Term Contracts

The more economically relevant segments are the contract markets where most GPU rentals by value are transacted. The 1-year segment captures the marginal demand from non-AI lab customers and spillovers from large buyers, making it the most sensitive indicator of tightening conditions. AI Natives and smaller AI Labs can be seen mostly in the 1-3y tenors, though increasingly these labs and AI Natives are looking to lock up compute by contracting for longer terms of more than 4 years and even agreeing to high prepays of above 20% which were previously atypical for 4y+ deals.

Long-Term Offtakes

Lastly, the 4–5 year segment is dominated by AI labs locking in huge quantities of capacity early. These deals involve large clusters of 50MW or 100MW or even larger – equivalent to about ~24,000 to 48,000 GB300 NVL72 GPUs. In aggregate – these deals represent a very large proportion of the overall Neocloud GPU rental market.

AI Labs like these contracts as they can lock in a huge amounts of compute in one go to cater to rapidly accelerating end demand. The AI Labs also have considerable influence on the cluster design, making decisions regarding storage, networking, CPU compute and more. These are very often bare metal deals as AI Labs have the engineering expertise to customize more layers of the tech stack and extract the highest performance per TCO possible.

Neoclouds like these deals as they can focus their sales resources on just a few large offtake deals rather than dozens of smaller deals which would end up generating the same amount of revenue. Longer-term contracts are also great for Neoclouds as they can then use these contracts to arrange debt financing on favorable terms that will match the tenor of the contract, removing most duration and GPU rental price risk from the equation and locking in a teens project IRR in most cases. It is also common to see Hyperscalers backstop these deals – serving as the direct offtaker from these Neoclouds but then on-selling the compute to an AI Lab. This structure is a win-win for everyone involved – Neoclouds can lend on very favorable terms since their offtaker is a AAA rated Hypercaler, while Hyperscalers can collect a slice of the project revenue by offering the backstop of their balance sheet without actually expanding the balance sheet.

The below table shows a number of large offtake deals we have been tracking. We analyze these deals closely to calculate the implied pricing in $/hr/GPU and profitability metrics like Project IRR, EBIT Margins, etc.

In today’s market climate, most of the large AI Clusters that are ramping are captive to AI Labs, who nonetheless have been dipping into the sub 4y market to supplement existing capacity ramps. They are also preventing supply from re-entering the sub 4y contract market by simply renewing their existing H100 and H200 clusters, indirectly playing in these markets. It will be interesting to see how supply-demand dynamics play out in the 1-3y contract market as AI Labs ramp their GB200 and GB300 megaclusters.

Where The Puck is Going

What is most striking is the disconnect between these underlying dynamics and broader market sentiment. Despite clear evidence of tightening supply and rising prices - conditions that should directly benefit Neocloud providers through margin expansion and stronger arguments for higher useful lives - public market sentiment has turned increasingly negative on names like CoreWeave, Nebius, IREN, and these companies’ share prices are currently at the low end of the 6-12mth trading range.

The market is still anchored to a narrative of eventual oversupply and commoditization, and the developments described above have done little to assuage concerns on GPU terminal value. While, the reality on the ground points to sustained scarcity and pricing power across all Neocloud providers, that suggest that all compute will be in-demand - no matter the relative performance given such an aggressive shortage.

Looking forward, there are three check points to monitor to determine if GPU rental prices will continue to remain elevated.

Pricing is Only Going One Way For Now, and ROIC Follows

Taken together, these factors point to a clear conclusion: GPU rental pricing is more likely to continue rising than falling.

The dynamic is self-reinforcing. As Neoclouds see supply tighten and prices rise, they move to secure more hardware ahead of further price increases, which only tightens supply and pushes pricing higher still. This echoes the 2023–2024 GPU shortage, when tight supply allowed OEMs to push through outsized margin expansion and drove a sharp spike in server pricing, though we think the server market is mature enough this time around that this may not repeat itself.

As we argued in a recent note to our institutional clients, the re-acceleration in GPU rental pricing improves Neocloud ROIC by expanding margins on already-deployed capital. At the same time, higher rental rates extend the economic useful life of existing GPUs, meaning invested capital generates cash flows for longer before requiring reinvestment.

Neoclouds with shorter contract tenors will see capacity roll off and reprice into the current environment, capturing immediate margin expansion. At the same time, Hyperscalers and Neoclouds locking in next-generation capacity over multi-year terms will benefit.

This inflection point means that demand has spiked and there’s been a run on GPUs at the hyperscalers and Neoclouds.

This new source of demand has spiked pricing for products and services across the supply chain, from DRAM and NAND memory to fiber optic cables, datacenter colocation and gas turbines.

Surge Pricing Comes to the GPU Rental Market

We have even heard of renters of compute subdividing their clusters and subletting the compute just like an apartment during the Monaco Grand Prix. Coming soon – the rise of Neocloud slumlords?

GPU Rental Prices – The Comeback Kid

Introducing the SemiAnalysis 1Y H100 contract price index

Today, we are making our SemiAnalysis H100 1-year rental contract price index freely available to create greater awareness and transparency around trends in the GPU rental market.

The SemiAnalysis H100 1-year contract price index is additive to currently available GPU indices for a few different reasons:

GPU Rental Market Structure Today

The GPU rental market structure is best understood by dividing it into three primary market segments, each of which tends to cater to different types of customers:

Short-term Rental: On-Demand, Spot, Less than 3-Month Contracts

Mid-Term Contracts

Long-Term Offtakes

Where The Puck is Going

Looking forward, there are three check points to monitor to determine if GPU rental prices will continue to remain elevated.

Pricing is Only Going One Way For Now, and ROIC Follows

Taken together, these factors point to a clear conclusion: GPU rental pricing is more likely to continue rising than falling.

Nvidia’s Datacenter Blackwell GPU (SM100) represents one of the largest GPU microarchitecture change in a generation, yet no detailed whitepaper exists. Until today, there is no public datacenter Blackwell architecture microbenchmarking study on PTX and SASS instructions, such as UMMA and TMA, with a focus on AI workloads.

After our in-depth Nvidia Tensor Core Evolution: From Volta To Blackwell article, SemiAnalysis has spent months of engineering time, tearing into the Blackwell architecture and measuring the raw PTX instruction performance, to establish hard practical performance upper bounds and compare them with the theoretical peaks. We do this to discover unit- and instruction-level hardware throughput and latency limits, providing a useful characterization from an ML systems and kernel development perspective. We focus on deep learning workload configurations, such as benchmarking asynchronous memory copy setups used in popular deep learning library FlashInfer.

We open sourced our Blackwell micro-architecture-level benchmarking repo here. Please drop a star if you find it useful.

Acknowledgement

Future Work

This article is the first in a series exploring low-level assembly and kernel code of AI accelerators. In future installments, we will expand the effort by benchmarking additional Blackwell and Blackwell Ultra PTX instructions, including EXP2 and TensorMap update latencies. Furthermore, we have concrete plans to benchmark TPU Pallas kernels, Trainium NKI kernels, and AMD CDNA4 assemblies. For AMD CDNA4 in particular, benchmarking is within reach in the near term since many of the instructions are already well-documented.

Join us if you want to work on low level benchmarking, ClusterMAX, inference simulators, or other interesting technical work. Send your resume to letsgo@semianalysis.com with 5 bullet points demonstrating your exceptional engineering abilities. Please attach GitHub repo links, YouTube demos, websites, blogs, etc to support your bullet points.

Blackwell Features

From Hopper to Blackwell, NVIDIA made several incremental improvements to the architecture and changes to the PTX abstractions for MMA-related instructions. We cover most of these in our article NVIDIA Tensor Core Evolution. The major notable changes are:

Clusters, GPCs, and Floorsweeping

Since Hopper, Nvidia datacenter GPUs have supported an optional feature which is known by several names, such as “thread block clusters”, “CTA clusters”, and “cooperative grid arrays” (CGAs), all of which refer to the same feature. A cluster is a logical grouping of CTAs, the shape and size of which can be statically or dynamically specified per-kernel. Clusters are visible to the programming model in some useful ways, one of which allows for multicast loads to multiple CTAs in the same cluster; we discuss this later in the context of TMA multicast.

Importantly, CTAs in a cluster are guaranteed to be co-scheduled on the same GPC. This has an important consequence in the 1-CTA-per-SM “persistent CTA”-style Blackwell kernel: if the cluster size does not evenly divide the number of SMs in a GPC, some of the SMs will be left idle. This behavior can be confusing for kernel authors who, unaware of the sparsely-documented GPC, naively launch a number of persistent CTAs equal to the number of SMs with clusters enabled, resulting in serialized execution for some CTAs.

The number of yielded SMs per GPCs is not fixed, not the same between GPCs on the same chip, and may not even be symmetrical between dies in the same package. Manufacturing of semiconductors results in defects and those defects can land all over the chip. As such Nvidia has to engineer their chips in a way such that they can still have those yielded units still exposed to software in a relatively uniform way.

We prompted Claude to write a utility to reverse-engineer the mapping of SMs to GPCs by launching clusters of various sizes and using PTX %%smid to record which SMs appear in the same GPC. The result is a list of logical groupings of TPCs into GPCs. The list is longer than the 8 GPCs present in Hopper/Blackwell because there are some TPCs which seem to occupy their own logical GPC, and are never co-scheduled with any other TPCs.

As of SM100, NVIDIA has provided a solution to this quantization issue so that kernels can get the benefit of larger clusters while still making use of all the available SMs. Kernels can be launched with two cluster sizes: a preferred cluster size and a fallback cluster size. In general, to use the whole GPU, the fallback cluster should be size 2 or size 1.

Logical vs. Physical GPC

The groupings of TPCs into GPCs we presented above are logical groupings. They represent software’s view of the GPCs, with no information about which of the 20 actual physical SMs in each GPC are enabled, or where each physical GPC is located on the two dies. In reality, B200 chips with the same logical configuration need not have exactly the same physical SMs yielded in each GPC. This can be a potential source of performance non-determinism between GPUs which otherwise might look the same from the view of software. Additionally, the logical groupings of SMs into GPCs tells us nothing about which GPC is on each of the two dies in the B200 package.

To discover more information about the physical layout of the SMs, we have every SM traverse a pointer-chase array that fills L2 cache, measuring the latency of each load. For each address, we compare the latency seen from each SM to the latency seen by every other SM, to produce an SM<->SM distance matrix. X and Y axes are SM ID.

We can see two clear sets of SMs, separated by >300 cycles average distance to L2; this must be the die-to-die crossing. We’ve also labeled the SMs with their logical GPC groupings as identified in the last section; interestingly, the singleton TPCs are close together and seem to correlate well with GPC0 in this benchmark, so one might guess that those TPCs physically reside on GPC0.

Based on this information, we can refine the list of yielded TPCs for each GPC, though the 5+3 is still just a guess.

Additionally, though in a roundabout way, we can conclude that the die-to-die latency penalty is roughly 300 cycles. This is also evident when looking at the latency profile for a singular SM from the benchmark (which also includes a lot of L2 congestion):

Memory Subsystem

In this section, we discuss the memory subsystem: the hardware units that move data between computation units. Memory copy instructions are operations that use the memory subsystem, and newer generations feature asynchronous copy instructions (Read the previous article for the asynchrony evolution). Here we focus on the two variants of asynchronous copy instructions: LDGSTS and TMA (Tensor Memory Accelerator).

Asynchronous Copy

Async copy (PTX: cp.async , SASS: LDGSTS) was introduced in the Ampere generation, and the instruction moves data from global memory to shared memory asynchronously. Async copy is non-blocking, allowing memory loads to overlap with computation. It also writes directly to shared memory without going through registers, reducing register pressure.

Referencing FlashInfer Multi-head attention (MHA) kernels, we benchmark async copy with the following configuration:

We plot throughput versus bytes-in-flight per SM, the total number of bytes concurrent memory loading instructions are loading.

Although different load sizes converge to similar throughput at the same bytes-in-flight, we prefer 16-byte loads. 16-byte loads achieve slightly higher throughput at similar bytes-in-flight while using less execution resources. For example, at 32 KiB in flight, 8B load uses 4 stages, while 16B load uses 2 stages. This saves the memory space for 2 memory barrier objects and reduces instruction issue pressure.

Overall, we see memory throughput with LDGSTS saturating at around 6.6 TB/s at 32 KiB in flight.

Our experiments show that increasing the number of stages achieves higher throughput at higher bytes-in-flight, and that increasing threads per CTA strictly improves performance across all configurations. Interestingly, MLA uses 2 warps and 12 stages, landing at about 2.2 TB/s . We believe this is due to softmax warps needing the most registers, and increasing warp count reduces register allocation per thread.

We benchmarked the latency of the same set of configurations. We see that LDGSTS has a baseline latency of ~600 nanoseconds and nearly doubles after 8 KiB in flight. This is because we need to use a large number of threads for LDGSTS to achieve high bytes in flight, leading to a high number of warps stalled due to MIO (memory input output) throttle.

Tensor Memory Accelerator (TMA)

TMA (PTX: cp.async.bulk.tensor, SASS: UTMALDG) is an asynchronous data copy engine introduced in the Hopper generation, specialized for moving large amounts of data from global memory to shared memory. A single thread can initiate TMA to perform address generation, memory swizzling, and out-of-bounds handling, freeing up other threads to execute independent work. Here we benchmark the 2D tensor version (cp.async.bulk.tensor.2d) to represent typical TMA usage.

Referencing FlashInfer attention kernels, we benchmark TMA, assigning only one CTA per SM but using one thread for each of 1 to 4 warps per CTA to issue TMA instructions of varying box sizes. The below graph shows the best-case throughput for each bytes-in-flight.

Async Copy vs. TMA Comparison

Deep learning kernel libraries like FlashInfer use both TMA and async copy for loading data. TMA and async copy have different performance characteristics: TMA is good for large loads with regular access patterns but has higher latency, while async copy can handle irregular memory access patterns but has size limits. We explain under what conditions we should pick one over the other. Here we benchmark the configurations FlashInfer uses for MHA and MLA kernels.

We see that throughput-wise, async copy slightly outperforms TMA at less than 32 bytes in flight, but TMA catches up after that and can continue scaling to 128 KiB. Latency-wise, we see async copy having slightly lower latency than TMA before 12 KiB in flight, but TMA latency greatly increases after that.

In reality, Blackwell MLA kernels use async copy for dynamically loading pages, while its MHA kernels use only TMA. Most of FlashInfer’s Blackwell MHA kernels are contributed by TRT-LLM, so we can only speculate what the kernels do by investigating the binaries. We found that similar to Hopper, all Blackwell TRT-LLM kernels use TMA. We suspect that for dynamic page loading, those kernels follow Hopper kernels, where they use 4D TMA with page index as the last dimension and index into the TensorMap object when needed. To understand the exact mechanics of the kernels, we urge NVIDIA to open source the FlashInfer TRT-LLM kernels for the benefit of the community.

TMA Multicast

TMA supports a multicast mode, where a single load copies data to the shared memory of multiple SMs, specified by a CTA mask. Multicast is commonly used in GEMM-like patterns, where input tiles are shared between SMs working on different output tiles. For example, multicast is useful for the activation function SwiGLU, which uses a dual-GEMM pattern of two GEMM operations sharing one input matrix. The major benefit is reducing HBM loads, which lowers effective bandwidth usage. It also significantly reduces L2 traffic, because requests for shared data for multiple CTAs are coalesced into one request.

According to NCU, the unit responsible for serving TMA multicast requests is called the L2 Request Coalescer (LRC):

The L2 Request Coalescer (LRC) processes incoming requests for L2 and tries to coalesce read requests before forwarding them to the L2 cache. It also serves programmatic multicast requests from the SM and supports compression for writes.

It sounds like the hardware might provide some multicast behavior, even if it isn’t explicitly requested, like a miss status holding register. We test this by running the same TMA multicast benchmark, except instead of one CTA issuing a multicast load, all CTAs issue independent TMA loads to the same data.

TMA multicast allows for much higher load bandwidth to fill SMEM buffers, even if data is not already in L2. For known traffic patterns, explicit TMA multicast instructions perfectly eliminate L2 traffic, resulting in the ideal “1 / cluster_size” L2 bytes per SMEM byte. We also observe that for this simple benchmark, we achieve nearly the same SMEM fill throughput in both the explicit and the implicit case. However, we can see the LRC is not perfect; the L2 receives a bit more traffic in the implicit case, especially as the total volume increases.

DSMEM vs. SMEM

NVIDIA introduced distributed shared memory (DSMEM) in the Hopper architecture. DSMEM allows CTAs within a cluster to access each other’s shared memory. This is useful for patterns like inter-CTA reductions. Reading peer-CTA memory through DSMEM has significantly lower throughput than SMEM’s 128 bytes per clock cycle.

We experimented with a few different PTX patterns for interacting with DSMEM. An important difference when writing code for DSMEM vs. SMEM is that DSMEM loads are packetized similar to global loads, so the optimal access pattern looks nothing like the interleaved accesses that avoid bank conflicts in local SMEM, and more like a typical coalesced access to contiguous locations in GMEM. Additionally, we observed that in order to get the full 128B/cycle for local SMEM, `ld.shared` must be used without `::cluster`. This was a pitfall we ran into when we wrote a benchmark which simply used `ld.shared::cluster` to local and remote SMEM addresses. With `ld.shared`, the compiler emits `LDS` instead of a generic `LD` emitted with `ld.shared::cluster` which does not seem to be able to achieve peak throughput for local SMEM. We also struggled to push the achieved throughput further with `ld.shared::cluster`, and only achieved slightly higher throughput through DSMEM after switching to `cp.async.bulk`(PTX) / `UBLKCP`(SASS) to move higher volume of data per instruction.

The peak throughput we achieved when using each PTX pattern is below, expressed as bytes per clock cycle (B/clk) to align with the known max achievable in SM-local SMEM.

Tensor Core 5th Generation MMA

The MMA instruction is the core operation that performs matrix multiplication. MMA performance has grown increasingly shape-dependent from Hopper to Blackwell. Here we investigate this phenomenon, sweeping through different shapes and data types to quantify the performance differences.

Blackwell comes with 2SM MMA, a new type of MMA instruction (.cta_group::2) where a CTA pair collaboratively executes one MMA operation across 2 SMs. Specifically, the input matrix A is duplicated while matrix B and D are sharded across the 2 SMs, and the CTA pair can access each other's shared memory. This enables even larger MMA shapes. We investigate whether 2SM MMA exhibits weak scaling, strong scaling, or both.

Throughput

NVIDIA claims specific throughput performance for different input data types, and here we show their claims for each (format + CTA group) and compare them with the max achievable throughput. We show that UMMA achieves near peak throughput for all formats and CTA groups, and even on 2SM versions where coordination overhead may be a concern.

For 1SM MMA across all N sizes, we see that the smaller M=64 achieves max 50% theoretical peak throughput, and the larger M=128 achieves near 100%. This confirms that M=64 is utilizing half of the datapath. For 2SM MMA, we see that M=128 throughput starts at 90% peak for N=64 and reaches near 100% for all other N sizes. M128N64 throughput must be bound at a different hardware unit such as TMEM, L2, SMEM, etc. Meanwhile, M=256 sustains near 100% peak throughput across all configurations, this is because M=256 is M=128 per SM, which can utilize the full datapath. We note that throughput is identical across formats with the same data type bit width, and micro-scaling data types have virtually no overhead.

MMA supports two different AB layouts: Both input matrices stored in SMEM (SS), and matrix A stored in TMEM and matrix B stored in SMEM (TS). We observed that for M=128, while ABLayout=TS achieves near peak throughput, ABLayout=SS underperforms in smaller N sizes and catches up at N=128.

We can show that this is because the instruction itself is SMEM bandwidth bound below N=128 for SS mode. For example, for FP16 we know the hardware can do 8192 MMA FLOPs per cycle per SM, and the SMEM bandwidth is 128 B/cycle (per SM). So for M=128 N=64 K=16, we have:

We compute this for increasing N and find we are finally Math limited starting from the N=128 instruction.

The same is true for other datatypes - MMA instructions with both operands in SMEM are SMEM-bound below N=128.

To further illustrate the point, we plot the roofline for all shapes of FP8 1SM MMA. We see clearly that the N < 256 is at the memory-bounded region, and the slope is roughly 128 bytes / cycle, the SMEM bandwidth.

2SM MMA achieves perfect weak scaling across all formats and shapes, reaching 2x speedup when using 2x the amount of compute resources than 1SM MMA. In smaller shapes of ABLayout=SS, we observe over 2x speedup, which again happens because the instruction is SMEM bound below N=128 for SS and the 2SM version splits operand B between the two SMs.

These experiments show that you should always use the largest instruction shape available for a given SMEM tile size to get maximum throughput.

Latency

We benchmarked single MMA instruction latency, and we plot the comparison below. Across all configurations, we see latency linearly increases from N=64 to 128, and the spike at N=256 is likely due to the jump from 128 to 256. For individual CTA group MMAs, 1SM MMA M=64 and M=128 have similar latencies across N sizes, whereas in 2SM MMA, M=256 latency grows slightly faster than M=128, which matches our theoretical estimations. Comparing data types, we see little difference for 1SM but clear separation for 2SM MMAs.

We believe integer operations being more power efficient leads to S8 being the fastest, and scale factor computation introduces a minor overhead for MXF8 and MXF4.

Throughput at Different In-Flight Instruction Count

In our throughput benchmark, we set high numbers of in-flight instructions to amortize instruction issuing and commit wait overheads, ranging from 256 to 1024. However, kernels typically use 1 to 4 in-flight MMA instructions. We benchmarked the throughput at 1 to 10 in-flight instructions, and we discuss the changes in throughput here.

Across all configurations, we see the same N and in-flight MMAs achieve similar percentages of Speed-of-Light (SoL). Notably, only the largest N reaches 90% SoL, while the smallest N achieves only about 70%. Comparing 1SM and 2SM MMA, we see 1SM achieves around 5% higher SoL throughput than its 2SM counterpart. For the same data format and CTA group MMA, the throughput for larger N is always higher than smaller N sizes. Finally, we observe that the throughput SoL percentages for 4 in-flight MMAs caps out at 78% - 80%.

Below we discuss real-world use cases with kernel writing library CUTLASS. We also discuss throughput, multi-cast, and floorplans.

We open sourced our Blackwell micro-architecture-level benchmarking repo here. Please drop a star if you find it useful.

Acknowledgement

Future Work

Blackwell Features

Clusters, GPCs, and Floorsweeping

Logical vs. Physical GPC

Based on this information, we can refine the list of yielded TPCs for each GPC, though the 5+3 is still just a guess.

Memory Subsystem

Asynchronous Copy

Referencing FlashInfer Multi-head attention (MHA) kernels, we benchmark async copy with the following configuration:

We plot throughput versus bytes-in-flight per SM, the total number of bytes concurrent memory loading instructions are loading.

Overall, we see memory throughput with LDGSTS saturating at around 6.6 TB/s at 32 KiB in flight.

Tensor Memory Accelerator (TMA)

Async Copy vs. TMA Comparison

TMA Multicast

According to NCU, the unit responsible for serving TMA multicast requests is called the L2 Request Coalescer (LRC):

DSMEM vs. SMEM

The peak throughput we achieved when using each PTX pattern is below, expressed as bytes per clock cycle (B/clk) to align with the known max achievable in SM-local SMEM.

Tensor Core 5th Generation MMA

Throughput

We compute this for increasing N and find we are finally Math limited starting from the N=128 instruction.

The same is true for other datatypes - MMA instructions with both operands in SMEM are SMEM-bound below N=128.

These experiments show that you should always use the largest instruction shape available for a given SMEM tile size to get maximum throughput.

Latency

We believe integer operations being more power efficient leads to S8 being the fastest, and scale factor computation introduces a minor overhead for MXF8 and MXF4.

Throughput at Different In-Flight Instruction Count

Below we discuss real-world use cases with kernel writing library CUTLASS. We also discuss throughput, multi-cast, and floorplans.

At GTC 2026, Nvidia delivered an event packed full of ground breaking announcements. Nvidia’s pace of innovation is not showing any signs of slowing, as they introduced three entirely new systems this year: Groq LPX, Vera ETL256, and STX. Also announced were updates to Nvidia’s Kyber rack architecture system, CPO making its debut for scale-up networking with the unveiling of the Rubin Ultra NVL576 and Feynman NVL1152 multi-rack systems. Early hints on Feynman’s architecture was also a key topic. A Jensen callout for InferenceX during the keynote was a highlight.

This is our GTC 2026 recap, and we will address many of the key questions that have been left unanswered by Nvidia. Specifically, we will go through the LPX rack and LP30 chip and explain how attention and feed forward network disaggregation (AFD) works; more details on the various rack architectures behind NVL144, NVL576, and NVL1152 and clarify just how much optics will be inserted as well as the rationale behind the dense Vera ETL256. The next generation Kyber rack had some big updates and some hidden details.

Groq

First up is the Groq LPU. One of the most significant recent events in AI infrastructure was Nvidia’s “acquisition” of Groq. Strictly speaking, Nvidia paid Groq $20B to license their IP and hire most the team. This functions almost as an acquisition, though its structure technically falls short of it being legally considered as one, thereby simplifying or obviating the need for regulatory approvals. Given Nvidia’s market share, if this transaction were structured as a full acquisition and were put to anti-trust review, such a transaction would likely not go through. The other benefit is that it avoids a drawn-out transaction closing process. Nvidia got instant access to Groq’s IP and people. This is why, less than four months after the deal was announced, Nvidia already has a system concept that is being integrated into the Vera Rubin inference stack.

Let’s now go through a refresher on the LPU architecture to see how Groq’s LPU complements Nvidia’s GPU. For more details see our original Groq piece. The premise from that piece remains unchanged: the standalone Groq LPU system is not economical for serving tokens at scale, but it can serve tokens very quickly which can demand a large market premium. This is the premise behind how LPU fits into a disaggregated decode system.

LPU chip

Groq’s first and only publicly announced LPU architecture was detailed in their ISCA 2020 paper. Unlike typical hardware architectures connecting many general-purpose cores, Groq re-organized the architecture into groups of single-purpose units connecting to other groups of different purposes, and they named the groups “slices.” Between functional units are streaming registers, scratchpad SRAM for functional units to pass data to each other. Groq opted for single-level scratchpad SRAM instead of multi-level memory hierarchy to make the hardware execution deterministic.

Concretely, LPU architecture has VXM slices for vector operations, MEM slices for loading/storing data, SXM slices for tensor shape manipulation, and MXM slices for performing matrix multiplication. Spatially, the slices are laid out horizontally, allowing the data to stream horizontally. Within a slice, instructions are pumped vertically across units. Conceptually, LPU resembles a systolic array that pumps instructions vertically and data horizontally.

The data flow and instruction flow design requires fine-grained pipelining to achieve high performance. Since LPU architecture makes computation deterministic, the compiler can aggressively schedule and overlap instructions to hide latency. The LPU’s use of high bandwidth SRAM and aggressive pipelining are the two main factors that enable LPU’s low latency.

LPU gen 1 was designed on a legacy Global Foundries 14nm process, with Marvell responsible for the chip’s physical design. This was a much more mature node compared to peers when it taped out in 2020, with the incumbent AI chip platforms mostly on TSMC’s N7 platform. This made sense for an early product focused on proving out Groq’s architecture and bringing its inference-centric design to market. The 14nm node was mature, relatively well understood, and suitable for an initial chip where architectural differentiation mattered more than pushing its silicon to the leading edge.

One of the selling points is that the chip can be manufactured and packaged entirely in the United States compared to their competitors being heavily reliant on the Asia semiconductor supply chain: logic and packaging in Taiwan, with HBM from Korea.

Since then, Groq’s roadmap has stalled due to execution, with no LPU 2 having been shipped. This leaves the Groq LPU looking even more dated against competing roadmaps. What was once a meaningful but still manageable node disadvantage versus 7nm-era peers has widened into a far sharper gap, with all leading accelerator platforms now moving onto 3nm-class processes in 2026.

The follow on Groq LPU 2 was designed for Samsung Foundry’s SF4X node, specifically at Samsung’s Austin fab, allowing them to extend the pitch that Groq is fabricated domestically in the USA. Samsung would also provide support for the back-end design. The choice of Samsung was driven by favorable terms / investment, with Samsung Foundry struggling to find customers for its advanced nodes and missing out on an AI logic customer. Unsurprisingly, Samsung was a key investor in Groq’s subsequent Series D in August 2024, and most recently in September 2025 before the Nvidia “acquisition.”

However, the Groq LPU 2 was never productized because of design issues. The C2C SerDes on the chip couldn’t hit the advertised 112G speed which caused the design to malfunction, as we detailed long ago in the Accelerator model. The third generation Groq LPU is the one that Nvidia will be productizing.

SRAM and Memory Hierarchy

We have written about the role of SRAM in the memory hierarchy, but the quick recap is that SRAM is very fast (low latency and high bandwidth) but this comes at the expense of density and therefore cost.

SRAM machines such as Groq’s LPU therefore enable very fast time to first token and tokens per second per user but at the expense of total throughput, as their limited SRAM capacity quickly gets saturated by weights, with little left over for KVcache that grows as more users are batched. GPUs win for throughput and cost as we have shown. This is why Nvidia has decided to combine these architectures to get the best of both worlds: accelerate parts of decode that are more latency sensitive and are not as memory heavy on a low-latency SRAM-heavy chip like the LPU, while memory hungry attention is performed on GPUs that come with a lot of fast (but not SRAM fast) memory capacity.

This brings us to the Groq 3 LPU or LP30, with LPU gen 2 being skipped over. This chip has no Nvidia design involvement. The SerDes issues affecting v2 appear to be fixed. Behind the paywall, we will reveal the SerDes IP vendor which may come as a surprise. Nvidia also announced an LP35 which is a minor refresh of the LP30 which will remain on SF4 and will require a new tapeout. It will incorporate NVFP4 number format but given Nvidia is prioritizing time to market we don’t expect any other drastic design changes.

LPU 3’s near reticle size die layout is very similar to LPU 1. a significant amount of area taken is up by the 500MB of on-chip SRAM, with a very small amount of area dedicated to MatMul cores that offer 1.2 PFLOPs of FP8 compute – a fraction of compute compared to Nvidia GPUs. This compares to LPU 1 with 230MB of SRAM and 750 TFLOPs of INT8, with the performance increase mostly driven by node migration from GF16 to SF4. As a single monolithic die, advanced packaging isn’t required.

Since Nvidia has taken over, the next generation LP40 will be fabricated on TSMC N3P and use CoWoS-R, and Nvidia will contribute more of their own IP such as supporting the NVLink protocol rather than Groq’s C2C. This will be the first LPU to be extremely co-designed alongside the Feynman platform. Groq’s original plans for LPU Gen 4 was also with TSMC and Alchip as the back-end design partner. Alchip’s involvement is now redundant with Nvidia able to perform backend design on their own. One of the technical innovations planned is hybrid bonded DRAM to extend on-chip memory with only a slight decrease in latency and bandwidth vs SRAM, but much higher performance compared to DRAM. SK Hynix was tapped as the supplier of the DRAM to be used for the 3D stacking. All of this and more was detailed long ago in the Accelerator model.

GPU and LPU Integration: Attention FFN Disaggregation (AFD)

Now with an understanding of what LPUs are good for we can understand how they fit into inference setups. NVIDIA introduced LPUs to improve the performance of high interactivity scenarios. In those scenarios, LPUs can leverage their low-latency capabilities to improve the decode phase latencies. One way LPUs can improve decode phase latencies is by applying the Attention FFN Disaggregation (AFD) technique, introduced in MegaScale-Infer and Step-3.

As we explained in our InferenceX article, LLM inference involves two phases: prefill and decode. Prefill processes the full input context: It is compute-intensive, which is suitable for GPUs. On the other hand, decode predicts new tokens and is memory-bounded. Decode is latency-sensitive because the model predicts new tokens one by one, and LPU’s high SRAM bandwidth and low-latency capabilities can help accelerate this iterative process.

Attention and FFN are subsets of operations in a model. In a model forward pass, attention’s output feeds into a token router, and the token router assigns each token to k experts, where each expert is an FFN. Attention and FFN have very different performance properties. During decode phase, the GPU utilization of attention barely improves when scaling batch size due to being bounded by loading KV cache. In contrast, the GPU utilization of FFN scales with batch size comparatively better.

As state-of-the-art mixture-of-expert (MoE) models grow increasingly sparse, tokens can choose experts from a larger expert pool. As a result, each expert receives fewer tokens, leading to lower utilization. This motivates attention and FFN disaggregation. If a GPU only performs attention operations, its HBM capacity can be fully allocated to KV cache, increasing the total number of tokens it can process, which then increases the tokens each expert processes on average.

Comparing the two operations, we see attention is stateful due to dynamic KV cache loading patterns, whereas FFN is stateless since the computation only depends on the token inputs. Thus, we disaggregate the computation of attention and FFN. We map attention computations to GPUs, which handle dynamic workloads well. For FFNs, we map them to LPUs, since LPU architecture is inherently deterministic and benefits from static compute workloads.

With AFD, token routing from GPUs to LPUs can become the bottleneck, especially under strict latency constraints. The token routing flow involves two operations: dispatch and combine. In the dispatch step, we route each token to their top k experts with an All-to-All collective operation. After experts complete their computation, we perform the combine step, where the outputs are sent back to the source location with a reverse All-to-All collective, continuing the next layer’s computation.

To hide the communication latency of dispatch and combine, we employ ping pong pipeline parallelism. In addition to splitting batches into micro-batches and computation pipelining like standard pipeline parallelism, the tokens dispatched to the LPUs are combined back to the source GPUs, so they ping pong between the GPUs and the LPUs.

Speculative Decoding

A different way LPUs could improve decode phase latencies is by accelerating a speculative decoding setup, where we deploy draft models or Multi-Token Prediction (MTP) layers onto LPUs.

For a decoding step of context N tokens, adding k additional tokens during forward pass (a warm prefill of k new tokens) marginally increases the latency when k << N. Using this property, speculative decoding uses a small draft model or MTP layers to predict k new tokens, saving time since small models have lower latency per decode step. To verify the draft tokens, the main model only needs one warm prefill of k new tokens, at the latency cost of roughly a single decode step. Speculative decoding usually boosts output token per decode step by 1.5 to 2 tokens, depending on the draft model / MTP accuracy. With its low latency capabilities, LPUs can further increase the latency savings and improve throughput.

For LPUs, deploying a draft model or MTP layers is quite different from applying AFD. FFNs are stateless, while draft models and MTP layers require dynamic KV cache loading. Each FFN is around hundreds of megabytes, whereas draft models and MTP layers take up tens of gigabytes. To support this memory usage, LPUs can access up to 256 GB of DDR5 per Fabric Expansion Logic FPGAs on the LPX compute tray.

LPX Rack System

Let’s look at the LPX rack system, which has interesting details. Nvidia has displayed an LPX rack with 32 1U LPU compute trays with 2 Spectrum-X switches. This 32 tray 1U version that Nvidia has shown off at GTC is very close to Groq’s original server design before the acquisition. We believe that this server configuration is not the version that will be shipped in 3Q, with Nvidia implementing changes. Here, we will detail what we know about the actual production version. This was already detailed in the Accelerator model.

LPX Compute Tray

Each LPX compute tray or node has 16 LPUs with 2 Altera FPGAs, 1 Intel Granite Rapids host CPU and 1 BlueField-4 front-end module. As with other Nvidia systems, hyperscalers customers can and will use their own Front-end NIC of choice rather than paying for Nvidia’s BlueField.

The LPU modules are mounted in a belly-to-belly on the PCB, meaning 8 LP30 modules on the top side of the PCB and the other 8 LP30 modules on the bottom. All of the connectivity that comes out of the LPU are via PCB traces and given the dense all-to-all mesh for intra-node connections this requires a very high spec PCB to support the routing. The belly-to-belly mounting is used to reduce PCB trace lengths across the ‘X’ and ‘Y’ dimensions.

Something interesting about the system is the important role the FPGAs play. Nvidia refers to the FPGAs as “Fabric Expansion Logic” which serves multiple purposes. First, they act as a NIC which converts the LPU’s C2C protocol into Ethernet to connect to the Spectrum-X based ethernet scale-out fabric. It is this scale-out fabric through which the LPUs connect to GPUs in the decode system.

Second, the LPUs also traverse through the FPGAs to reach the host CPU, with the FPGAs converting C2C to PCIe to the CPU.

Third, the FPGAs are connected to the backplane to talk to other FPGAs in the node, we believe this is to help manage control flow and timing of all the LPUs. The FPGAs also bring extra system DRAM of up to 256GB each. This pool of memory can be used for KVCache if the user wants the entire decode process served by the LPX.

On the front panel there are 8 x OSFP cages for cross-rack C2C, while there will be 2 cages (likely QSFP-DD) that goes to the Spectrum-switches that is used to connect the LPUs and the GPUs for the disaggregated decode system. We will share more about this when we describe the network.

LPU Network

The LPU network can be divided into the scale-up ‘C2C’ network and scale-out network which interacts with the Nvidia GPUs through Spectrum-X. First let’s discuss the scale-up network which can be divided into 3 portions: intra-node, inter-node/intra-rack, inter-rack. For C2C within the rack Nvidia announced a total of 640TB/s of scale up bandwidth per rack which comes from 256 LPUs x 90 lanes x 112Gbps/8 x 2 directions = 645TB/s. Note that Nvidia uses the total 112G line rate rather than 100G of effective data rate.

Intra-Tray Topology

Within each tray or node, all 16 LPUs are connected to each other in an all-to-all mesh. Each LPU module connects to the 15 other LPUs within the node with 4x100G of C2C bandwidth. Note that this ‘C2C’ is not related to NVLink, but Groq’s own scaleup fabric. These connections are all via PCB trace, which necessitates an extremely high spec PCB to support this routing density. This is why the belly-to-belly layout is used: it reduces the ‘X’ and ‘Y’ distance between all the LPUs and instead have routing go in the ‘Z’ dimension.

The LPU also has 1x100G going to one FPGA, with each FPGA interfacing with 8 LPUs. The 2 FPGAs each have 8x PCIe Gen 5 going to the CPUs. The LPU needs to traverse through the FPGA to interface with the CPU as LPUs don’t have PCIe PHYs to interface directly.

Inter-node/Intra-rack

Each LPU connects to one LPU from each of the 15 other nodes in the server. Each of these inter-node links is 2x100G so there are 15x2x100G inter-node links coming out of each LPU. These inter-node links are via a copper cable backplane. In addition, each FPGA also connects to an FPGA in every other node at either 25G or 50G per link for 15x25G/50G. This also goes through the backplane. This means that each node has 16 x 15 x 2 lanes for inter-node C2C and 2 x 15 lanes for inter-node FPGA which is a total of 510 lanes or 1020 differential pairs (for Rx and Tx). Therefore, the backplane is 16 x 1020/2 = 8,160 differential pairs – we divide by 2 as each device Tx channel is a corresponding device’s Rx channel.

Inter-rack

Lastly, there is the inter-rack C2C. Each LPU has 4x100G lanes that go to the OSFP cages to connect LPUs across 4 racks. There are various configurations that can be used for this inter-rack scale up. One option is 4x100G from each LPU going to one OSFP cage, each OSFP escaping 800G of C2C from 2 LPUs. However, for greater fan out the preferred configuration seems to be each 100G lane from the LPU going to 4 individual cages, with each cage escaping 800G of C2C from 8 LPUs. In terms of how the racks are networked together it appears to be a daisy chain configuration, with each Node0 connected to 2 other Node 0. This can all be achieved within the reach of 100G AECs, though optics can be used if necessary.

Nvidia’s CPO Roadmap

NVIDIA revealed its CPO Roadmap at the GTC Keynote 2026, with Jensen following up with additional commentary in the Financial Analyst Q+A meeting held the following day. Though many had their hopes up for CPO to be used for scale-up within the rack for Rubin Ultra Kyber, Nvidia’s focus was instead on using CPO to enable larger world size compute systems.

In the Rubin Generation, Nvidia will offer the Rubin GPU in an Oberon NVL72 form factor with an all-copper scale-up network. For Rubin Ultra, as we expected, there will only be a copper scale-up option for Rubin Ultra in the Oberon and Kyber Rack form factor. Rubin Ultra will also be offered in a larger world size system that connects 8 Oberon Racks of 72 Rubin Ultra GPUs to form what will be referred to as NVL576. CPO scale-up will be used to build the larger world size, connecting between the racks in a two-tier all to all network, though scale-up inside the racks will remain copper-based.

When we reach the Feynman Generation, CPO usage will expand via another large world size rack, the NVL1152 which is formed by combining 8 Kyber racks. While the Nvidia Technical Blog that outlines the rack configuration roadmap states that “NVIDIA Kyber will scale up into a massive all-to-all NVL1152 supercomputer using similar direct optical interconnects for rack-to-rack scale-up”, Jensen Huang in a Financial Analyst Q+A session did say that NVL1152 in Feynman would be “all CPO”. There is some disagreement on whether copper will still be used for scale-up within the rack or whether CPO will replace copper.

Nvidia’s approach has been to use copper where they can, and optics where they must. The architecture of NVL1152 in the Feynman generation will follow the same principle. It is clear that the NVL1152 will adopt CPO to connect between racks, but from GPUs to NVLink Switches is currently copper POR. Nvidia is unable to achieve another doubling of electrical lane speed from 224Gbit/s bi-di to 448Gbit/s uni-di means bandwidth isn’t that amazing.

While 448G high speed SerDes have big challenges for shoreline, reach, and power versus using a die-to-die connection to an optical engine, the manufacturing challenges, cost, and reliability for Feynman necessitate using copper to the Switch.

With that said, the NVL1152 SKU is years out – and the roadmap is highly likely to shift. For now, our base case stands at copper being used within each rack and CPO between the racks, but this could easily change.

Oberon and Kyber Updates, Larger World Sizes Introduced, More Networking Updates

Nvidia provided a long-awaited update on its Kyber rack form factor, the latest addition to the lineup after Oberon having first been previewed as a prototype at GTC 2025. As a prototype, the rack architecture has continued to evolve, and we notice some changes. First, each compute blade has densified, with 4x Rubin Ultra GPU and 2x Vera each. There are a total of 2 canisters of 18 compute blades which amounts to 36 compute blades total for 144 GPUs in a rack. The initial Kyber design featured 2 GPUs and 2 Vera CPUs in one compute blade, with a total of 4 canisters of 18 compute blades each.

The details below are based on the Rubin Kyber prototypes, but Rubin Ultra will be redone.

Each switch blade is also double in height vs the GTC 2025 prototype, with 6 NVLink 7 switches per switch blade, and 12 switch blades per rack, amounting to a total of 72 NVLink 7 switches per Kyber rack. The GPUs are connected all-to-all to the switch blades via 2 PCB midplanes or 1 midplane per canister.

For Rubin Ultra NVL144 Kyber, there will be no CPO used for scale up as we have told clients multiple times, despite rumors from other analysts suggesting scale-up CPO introduction for Kyber. However, optics for NVLink are coming and will be progressively phased in. Scale-up CPO will first be used for the Rubin Ultra NVL 576 system to connect between 8 Oberon form factor racks, forming a two-layer all-to-all network. A copper backplane will still be used for scale-up networking within the racks however. This is still for low volume / testing purposes.

Moving back to the Kyber Rack, each Rubin Ultra logical GPU offers 14.4Tbit/s uni-di of scale-up bandwidth, using an 80DP connector (72 DPs used x 200Gbit/s bi-di channel = 14.4Tbit/s) per GPU for connectivity to the midplane board. Connecting all 144 GPUs in an all-to-all network will require 72 NVLink 7.0 Switch Chips running at 28.8Tbit/s uni-di of aggregate bandwidth each.

In the Kyber Switch Blade picture below, we can see that there are 2 separate PCBs carrying 3 Switches each. The switch blade should have 6 152DP connectors, 3 connectors serving each midplane board. The picture is a prototype blade using less dense connectors, which is why there are 12 connectors instead of the 6 that we expect in the production version.

Each 28.8T NVLink Switch has 144 lanes of 200G (simultaneous bi-directional) which means each Switch has 24 lanes of 200G going to each connector. Copper flyover cables are used to connect each switch to the midplane, as the distances involved are too long for PCB traces. This is also why the switches are further away from the midplane, to provide space for the routing of the flyover cables.

Each NVLink Switch Chip connects via flyover cables to the connector (144 DPs used x 200 Gbit/s bi-di channel = 28.8Tbit/s) connectors at the edge of the switch blade, and these connectors plug into the midplane board. Nvidia is looking into using co-packaged Copper to reduce loss further, in case NPC doesn’t work. As far as we know the Nvidia is telling supply chain to go for fully co-packaged copper.

Rubin Ultra NVL288

Though not officially discussed by Nvidia at GTC 2026, an NVL288 concept has been explored within the supply chain. This would entail two NVL144 Kyber racks placed adjacent to each other, with a rack-to-rack copper backplane used to connect the two racks. One possibility is that all 288 GPUs are connected all to all, but this would require higher radix switches than the current NVLink 7 switches which only offer a maximum radix of 144 ports of 200G.

If Rubin Ultra NVL288 is deployed, each Rubin Ultra GPU will have a scale-up bandwidth of 14.4Tbit/s uni-di, requiring 144 DPs of cables to connect the NVLink 7 switches. 72 DPs per GPU times 288 GPUs means a total of 20,736 additional DPs required to connect this larger world size domain. This entails a lot of cables, so it is an upper bound of how much cable content could be used.

The radix of the 28.8T NVLink Switch limits the number of GPUs that each switch can connect while still providing for cross-rack connectivity. Either a higher radix switch will have to be used - or there will have to be a degree of oversubscription in this architecture while potentially adopting a dragonfly-like network topology. This would also require fewer DPs worth of copper cables.

All current evidence in the supply chain points to NVSwitch 7 being the same bandwidth as NVSwitch 6, but that is seems a bit illogical to be frank. Our belief is that NVSwitch 7 is actually 2x the bandwidth and radix of NVSwitch 6, so all-to-all can be done, and architecturally that makes the most sense from a systems perspective.

Rubin Ultra NVL576

To push the scale up world size beyond 144 GPUs and across multiple racks, optics are needed as we are approaching the maximum compute density that is within the reach of copper. Rubin Ultra NVL576 is now on the roadmap with 8 racks of lower density Oberon.

Optics will be required for the inter-rack connections, though strictly speaking it isn’t confirmed whether this will be with pluggable optics or with CPO, though CPO seems much more likely. The current Blackwell NVL576 prototype “Polyphe” uses pluggable optics.

We have shown a concept of NVL576 for GB200 previously with pluggable optics to interconnect the second layer of NVLink switches. The use of pluggables contributed to an enormous increase in BOM cost that made the system untenable from a TCO perspective for a switched all-to-all. However, it is plausible that Rubin Ultra NVL576 will be rolled out in test volumes before Feynman NVL 1,152, where we will see actual volume ramp of scale-up CPO.

The downstream implications of this are exposed in our institutional research, trusted by all major hyperscalers, semiconductor companies, and AI Labs, at sales@semianalysis.com

Feynman

While not much is known about Feynman, the Keynote sneak peek was enough to tell us Feynman will be exciting, with three major technical innovations all being pushed in a single platform: Hybrid bonding/SoIC, A16, CPO, and custom HBM.

While Feynman adopting CPO is on the roadmap, the question is to what extent? Will in-rack interconnectivity be copper based or optical? We will show possible configurations behind the Paywall. Vera ETL256

CPU demand is rising as AI workloads require more data handling, preprocessing, and orchestration beyond GPU compute. Reinforcement learning further increases demand, with CPUs running simulations, executing code, and verifying outputs in parallel. As GPUs scale faster than CPUs, larger CPU clusters are needed to keep them fully utilized, making CPUs a growing bottleneck.

The Vera standalone rack addresses this directly, achieving unprecedented density by fitting 256 CPUs into a single rack — a feat that necessitates liquid cooling. The underlying rationale mirrors the NVL rack design philosophy: pack compute tightly enough that copper interconnects can reach everything within the rack, eliminating the need for optical transceivers on the spine. The cost savings from copper more than offset the additional cooling overhead.

Each Vera ETL rack consists of 32 compute trays, 16 above and 16 below, arranged symmetrically around four 1U MGX ETL switch trays (based on Spectrum-6) in the middle. The symmetric split is deliberate: it minimizes cable length variance between compute trays and the spine, keeping all connections within copper reach. From each switch tray, rear-facing ports connect to that copper spine for intra-rack communication, while 32 front-facing OSFP cages provide optical connectivity to the rest of the POD.

Networking within the rack uses a Spectrum-X multiplane topology, distributing 200 Gb/s lanes across the four switches to achieve full all-to-all connectivity while maintaining a single network tier. With each compute tray housing 8 Vera CPUs, the result is 256 CPUs per rack, all interconnected over Ethernet through a single, flat network.

CMX and STX

We have written extensively on Nvidia’s CMX, or ICMS platform in our last Rubin piece and Memory Model. Nvidia introduced the STX reference storage rack architecture.

CMX

CMX is NVIDIA’s context memory storage platform. CMX addresses a growing bottleneck in modern inference infrastructure: the rapid expansion of KV Cache required to support long-context and agentic workloads.

KV cache grows linearly with input sequence length and number of users and is the primary tradeoff when it comes to prefill performance (time to first token). At scale, on-device HBM does not have enough capacity. Host DRAM extends beyond HBM capacity with an additional tier of cache, but also hits limits on total amount per node, memory bandwidth, and network bandwidth. Enter NVMe storage for additional KVcache offload.

NVIDIA introduced a “new” intermediate storage “tier G3.5” within the inference memory hierarchy at CES in January. Tier G3.5 NVMe sits in between tier G3 DRAM and tier G4 shared storage (also NVMe, or SATA/SAS SSD, or HDD). Previously referred to as ICMS (Inference Context Memory Storage) and now branded as the CMX platform, this is just another re-brand of storage servers attached to compute servers via Bluefield NICs. The only difference from NVMe architectures is the swap from Connect-X NICs to Bluefield NICs.

STX

To expand the scope of CMX, NVIDIA also launched STX. STX is a reference rack architecture using Nvidia’s BF-4 based storage solution to complement VR compute racks. The reference architecture effectively specifies exactly how many drives, Vera CPUs, BF-4 DPUs, CX-9 NICs, and Spectrum-X switches are needed for a given cluster.

Unlike the BF-4 in the VR NVL72, which consists of a Grace CPU and a single CX-9 NIC, the BF-4 in the STX reference design includes one Vera CPU, two CX-9 NICs, and two SOCAMM modules. Each STX box contains two BF-4 units, totaling two Vera CPUs, four CX-9 NICs, and four SOCAMM modules. For the whole STX rack, it has a total of 16 boxes, implying 32 Vera CPUs, 64 CX-9 NICs, and 64 SOCAMMs.

The STX announcement included a typical Nvidia show of strength where they named all major storage vendors as supporting STX, including AIC, Cloudian, DDN, Dell Technologies, Everpure, Hitachi Vantara, HPE, IBM, MinIO, NetApp, Nutanix, Supermicro, Quanta Cloud Technology (QCT), VAST Data and WEKA.

Put together, BlueField-4, CMX, and STX represent NVIDIA’s broader effort to standardize how clusters are designed at the storage layer. NVIDIA has captured the compute and network layer, and is actively moving into the storage, software, and infrastructure operations layers over time.

Now behind the paywall, we will share some more details on how all of this impacts the supply chain. Including beneficiaries of the LPX system, and the updated Kyber racks. We will also reveal a rack concept that Nvidia has yet to announce.

Groq

LPU chip

SRAM and Memory Hierarchy

GPU and LPU Integration: Attention FFN Disaggregation (AFD)

Speculative Decoding

A different way LPUs could improve decode phase latencies is by accelerating a speculative decoding setup, where we deploy draft models or Multi-Token Prediction (MTP) layers onto LPUs.

LPX Rack System

LPX Compute Tray

Second, the LPUs also traverse through the FPGAs to reach the host CPU, with the FPGAs converting C2C to PCIe to the CPU.

LPU Network

Intra-Tray Topology

Inter-node/Intra-rack

Inter-rack

Nvidia’s CPO Roadmap

Oberon and Kyber Updates, Larger World Sizes Introduced, More Networking Updates

The details below are based on the Rubin Kyber prototypes, but Rubin Ultra will be redone.

Rubin Ultra NVL288

Rubin Ultra NVL576

The downstream implications of this are exposed in our institutional research, trusted by all major hyperscalers, semiconductor companies, and AI Labs, at sales@semianalysis.com

Feynman

CMX and STX

We have written extensively on Nvidia’s CMX, or ICMS platform in our last Rubin piece and Memory Model. Nvidia introduced the STX reference storage rack architecture.

CMX

STX