Since the video’s release on Youtube late last week, shock clips have appeared on social media that take the whole conversation out of context. What actually follows in the whole 35 minute discussion is an insightful discussion into the future of memory, how it is an exciting time for memory innovation, where memory is going in the next few years, and what the industry is looking at today.

This article discusses these points in greater detail, providing necessary color where helpful so that the whole discussion can be viewed in a more helpful and positive context for the future of memory.

By reading this post, you agree to the terms and conditions. Also see the full ethics statement.

If you’re new, check out the About page. A lot of readers expense the subscription to this newsletter as it helps their professional work. Group subscriptions (3+) are 20% off. If you have any questions, reply to this email and let me know!

If you are not a paid subscriber, you can purchase just this article using the button below. You can find the whole catalog of articles for purchase at this link.

Buy the article

Also check out the Semi Doped podcast with and myself, and our daily free newsletter with latest semi news.

If you would like to engage independent research services for your project or need expert calls, check out my boutique consulting practice at SemiExponent.

The Three Fold Problem

Gelsinger points out that there are actually three major problems with HBM today: (1) insatiable demand, (2) larger capacity demand, and (3) greater memory bandwidth requirements.

When these three elements are put together, only one technology really emerges to satisfy the requirements of the industry: high bandwidth memory, or HBM. Stacking DRAM chips to make HBM ticks the higher capacity with high bandwidth boxes, while creating a massive appetite for memory. Today, AI is a memory-limited problem.

The consequence of this is that memory makers enjoy a near 90% margin, which Gelsinger points out is something that should never happen. Memory companies used to have one good year out of four as they usually built out capacity too quickly, and were left holding the bags in a downturn. Today, the memory cycle is looking strong several years into the run up, with no end in sight.

Unfortunately, the memory crunch is here for at least another 2-3 years, as Gelsinger points out, because capacity buildouts take time. He also points out that in that time frame, Apple may increase prices yet again. The consumer market will continue to be choked by AI demand in the near future. The hardest question to answer today is: When capacity does come online, will we still need all this memory?

Vijay Shilpiekandula from Dilation Capital makes an interesting point: token growth is exponential, while capacity buildout is linear. He mentions that memory intensity per token is on the rise - a framing that I particularly like. As context lengths continue to grow, and the user base (human and agent) increases, the amount of memory that the system has to support only increases, and from what we can tell today, there is demand inflection on the horizon.

HBM: The Ugly Baby

The said ugly baby was actually born out of a collaboration between AMD and SK Hynix as early as 2013, when they were trying to solve the memory bandwidth problem for supercomputing applications. HBM1 stacked four 1 GB DRAM chips for a total capacity of 4 GB, with a memory bandwidth of 128 GB/s. Today’s HBM4 stacks sixteen 4 GB dies for 64 GB capacity and a total bandwidth of 2TB/s.

The baby is all grown up now, but has become increasingly difficult to handle. Stacking to increase capacity causes thermal, yield, and capacity issues. DRAM chips generate heat, and it is becoming difficult to ensure that it is effectively removed. Yield is also becoming difficult as stack heights increase. Capacity-wise, HBM is incredibly inefficient because the bit-density of a DRAM die for HBM is 1/3rd of a standalone DRAM chip - a consequence of putting through-silicon-vias (TSVs) through the DRAM die. Making HBM now pulls all the capacity out of the DRAM supply chain because it takes 3x as many wafers to make the same HBM capacity, as you would with DRAM.

The problem gets worse with HBM5. JEDEC, the standard body that defines how high these stacks can physically be, has been relaxing the specs because keeping the physical dimension constrained would involve a shift to hybrid bonding - a technology still new to the memory industry at scale. Concept demonstrations for HBM5 from the big memory makers now include integrated cooling solutions co-packaged with HBM to extract heat. In addition, the base die that sits under the DRAM stack is going to a custom design that makes memory no longer a commodity, off-the-shelf product that is fungible between different designs.

The memory game has fundamentally changed, and Gelsinger’s main point is that the next few years will see the rise of new architectures and technologies if we were to fulfill our AI dreams, giving new life to a technology that has stopped scaling for the good part of a decade. It is the greatest time in history to be in the memory business.

After the paywall:

• The Era of Memory Innovations

  • Bandwidth solutions (SRAM, 3D-DRAM)

  • Capacity solutions (SOCAMM/LPDDR, Flash)

• Nobody knows how this memory cycle ends

Read more

In Part 1 of this series, we built up the fundamental physics of ultra high power (UHP) lasers. We concluded by identifying the four walls that make it challenging to implement a UHP laser: managing heat, maintaining narrow linewidth, getting light out, and catastrophic optical damage.

In Part 2, we will explore exactly how Lumentum overcame these challenges via their 2007 paper as JDS Uniphase, and via their later 2022 CLEO paper on UHP lasers. We will touch on reliability and single mode operation, and dig into how defensible their UHP laser platform is, including what it means for the competition.

Contents:

• Suppressing Re-Broadening for sub-MHz Linewidth: How to choose the key design parameter that makes UHP lasers work

• Precisely Engineering the Cavity Length: Are longer lasers necessarily worse? How do laser designers choose the laser cavity length?

• Pros and Cons for Longer Cavity Length: What does cavity length help with, and what it means for reliability and mode-stability

• The Moat isn’t the UHP Laser design: The engineering mindset is to out-design the competition in UHP lasers, but reality is different: is it good enough?

By reading this post, you agree to the terms and conditions. Also see the full ethics statement.

If you’re new, check out the About page. A lot of readers expense the subscription to this newsletter as it helps their professional work. Group subscriptions (3+) are 20% off. If you have any questions, reply to this email and let me know!

If you are not a paid subscriber, you can purchase just this article using the button below. You can find the whole catalog of articles for purchase at this link.

Buy the ebook

Also check out the Semi Doped podcast with and myself, and our daily free newsletter with latest semi news.

If you would like to engage independent research services for your project or need expert calls, check out my boutique consulting practice at SemiExponent.

Suppressing Re-Broadening for sub-MHz Linewidth

Even in 2007, Lumentum (then JDSU) realized that increasing the output power of a DFB past a few 10s of milliwatts resulted in the laser linewidth getting broader than 1 MHz even if the “Schawlow-Townes-Henry” equation predicts that it should get narrower as the laser power is increased (explained in part 1). Good lasers should have narrow linewidth because the transmission of an optical pulse on an optical fiber broadens the pulse anyway; no need to have that problem at the source.

Pulse broadens as power gets higher. Source: Gemini, for conceptual purposes only.

They explain that there were two things causing this:

1. With so much optical power in the laser, photons are not distributed uniformly in the physical structure under the DFB grating. We explained this effect as spatial hole burning in part 1. This non-uniformity also causes a non-uniform refractive index, which in turn makes the linewidth broaden.

2. Within the InP multiple quantum well (MQW, where electricity is converted to light), there are random charge recombinations always occurring. This added to the “phase noise” of the system and also caused linewidth to broaden.

The former mechanism was more dominant, but the latter is significant enough to not be ignored. It was already known through other published work at the time that the easiest way to avoid re-broadening is to make the cavity length (L) longer. This generally helps distribute the photons in the cavity more evenly and keep the refractive index as even as possible under the DFB grating.

But it is not quite as simple as just increasing length because how evenly the photons are distributed depends on the DFB grating design. In part 1, we explained how there is a quantity κ – the grating’s reflection strength per unit length – combined with the length of cavity L, determines how evenly the photons are distributed.

As a recap from part 1:

κL is the key knob that decides how light is distributed along the chip.

• If the κL is too low, then light spreads out and leaks towards the ends. Insufficient feedback to hold the light in the channel evenly, and wavelength selection is weak.

• If κL is too high, then light gets trapped and piles up in the middle of the chip and causes spatial hole burning.

Engineering the right UHP laser now came down to picking the right cavity length L, for a given grating reflection strength κ set by the grating design. There is a goldilocks range for κL where the photon field was the flattest – which JDSU determined to be 0.7. Look at the pink lines in the graphs below, for different phases of the optical signal. κL=0.7 provides a flatness over nearly 200 degrees of phase – much better than κL=0.5 or κL=1.

Precisely Engineering the Cavity Length

Now that κL is set, the crux of the problem is that both κ and L are design parameters that can be chosen to keep κL=0.7. How do we choose?

Read more

The goal of this article series is not to dethrone Lumentum or advocate for an alternative provider in any way. It is to understand the true difficulty of making UHP lasers for CPO from first principles, so that the reader is well informed enough to cut through the noise and think for themselves.

Instead of examining generic laser architectures, we will directly address the one that Lumentum says their structure is most similar to, in their CLEO 2022 paper.

In their CLEO 2022 paper, it seems interesting that Lumentum would quote a paper from 2007 as the basis for their high power laser design. These authors at the time were affiliated with JDS Uniphase, which spun off in 2015 to become the publicly traded, and well loved company that we know as Lumentum today.

Interestingly, none of those authors work at Lumentum any more. Another important takeaway is that Lumentum has been working on UHP lasers since before they were Lumentum. As it turns out, 20 years later, this precise laser architecture is what the world wants for CPO and Lumentum just happens to have a ton of experience with it.

This 2007 paper describes a 1310 nm InGaAsP/InP DFB doing 600 mW on-chip and 350 mW ex-fiber with sub-500 kHz linewidth, and is spec-for-spec the ancestor of the UHP laser Lumentum sells for CPO today. We’ll use this as a case study to understand UHP lasers and how its related challenges were solved, but is still by no means an endorsement for investments in Lumentum. Do your own research.

To understand how to make a good UHP laser requires a background in how these devices work. This first part is groundwork: how an InP DFB laser works and the physical walls that make high power hard. Where Lumentum’s advantage actually lies, and how deep the moat runs, needs this physics first, so that verdict waits for Part 2.

Contents:

• InP laser diode fundamentals

  • Multiple Quantum Well structure

  • L-I curve, spontaneous vs stimulated emission

• Distributed Feedback (DFB) grating design

  • Mode-hopping

  • How to space gratings

  • How DFB works

  • Refractive index changes

  • Reflection strength and length

• Laser Linewidth and DFB Cavity Length

  • Calculating linewidth: Schawlow-Townes-Henry equation

  • Impact of power, mirror loss and dependence on length

• Low Power DFB Lasers for Telecom/EML

  • Direct modulated lasers (DML)

  • CW lasers and EMLs

• High Power DFB Lasers for CPO

  • How to increase output power

  • The 4 walls of UHP lasers

    • Heat

    • Linewidth widening

    • Getting light out

    • Catastrophic optical damage

By reading this post, you agree to the terms and conditions. Also see the full ethics statement.

If you are not a paid subscriber, you can purchase just this article using the button below. You can find the whole catalog of articles for purchase at this link.

Buy the report

If you would like to engage boutique research services for your project, reach out to us.

Contact SemiExponent

The Indium Phosphine (InP) Distributed Feedback (DFB) laser is the fundamental component of optics today, so it is best we understand how it works. It consists of two basic parts built on the same piece of material: The InP laser diode and DFB grating.

The InP laser diode

This is often called the gain section and its job is to convert electrical current into optical power. The light is generated by Multiple Quantum Wells (MQW) in an “active region” which are essentially alternating layers of InGaAsP (Indium-Galium-Arsenic-Phosporus) sandwiched together. The alternating layers have slightly different compositions of these fundamental elements, so that they have slightly different energy levels. The purpose of this structure is to trap electrons in a low-energy layer between two high energy layers so that the electrons are trapped in a “well” and can’t get out. Multiple wells means more electrons, which means stronger laser output.

Source: Effects of multiple quantum well width on InGaAs/InP laser diode, doi:10.1088/1742-6596/2937/1/012007. (link)

If you want to understand more about what energy levels in a semiconductor mean, I have several articles on this.

To generate light, current is applied to the laser diode by applying a voltage across the anode and cathode. When the current applied is low, the emission of light is a messy stream of light waves out of phase with each other, each one behaving independent of the other. This is called spontaneous emission, and is not a focused beam of light; it is much like LED light. If photons were people, they would just wander around in a park without purpose.

Beyond a threshold of current, stimulated emission kicks in. The photons generated in the presence of a highly concentrated “well” of electrons kick up more photons themselves resulting in an avalanche of more photons. These photons now march in step with their phases correlated to each other. Now the photons are on a military parade in complete lockstep while being highly directional and focused.

Source: Wide-Band and Scalable Equivalent Circuit Model for Multiple Quantum Well Laser Diodes, Jae Hong Kim, PhD Thesis, Georgia Tech 2005.

This relationship between light and applied current is represented by the L-I diagram shown below. The threshold of current needed to make photons march together depends on the temperature; higher temperatures need more drive current. This is an important observation that will later drive a lot of our understanding of UHP lasers.

Source: Wide-Band and Scalable Equivalent Circuit Model for Multiple Quantum Well Laser Diodes, Jae Hong Kim, PhD Thesis, Georgia Tech 2005.

Read more

Beyond the midplane PCB, SemiAnalysis notes that:

• The NVL576 multi-rack system is also likely delayed or limited to small volumes due to CPO challenges, leaving Nvidia without a proven way to expand the scale-up world size for Rubin Ultra.

• The NVL72x2 back-to-back architecture was cancelled after pushback from hyperscalers over its odd design and operational burden.

• The Oberon rack will remain the mainstay, and Nvidia will sell more of them in the Rubin era to cover the Kyber shortfall. SemiAnalysis still sees data center compute revenue running 20 percent above consensus in the second half of FY27, so views are not uniformly bearish.

In this post, we will view all these claims through an engineering lens to understand the underlying physics, draw conclusions where possible, and suggest solutions that might be deployed.

Table of contents:

1. Is the midplane PCB really too hard to manufacture?

2. NVL72x2: Why It Might Have Really Been Cancelled

3. NVL72x2: How to Make it Work

4. CPO for Scaling up World Size

5. Does this really create an opening for Google and AMD?

At the end of this post, you will walk away with greater nuance as to what roadblocks are in the way of next generation hardware, how the big players are working around them, and where the pieces might fall in the future.

By reading this post, you agree to the terms and conditions. Also see the full ethics statement.

You can find the whole catalog of articles for purchase at this link. This article will be available for paid digital download shortly after publication.

This is how we think about market claims at SemiExponent. If you have a bet you'd like tested this way privately, get in touch.

Contact SemiExponent

Is the midplane PCB really too hard to manufacture?

Let’s first introduce the midplane PCB and explain what problem it aims to solve. Here is a picture of it:

In all the generations of racks till date, the scale-up domain in NVIDIA racks has always been copper. There is a cable cartridge that contains just over 5,000 passive copper cables that hook up the GPUs in an all-to-all configuration. As the interconnect speeds in the scale-up domain increased in every successive generation, copper interconnect reach became an issue. Copper’s losses meant that you needed something much shorter than long copper cables running through a spine. Enter the PCB.

The idea of the PCB was simple: hook up all the GPUs and network switches on a board level, instead of cables running through a spine. The compute trays would hook into one side and the network switches on the other. Then, by simply running traces on the PCB, you can get GPUs hooked up to each other. Simple and elegant right?

No. Turns out that this is a routing problem of gargantuan proportions (literally) because you can’t just draw metal lines between GPUs like in a picture. As a result the routing problem only increases. You would need to route over or under metal connections to prevent them from intersecting.

Think of a freeway intersection: to keep the flow of traffic moving smoothly, you have different levels of roads, each level adding a significant amount of complexity to the routing problem.

It’s the same problem on the PCB, except that you have about 78 layers. In addition, the Kyber rack has 144 GPUs compared to the Blackwell era “Oberon” rack which has 72 GPUs. The exact PCB layer count seems to differ based on who you ask, but the takeaway remains: it is a very complex PCB. The midplane PCB makes the equivalent of 20,000 connections that would otherwise need to be copper cables which would be too many to fit in the spine. These boards are built on separate 26 layer boards each nearly a square meter in size and then laminated together to form a massive PCB required for Kyber midplane.

Anybody who has worked with high speed interconnects knows that PCBs are not automatically better. Sure, you can pack a higher routing density in a smaller space cutting the reach distance, but that says nothing about the losses. In general cables have lower losses for high speed signals than metal lines on a PCB. To keep the midplane PCB losses in check, it is built on an advanced high-frequency material consisting of an M9-grade copper clad laminate (CCL) with Quartz glass (Q-glass) reinforcement. The use of these specific materials is so that the dissipation factor (~0.0007) is as low as possible. Dissipation factor is a measure of how much of the signal is lost in the PCB dielectric material. When you have so many layers, it is best to lose as little as possible.

The sheer size, number of layers, and specialized materials required for the midplane PCB make it an engineering marvel. When deploying this in production, the yield of the PCB and reliability in the field have to be exceptionally high. The obstacles are plenty:

• No de-lamination can occur at any layer across a panel near a square meter.

• Quartz glass is abrasive and hard to drill cleanly; connections between layers (through what are called vias) and drill bit wear become yield limiters.

• Since yield compounds per layer, it becomes extremely difficult at 78 layers. A defect anywhere scraps the whole board.

• A board-level failure affects the operation of 144 GPUs (in an NVL144 configuration) all at once.

These factors make the engineering marvel hard to build in volume. The reason we examined the underlying engineering is to ground rumors of schedule delays in engineering reality. Yes, building these boards is hard but not impossible. AI PCBs already span tens of layers easily, the midplane PCB takes it to another level.

Confirming or denying the delay outright requires tracking the supply chain and orders closely, which is out of scope here. Additionally, Nvidia has responded saying that their roadmap is intact – so we’ll take their word for it. But, if you are NVIDIA, and your best engineers say that the midplane PCB is hard, what’s the backup plan? The answer is the NVL72x2.

NVL72x2: Why It Might Have Really Been Cancelled

144 GPUs are difficult to fit into a rack for a couple of reasons:

1. They draw a lot more power per rack which would require the use of 800V DC systems to handle the power distribution and copper cable weight within the rack (see earlier detailed post on this below)

2. The routing problem as we discussed is complex, and requires an enormous PCB that is hard to make.

The logical stopgap solution would be to instead put in 72 GPUs per rack, just like in the Blackwell era, and put in two such parallel racks back-to-back. This solves the scale-up interconnect problem within a rack because you only need to connect 72 GPUs in the rack just like in the Blackwell “Oberon” rack. NVIDIA can simply use existing racks and not have to worry about it. The 144 GPU implementation would then involve putting two such racks together.

The problem now is how to connect the two racks together so that they behave as if they were one rack.

While the Kyber issues were being sorted out, SemiAnalysis stated that NVIDIA considered a back-to-back configuration as shown below. Perhaps this configuration was known to their institutional clients, but a detailed check of my SemiAnalysis newsletter archives shows that this was not mentioned anywhere before – including in their Vera Rubin deep-dive. Perhaps it was not in consideration at the time of writing.

Regardless, this configuration was allegedly disliked by the hyperscalers for its “odd” configuration. This argument I do buy because the data halls are not laid out to accommodate racks in any orientation. There are cable raceways, cooling, and power all laid out to accommodate racks in a particular orientation, and going back-to-back will require changing all the data hall infrastructure. If this was actually suggested by Nvidia as a configuration, no wonder hyperscalers and CSPs pushed back. SemiAnalysis stated in their tweet that this configuration was cancelled.

However that does not mean the NVL72x2 will never work, or that there aren’t other ways around current challenges. Beyond the paywall, we discuss:

• How to make the NVL72x2 work

• Why CPO/NPO is the right answer for scaling to larger world size

• Does this really create an opening for AMD and Google?

Read more

HBC is a platform technology, the foundation Qualcomm means to run through every business unit and not only the data center, and the whole thing rests on one move in packaging: it does not remove the hard part, it shifts it from the advanced substrate bracket (CoWoS) to a stacked-die approach that brings its own complexities.

Contents:

• HBC as the Unit of AI Intelligence

• Memory-on-Logic Packaging as a Key Risk Factor

• Logic Die Function in HBM versus HBC

• The Power Budget Under the Stack

• Who Builds the HBC Stack

• What to Watch For

Disclaimer: I am an ex-Qualcomm employee but hold no MNPI regarding their line of datacenter products. I worked as an RF engineer dealing with transistor devices – far from the sexy world of AI. The facts in this article are from public sources, and include my own inferences. I currently do not hold any QCOM stock. Not financial advice. Do your own research.

By reading this post, you agree to the terms and conditions. Also see the full ethics statement.

You can find the whole catalog of articles for purchase at this link. This article will be available for paid digital download shortly after publication.

If you would like to engage boutique research services for your project, reach out to us.

Contact SemiExponent

HBC as the Unit of AI Intelligence

The figure shows a standard packaging substrate with an System-on-Chip (SoC) and an low-power DDR (LPDDR) memory stack sitting on top of an XPU. At first glance it looks like an XPU next to HBM, but it is not that. The DRAM is stacked on the XPU, the die that runs the matrix multiplications, and the SoC beside it is a separate chip whose identity changes with the end market.

That second chip is the part Qualcomm has always built. It has been a modem, an automotive ADAS processor, an IoT SOC, and even a robotics chip. The XPU with the DRAM on top is the new part that provides Qualcomm with an AI accelerator. The main idea behind HBC for Qualcomm is for it to serve as the unit of intelligence, and pairing it with one of Qualcomm’s business-unit SoCs is what makes that line AI-enabled. As such, its applicability and importance extends to its entire business.

Memory-on-Logic Packaging as a Key Risk Factor

Qualcomm’s claim is that HBC removes the need for advanced packaging, whose supply chains are hard to get hold of right now, and that HBC therefore runs on standard packaging substrates, which are cheaper, more available, and easier to integrate on. HBC does avoid the 2.5D interposer that an XPU-plus-HBM design needs, and the gains it buys are real.

Stacking the memory on the logic die swaps the HBM path, a high-speed interface running along the die edge and out through an interposer, for short vertical links through the die area. That lifts the connection count from about 2,048 (in HBM4) along the edge to the order of 10,000-100,000 through the area, and it brings down the energy needed to move bits by an order of magnitude.

Qualcomm however glossed over the fact that packaging difficulty has not gone away. It moves from the advanced substrate (CoWoS) to 3D die-to-die stacking, which is also advanced packaging, and on a compute die, it is the harder version of it. Qualcomm is not removing the packaging step but instead trading a supply-constrained one for a harder and less proven one.

Stacking DRAM on logic is not new in itself. HBM already does it via a core logic die sitting under a stack of DRAM connected through silicon vias. HBC looks similar, but the logic die does a different job, and that difference is the whole risk.

Read more

Here’s what you might have missed this week:

• What AI Inference Actually Demands from a NAND SSD

I was traveling in the UK last week, and Austin was in NYC for Qualcomm’s event. We got to recording the podcast later than usual this week, but our entire breakdown of the Qualcomm investor event should be out on the podcast soon.

Reminder that you can keep up with the daily news on the Semi Doped Substack, for free.

Now, on to the news.

This edition of TWiC is presented by … SambaNova

Run frontier models. Skip the GPU rack.

SambaCloud runs MiniMax M2.7, the fastest provider for this model, at 435 tokens per second via API. M2.7 ranks alongside Claude Opus 4.6 and GPT-5.4 on agentic and software engineering tasks, at a fraction of the cost. No hardware procurement, no lead time, no sharding across hundreds of GPUs.

Texas Advanced Computing Center, OVH Cloud, and Hume.ai are already on the platform. If you’re running long-horizon agent workflows in production, M2.7 is available today on SambaCloud’s Developer and Enterprise tiers. Benchmark your workload against your current provider in an afternoon.

👉🏽 Start benchmarking today.

(I tried it for free, and its always using something while knowing what the hardware behind it looks like).

OpenAI’s Jalapeno shows that ASIC design is commoditized

There has been a lot of generic coverage about the fact that OpenAI has their own chips, but I want to point out one thing: this chip was made with Broadcom doing the ASIC design from start to finish in 9 months. That means one of two things: (a) Broadcom’s silicon prowess is what Hock says it is — amazing, or (b) ASIC design has become a commodity business, with no moat after all.

I am willing to bet on the latter. The process of wafer fab and packaging usually takes about 3-4 months from start to finish. If you can one-shot a cutting edge inference chip in a mere 5-6 months, it shows that its probably easier to make than most people think.

I have personally never done ASIC design so take my words with a grain of salt, and feel free to correct me. But I do know that engineers working on that largely deal with software only. Coupled with the fact that AI-enabled design flows are now being pushed by every major EDA company, maybe chip design is getting commoditized after all — the press release does say “accelerated by OpenAI models.” But, not all kinds of chip design are easy: optics, power, analog, and anything that does not rely on digital transistors turning on and off, are still challenging to make and require specialized expertise. CPU design is still challenging perhaps — I don’t hear of anybody doing that in 9 months. Just some observations.

What does Modular bring to the table for Qualcomm?

Qualcomm recently announced that they will be acquiring Modular in their recent Investor day. But why is a company like Qualcomm buying a software platform like Modular?

Qualcomm is betting on disaggregated inference, which means that the future of AI involves a mix and match of hardware, each suited to the best function and workload. Imagine that you can get a DragonFly rack from Qualcomm, a low latency inference rack of LPUs from Nvidia, and a CPU rack from AMD. How do you ensure that they all work together?

This is what Modular brings to the table. One software platform for different kinds of hardware. To work with the Modular layer means that the software stack is written with Mojo. If NVIDIA CUDA is like Apple’s iOS, then Mojo is like Android. No need for specialized hardware tied to the software, but instead work with every hardware provider possible. These kernels written up with Mojo is then served with MAX on your own cloud platform, or you could use Modular’s.

But if Modular is owned by Qualcomm, how does a purchaser of a Qualcomm DragonFly rack deploy actual services? Do they pay Qualcomm a royalty fee for the software layer? Are they even willing to adopt the “Android of AI Inference” in order to deploy Qualcomm racks? A lot of these questions remain and remains unclear until the hardware actually lands.

The Gatekeeping of Frontier Intelligence Models

ChatGPT 5.6 “Sol” was released this week, and the full capabilities of the model are only available to a select few. The Trump Administration has “requested” a staggered roll out, ask that it approve its use on a case-by-case basis, as reported by The Information. This is a much more favorable deal for OpenAI than what Anthropic got even with its nerfed Fable model. Mythos was never meant for public consumption from the get go.

Gatekeeping frontier intelligence models creates an opposing tension; on one hand, it creates a layer of resistance to monetize the newly trained model in an AI token factory, but on the other, the industry does not have enough compute to serve frontier intelligence at scale. The path to usefulness of AGI is always limited by how many chips we can actually make. These opposing tensions place frontier models in a neutral equilibrium: a restricted user base on a finite supply of compute.

This disturbing trend of AI governance opens the door to many future questions:

• Would OpenAI or Anthropic dedicate enough resources to develop the next frontier model knowing a-priori that its full capabilities are going to be restrained?

• Who decides who gets to use cutting edge models, and does that give them an unfair advantage over competitors?

• If model releases depend on the benevolence of political leaders, what levels of quid-pro-quo can we expect to see between Silicon Valley and the White House?

Model scaling laws still hold: adding more compute to training a model creates better models, but human laws are a limiter. The imperative now shifts to model labs to instead continue to monetize via AI inference where the real economic value lies. The bodes well for the smorgasbord of interference accelerator companies who rely on interference to be the main driver of silicon demand. Today, NVIDIA is the dominant provider of training chips, and a slow down in frontier model progress due to governance will also slow down the consumption related training hardware.

However, at the recent Computex event, I had an interesting chat with some investors who were interested in knowing if SLC NAND was making a comeback because write endurance of TLC/QLC NAND was worse, even if it had more capacity. The answer to this is nuanced because it is no longer sufficient to think of NAND as a single tier, but instead how NAND is starting to split into multiple sub-tiers based on its place within the memory hierarchy, each with its mix-and-match of technologies.

The broader question that it encompasses is what the performance expectations of NAND in these newly forming performance tiers are, and more importantly what products from which companies are well positioned to capture the upside.

In this post, we will explore how NAND SSDs fit into context memory storage (or CMX, as NVIDIA calls it), and look at which SSD vendors are best positioned to serve this market.

Here is what we will cover:

• Why NAND SSDs are a viable option for AI inference

• Tuning SSD Performance for Memory or Storage

• CMX-Specific SSD Optimizations

• I/O size and the indirection unit (IU)

• Asymmetric read/write bandwidth

• Write Amplification and FDP

• NAND SSD Competitive Landscape

By reading this post, you agree to the terms and conditions. Also see the full ethics statement.

If you are not a paid subscriber, you can purchase just this article. This post will be updated with a direct purchase link shortly; check back on Substack.

You can find the whole catalog of articles for purchase at this link.

If you would like to engage boutique research services for your project, reach out to us.

Contact SemiExponent

Why NAND SSDs are a viable option for AI inference

Relying solely on DRAM (including HBM) to hold context is becoming untenable in the agentic era of inference due to cost and capacity. As users delegate increasingly complex tasks to AI, inference is no longer a process that takes minutes; it is extending to hours and days, and if we could manage it, to weeks and months. “Hardware-utilization-factor” will become an increasingly important metric as neoclouds and hyperscalers grapple with large capital expenditure costs while trying to translate investments into revenue. Underutilization due to memory capacity constraints is an unacceptable scenario.

NAND SSDs are a viable alternative because their performance can be finely tuned to act more like memory, or more like storage, depending on the underlying technology used. What was once a means to just faster storage and an alternative to HDDs, is now becoming a memory tier that lies somewhere between storage SSDs and DRAM.

The primary function of NAND in the agentic AI era is to hold KV-cache, which can often balloon to tens of TBs over the course of a large agentic task, eliminating the possibility of storing it on any DRAM tier. During the decode step of inference, the KV-cache needs to be accessed as fast as possible to provide a high token throughput. Speed is especially important because the decode step generates one token at time due to the autoregressive nature of the transformer architecture. NAND SSDs, with the right modifications, are capable of serving as a KV-cache store while streaming data fast enough for decode to still work well.

DeepSeek v4, for example, has already made NAND a first class citizen by ensuring that context is offloaded to SSDs and cache-hits are maximized for the lowest possible token cost. It is highly likely that similar architectures will follow in other models.

Aside: CXL pooled DRAMs are equally viable, and some would argue a better alternative, but that’s a different post. In reality, it occupies its own tier in the memory hierarchy.

Tuning SSD Performance for Memory or Storage

The feasibility of NAND for AI inference stems from the design choices of the underlying flash cells. While my original NAND article covers the in-depth workings of the various types of NAND, we’ll address it briefly here, and focus more on how to slot them into different NAND performance tiers.

Flash memory essentially operates by storing or releasing charge, which represents the bit representation of 0 or a 1. An N-level NAND cell has 2^N states in which a memory cell can exist in. For a single level cell (SLC, N=1), there are just two states. For a quad-level cell (QLC, N=4), there are sixteen states. The capacity increases when more states are stored in the same cell because the cell physically does not take up any more space where it stores a 0/1 in SLC or 0000—1111 in QLC. In other words, you can store more bits in the same physical area.

Capacity increase comes with other trade-offs, namely endurance and speed – and both of which are tied to the number of cell states. It comes down to distinguishing between either just two cell states or sixteen, and how that can be done quickly and reliably over the life of the NAND SSD under all operating conditions it is subject to.

• Endurance: Over time, the voltage levels that distinguish between the cell states start to move due to constant cell writes and temperature effects over time. It is just part of the wear and tear induced in the oxide layers within the cell when it is subject to constant use over its lifetime. For a single voltage level that distinguishes between cell states (like in SLC), the shift can be higher and we can still tell if the cell holds a zero or one. For a QLC with 16 voltage states, it becomes much harder, and guessing wrong results in a bit error. Once bit errors increase beyond a certain threshold, the NAND SSD is deemed worn out.

• Write Speed: Writing to a cell involves finely adjusting the voltage applied to the cell, and closely monitoring the resulting current from the underlying cell. To determine what the state of the cell is, a process called incremental step pulse programming (ISPP) is used to recursively adjust the voltage to just the right level (more in the original NAND article). This iterative process of adjustment affects the write speed because it takes many recursive steps to set the right voltage. With fewer cell states, the iterative process is quicker which means SLCs are faster than QLCs.

By now it should start becoming clear how NAND SSDs can be fine-tuned for capacity and performance, at least to the first order. Use more cell states (higher level NAND) to get more capacity, but fewer cell states (lower level NAND) to get more speed/endurance. A hybrid approach is also possible: use a SLC-cache with a TLC NAND, for example, to get the best of both worlds with the right speed/endurance/capacity tradeoffs.

Next, we will discuss how NAND SSDs are better optimized for AI use-cases, and conclude with a competitive survey of the NAND landscape for agentic AI.

Beyond the paywall:

• CMX-Specific SSD Optimizations

• I/O size and the indirection unit (IU)

• Asymmetric read/write bandwidth

• Write Amplification and FDP

• NAND SSD Competitive Landscape

Read more

Other stuff in our universe has been centered around advanced packaging, in case you missed it.

As part of Semi Doped we also publish a daily newsletter with the hottest takes, straight from the oven — all for free. Sign up at semidoped.com.

Now, on to the this week’s explorations on TWiC.

How MEXT Makes Flash Work Like DRAM

This week AMD announced that they are acquiring mext.ai. The chart below is unrelated to the announcement, but also highly relevant. I’ll explain why.

This memory share estimate of CapEx is 36% two months ago, for 2027. Memory prices keep increasing, so let’s just round off to 40% for simplicity and assuming a $1T AI CapEx spend in 2027 because it makes math easy. That’s $400B dollars spent in just paying for memory. We had a whole episode on Semi Doped explaining why this crazy pricing is not useful to anybody. That much money can be spent on something much more useful — like trying to improve communication bottlenecks, or building better power delivery systems.

Any CFO worth their salt will try to reduce this spend by asking “Do you really use all that memory we pay for?” MEXT’s solution what is meant to answer this question.

MEXT is a software company that helps you deal with storage more effectively, whether it is DRAM or NAND. Their key insight is this: not everything is DRAM is immediately useful — some stuff is hot, some stuff goes cold. Then the sensible thing to do is to boot the cold data from DRAM to NAND flash which has much more storage capacity. Next, they predict using “AI” (it’s like AI ouroboros here) what needs to be in DRAM, and do the reverse as well — load data from NAND to DRAM before the compute asks for it. It’s the memory management method that sweeps data under the rug, and pulls out only what is needed at the right time.

NAND is much slower than DRAM, but when used together in a form of deliberate data dance, it becomes very effective. These kinds of storage optimizations are essentially bullish for NAND. This bottom line is: (a) Hardware builders cannot go on indefinitely adding DRAM without asking if we are using it effectively, (b) Memory companies cannot raise prices indefinitely without the industry trying to find ways around it, and (c) We will see the emergence of a lot more of software level optimizations like this in the future that will help with memory management in a clever manner.

PicoJool and VCSELs for AI Scale Up

This week a startup called PicoJool announced that they are introducing 200G VCSEL lasers for AI scale up. VCSELs have been famously difficult to make work beyond 100G due to modulation bandwidth limitations in the electrical-optical conversion. In simple terms, a high bandwidth means that the optical signal changes fast enough to track the changes in electrical signals.

The chart above is from a 2022 paper showing the modulation bandwidths of 850nm VCSELs. As you can see, VCSELs have been continuously getting faster, and capable of supporting higher modulation bandwidths. PicoJool’s stated bandwidth is better than anything on this chart — at nearly ~38 GHz. Their measurement is shown below.

The way to read this is to think of a continuously varying RF signal being input to the electrical-control of the VCSEL. At say 10 GHz, there is little loss from the conversion to optical — ie, the S21 (transmission scattering parameter) is 0 dB. That means just rapidly changing the electrical input doesn’t make the VCSEL light any dimmer. At ~38 GHz, the signal is so fast that only half the optical power is produced at the output. Beyond this speed, most of the energy is lost in the conversion because the VCSEL can’t keep up. At this point the link budget gets worse, and errors rates go up for a given reach.

The big thing about using VCSELs is that they are made from the widely available Gallium-Arsenide (GaAs), as opposed to the much sought after InP, and GaAs for scale up has a big impact on TAM of InP lasers should this technology actually work out (yet to be proven). However, note that not all VCSELs are Indium free. The quantum well could be engineered with GaAs/InGaAs too. It really depends on the laser implementation — so be sure to ask for device details before jumping into supply chain calculations.

One objection that people have around VCSELs is that they are not reliable. The degradation often comes from the oxidation of aluminum in the quantum junction (it is actually not purely GaAs — it is a mix of GaAs and AlGaAs). However, if you switch to 980nm/1060nm VCSELs (like Lumentum is doing), then the bandgap for that laser wavelength does not require aluminum — which inherently eliminates one major source of failure. The elimination of aluminum also helps with defect propagation, but we won’t get into that here.

There are a lot of different aspects to whether VCSELs will ultimately become feasible. It is interesting to follow the developments here.

Intel had a Very Good Thursday

Stock is up 10% into new all time highs. It’s mainly because of the news below, and also the fact that they hired Seok-Hee Lee, an SK Hynix veteran, who was ex-Intel incidentally, to lead advanced packaging and back-end engineering.

This is a major turnaround since Intel famously refused to manufacture iPhones for Apple because they didn’t think the market would be big enough. This primarily led Intel to miss the wireless boom, and handed over the keys to the mobile chips kingdom to TSMC.

Apple and TSMC have since built a massive business due to a long term partnership, where the business from just Apple alone was so much that it has made TSMC into what they are today. Apple going to Intel again for chips is opportunity knocking twice, at a time where Intel Foundry needs volume to pick up its foundry business whose near-death experience had many worried. TSMC’s capacity issues are now blowing fair winds Intel’s way again, and it’s time for Intel to orient their sails just right this time.

According to a recent TrendForce estimate, the industry’s CoWoS capacity could reach 200k wafers-per-month (wpm), with TSMC running ~120k wpm by the end of 2026, while ASE and Amkor get a seat at the table only as overflow, soaking up something like ~80k wpm. TSMC has been rapidly building out capacity to support CoWoS demand. Most of the big GPU chips are packaged by TSMC, while simpler flows like Vera CPUs and automotive parts overflow to Amkor/ASE facilities.

Intel’s EMIB is the first credible second source to show up with real orders behind it. AWS and Cisco are already shipping external silicon on EMIB. SpaceX and Tesla signed on around the Q1 2026 earnings call as partners in Intel’s Terafab project. Google’s TPU v8e is reported to land on EMIB in the second half of 2027, with Apple, Microsoft, and NVIDIA sitting in the talks pile. CFO David Zinsner moved Intel’s external advanced-packaging outlook from a few hundred million to north of a billion dollars across the last two calls, and told CNBC he expects billions of dollars per customer. For a business that small, packaging alone going past a billion is a step change.

However, as orders flow to Intel, TSMC is continuously expanding CoWoS capacity. The increasing sizes of advanced packages for AI accelerators puts the two competitors at a crossroads again -- and this time the outcome will be decided based on what happens in the next few years as the industry moves to adopt glass core substrates.

Contents:

1. EMIB Comes for the CoWoS Monopoly — the three-year packaging monopoly, and the orders that are breaking it.

2. Deeply Understanding CoWoS-L vs EMIB — the three CoWoS flavors, and how Intel’s bridge differs from TSMC’s.

3. Where the Size Limitations Lie — the round-wafer-versus-panel math that caps CoWoS-L and frees EMIB.

4. The Convergence on Glass Core Panels — why warpage and CTE force both foundries onto glass.

5. Where This Leaves Investors — the winners and losers, and the OSATs that win either way.

6. The Taiwanese Glass Partner Already Inside TSMC’s CoPoS — the name the glass maps miss.

By reading this post, you agree to the terms and conditions. Also see the full ethics statement.

If you are not a paid subscriber, you can purchase just this article using the button below. You can find the whole catalog of articles for purchase at this link.

Buy the ebook

If you would like to engage boutique research services for your project, reach out to us.

Contact SemiExponent

Deeply Understanding CoWoS-L vs EMIB

Many comparisons online of CoWoS to EMIB are not precise about the exact flavors of each technology, and opens doors to misinterpretation. We’ll clarify that aspect first.

• CoWoS-S is the original, a single monolithic silicon interposer. It gets really large and expensive beyond a certain size. TSMC’s limit is about 3.3x reticle size (or 2,380 mm²).

• CoWoS-R swaps that silicon interposer for a cheaper organic RDL one, which is what AWS uses for Trainium 2/3 chips. They do not have the interconnect density of silicon interposers, and are rather limited in size too.

• The variant carrying the biggest AI packages today, the Blackwells and the Rubins, is CoWoS-L, which scales to 5.5x reticle size (or 4,720 mm²) today, with plans to expand to 9.5x reticle size (or 8,150 mm²) in 2027. TSMC’s roadmap extends to 14x reticle size (or 12,000 mm²) by 2029.

Both TSMC’s CoWoS-L and Intel’s EMIB are built on the same idea of giving up a full silicon interposer for silicon bridges only where you need fine pitch wiring, such as die-to-die and die-to-HBM, and use cheaper routing for everything else.

• TSMC calls its bridge an LSI (local silicon interconnect, and hence the ‘L’) and embeds it in an Re-Distribution Layer (RDL) interposer.

• Intel calls its bridge EMIB (embedded multi-die interconnect bridge) and embeds it straight into the organic package substrate.

Note the difference in where the bridges live. CoWoS-L’s RDL interposer is built on a wafer, so it stays bound to the 300mm wafer and the reticle limit (the whole reason CoPoS and glass panels are coming). EMIB embeds its bridges in the organic substrate with no wafer carrier, so it clears the wafer-size wall, and scales to bigger interposers due to the use of rectangular panels.

Here is Intel’s EMIB roadmap in terms of size.

Today, Intel’s EMIB holds a size advantage by being able to deliver on 8x reticle size (or 6,860 mm²), which is ahead of TSMC’s 5.5x capability, due to the choice of rectangular formats for organic substrates rather than RDL interposers on round wafers. Intel promises 12x by 2028 as opposed to TSMC’s 14x, but we’ll have to wait and see who actually delivers, and at what reticle-size multiple.

Meanwhile, we can try and guess.

Where the Size Limitations Lie

CoWoS-L runs into a size ceiling that comes straight from the round wafer it is built on. The LSI bridges are embedded into RDL, but the RDL itself is fabricated on a 300mm silicon wafer and then mounted on an organic package substrate. While the bridges themselves are small, the RDL interposer carrying them is one big rectangle, and a round wafer yields only a few per wafer.

Let’s actually calculate who is capable of delivering a 14x reticle by 2028. On the left is a 300mm round wafer with a total area of 70,600 mm^2, representative of TSMC’s process. On the right is a 310mm x 310mm organic panel, with a total area of 96,100 mm^2, representative of Intel’s EMIB.

The rough excalidraw picture above shows that a round wafer fits just about three 14x-sized packages, while the rectangular panel provides six. The area utilization goes from ~50% to ~75%. A transition to bigger 510mm x 515mm panels in the future will yield ~20 interposers.

It is clear that EMIB’s use of rectangular panels is vastly superior when it comes to large package sizes. It is the same round-versus-rectangular problem pushing TSMC toward CoPoS, except EMIB already uses the rectangular format. Intel is starting where TSMC is trying to get to. What keeps TSMC ahead in practice is volume, since it ships these packages today. Intel’s advanced packaging story is just getting started, but it holds a structural advantage in large packages in the future.

The Convergence on Glass Core Panels

Unfortunately for Intel, at 14x reticle sizes, it is not just the advantage of rectangular vs round formats that matter. For such large packages, organic substrates inherently become poor candidates due to their tendency to warp, and the co-efficient of thermal expansion (CTE) mismatch with silicon. Regardless of the format, the physical properties of the material starts causing yield issues which strips away some of Intel’s lead in large size packages, and instead levels the playing field with TSMC.

The answer for both foundries is the same: large rectangular panels on glass core substrates. It has unique properties that make well suited as a core material:

• It has a high degree of flatness

• It has a CTE that is closer to silicon than organic materials

• It has low dielectric loss (good for interconnect)

• It has a high Young’s Modulus, which means that it is very stiff and needs a high degree of force to stretch it even a little.

• In glass, through glass vias (TGVs) can be made much closer, which results in better power delivery due to lower parasitics.

• Light propagates through glass, which opens the doors to optical integration in the future.

Intel is the first mover on glass substrates, demonstrating a 78mm x 77mm, which is equivalent to 7x reticle size. This houses 2 full reticle-sized GPUs, for example, with room for interconnections with EMIB.

More importantly, Intel has solved the issue of “SeWaRe” - a Japanese term that refers to micro-cracking in glass substrates. It usually happens right through the lateral cross-section of glass -- kind of like when you pull apart the biscuit halves in an Oreo. Micro-cracking while drilling TGVs is an important problem that requires special techniques to deal with. One company holding a dominant position here is LPKF Laser and Electronics - a German company - whose Laser Induced Deep Etch (LIDE) process is capable of forming TGVs at scale. We’ll introduce another contender after the paywall.

The race to glass panels is just not an Intel vs TSMC race. Samsung and Rapidus are expecting to roll out glass interposer solutions as well.

For paid subscribers:

• What this means to investors, and how the advanced packaging scene will play out

• Two other supply chain names outside LPKF that are worth a look

Read more

It is arguably a provocative title, but going purely off the heading is akin to judging the book by its cover. The institutional note “automagically” landed in my inbox and I had a long, quiet read to process it. I also read all the notes by FundaAI around this, a few other Substack notes, and tried to keep up with the flurry of takes on X.

The now infamous note is largely post-Computex, where we identified optics and power as the major themes as well. Here is our coverage this week:

• 📚 Computex 2026 Debrief

• 🎙️ Computex 2026: Decoding Optics and Power Content

The long and short of it that CPO is coming eventually, as is 800V; nobody denies this, but the controversy is around the when. We’ll focus on CPO today.

The SemiAnalysis claim worth focusing on is that Spectrum-6 CPO is slipping by 2+ quarters due to: (1) an unusual level (>3.5 dB) of optical loss in Spectrum-6 that eats link margin and notably higher than Spectrum-5 that uses a similar port count forcing a redesign, and (2) a yield constraint that causes issues at scale.

We have no way of verifying actual numbers here, unless the reader is the rare case of the one actually working on it. However, technical discussions around these two claims are light, and rebuttals mostly come from the supply chain argument side. There is little dialogue around understanding why this could happen, and if the engineering side holds up.

We will try to fill in this gap today.

This edition of TWiC is presented by … SambaNova

Run frontier models. Skip the GPU rack.

SambaCloud runs MiniMax M2.7, the fastest provider for this model, at 435 tokens per second via API. M2.7 ranks alongside Claude Opus 4.6 and GPT-5.4 on agentic and software engineering tasks, at a fraction of the cost. No hardware procurement, no lead time, no sharding across hundreds of GPUs.

Texas Advanced Computing Center, OVH Cloud, and Hume.ai are already on the platform. If you’re running long-horizon agent workflows in production, M2.7 is available today on SambaCloud’s Developer and Enterprise tiers. Benchmark your workload against your current provider in an afternoon.

👉🏽 Start benchmarking today.

The industry agrees as a whole that scale-out is where CPO lands first. NVIDIA Spectrum-X CPO (Ethernet) and Quantum-X CPO (Infiniband) are slated to enter mass production in 2026. The SemiAnalysis claim is that the Ethernet version is due to slip by 2+ quarters, and is materially important because the Ethernet version ships more volume than the Infiniband one.

Both the increased optical loss and yield issues have their roots in the sub-micron accuracy alignment requirements of optical engines for CPO, and requires a procedure called “active alignment.” It involves optimizing the fiber attach unit (FAU) to the photonic chip across 6 degrees of freedom (position+angle across x,y,z). The alignment is done by actively monitoring the amount of optical power being coupled, and a few degrees of freedom are adjusted till maximum power is coupled.

This has to be repeated for each optical engine in the whole Spectrum-X switch, and is required at several points along the CPO build process. Even a single CPO module can require 100+ active alignment operations from start to finish, each one taking only a few seconds from start to finish in high volume production.

CPO testing is an interesting aspect that deserves its own deep-dive, but key takeaway is that co-packaging an optical engine next to the silicon is notoriously difficult, and the difficulty arises when all of them need to work when attached. Quantum X actually implements CPO differently from Spectrum X as shown in the pictures below.

Quantum-X is designed for routing supercomputing traffic with ultra-low latency. It has a total of 18 optical engines (6x optical subassembly with 3 engines each) to provide 18 ports in an NVswitch supporting Infiniband. Quantum X uses a replaceable module of 3 optical engines that can be swapped out as a whole if any one of the three engines in the module do not work. The complexity of active alignment is limited to just the optical subassembly now. If the yield of each attachment is 95%, the overall yield of the subassembly is 0.95³ = 0.853, or 85.3%.

Spectrum-X handles larger, multi-tenant cloud and Ethernet AI environments and thus requires more port counts, and more optical engines per switch. As a result, the “swappable module” approach does not work for this product line due to sheer lack of space. You can see from the picture that there are 32 optical engines directly co-packaged next to the silicon. A single non-functioning optical engine poses a danger to the entire switch.

Here the math is that if each one has a 95% chance of success, then the chance of all 32 engines being successful is (0.95³²=0.193, or ~19%). This makes it significantly harder to deploy Spectrum-X switches at scale if the yield is so low. If you want ~85% yield like we saw in the 3-engine optical subassembly, the required attachment yield rate skyrockets to 99.5%. There are some counter-arguments floating by to this method.

1. Is the yield really 95%? This number is genuinely difficult to come by, and if based purely on hearsay, is something to take with a grain of salt. The exact yield number matters a lot, arguably to the last decimal point, when you take a power of 32. Hand wavy napkin math does not work here.

2. Failing parts are binned. One post I read argues that the napkin math is bad because optical engines are binned and only functional ones will be attached and therefore the failure rate cannot possibly be that high. Yes, failing parts are binned, but the actual failures come during/after the attach process. As a result, there is no way to avoid the power of 32.

Most of the commentary on this matter has been a rebuttal on the basis that the bottleneck is on the supply-side, and not the demand side. Lumentum’s comments at the Mizuho conference also stated how much demand is waiting for them, if they could only ship. Morgan Stanley released a note with wafer shipments, while acknowledging TSMC yield numbers in SoIC and downstream assembly yield.

Subscribe now

The loss and yield argument makes “qualitative” engineering sense given how Spectrum-X and Quantum-X CPO architectures work, and the crux of the SemiAnalysis argument reduces to finding the right yield number. If there is a fundamental problem with the active alignment process, it is easy to drop half the optical power (which is what 3 dB of loss is) due to misalignment issues; remember that the alignment accuracy requirements are sub-micron for thousands of optical fibers per switch. The loss number itself is within the realm of reason given how sensitive CPO is to alignment issues.

Given how new the industry is to CPO at scale, loss low yield could very well be growing pains as the technology ramps. It is not an unsolvable problem; just one that takes its time as assembly processes for CPO mature. It is entirely reasonable that the alternative while the industry grows to adopt CPO, is NPO.

In the case of NPO, the assembly and packaging process is much simpler because only the optical engine in that NPO module needs active alignment — it is not an exponent-stacking problem like in CPO. This makes it much more conducive to the short/medium term while the CPO production process matures.

What the Street missed in the whole controversy is that whether it is NPO or CPO, the optical content remains unchanged. It is only a question of how the packaging works and the demand for optics is higher than ever, just in different form factors.

But first, Computex is a very large show, and impossible for one person to cover all what it has to offer in any reasonable depth. What I’ll discuss here is by no means comprehensive but I’ll provide a stream-of-consciousness download of the show. I hope you get something useful out of it.

I’ll discuss three major areas:

• Memory

  • The trade

  • HBM4 base die and cooling

  • What would actually break memory

• Optics

  • Why CPO is still early

  • The demos

  • NPO and pluggables

• Power

  • 800V and the conversion chain

  • Delta’s system play

  • When it’s real

  • Moving BBUs out of the rack

By reading this post, you agree to the terms and conditions. Also see the full ethics statement.

If you would like to engage boutique research services for your project, reach out to us.

Contact SemiExponent

Read more

What’s left of my grey cells have been deployed into thinking about interconnects. I have a deep dive piece on how power density is driving up active copper cable use; link below in case you missed it. I believe we’re still in the early phase on this one.

Today’s short post is about a strange contradiction in the argument about what the right laser source for CPO is: a single high power laser, or a wavelength multiplexer, multi-color laser. I believe either one is viable, and we will explore why.

A quick note: I will at Computex 2026 in Taipei next week, and posting might be light while I attend the conference. If you are attending and happen to see me, say hi! 👋🏽

There has been a lot of recent news about Sivers Semiconductor, whose stock has shot up about 2,000% based on their optics business lines for CPO. First thing to note is that their optics business is a minority; their main business is RF, wireless, and phased array antennas for Satcom. I’m surprised more people aren’t talking about them as a potential acquisition target by SpaceX post-IPO – because that’s where their real strengths lie. I’m still not convinced that there’s going to be a Satcom boom post-IPO, so let’s move on.

The other argument I’ve seen is that Sivers’ approach to CPO is inherently flawed, because the use of wavelength division multiplexing (WDM) is worse than using a single powerful external laser.

To explain this further, you should know that there are two schools of thought:

1. Ultra High Power Laser: The idea here is to just create a really powerful laser and keep it external to the rack. Once it reaches the optical engine, you split it up into parallel lanes and modulate each lane with data on a SiPho chip. Coherent showed a 400mW UHP laser, and Lumentum showed an 800mW-1W “Super High Power” (SHP) laser.

2. CW-WDM Laser: The other approach is what folks like Sivers and Ayar Labs are adopting. The light source itself is multi-color from the start with each color having about 50-65mW. Across 8 lanes, that is still 400mW total but you don’t have to split it up at the optical engine. Since you have different light colors, you can modulate each one with data.

But is the WDM approach to CPO really worse than UHP lasers? It is also important to note that Lumentum, often touted for their SHP lasers, also have a CW-WDM laser which is really very good: 8 wavelengths, each up to 100 mW. There is a whole multi-source agreement (MSA) around CW-WDM too.

If CW-WDM is so flawed from the start, why does Lumentum have a product here, and why are they part of the MSA? Note that Coherent is on here too. See the contradiction here?

The real answer is that UHP lasers vs WDM lasers are just two different ways of solving the same problem, and Lumentum is hedging its bets on which architecture will win by having both. Here’s why:

1. The SHP laser approach needs 800mW because it is going to be split up into multiple lanes on the other side. For this, there should be substantial power available for each lane after the split. Let’s take a more “ordinary” UHP laser like the one from Coherent for comparison running at 400mW, single color. Assuming no loss through the channel for simplicity, 400mW gives 50mW per channel across 8 lanes.

2. If you see Sivers’ WDM lasers across 8 lanes, each one has 50mW per channel. No difference at the modulator level compared to the single color UHP laser approach.

You can roughly see how they are equivalent. If you really want to take loss into account, remember that a single UHP laser needs a 1:8 which a WDM approach does not. This adds to the loss budget and requires a higher power laser to start with. WDM instead has to deal with multiplexer/demultiplexer loss. Problems on both sides.

Using a WDM approach means that you can transmit more data on a single fiber, whereas UHP lasers after splitting need different fibers. WDM has bandwidth density that split UHP lasers do not.

What matters is how much laser power is available at the modulator ultimately, and not how much power is available at the external laser source. Running 800mW in a single laser also presents cooling problems, while running 8 lanes at lower power makes thermals easier to manage on a per laser basis.

Higher laser power isn’t a virtue in itself as a lot of commentary on optics makes it out to be. Higher power is good for SNR and BER. But more power also means more heat, which is one thing that lasers explicitly dislike because their lifetimes drop like a rock at higher temperatures. Even if the laser is external to the rack, reliability is one thing that CPO adopters still worry about the most (data from conferences shows that lasers do okay, but no info is available at scale yet).

The real danger to Sivers is not the use of their WDM approach, but instead Lumentum beating them at their own game with higher quality WDM lasers.

Comparison table with reported numbers

Have a great weekend!

This can be done across many axes, with: hybrid bonding for advanced packaging, Unified Bus for memory protocols, near-packaged optics for interconnects and software optimizations overall. We will just focus on the transistor layer here, and address the question: What if the route to leading-edge density did not require a leading-edge transistor at all, but instead came from folding a 7nm-class design back on top of itself until the footprint density rivals a node three years ahead? This is what Huawei calls LogicFolding.

In this post, we will discuss:

• What Tau Scaling and LogicFolding actually are: chasing time as an axis instead of scaling dimension.

• Why hybrid bonding is the hard part: the challenges that decide whether fine-pitch logic stacking actually yields.

• Who builds the bonders: the short, concentrated supply chain and why its geography matters for China.

• How far along China’s own bonder makers are: Where is the current capability, and the gap that still leaves.

• What it means for the US lead: why the same stacking on an EUV-class node widens the gap instead of closing it.

If you need a refresher on hybrid bonding, see earlier post below.

We also have a deep dive on lithography on the Semi Doped podcast. See it on YouTube.

By reading this post, you agree to the terms and conditions. Also see the full ethics statement.

If you are not a paid subscriber, you can purchase just this article using the button below. You can find the whole catalog of articles for purchase at this link.

Buy the ebook

If you would like to engage boutique research services for your project, reach out to us.

Contact SemiExponent

Tau Scaling and LogicFolding

Huawei’s Tau Scaling takes its name from the Greek Letter τ, which represents the delay through a signal path. Historically, making transistors smaller and faster has reduced this delay. But past the 7nm node, making the delay smaller has become increasingly difficult without the use of EUV lithography, which China does not have access to due to export control laws.

Huawei’s solution is practically sound: if you can’t improve the signal delay through the transistor without EUV, why don’t we optimize everything else we can? Their goal is to shrink that delay across four levels at once: the device, the circuit, the chip and the full system.

At the chip level, the way to do this is to take a chip built on a trailing DUV node, like SMIC’s 7nm-class process (incidentally, Huawei has never named the foundry, but everyone assumes it is SMIC). If you can stack logic chips built on this one on top of the other and hook them up, you’ve just doubled transistor density without EUV. Think of it like printing two chips on a napkin, and folding them in half so that they align, and then hooking them up. This is what Huawei calls LogicFolding, and hooking them up requires a packaging technology called hybrid bonding.

By folding chips like a taco, you’ve just reduced the interconnect distance massively, and the unwanted resistance and capacitance that come from longer wiring is dramatically reduced. This makes the silicon run faster with lower delay, and Huawei has effectively made chips faster and denser by stacking them on top of each other.

Huawei reports the Kirin 2026 part, its first LogicFolding product shipping in fall 2026, moving from 155 to 238 MTr/mm² of transistor density (roughly +53.5%), alongside a 41% gain in performance-core power efficiency and a clock bump to 3.1 GHz. The caveat here that many experts are pointing out is that Huawei spent twice the amount of silicon to double the transistor density, which is not the same as using EUV to make denser transistors.

Read more

The rise of power density within a single rack as hyperscalers pack in more GPUs is causing the networking domains to blur. As GPUs per scale-up domain climb, the domain outgrows a single rack and operators split it across racks to keep power density in check. Meta’s Catalina is a prime example where the scale-up domain spreads across racks at lower power density, linked with active copper.

A dense rack like NVIDIA Rubin pulls so much power that many brownfield datacenters (a term used to refer to existing datacenters repurposed for the AI age) cannot host one because the installed racks cannot support the power density. Short of swapping in NVIDIA reference racks as-is, the only way to deploy power-dense hardware is to spread the same GPUs across multiple lower-density racks.

Once power pushes scale-up across racks, passive copper does not have enough reach. But the gap is not wide enough for fully retimed copper with DSPs, which are too power hungry and too high latency, and optics for a few meters is overkill at this networking generation. Active copper fills the gap as a “goldilocks” choice to keep power budgets in check and opens up a new market for AI interconnects.

In this post, we will look at:

• A tour of the interconnect technologies inside an AI datacenter, and where passive copper, active copper, DSP cables, and optics each fit.

• Why rising rack power is forcing scale-up networks to split across multiple cabinets, with Meta’s Catalina as the worked example.

• Which interconnect actually wins the short links between those split racks, and why it turns out to be active copper.

• For paid subscribers, a full competitive teardown of the five vendors fighting for ACC: Semtech, Marvell, MACOM, Ciena/Nubis, and the conspicuous one sitting it out.

By reading this post, you agree to the terms and conditions. Also see the full ethics statement.

If you are not a paid subscriber, you can purchase just this article using the button. You can find the whole catalog of articles for purchase at this link.

Buy the ebook

If you would like to engage boutique research services for your project, click below.

Visit SemiExponent

How AI Racks Move Data Today

All scale-up networking inside NVIDIA racks has been passive copper. The NVLink spine holds roughly 5,000 copper cables. Passive copper, or direct attach copper (DAC), works because its reach inside a single rack covers the generation it sits on. The speeds across three NVIDIA generations:

Hopper, at 100G per lane, gave DAC about 3m of reach which is plenty. Blackwell’s NVLink 5 went to 200G per lane and passive reach fell to about 1.5-2m, but NVIDIA kept the scale-up domain inside the spine by packing all 72 GPUs and the NVSwitch trays into one rack with a copper cable spine, so no link runs far.

Rubin’s NVLink 6 was labeled 400G per lane at GTC 2026, though the industry read is 200G bi-directional. Running a transmit+receive pair at once leans hard on echo cancellation and tightens cable performance demands, but DAC for scale-up is alive and well in Rubin. The added BiDi complexity is handled by the SerDes on the switch silicon, not with external redrivers. Rubin Ultra moves to a mid-plane PCB approach, which is still passive copper.

Scale-out is different: connecting a row of racks needs real reach. It has leaned on active electrical cables (AECs), which drove Credo’s purple cable mania a few years back. Marvell and Astera Labs play on the silicon side; Credo is vertically integrated. As speeds climb, scale-out is migrating to optics, pluggable transceivers first, then CPO for lower energy. For linking datacenters across a campus or beyond, optics is already the mainstay.

Today, a new space for interconnects is emerging as the scale-up domain spans multiple racks; one whose reach is not high enough for pure optics, but also latency and power sensitive for pluggable optics and fully-retimed copper solutions using DSPs. In the 1.6T generation and possibly beyond, this is the space that Active Copper Cable (ACC) is looking to fill.

Why Meta’s Catalina Spans Two Racks

The cleanest scale-up domain is the one that never leaves the rack because passive copper cables without active silicon is cheap, low-latency, and low-power. This is the sole reason why hyperscalers would all keep scale-up inside one rack if they could.

The fundamental problem is that a frontier scale-up rack like NVIDIA’s Blackwell NVL72 uses more power that traditional racks. Once you cross roughly 100 kW in a single rack, you run out of cooling headroom, busbar capacity, or both, and the only move left is to spread out the GPUs across two racks, and run the scale-up fabric between them.

A single GB200 NVL72 draws around 120-130kW in one rack. If most racks in an existing datacenter top out around 60-70 kW per rack, then NVIDIA offers the same 72-GPU domain repackaged as NVL36x2: two cabinets of 36 GPUs at roughly 67kW each. But what if NVIDIA’s customers have their own racks already?

Meta’s Open Rack v3 High Power Rack (HPR) supports 93.5 kW per-rack, and is not suited to directly deploying a single NVL72 GB200 domain within it. At this point, Meta has to either replace all the racks in its existing datacenter with NVIDIA’s reference architecture, or expand the 72-GPU domain across two racks. This latter is what Meta did with its Catalina racks.

Catalina stretches the 72-GPU NVLink domain across two ORv3 HPR cabinets, roughly 36 GPUs and ~67kW each, so both racks sit comfortably inside the 93.5kW power envelope Meta already knows how to cool and power. The Blackwell silicon is NVIDIA’s, but the rack around it is Meta’s implementation so that the cabinet drops into Meta’s existing data halls instead of forcing a new facility build.

The tradeoff lands on the scale-up traffic which has to jump across the split rack domains. In the picture below, you can see a thick bundle of black cables running between the two racks, tying the GPUs on both sides together through the NVLink switches into a single 72-GPU domain. The backend network within each rack is in-rack scale-up. Holding 72 GPUs non-blocking across that split takes 162 cross-rack links (18 ports/NVswitch × 9 NVswitches/rack). The saving grace is reach because the interconnect just has to go to the neighboring rack scale-up, and not run across an entire row like in scale-out networks.

Which Interconnect Wins Multi-Rack Scale-Up

The cable only has to reach the rack next door, which works out to about 3 meters once you allow for slack. That is a short, fixed reach, and at 200G/lane it is enough to rule most of the options in or out.

Passive copper is the first to go. At 200G/lane a plain DAC runs out of reach well short of 3 meters, so it cannot bridge the gap on its own. At the other extreme, a DSP-based AEC reaches far past 3 meters, but it burns around 20 watts per end and carries the latency of a full retimer, which is overkill when all you have to do is cross to the neighboring rack. Pluggable optics runs into the same wall from the other side. It clears the reach and data rate easily, but tens of watts of transceiver power per end is wasteful on a link that only has to span a few meters.

Co-packaged optics is the interesting one. It is the lowest-power and most future-proof option, and NVIDIA is pushing it for multi-rack scale-up in Rubin Ultra. But it is still unproven at scale and ties you to a handful of vendor ecosystems, so it reads as a next-generation answer rather than something to deploy at 1.6T today. CPO is not the only future bet either, as several players are investigating microLEDs and VCSELs for the same short-reach role, though none of these are in large-scale deployment yet.

That leaves the option built for exactly this gap. Active copper cables add a linear redriver, no DSP, which pushes copper out to 3 meters at 200G/lane for just a couple of watts per end. Most of the reach of an AEC, a fraction of the power, and none of the DSP latency.

After the paywall:

• Semtech’s proven design wins, including the marquee platform already running its silicon.

• Why Marvell is every bit as strong in ACC as Semtech, for completely different reasons.

• Why MACOM has the silicon and a standards seat but no ACC wins to show.

• How Ciena bought Nubis for optics and picked up a useful ACC play.

• Why the best-known copper-cable company is sitting ACC out.

Finally, a single table to summarize the competitive ACC landscape.

Read more

Thoughts swimming in my head this week have been entirely around the reality of “Orbital Compute” that Gavin mentioned on the podcast as the next big thing. I just wanted to pen down major themes swirling around this topic, on this week’s TWiC, so that I can revisit this when it becomes The Big Bottleneck™, but hopefully sooner than that.

In other news, I have content published around inference accelerators this week.

• 📚 Inside SambaNova’s Inference Architecture

• 🎙️ Cerebras IPO

Ok, lets get to the space stuff.

There has been a lot going on around the idea of putting datacenters in space. Some newsworthy items in no particular chronological order.

• Startcloud put an Nvidia H100 in space — they have a nice whitepaper worth reading.

• Elon Musk explains on X how datacenters should ideally go to space.

• Google’s moonshot Project Suncatcher explores putting Google TPUs in space.

• Axiom Space is a company working on orbital datacenters.

• Last week, Google and SpaceX are in talks to launch datacenters into orbit.

There’s probably a lot more (leave it in the comments), but 5 bullets will do for now.

With the impeding SpaceX IPO, and the fact that SpaceX has an absolute stronghold in the reusable rocket market while also running datacenters here on Earth, the interest in orbital compute is on the rise. In the interview, Gavin thinks orbital compute is just around the corner — in the 2027/28 timeframe.

The one thing Gavin clears up immediately is — “don’t think about it as a Pentagon-shaped datacenter in orbit.” Instead, he asks you to think about racks in space, not entire data centers floating in sun-synchronous orbit.

Today, the power onboard satellites is around 20 kW, and SpaceX engineers seem to think they can scale that up to 100 kW, which should fit a Blackwell-class rack in orbit. The networking aspect is most interesting. Scale up would remain in copper within the rack for sure, but scale out would use free space optics to connect between orbital racks. Most amazingly, SpaceX satellites today already have the ability to communicate with each other to operate in a mesh using free space optics. This technology isn’t in a lab; it’s actually up in space today.

For the industry itself, running data centers in space means that there are four markets that open up to make this happen.

1) High power free space optics

Connecting racks with free-space lasers isn’t something we do on earth today but constellation operators already have optical satellite links built into them, including SpaceX and Amazon, with Telesat and OneWeb still considering adding laser comms to their satellites. The use of free-space optics would open up an entire segment of the optics industry that isn’t directly involved with AI data centers today such as Tesat (privately held) who are widely known for inter-satellite optical communications, CACI () for their optical terminals for defense work, General Atomics, and others such as Skyloom (bought by IonQ) and Bridgecomm (acquired by Voyager Technologies). If optical comms in space are going to become a thing, only a few people really know how thats done today.

Imagine, if NASA’s Terabyte Infrared Delivery Subsystem (TBIRD) could run at 200 Gbps from Earth to orbit in a system the size of a tissue box, we can connect racks in space alright. I’ve written about this before; see post below.

2) RF SATCOM Companies

If you want to communicate with data centers in space from Earth at scale, the best-known approach is through satellite communication systems operating at Ku or Ka band. Starlink terminals currently use this frequency range and typically require their own satellite terminal to communicate with satellites at sufficient bandwidth.

Companies have tried going direct to cellular, but the bandwidth is usually not high enough. The idea of data centers in space could give second wind to companies working on RF-based SATCOM, especially since their forays into millimeter-wave 5G failed spectacularly.

The key technology here is mmW phased arrays, which use an array of antennas that can be cleverly controlled to form a pencil beam aimed directly at the satellite for maximum data throughput. There’s a GaN angle here too in power amplifiers for satellites. Qorvo, MACOM, Analog Devices are among some companies with this expertise.

Since I have actually worked in this field, I’ve written a bunch of posts about this, if you’re interested.

3) Radhard Specialists

If you’re not familiar with silicon in space, you need to know that silicon has to be space qualified and radiation hardened (Radhard) to work in space. Low Earth orbit deployments are easier from a radiation-hardening perspective because the Earth’s atmosphere shields a lot of radiation. As you move farther from Earth, radiation requirements get much worse.

Aside: I attended a talk by the Voyager mission’s chief engineer at IMS 2025, and he explained how insane it was to design electronics that work in the radiation environment of Jupiter. He was cracked AF and a great storyteller! At 15 billion miles away, the data rate from the Voyager is 160 bytes-per-sec. 😭 A 1-way message takes a day!

The ability to test and qualify parts for space is something a company builds over time, and it typically requires a unique end market. Not every silicon supplier can serve these markets, but defense providers and companies in the satellite space often have this capability, so it is worth watching for. Also, it would be wrong to assume it’s only the GPUs that need space qualification. Every component, from memory up through lasers, PCBs, connectors, and cabling, needs to be qualified for space operation. It’s a different ball game the industry is not entirely ready to play at the scale of orbital datacenters today.

4) Cooling Technology

A common mistake is assuming data centers are naturally suited to space because it’s so cold. Convection is the best way to extract heat from surfaces via air- or liquid cooling, and space naturally has neither.

The only way to remove heat from chips in space is through radiation, which is much less efficient. Cooling is a hard problem for data centers in space, even if liquid cooling is used because heat still has to be radiated away with large radiators to keep the chips cool. The ISS rejects on the order of 70 kW with very large panels, so cooling a single rack in space seems feasible in theory.

The best part is that space, well has “space”. You can make the radiators hundreds of feet in size, so a lot of heat can be exchanged with the cold sink outside kept at a steady 2.7K. NASA studies show radiators can account for more than 40% of total power system mass at high power levels. That is the real reason this is hard.

Some companies to watch for in this space are Sierra Space, Paragon Space Development, Redwire Space, ThermAvant Technologies, and Advanced Cooling Technologies.

If AI is going to space, then I’m going to write deep dives about its various aspects.

Meanwhile, have a good weekend!

• GPUs are packed with HBM, and during inference, the model is sharded across multiple GPUs.

• Groq’s LPUs with their hundreds of MB of SRAM require thousands of chips to run a large-scale model.

• Cerebras can hold up to 44 GB per wafer in SRAM, but requires the problem to be distributed to multiple wafers for large-scale model serving.

SambaNova has a different take on the problem.

What if, instead of a single large Mixture-Of-Experts (MoE) model, you built a composition of many smaller experts, each trained to be competitive with monolithic MoE model experts? Think of 100+ 7B parameter models all doing one thing really well. This is what SambaNova calls Model Bundling, which was the initial motivation for the architectural choices that Sambanova’s hardware is built on.

In this post, we will discuss SambaNova’s 3-tier memory architecture, their Reconfigurable Dataflow Unit (RDU), and how they compare to more recent hardware architectures in the news like Nvidia’s Rubin, Groq LPUs, and Cerebras wafer scale engines.

Specifically, we will discuss:

• The Three-Tier Memory Architecture

• Understanding how RDUs work

• Scaling up to the rack level

• Where SambaNova fits

  • SambaNova vs Rubin

  • SambaNova vs Groq vs Cerebras

• Where does this leave SambaNova

Disclosure: This is not a sponsored post. SambaNova provided their inputs on a draft version of this post, primarily on their use cases and nomenclature which has been incorporated into the post. Opinions are my own.

If you are interested in other inference hardware deep-dives on this newsletter, check these out:

If you are not a paid subscriber, you can purchase just this article using the button below. You can find the whole catalog of articles for purchase at this link.

Buy the ebook

The Three-Tier Memory Architecture

Let’s say you have a bunch of small models tuned to do one thing really well. Each “expert” could be a coding model, a writing model, a sales model, and so on. To quickly switch between these experts, each expert’s weights need to be stored in high-capacity memory (like DDR) that can be quickly swapped into a faster memory tier (like HBM) with latencies in the microsecond range, and the hottest weights loaded into SRAM. This is what SambaNova’s 3-tier memory architecture is built to do; all tiers are addressable directly from the chip, each with their specific function.

• Per model layer scratchpad: On-chip SRAM (several hundred MB per chip)

• Working model weights: CoWoS-attached HBM (several tens of GBs per chip)

• Cold storage of model weights: DRAM (1 to 2 TB) on the board level per socket.

Loading these weights from network storage over InfiniBand is too slow for frequent back-and-forth trips between different expert models serving different tasks. The three-tier hierarchy gives you a lot of memory and keeps all of it close enough to compute to be useful during inference. SambaNova is not the only startup to envision multiple memory tiers for inference. MatX, for example, uses a mix of SRAM and HBM. d-Matrix uses a combination of SRAM and 3D DRAM. All of them have different approaches to solving the memory bandwidth problem.

The table below shows the specifications of different memory tiers in SambaNova’s last two chip generations: SN40L and SN50. A key point to note is that the SN50 actually went back a generation in HBM: from 3 to 2E. Edit: SambaNova pointed out that both SN40L and SN50 are on HBM2E while also being on TSMC 5nm; the table below reflects that. But, there is an actual architectural reason why HBM bandwidth does not always matter for SambaNova. We’ll get to this later.

To really understand how this memory architecture works with either model bundling or true MoE models, we need to understand how RDUs work first at the chip level, and then at rack scale. This really matters for how data flows through the system.

Understanding How RDUs Work

A reconfigurable data unit or RDU is just two primitives that are repeated many times on the chip, and wired together with a programmable switch fabric. The two primitives are:

• Pattern Compute Units (PCUs): These are the arithmetic engines. Each PCU can perform multiply-accumulate operations, perform softmax operations for attention calculation, or layernorm operations. The compiler decides ahead of time what the PCU does, and what flows in and out. SN40L has 1,040 PCUs per socket (across two dies); SN50 has roughly 2,080 PCUs per socket (across two chiplets).

• Pattern Memory Units (PMUs): These are the on-chip SRAM scratchpads. PMUs hold the weights and activations that feed the surrounding PCUs. The 520 MiB of on-chip SRAM on SN40L is spread across 1,040 different PMUs. Same arrangement on SN50 (432 MB across ~2,080 PMUs). Each PMU sits physically next to the PCUs that read from it, so the bandwidth between scratchpad and compute is local and short.

The switch fabric is a programmable interconnect that wires PCUs and PMUs together. The compiler determines what data moves from one PCU to another over the fabric at any given time.

To show how the RDUs sharply differ from a general purpose GPU, we must understand how computations are done on both, and where the RDU shines.

Here is how the process of inference works in a GPU. Each operation (or kernel) in the process of inference is written to HBM and then read back for the next step in the calculation. The figure below shows this in more detail. First, RMSNorm is calculated and the result is sent to HBM. Then, all the info along with weights is read back for QKV calculation and so on. The problem with this approach is that there are a lot of memory accesses along the way which increases latency, and burns more power shuffling bits in and out of memory.

Here is how SambaNova does it: They assign all the PCUs and PMUs on the chip to various parts of the inference process. As shown in the image below, you can literally gerrymander your way through memory and compute cores via the software compiler. The computations are done on the PCUs, and the intermediate weights and results are stored in the PMUs. In a sense, the SambaNova compiler predetermines the “dataflow” across the chip – that is, how to split up the available data units to perform different functions required for inference.

By keeping the entire calculation process on the chip, the data never leaves the chip to external HBM memory. HBM is only present to load up the weights into PMUs, and is not really used in the computation process. This explains why SambaNova can get away without using the latest HBM4 or HBM4e because their architecture simply does not demand extreme memory bandwidth from HBM. All relevant data for inference stays on the chip, and when something more is needed, it is read from HBM. They can however, still benefit from HBM4, as we will see later.

Scaling up to the Rack Level

Now this is just one chip, and you might be thinking that a few 100 MBs of SRAM is not going to cut it. The real picture starts coming together when you look at it at scale. Each “SambaChassis” (my name for it, not SambaNova’s) has 8 RDU chips, and two x86 CPUs – likely Intel Xeon 6s, from the looks of it, based on recent partnership announcements. You can put 2 chassis per rack, for a total of 16 RDU chips.

This entire rack acts as one giant chip, and you can basically break up the process of inference across all of them in any slice-and-dice way you want because they are all on the scale-up fabric. You can route your datapath through any of the RDUs across the entire rack.

Across a rack, you get ~7GB of SRAM and 1TB of HBM. Arguably, this is significantly less than Cerebras WSE’s 44GB per wafer, and you get two WSE per rack, which is 88GB of SRAM per rack. Also, 1TB of HBM is on the low-side on rack scale because just 4 Rubin chips will get you the same 1TB of HBM. A Rubin NVL144 (with 144 GPU chiplets) has 20.7TB of HBM in a single rack. We will make more detailed comparisons after the paywall.

Future SambaNova chips would surely benefit from the use of HBM4 instead of HBM2E, but the short supply and pricing is always a concern for smaller players.

You can also run different kinds of parallelism: tensor, pipeline, or expert, on the entire rack of 16 chips. When you scale up to multiple racks, the ability to slice up the data units across all 256 RDU chips makes it quite versatile as to how you can split up a large model into various pieces of hardware in the system. SambaNova’s video below explains a lot of this quite well:

Where SambaNova Fits

In terms of a deterministic compiler and SRAM capacity, SambaNova resembles Groq more than Cerebras or Rubin, but with one major difference: the ability to shape the performance based on the workload due to availability of both HBM and DDR memory tiers. This allows KV cache storage within the memory hierarchy without the immediate need for external appliances. Unlike Groq which is a one-trick pony, and Cerebras which suffers when handling large models, SambaNova is a swiss army knife due to compiler driven scheduling and high capacity memory tiers.

After the paywall, we discuss:

• How SambaNova compares to Rubin

• How SambaNova compares to low-latency inference hardware like Groq LPU and Cerebras WSE

• The real benefits of SambaNova: 3-tier memory hierarchy, RDU’s versatility, and TCO

Read more

Here’s a recap of content this week:

• 📖 Cerebras IPO and the Four Bottlenecks in Its Custom-Everything Architecture

• 🎙️Power as the next physics wall for AI

• 🎙️Gimlet’s cross-vendor inference cloud

As part of the Semi Doped ecosystem, we also have a daily newsletter where we post our takes on industry news in the semi space . We are now 1,000+ subscribers strong. So if you haven’t signed up for that, get in on it now.

Semi Doped

Semis and AI commentary from Vik Sekar and Austin Lyons

I want to use this TWiC format going forward to explore undeveloped themes that I am thinking about that will eventually convert into deep dives on this newsletter. Let me know what you think.

GaN or SiC for Power Devices?

There is a question among investors about whether gallium nitride (GaN) or silicon carbide (SiC) is better for power devices, particularly as the industry pushes toward megawatt-scale racks and an impending power wall in AI data centers.

At the fundamental physics level, it does not matter. Both are wide band gap devices, which is a technical way of saying that transistors built from these materials can handle a lot of power before they break down. Depending on the voltage level the transistor can handle, both transistors can be used in power converter applications.

Broadly classifying transistors by material they use is reductive. Take GaN for example: it really depends whether we are talking about GaN-on-Si, GaN-on-GaN, or even vertical GaN. All these affect the voltage handling, and switching speed of these devices (how fast you can turn them on or off).

A lot also depends on which stage of the conversion process you are talking about; closer to the GPU or closer to the grid. A lot of things matter here. We’ll explore all this in a future deep dive. But at the surface level, just know that it's not like one is automatically better than the other.

Why is everyone talking about Wolfspeed?

Remember that this was a company that was in the brink of death just a few months ago. You can check their stock price: it hit $0.5 at some point in the recent past. And, now it’s at about $70. So what’s going on here?

Wolfspeed basically overbuilt SiC manufacturing for EVs and when demand took a downturn, primarily due to massive capacity from China, they were left holding the bags. They even filed Chapter 11 bankruptcy. Mind you, the company is still running at a GAAP net loss of $120M according to recent earnings.

The recent interest in the company, I think, is coming from this one SiC transistor that no one else has. Picture shown below.

This is SiC MOSFET that holds off 10kV of voltage, which is something that only SiC can really do (not sure if GaN can; I haven’t looked). The other alternatives are 6.5kV Insulated Gate Bipolar Transistors (IGBTs) which are silicon based. SiC also switches at a higher speed compared to IGBTs which allows the use of smaller magnetic components in power conversion.

So why is this component so important? You see Medium Voltage (MV) transformers are in very short supply, with backlogs going back 3 years. See this Reuters article, from where I took the screenshot below. This is becoming a serious problem for datacenter buildouts.

What power SiC does here is replace these power transformers that use an iron core and windings with SiC based voltage converters that operate by switching the transistors on and off. This makes for a very interesting use case that deserves looking into. Also, read talk about this. The whole interview is great. also wrote a memo about power semis that really kicked up WOLF stock a notch.

If you’re interested, read this whitepaper from Wolfspeed. We’ll break the tech down in a much simpler fashion in a future deep-dive with wider context.

Semtech CopperEdge

This one is another Irrational Analysis pick from the interview link above, and I’ve had a few people ask me about it. Really haven’t gotten deep into it, but at the surface, it seems like Semtech is a company that was typically known for some IoT and LoRa (long range) stuff that few really care about these days. As it turns out they have a bunch of people who really know how to do analog semis and design some equalizers/redrivers for copper interconnects — the real magicians of semis.

Note that cables when used with Semtech chips will not have DSPs — which puts them more in the class of Active Copper Cables (ACCs) rather than Active Electrical Cables (AECs). I have a post on this from a while ago, if you want to understand it better.

This is crazy to me, because at 200G/lane, I’d imagine you’d need DSPs and pay the power-price for this stuff to actually work. But no DSP? That means straight up power savings — I need to understand this analog magic better. In simple terms, their approach goes something like this:

• The degraded analog PAM4 signal comes in.

• The chip selectively boosts the weakened high-frequency components (while keeping the overall waveform linear/transparent).

• It outputs an amplified, equalized analog signal that the receiver can still understand.

• No ADC, no DSP core, no DAC, no re-clocking.

This means, each cable end is <2W or ~1.25 pJ/bit at 1.6T. Most AECs like Credo HiWire are closer to the 20W range per cable end or 12.5 pJ/bit. I am not sure what the power efficiency of Marvell Alaska AECs at 1.6T is, but even if its 2X better than Credo, or 10W per cable end, it’s still 5X worse than what Semtech is saying they can do.

No DSP also means near real time analog latency that is in the 10s of picoseconds (versus nanoseconds with DSPs). This is meaningfully important for the scale up domain where latency matters a lot, in addition to low power.

I’ve got to look into this stuff in a deep dive. Just some off the cuff reaction thoughts when I looked up this thing, and is not an endorsement for the company.

I clearly have lots of work to do here. Let me know how you like the “unfinished thoughts” approach to this edition. Oh, and have a great weekend!

What this is not, in my read, is a technology displacement story. Nvidia and Groq LPUs serve the low-latency segment of inference, and can hold its place in the overall inference market. It’s not always zero-sum as the market likes to think it is. Cerebras on the other hand, has massive customer concentration risk in G42, has compute (not hardware) purchase agreements with OpenAI, and lacks the overall software and hardware ecosystem that Nvidia has for its LPUs from Groq.

The investor question is not “does Cerebras affect Nvidia,” but whether the custom-everything architecture can scale revenue past strategic deals with OpenAI without breaking on a single design choice or supplier. We’ll work through four risk vectors in this piece.

If you are not a paid subscriber, you can purchase just this article using the button below. You can find the whole catalog of articles for purchase at this link.

Buy the ebook

How Wafer-Scale Inference Actually Works

The central idea to Cerebras’ silicon is the Wafer Scale Engine (WSE). While GPUs/TPUs are often diced out of a whole 300mm wafer and then packaged, Cerebras keeps the whole wafer intact – the wafer is the chip and it is 46,225 mm2 in size. The latest iteration WSE-3 has 900k processing cores, 44GB of SRAM per wafer, and a memory bandwidth of 21 PB/s per wafer. For comparison, Groq LPUs “only” have low 100s of TB/s per chip.

To be clear, Cerebras built this enormous wafer-scale chip for training so that the model can remain on the wafer to speed up GPU cluster training by skipping the slow network links between GPUs. But this idea never caught on for training for a couple of reasons:

1. Memory bandwidth was never the constraint for training.

2. CUDA was entrenched in the training phase, and there were no incentives for AI labs to break convention and risk falling behind.

But the arrival of disaggregated inference (prefill+decode, attention+ffn) presented a new opportunity. While the prefill stage was mostly compute bound, the decode stage is entirely memory bound. SRAM is the highest bandwidth memory tier possible, and what better way to get as much of it as possible than to make a chip the size of the wafer. If you want a deeper look at SRAM for inference, see this earlier post.

The end result is that the decode phase of inference is sped up enormously due to the WSE’s memory bandwidth. Since large models require more than the 44GB of SRAM available per wafer, a trillion-parameter model will require multiple WSEs to hold the weights and KV-cache. Since only activations flow between different WSEs, the need for high speed interconnects between wafers is not as much. Various forms of parallelism – model, tensor, pipeline – can be implemented between the different WSEs to implement a full scale model.

While the technology is certainly cool, this level of inference performance is not always necessary. In February this year, OpenAI was reportedly looking to replace 10% of its inference compute with low-latency hardware. The ultra-fast decode only makes sense in a few applications that require real-time performance, or where faster inference literally means money. For most of the inference market, GPU-based inference is the right answer. So like all low-latency inference hardware, this is not a “GPU/xPU killer.”

Yield, Stitching, and the TSMC Lock-In

Wafer scale chips are not a new idea. In the 1980s, Trilogy Systems led by Gene Amdahl famously tried it and failed, concluding that wafer yields need to reach extraordinary levels for this idea to work. The Chip Letter on Substack has a nice series of posts on the history of wafer scale chips (Part 1, Part 2).

Cerebras made it work with a combination of clever engineering that reroutes around to processing units that have defects, to spares on the chip. There are 900k working cores, but closer to a million physical cores on the wafer that are configured at the software level to appear “defect-free.” Cerebras has spent close to a decade with TSMC working out the redundancy schemes, the wafer-level interconnect, and the packaging that makes the WSE physically viable, none of which a competitor can replicate easily. Cerebras has solved wafer-scale chips. You can read all about it here.

The key point to note here is that the foundry lock-in with TSMC for Cerebras is real. The entire process of stitching multiple reticles together to create a wafer-scale chip is not something that easily translates to another fab that may have better capacity for scale.

The OpenAI Deal: Cloud Service, Not Hardware Sale

The agreement with OpenAI announced January 14, 2026 works like this. 750 MW of compute capacity will be deployed over 3 years, with an option for OpenAI to expand to 2GW total through 2030. The S-1 values the base 750 MW commitment at more than $20 billion.

But OpenAI is not buying hardware. Cerebras is building or leasing the data centers, filling them with WSE-3 systems, and operating them as a cloud service. Several people I’ve spoken to have said that running your own token factory with your own hardware is not where an inference hardware provider wants to be. They’re forced to do so only if people are not buying their hardware directly. Remember that Groq was pushed into selling tokens via GroqCloud as well, until Nvidia came in and paid an extraordinary amount of money to acqui-hire them.

So in reality, OpenAI pays Cerebras for compute time, not for equipment. Cerebras keeps the infrastructure on its balance sheet, recognizes recurring service revenue, and is operationally on the hook for running AI data centers at scale. The financial structure has a bit more detail worth looking at:

1. OpenAI advanced Cerebras a $1 billion working capital loan at 6% interest, with the interest waived if the loan is repaid through capacity delivery rather than cash. This is effectively prepayment for compute dressed up as a loan, and it funds Cerebras’ upfront capex for building out systems.

2. Cerebras issued OpenAI a warrant covering ~33M non-voting shares at a near-zero strike price ($0.00001, ok basically free), vesting as OpenAI actually purchases capacity, with full vesting only on the full 2 GW deployment.

As it stands today, Cerebras’ fortunes are tied to OpenAI’s success, and their own ability to deploy and operate at large-scale, which is something they have never done before. Their other source of revenue comes from direct hardware sales to G42 and MBZUAI (in the UAE), and represented a disproportionate share of 2025 revenue (86%). But revenue from running cloud based hardware is expected to be the real growth vector for the future.

The architecture, the deal, and the moat are the publicly debated parts of the Cerebras story. The four risk vectors after the paywall are less discussed and matter more to what puts Cerebras’ ability to scale, in danger.

Read more

Here’s what you might have missed this week:

This article has been very popular because there are good reasons why nature pushes us up against a wall when it comes to power delivery. Give it a read; lots of tech content even before paywall.

Memory is just stupid expensive, but Deepseek V4’s SSD focus has implications for NAND. More on the Semi Doped Podcast.

Finally, if you want to keep up with daily news updates fresh and hot off the press, sign up for free to the Semi Doped Substack. Austin and I both post our takes there; easy and quick to read with your morning coffee. ☕

Semi Doped

Semis and AI commentary from Vik Sekar and Austin Lyons

Now, on to the news.

GlobalFoundries SCALE vs TSMC COUPE

GlobalFoundries’ SCALE (Silicon photonics Co-packaged Advanced Light Engine) is an optical module solution for co-packaged optics deployment in AI data centers, while being compatible with Optical Compute Interconnect Multi-Source Agreement (OCI MSA).

TSMC is the king of CPO today and COUPE goes to mass production this year with NVIDIA, Broadcom, and Ayer Labs as customers. But TSMC integrates the EIC and PIC on SoIC-X with hybrid bonding under one roof, and ties the whole stack into CoWoS. GF makes the PIC, but the EIC has to come from a different fab on a “single-digit advanced node,” which means TSMC or Samsung.

The new piece is the OCI MSA badge: Hyperscalers want optionality. The interesting angle is that OCI MSA changes the structural opportunity. If hyperscalers force multi-source qualification for scale-up CPO (the explicit goal of the MSA), GF’s TAM expands because customers are required to qualify a second supply chain. The bigger winner in this story isn’t GF. It’s the hyperscalers, who have the option to move scale-up optics from a proprietary fight to an open spec.

(via GlobalFoundries)

Apple, Intel Have Reached Preliminary Chip-Making Agreement

Very big news for Intel, and stock is up a lot . While still not clear what product line from Apple will be manufactured at Intel, TSMC’s underinvestment in chip capacity is systematically giving away the keys to the kingdom to its competitors. For Intel, this is a much needed big customer for its Foundry services. This deal also brings back chip manufacturing back into American soil, possibly in a large way, depending on the order volumes Apple places with Intel. Intel has been killing it recently with CPUs, advanced packaging capacity via EMIB (another TSMC kingdom key), and now Apple business. Can’t stop winning.

(via WSJ)

Nvidia Invests $2.1B in IREN for AI Infrastructure

Nvidia announced a strategic partnership with infrastructure provider IREN, investing up to $2.1 billion to accelerate large-scale deployment of AI data center infrastructure. The partnership targets building up to 5 gigawatts of AI infrastructure capacity, combining Nvidia’s computing platforms and GPU accelerators with IREN’s infrastructure development, deployment and operational expertise.

This is now the standard NVIDIA neocloud financing playbook. NVIDIA stitches together a GPU buyer’s capital stack by:

• Selling them GPUs (revenue)

• Becoming an anchor tenant (revenue for the cloud, recycled back to NVIDIA)

• Taking equity exposure to participate in upside

The capital stack of every major neocloud now includes NVIDIA either as direct equity, warrants, anchor customer, or all three. That concentrates systemic risk, but it also means NVIDIA can keep volume flowing even when traditional financing tightens.

(via Nvidia News)

Optics Industry Update

Biggest optics names reported this week: . All either beat or printed strong growth. All four sold off.

Main thing happening here: When everything is up 1,000%+ YoY, anything short of perfection becomes a sell signal. Hurlston during the LITE call: “We are significantly undershipping demand. And we’re having to make choices as to who we support.” Usually that’s bullish but now it tells investors you are capacity-bound, which means upside is now governed by how fast you can add capacity, not how fast demand is growing.

Also keep a close watch on microLEDs and VCSELs for short reach interconnects. More companies are optimizing microLEDs for datacenter applications, and not inheriting tech from display industry. Longer wavelength VCSELs at 980nm and 1060nm have distinct benefits at 200G/lane (needs a deep dive - interesting physics). ams OSRAM making moves into AI interconnects as well.

Optics is still important but a lot of the big names are priced in already. While they should still grow, just don’t expect 1,000%+ returns again.

Evercore ISI: Cisco Silicon One is Underrated

Silicon One is Cisco’s family of networking chips. It started in 2019 inside Cisco’s own routers, and over time Cisco opened it up to sell directly to hyperscalers as merchant silicon. The latest parts are the G200, a 51.2T switch chip, and the P200, a deep-buffer router chip. Microsoft and Meta are the named hyperscaler wins. For most of its life, Silicon One has been a small line item buried inside Cisco’s revenue base, and the buy-side has indexed Cisco as a slow-growth networking vendor rather than an AI silicon play.

This week, Evercore put a number on it. Silicon One could ramp from ~$3B in FY26 to $12-15B in three to four years, with roughly $7-8B in silicon and $5-7B in attached optics. The silicon number would put Cisco at roughly Marvell-scale and credibly third behind Broadcom in merchant networking ASICs.

Have a great weekend!

Power density in a datacenter is an important metric for two reasons:

1. Land usage: There are geographical, environmental, social, and economic limitations to how big a datacenter campus can be. Beyond that limit, multi-datacenter campuses become necessary which complicates infrastructure.

2. Networking complexity: It is better to put as much compute in a single rack as possible so that networks span short physical distances, for which copper will suffice. Spanning different racks, buildings, or campuses increases power cost due to the need for optics at different levels, or active electronics for copper interconnects.

In an earlier article, we covered why datacenters need to shift to 800V DC in quite some depth, including an in-depth discussion of electrical systems in datacenter campuses.

What may not be obvious is that deploying high voltage datacenters is not as simple as retrofitting existing buildouts; true high voltage infrastructure needs greenfield buildouts. This is why early implementations of HVDC architectures by OCP Mt. Diablo and Nvidia’s Kyber rack have DC sidecar racks that handle 800V power supplies instead of natively routing it through the datacenter.

The essence of power delivery to a datacenter relies on keeping voltages as high as possible, for as long as possible, before it is delivered to a GPU. There are good physics reasons for this, which we will cover.

The question of how to generate power, where to convert the voltage, what kinds of devices are needed to do so, what is the most efficient method, and what is the most scalable approach as power in datacenters continuously increases, is where the whole discussion lies.

Several notable analysts are pointing out that power delivery is becoming increasingly important. Recently Wood Mackenzie reported that there is a structural shift in the demand for electrification, and a string of shortages in the supply chain. ON Semi CEO stated that power semi content will be 10x with the move to 800V DC racks, with additional semi content in power conversion outside the rack. Notable analysts like SemiAnalysis and Citrini Research are also pointing out that the “power plumbing” of AI datacenters, while seemingly boring, is showing a real inflection point in demand.

In this post, we will map out the landscape of power semiconductors. We will not cover power generation, industrial hardware, and delivery systems at the grid level here, or even outside the rack; that is a different discussion. We will look at the physics reasons for higher voltages, voltage conversions within the rack, and the engineering trade-offs in doing so. Finally, we will identify which architecture wins in the near and longer terms.

We will cover:

• Power physics that forces the rethink

• Power conversion architectures

• Which 800V architecture wins

If you are not a paid subscriber, you can purchase just this article using the button below. You can find the whole catalog of articles for purchase at this link.

Buy the ebook

Power Physics that Forces the Rethink

It’s instructive to draw parallels between power and interconnects.

The reason optics is becoming increasingly popular is because skin-effect in copper wires increases its resistance. This is an effect that occurs when alternating currents flowing through a wire only utilize the periphery and not the entire cross section of the metal wire. This gets progressively worse as the speed of the signal increases. Optics is immune to this effect because light propagates through glass fibers; electrons and metals are not involved.

The consequences of interconnect physics in copper and optics is that you want to stay in the optical domain as long as possible, and convert to electrons as close to the chip as possible – this is the fundamental idea behind co-packaged optics, or CPO.

Now let’s turn to power.

When electrical signals flow through grid transmission lines, they have losses due to skin effect too. But this can be mitigated by using high voltage lines often in the 100 kV range for power transmission. This ensures that the currents flowing are low in magnitude, thereby resulting in lower losses. The power is converted to medium voltage (10-30 kV) only at the substation in a datacenter, and distributed at 400-480V 3-phase AC within the datacenter halls. AC is converted to DC at the rack level to 48V by the power supply units (PSUs), then down to 12V by intermediate bus converters (IBCs), and finally down to 0.8V-1V by voltage regulator modules (VRMs).

The rack scale voltage of 48V is important. To deliver P=600 kW (rating of the Kyber rack used to house the Rubin Ultra) at V=48V requires I=12,500A of current (P=VI). This level of current has serious implications.

1. Resistance headroom: If only 1% of the voltage budget (1% of 48V = 0.48V) should be lost to power delivery, then the total resistance from the rack PSU to load should be 0.48V/12,500A = 38 micro-ohms (R=V/I). This covers the busbar, joints, connectors combined, leaving no headroom; that resistance is already very low.

2. Copper weight: Busbars are thick metal buses that carry current, and practically supports about 1,000A/sq. in, before heating becomes a problem. For 12,500A, the cross-section of copper needs to be 12 sq. in (say 4” x 3”) that runs a full rack height of 7’. This puts the copper busbar volume at ~1,000 cubic inches per rack. Assuming copper density of 147 g/cu. in, each busbar then weighs 147 kg/rack. The cost and weight of this much copper is unmanageable.

3. Power dissipation: The power dissipated by this much current is I²R, which means that a linear increase in current results in quadratic power losses. With 12,500A of current and 38 micro-ohms of resistance, the power dissipated is 6 kW, which requires significant cooling efforts.

One way to greatly mitigate all these problems is to move to higher DC voltages like 800V.

At this voltage, 600 kW of power needs only 750A of current in the rack. 1% voltage budget leaves 8V/750A ~ 11 milli-ohms, which is much better headroom than micro-ohms. Copper busbars can be 1” x 1” and still support up to 1,000A. Busbar volume drops to 84 cu in, or 12.3 kg/rack which is much more manageable. Power dissipation assuming 1 milli-ohm resistance (25x worse than 38 micro-ohm case) is 0.56kW, which is much 1/10th of the 48V case, and better to handle cooling-wise.

All these numbers show one important implication: use high voltages as long as possible, before converting to lower voltages near the chip.

Just like optics has CPO, the equivalent in power delivery is called “Point-of-Load” (PoL), where only specific implementation is called Vertical Power Delivery (VPD). The main idea is to convert the high voltage to low voltage, right under the GPU where it is actually used.

How exactly voltage is converted from the 48V or 800V down to 0.8V required by GPUs is where all the engineering lies, and is what defines the entire competitive space of power semiconductors in AI datacenters at the rack level. Now that we have established why higher voltages are necessary, the next question is how to step down efficiently from 800V or 48V to the GPU core voltage.

Power Conversion Architectures

The power conversion architectures at 48V and 800V are shown below. These do not cover every possibility, but are sufficient to understand the trade-offs in each case. For each architecture, the number of conversion stages and voltage choices at each stage determine the overall efficiency, cost, and dynamic behavior of the power conversion. The 12V or 5-6V stages in blue are often called Intermediate Bus (IB) voltages. We will discuss these trade-offs at a high level.

Number of conversion stages

Every stage of power conversion has some power loss, and hence fewer conversion stages means lower loss, to the first order. Even a 97% conversion at each stage results in a 91% overall efficiency for a 3-stage power convertor. The per-stage conversion efficiency depends on the actual voltage values at the input and output, and efficiency of each stage need not be the same. This is overall a simple concept: fewer stages, lower loss, higher efficiency. So why not just convert all voltages in a single stage?

Voltage Levels and Conversion Ratio

Every stage of voltage conversion typically uses a completely different kind of converter, which implies that there are different companies that capture varying amounts of value depending on the power conversion architecture being deployed. The output to input voltage ratio and the actual voltage values drive circuit topology choice.

Three constraints determine the voltage levels and the topology choice at each stage: isolation requirements, conversion ratio efficiency, and the need for tight voltage regulation at the final stage.

1) Isolation and the SELV threshold

SELV stands for Safety Extra-Low Voltage. The IEC 61140 standard defines a SELV circuit as one where the voltage between any two conductors, or between any conductor and earth, does not exceed 60V DC (often used in practice even if spec is higher) under normal and single-fault conditions. Below this threshold, current through a person making direct contact is unlikely to be fatal. Above it, the circuit must be galvanically isolated from any electrical conductor a human can touch, which usually means a transformer separating the primary and secondary sides of the converter (these are magnetically coupled, and have no wires from input to output, which isolates them galvanically).

This is why 48V and 54V rack voltages exist where they do because they sit just below the SELV ceiling. Anything downstream of the 48V busbar can be non-isolated, which removes the transformer from the converter and saves space, cost, and a percentage point or two of efficiency.

For architecture, the rule is simple. Any conversion stage with input above 60V DC needs an isolated topology. AC line to 48V is isolated. 800V to 48V is isolated. Once at 48V or below, no further isolation is needed, and the rest of the chain runs on non-isolated converters.

Beyond isolation, two more constraints determine which converter sits at each stage and which vendors compete for that socket. These constraints, and how they map to the 800V architectures, are where the investment thesis lives.

Read more