Without much ado, here’s my “big update.” After 15 years of working in professional semiconductor companies, I’ve decided to strike out on my own. My focus will be around researching and writing about semiconductors on Substack, building out the Semi Doped podcast, and expanding into consulting. That’s the grand plan at least.

Why the big change?

Having worked at small, medium, and large companies, I’ve realized that there is so much more to the world of semiconductors than I’m privy to, compared to my daily duties on the job. I started this newsletter in 2024 to explore my curiosity and find out what else is out there; to give myself an opportunity to learn and explore technology on my own, without concern for whether it results in career advancement.

Even with many years of work experience, I realized how little I knew about how the industry works. Every piece of technology has vast technical depth and nuances that are humbling, and many times I only scrape the surface of what lies beneath. Staring into the abyss of the unknown excites me more than it intimidates me. Today, my content channels are merely a way for me to explore that technological abyss and see what lurks beneath the surface.

I often relate to my PhD years as my best years but I never stopped to question why. Most people speak of it in tortured terms, but I’ve always looked back fondly at it. I did well too, by any conventional metric. I graduated with well over a dozen journal and conference papers to my name, including an award at an international IEEE conference. This, I naively mistook for intelligence - as if, somehow writing more papers makes me smarter. My fresh academic brain knew no better.

For a while, I considered getting into academia. I even applied to universities. My wife maintains to this day that I am an excellent teacher; I did the impossible by teaching her mathematics for her GRE. But I had seen the seedy underbelly of university life in my PhD years and it wasn’t something I was eager to jump into. Industry was the obvious alternative for someone with an engineering degree like me. I entered the workforce in 2011 and it was not the best of times (perhaps it was the worst - for some). I had applied to 77 jobs, got 3 callbacks, and 2 offers. I picked the job in California since the pay was more. The “spray and pray” approach worked for me luckily.

Ever since then, I have genuinely had a lot of fun in the industry for well over a decade, and have met a lot of smart people along the way. At some point, it became clear to me that it was always going to be the “same thing, different place.” Nobody tells you how hard it is to change course once you have experience in a certain thing. You’re stuck doing the same thing forever, even if you don’t like it anymore, because you’re the expert now. Some people do however change roles with chameleon-like versatility. I never figured out how.

I was lucky in a sense. My best professional years were dead centered around the wireless explosion. My skills were perfectly aligned for the 2010s even though everyone told me a decade before that doing electromagnetics for a living was a poor professional choice. When I joined my last employer, 5G was a matter of national concern. In the last 3 years however, the semiconductor industry has been … different.

The birth of transformers has led a semiconductor renaissance that I never imagined I would see in my lifetime. The last time semiconductors were this sexy were in the 1970s, when Robert Dennard taught us how to scale transistors. In the modern era, my skills started to seem dated, from a time gone by. I have started to believe that wireless technology is largely a solved problem today. The world of artificial intelligence however, appears ripe with opportunity. The highest agency action I could undertake was to just learn everything about it I could. Not for a job change, not to convince someone, but for the sole reason that it was cool.

Substack was the perfect motivator. I had to learn something new to write every week - a cadence that I have largely kept up since the birth of my Substack in Jan 2024. As I kept churning out articles, it struck me. The reason I enjoyed my PhD was because I expressed complicated technical knowledge in simple words. My language in papers was not particularly stuffy, and my guess is that reviewers found my papers easy to read. Nearly every paper I wrote was accepted with minimal objections. Perhaps complex technical ideas, curiosity, and language were my Ikigai somehow. An odd combination, but it only makes sense in hindsight.

Today, I work more than I ever have. But it comes from a place of happiness - of doing what I truly enjoy. Through my writing, I have made so many connections with people that were never possible before. The commonly touted mantra of “writing makes your life better” is actually true. I recommend everyone to try some form of it.

With the support of all my paid subscribers, the newsletter income has grown to a point where I can support my lifestyle by writing on Substack – to where I can say I have “enough” to continue doing this full-time and hopefully have a lot of fun along the way. Thank you to everyone who supports this publication.

It’s all about figuring out one thing at a time. I have location and time freedom to a large extent now – which I fully appreciate. Being my own boss is liberating, but also scary. You stop working, you stop getting paid. I am also hoping that the Substack will continue to grow.

The Semi Doped podcast with Austin Lyons has been a fun endeavor, and we plan to grow out this channel with sponsorships and such. The newly-minted Semi Doped substack is an extension of the podcast – quick hits on the semi industry, without the depth. We want to grow it to be the Morning Brew, or 1440, of semis. Free to read, but sponsor driven. Reach out to us if you’re interested in sponsoring the Semi Doped newsletter or podcast.

Finally, I’m slowly building out SemiExponent – the consulting arm of my semiconductor work as well. If you would like to work with me, feel free to reach out to me at vik (at) semiexponent (dot) com.

Regular programming for ViksNewsletter will resume next week. Paid subscriptions have been paused, but you can still sign up for the free tier. Everything will resume as normal from April 30th.

Till then, I’m off to the beach!

I am pausing subscriptions for just under a week (April 25th - April 30th) since I am leaving for a week’s vacation today. Your existing subscription will extend accordingly, and you will still have access to all paid articles. New paid subscribers cannot sign up in that time. There will be no deep-dive and TWiC next week, but there may be a Semi Doped podcast from the beach. 🌊 Not sure yet.

Substack deep-dive this past week was about a new optical pluggable format called XPO spearheaded by Arista Networks, and how that matters for CPO.

CPUs for agentic AI are becoming hotter, and demand far outstrips supply at the moment. In light of this, I have removed paywalls from my most widely read, and highest revenue generating post ever on this Substack, so more people may have access and get a sample of what kind of content lies behind paywalled deep dives.

I wrote about this just about two months ago, when other sharp analysts like were calling out CPU shortages. The deep dive outlines why agentic AI demands more CPUs. Also, the post has been updated to add comparisons to ARM AGI CPU as well.

On the Semi Doped podcast, we spoke about Credo acquiring Dust Photonics, more XPO discussions, and a new AI CPU company - NuvaCore.

We’re also starting up a Substack presence for Semi Doped where Austin and I will write our commentary on industry goings on as a complement to the podcast. Our intentions are to keep this entirely free, less crazy technical, and accessible to a wide range of people interested in keeping a pulse on the semiconductor industry. It’s meant to be easy reading. Do sign up and tell your colleagues!

Semi Doped

Semis and AI commentary from Vik Sekar and Austin Lyons

With all that housekeeping out of the way, let’s get to the news.

Google announces TPU v8 chips and new networking architectures

Google Cloud Next is the biggest piece of news this week where the company announced their newest line of TPU chips: v8t for training and v8i for inference. This is the first TPU generation where there are two SKUs with different chip specifications. It is now clear that training and inference have different hardware needs: in both chips and networking.

Few interesting things to note:

• HBM capacity on the inference chip is higher than training. Memory bandwidth is critical for inference performance as more people and agents use AI. Training is not really general public facing, and you can build out a datacenter with more GPUs even if lesser HBM exists. It’s still not clear to me why not max out everything.

• Look at the on chip SRAM on TPU 8i! It’s 3x higher than the training chip. Having SRAM allows you to automatically provide fast inference without having to use special chips like Groq LPU. The TPU 8i with its SRAM bandwidth can actually decode really fast. I’ve always maintained that having SRAM is the key here, not strange architectures like Groq’s VLIW architecture. In any case, Google TPU’s systolic arrays do provide deterministic behavior like Groq VLIW.

• Arm Axion CPUs as the head node. This is a chip developed internally by Google based on Arm architecture. More evidence that ISA is not really that important anymore.

Finally, the networking innovations.

• At the scale-out level, Google’s new Virgo architecture replaces their Jupiter network by using Optical Circuit Switches (OCS) with high switch radix. This allows a lot of chips to be connect with just two networking layers, greatly reducing latency.

• At the scale-up level, the training chips still use 3D torus. The interference chip networking now goes to a Boardfly approach.

  • 4 TPU 8i chips to a board — connected by copper

  • 8 boards to a rack (also called group) — connected by AEC

  • 36 groups to the pod — connected by OCS

  • Total 1,152 TPUs per pod

The underlying substrate for all of Google’s TPU networking is OCS. This is a structural change that makes future Google datacenters heavily rely on both optics and copper. I explain the entire networking aspect in-depth on Semi Doped. Look out for the next episode on YouTube.

Intel goes from surviving to thriving

Just a year ago Intel was struggling for existence. TSMC was about to get control of Intel Fabs and help resurrect the business - a brainchild idea of not-quite-loved ex-chairman of the board Frank Yeary. Even Qualcomm allegedly approached Intel for a takeover.

In a riches-to-rags-to-riches story - a comeback story that everybody loves - Intel is now in a position of dominance across three different assets: (1) x86 CPU franchise, (2) advanced packaging, and (3) their massive chip manufacturing network.

The rise of CPUs for agentic AI is a gift that fell in Intel’s lap because it precisely aligns with their core competence. It is up to execution at this point. TSMC’s CoWoS is so bottlenecked that Intel’s EMIB packaging is getting increasingly higher order backlogs. Seemingly, their 18A yields are improving as well. The AI story is playing out because, according to Intel’s latest earnings call, their AI-driven business represents 60% of revenue and grew 40% YoY.

It’s not just CPUs though. Lip-Bu stated on the earnings call that they are “quietly building up the GPU with a new hire” — which refers to Eric Demers being hired as Chief GPU architect earlier this year, who also led Qualcomm’s GPU efforts before he joined Intel.

Lip-Bu also has been smitten by the Elon bug as part of the Terafab project, stating on the earnings call:

We are excited to explore innovative ways to refactor silicon process technology, looking for unconventional ways to improve manufacturing efficiency that will eventually lead to a dynamic improvement in the economics of semiconductor manufacturing.

Sure, you do you — as long as we can get 18A yield up here on earth before we start building fabs in space.

Marvell announces acquisition of Polariton for Plasmonics

You can’t take your eye off the optics ball with all these CPU/TPU distractions. Marvell continues its spree of acquiring photonics companies as it continues its march toward next-gen 3.2T networking, by announcing the acquisition of Polariton Technologies — a Swiss startup.

It’s too late in the week for heavy physics, and we’ll do that in a future deep dive. Let’s look at this cool picture first. This is what Polariton does.

You can send an optical signal through an electro-optic dielectric material sandwiched between two metal electrodes. When you provide electrical 0s and 1s on one of the metal electrodes, the electrons on the metal interact with the light flowing through the dielectric waveguide causing Surface Plasma Polaritons (SPPs). These things affect the phase of the light creating highly effective, and very tiny, phase shift modulators.

The small size of this thing has two advantages:

1. Low parasitic capacitance, which means you can modulate at high speeds.

2. Much lower power dissipation

It’s so interesting when wave-particle interaction physics is hitting mainstream applications. I guess we’re really pushing technology now.

Have a great weekend!

The networking industry has been making rapid strides towards Co-Packaged Optics (CPO) for scale-out networking with external lasers, replaceable 3D optical engines, and per-lane data rates of 200G or more. Traditional arguments against CPO such as laser failures and serviceability are no longer show-stoppers, as explained later in this post. Still, CPO is an architectural reset that upends the traditional pluggable optics market and consolidates various networking components into the hands of a few suppliers like Nvidia, Broadcom, and Marvell.

This paradigm shift puts immense pressure on pluggable transceiver manufacturers like Eoptolink, and system integrators like Arista Networks. As optical functions shift to advanced silicon packaging using TSMC’s CoWoS and COUPE, the value chain moves away from traditional PCB assembly and rack integration. In a defensive play to protect their margins and market share, these players are responding with a new format of their own.

At OFC 2026, Arista Networks introduced eXtra-dense Pluggable Optics (XPO) for AI data centers. XPO keeps the existing pluggable approach, but scales it for higher bandwidth and better thermal management. The XPO multi-source agreement (MSA) now has >100 participating companies, including Marvell (playing both CPO and pluggable sides), Lumentum, Coherent, and Eoptolink - a sign that the industry is serious about this.

In this article, we’ll cover three things:

1. A primer on the XPO format and how it solves the faceplate bottleneck

2. The three-way trade-off between speed, power, and modularity that decides CPO versus XPO at each generation

3. Why CPO and XPO formats will coexist in scale-out at 1.6T but diverge as lane speeds climb to 3.2T and beyond.

We’ll conclude with a view on where each format lands across scale-up, scale-out, and scale-across, and why multiple strategies will likely coexist for years.

This post will have a lot of optical terminology. If you are unfamiliar with it, please see an earlier post below that explains most of what you need to know.

If you are not a paid subscriber, you can purchase just this article using the button below. You can find the whole catalog of articles for purchase at this link.

Buy the ebook

How XPO Solves the Faceplate Bottleneck

As switch ASICs scale toward 204.8T, the physical real estate of the switch faceplate becomes a severe bottleneck. The traditional pluggable model simply runs out of physical space. Consider the math for outfitting a 204.8T switch with standard 1.6T OSFP modules: a standard 1OU (Open Rack Unit) slot accommodates about 32 OSFP plugs, delivering 51.2T of bandwidth. Fully outfitting a 204.8T switch therefore requires 4OU of rack space purely for front-panel connections.

XPO alters this physical footprint entirely. An XPO module delivers 12.8T of bandwidth, or eight times the capacity of a 1.6T OSFP. While the XPO form factor is roughly 2.7x wider (allowing only 16 plugs per 1OU), those 16 plugs can deliver the full 204.8T in that single rack unit.

The core advantages of XPO:

• 4x Density Gain: It reclaims 3OU of highly valuable rack space per switch, reallocating it back to CPUs and GPUs at the datacenter level.

• Reduced Component Failure Domain: Just 16 XPO plugs replace the DSP, transimpedance amplifier (TIA - that amplifies the detected signal), driver, and photodetector components that would otherwise be spread across 128 OSFP plugs.

• Supply Chain Continuity: It retains the external, modular approach hyperscalers are accustomed to. MSAs ensure a diverse supply chain, and the format supports any optical interconnect reach (DR, LR, ZR, ZR+) and technology (IMDD (intensity modulation direct-detection), coherent, coherent-lite, micro-LEDs, and RF).

DSP Power and Cooling in XPO

The primary downside to XPO’s pluggable format is dealing with signal degradation over copper interconnects between the faceplace and switch ASIC using power-hungry DSP chips. XPO offers three implementation paths to manage this, in decreasing levels of power consumption:

1. Full Retimed: Utilizes a DSP on both the transmit and receive sides (highest power).

2. Linear Receive Optics (LRO / Half-Retimed): Utilizes a DSP only on the transmit side, saving power while maintaining signal integrity.

3. Linear Pluggable Optics (LPO): Eliminates the DSP entirely, relying only on drivers and equalizers, leaving the heavy lifting to the switch ASIC (lowest power).

Regardless of the DSP architecture, XPO’s immense bandwidth density makes traditional air cooling impossible. Liquid cooling is essential. Each XPO plug features an integrated cold plate designed to dissipate roughly 400W. While some legacy solutions attempt to bolt cold plates onto the exterior of OSFP plugs, XPO’s natively integrated liquid cooling is vastly more efficient. This also implies that the racks in the datacenter using XPO need to have liquid-cooling available, which might be difficult in legacy infrastructure.

In comparison, CPO boasts the lowest power consumption per port across the board. Because optical-to-electrical conversion happens directly adjacent to the switch silicon, CPO bypasses the need for heavy external DSPs. How XPO ultimately stacks up against CPO economically depends heavily on which of the three XPO implementations - Full, LRO, or LPO - a data center adopts.

Approximate numbers; varies based on vendor and implementation.

That sets up the real question: which format actually wins, and where? The answer depends on how hyperscalers weigh three trade-offs that diverge as we push to higher lane speeds: speed, power, and modularity.

After the paywall:

• A deep dive into the trilemma of speed, power, and modularity that dictates whether CPO or XPO wins in future networks.

• A detailed breakdown of CPO’s superior energy efficiency, how serviceability concerns are solved, and its market impact on the pluggable optics ecosystem.

• An analysis of the XPO-LPO and XPO-LRO “Goldilocks” approaches for 1.6T networking, and their limitations on lane speed and supply chain concentration.

• A look at the divergence point: why XPO becomes power-prohibitive at 3.2T and beyond, making CPO the more attractive future-proof architecture.

• The final verdict on where CPO makes the most sense (scale-out) and why XPO is the natural fit for scale-across networks.

Read more

Here’s what you might have missed this week:

• Tokenmaxxing and the Token Value Chain

• Is Intel Finally Back with a $300B market cap? OpenClaw can Dream?

Now, on to the news.

Credo agrees to acquire Dust Photonics

This is perhaps the biggest piece of news this week with many substacks covering it. We won’t get into the depths of the deal because a simple search will uncover a wealth of takes which you can read. I’ll only touch on a few important points here.

Dust Photonics provides photonic ICs which integrates nicely into Credo’s SerDes experience making them a full-stack interconnect provider — from active electrical cables, pluggable optics, microLED cables, and now integrated photonics. This latest acquisition makes Credo an interconnect powerhouse in the AI era.

I’ve also maintained that Credo’s “zero-flap” metrology for link health is one of their biggest strengths, and is a software layer that really makes the interconnect layer work at scale. You’ll also notice that Marvell has a lot of parallels in the interconnect space here; SerDes expertise, optical DSP, photonics, and telemetry systems. Credo is a smaller player, and gives the big dog (MRVL) a run for its money.

I really like Gavin Baker’s (who runs Atreides Management, an investor in Dust Photonics) explanation of the acquisition in his tweet which you should read in its entirety.

The most important take-aways from what Gavin said is that copper and optics is not a zero-sum game. Copper will remain relevant well into the 2030s, and optics will play an increasing role in datacenters. He identifies scale-up as the area for most networking growth (note that it need not be optical), followed by scale-across (this was surprising to me), and finally scale-out (which might involve CPO at some point). All this while laser shortage is getting worse!

He calls out that Credo’s Hyperlume acquisition was a smart bet, and that there is a lot of positive feedback about MicroLEDs as viable interconnect tech. We’ve started covering microLEDs on this newsletter (free post below). Also, the upcoming Semi Doped episode has a longer discussion on the Credo acquisition. Stay tuned!

(via Credo)

Thoughts on the CPU Shortage

We’ve covered the need for CPUs in agentic AI in quite some depth, and the article below has been the most popular, highest earning article YTD. We called out early that CPU:GPU ratios are going to go to one, or higher.

Since the two months that the article was published, the CPU shortage has gotten worse — to the point that cloud service providers that have nothing to do with AI inference are falling short of CPUs. The x86 vs ARM argument does not really even matter anymore. CSPs and hyperscalers will take anything they can lay their hands on, and port over software if they have to.

The situation is dire enough that the winner in CPUs is anyone who will execute on the manufacturing side and provide sufficient CPU supply. Things like core count vs IPC, or reasoning vs action workloads, or what CPU is best suited for an end application is all moot now. Sadly, most of the supply chain squeeze for CPUs also depends on what TSMC can deliver, and their ~$60B 2026 capex spend has been argued to still be too conservative for the number of chips we need. The ace in the hole would be if Intel can deliver server CPUs at scale purely on 18A.

Nuvacore: Engineered for Altitude

Speaking of CPUs, Nuvacore is a new CPU company funded by Sequoia capital, with the promise below:

A general-purpose CPU core designed to excel everywhere—from core infrastructure to advanced AI systems, including the continuous demands of agentic computing.

This is not iteration, it is a complete rewrite of the rules of silicon.

The brains behind this operation are Gerard Williams, John Bruno, and Ram Srinivasan, whom you may know from Nuvia - a company Qualcomm bought in 2021 and used their IP for their Oryon series of desktop ARM processors. The ARM licensing agreement used in this product was a matter of big contention, which was hotly contested in courts. All three founders left Qualcomm earlier this year, and Nuvacore was born.

Here is what is exciting about this news: Nuvacore promises to rethink how CPUs are built for everything including the agentic AI era. While an earlier post on this newsletter discussed the “most desirable” features of an agentic CPU across 9 different metrics, it is interesting what these CPU legends think is important for AI. More cores? More performance per core? Special architectural decisions? We have to wait and see.

(via nuvacore.ai)

NAND Shortage is Real; Phison fires warning shot

Digitimes reports that Phison CEO, Pua Khein-Seng, warned that NAND flash shortage is going to significantly worsen towards the end of 2026. It has led the company to break its 26 year old tradition of “no-debt” by fundraising a total of $1.4B for 2026. Supply tightness is expected to extend well into 2028, possibly beyond.

Note that Kioxia recently ended the production of older SLC and MLC NAND to prioritize higher margin TLC and QLC NAND SSDs for AI applications. I’ve had multiple discussions with professional investors this week about NAND, and I’ll be digging deeper into the demand for NAND in a future newsletter article. If you need a refresher, check out an older post.

(via digitimes)

Largan won’t make camera lenses for Apple, becomes CPO pilled

I like it when strange things happen — things outside the norm. In what sounds like a little guy finally standing up to the big bully, Largan Precision has said that it won’t increase order volume for the Apple iPhone 18 Pro, instead choosing to focus on optics for CPO.

Largan is a major provider of optics to Apple, but the recent move against Apple indicates an aggressive business pivot from a high volume, low margin smartphone business to high margin, potentially hyper-growth AI optics sector.

It’s a high risk gamble especially since Apple is buying up all the DRAM available at elevated prices to lock others out of the Android market. This shows that Apple is pretty confident that the memory prices are not going to affect their smartphone sales. Yet, Largan is willing to walk away, leaving the pieces to be picked up by competitors such as Sunny Optical (2382.HK) and Genius Electronic Optical (GSEO/TPE:3406). Incidentally, GSEO got a nice stock price bump right after the news of the rebuke broke. Not entirely sure if these companies can fill Largan shoes for iPhone orders.

(via wccftech)

Meta’s now-defunct “Claudeonomics” leaderboard ranked employees by token usage, while celebrating power users with titles like “Token Legend.” In a 30 day window, company-wide consumption topped 60 trillion tokens, with the top contender alone burning 281 billion. CTO Andrew Bosworth bragged publicly that his best engineer was spending the equivalent of his salary in tokens and getting up to 10x more productivity out of it. How productivity was measured remains unknown. On the All-In podcast at GTC, Jensen said that if his $500K engineer was not consuming at least $250K worth of tokens, he would be “deeply alarmed.” Silicon Valley engineers tell me their companies are pushing the same tokenmaxxing message internally. But if the number of tokens used becomes the target, it stops being a useful metric (Goodhart’s law).

Here’s what you have to pay attention to: the tokenmaxxing message applies to those who are able to establish a linear (preferably exponential) relationship between outcome and token usage. Tokenmaxxing can be rational, but not always.

Foundation model providers like Anthropic and OpenAI have a clean, understandable revenue metric (pay-per-token). Meta has massive internal workloads like serving ads that require internal token consumption. Garry Tan wants software startups built faster at YC. Nvidia and AMD simply want to sell more hardware.

In this post, we will examine the promise of the Token Value Chain, how reality is more nuanced, who stands to benefit, and who stands to lose. More importantly, we will discuss what a “smart token” strategy should look like for companies looking to go AI-native, and when tokenmaxxing is a viable strategy.

If you are not a paid subscriber, you can purchase just this article using the button below. You can find the whole catalog of articles for purchase at this link.

Buy the ebook

The Token Value Chain

The token value chain is simple: massive amounts of wealth injected into the top of the chip manufacturing supply chain will flow through token providers, and ultimately manifest as profit for enterprises adopting AI through dramatic increases in productivity (products shipped, revenue earned, etc.).

The Token Value Chain has three levels:

1. Token Supplier: Nvidia and AMD sit at the top of this tier selling GPUs, but the real supplier base is much wider: CPUs, memory/storage, interconnects (copper/optics), PCBs, and power/cooling. Neoclouds like CoreWeave and Nebius belong here too, renting out GPU capacity by the hour. Every single token burned is a pick-and-shovel sale for this tier. Revenue scales with volume, supply chain and power bottlenecks not included.

2. Token Provider: Anthropic and OpenAI do most of the work, with Google, xAI and a handful of others in the mix. These are the companies that take raw compute from the Supplier tier and turn it into tokens sold on a pay-per-use basis. Every token they produce is money, which is why Anthropic’s revenue run rate has gone from $1 billion to $30 billion in about 15 months. Revenue here also scales with volume. The more their customers tokenmaxx, the better the numbers look.

3. Token Consumer: Enterprises, startups, and increasingly every Fortune 500 company that has been told to go AI-native. It is also the only tier in the value chain whose revenue is not tied to token volume – at least not directly, and not always.

The promise of AI is that massive amounts of investments into the Supplier tier translates into more tokens generated by Providers, which ultimately leads to massive profitability at the Consumer level, which will then recursively feed the massive infrastructure spend at the Supplier tier. If you look closer, the Provider layer revenue is tied to token volume, which translates to volume in Giga-Watts deployed at the Supplier tier. But the Consumer pays the ultimate token price.

Gavin Baker puts this eloquently in an interview at Nvidia GTC.

If you are not a token producer, your fate as a business is going to be entirely determined by how efficiently and effectively you consume tokens as an organization.

In other words, while the cost per token used is pre-determined, the revenue generated per token used is anybody’s guess. While both the Supplier and Provider win on volume, the Consumer has to win on outcome. If you view tokens as the raw materials required to produce a product, then there are two approaches to run the business:

1. Use as little raw material as possible to produce products, and increase margins. A token-constrained consumer would use an intentional token deployment strategy and ride the efficient frontier of intelligence per token used. A number of “smart token strategies” can be deployed to get the “most bang for token buck.”

2. Use as much raw material as possible to generate as many products as possible and corner the market. A token-rich consumer would deploy an infinite number of tokens of a leading intelligence model without regard to token cost, to produce remarkable results; just like infinite monkeys with a typewriter can produce Shakespeare. In theory, infinite intelligence (and money) would automatically lock out competitors, leaving token-constrained entities unable to compete. This is tokenmaxxing as a strategy.

We’ll expand on these two ideas after the paywall.

Read more

We’ve covered a lot this week, and in case you haven’t had a chance to catch up.

• MicroLEDs vs Lasers: The Linewidth Tradeoff — why linewidth matters and how LEDs and lasers differ (no paywall this week, enjoy!).

• Busy week on the Semi Doped podcast (which crossed 10K downloads!):

  • NVIDIA’s Marvell Strategy, Is Memory Different This Time?, Intel’s Ireland Fab

  • $300M for 70K Viewers | Intel x Elon, OpenAI x TBPN, Citrini’s Strait of Hormuz Stunt

  • Austin interviews Reiner Pope from MatX

Alright, let’s dive into the news.

Intel Joins Musk’s TeraFab Project

Intel announced it will join Elon Musk's TeraFab project alongside SpaceX, xAI, and Tesla. The Terafab project aims to produce 1TW of compute capacity annually and Intel would bring its expertise in advanced nodes, packaging, and silicon manufacturing at scale.

If Musk actually delivers on even 10% of TeraFab's ambition, Intel secures a whale customer, a strategic win, and a major turnaround. For the Terafab project, Intel being involved actually brings in a practical level of feasibility than Tesla/SpaceX/xAI trying to build a leading-edge fab from scratch.

This is a win-win for both parties involved but will have material impact only when we see significant wafer starts per month from Intel towards the Terafab project.

Heterogenous systems for Agentic AI: Intel + Sambanova

On other Intel news, from the press release,

SambaNova today announced the next phase of its collaboration with Intel: a heterogeneous hardware solution that combines GPUs for prefill, Intel® Xeon® 6 processors as both host and “action” CPUs, and SambaNova RDUs for decode to deliver premium inference for the most demanding Agentic AI applications.

Just like we saw Nvidia pair Rubin GPUs with Groq LPUs and Vera CPUs, Intel and Sambanova are collaborating to provide specialized decode hardware and CPUs. Their comparison table below is interesting and points out something important: the number of decode chips required.

One of the major limitations of LPUs is its limited SRAM and the need to deploy thousands of chips for large models. The complex compiler architecture of Groq chips means its not easy to add HBM/DRAM to expand capacity at the expense of speed. This inelasticity is why I have believed that Groq LPU was never a good fit long term, regardless of how much Nvidia paid for them.

Sambanova’s SN50 RDU architecture uses SRAM, HBM and DRAM, but their secret sauce is to “predetermine” where each piece of the information is going to come from in advance. The RDU stands for Reconfigurable Data Unit, and it establishes a data path between different heterogeneous storage media to access data. This makes it vastly more flexible than Groq’s static approach. MatX also uses a hybrid SRAM+HBM system that is super interesting (check out Austin’s interview with the CEO linked above). We discussed all this in a previous TWiC edition.

This is just the first of many such heterogeneous solutions that will emerge. The effectiveness of each solution will depend on how data movement is architected between various pieces of disparate hardware. Exciting times.

(via Sambanova)

Intel + Google partner on Xeon CPUs and IPUs

In even more Intel news, Google and Intel partner to deploy Xeon CPUs and Infrastructure Processing Units (IPUs) for AI workloads. IPUs handle networking, storage, and security, much like Bluefield Data Processing Units (DPUs) from Nvidia. The main idea is to offload some admin functions from the main CPU so that it can focus on more critical AI related orchestration tasks.

In a way, this shows that not all CPUs are headed the ARM way, and x86 retains its stronghold in AI datacenter deployments. Google’s own ARM-based Axion CPUs were deployed for cloud infra and were focused on energy efficiency. In the AI era, raw power still matters and x86 Xeons still crush. If you want a head-to-head comparison CPUs in the market, check out our CPU yellow pages.

(via Intel)

Samsung Projects Record Q1 Profit, Driven by Memory

Samsung is projecting a blowout quarter with a 755% YoY increase in profits of 57T won ($38B) compared to 6.7T won ($5B) in Q1 2025. 95% of the profits are from the memory sector, while the rest is from other sectors like mobile which are flat or declining. Revenue is up 68% YoY. It is anticipated that other major players like SK Hynix and Micron will post spectacular results in the coming quarter too.

We are now deep in the memory supercycle since we first spoke about it on this newsletter six months ago.

In this week’s Semi Doped podcast, we discussed how spot DRAM prices are so high due to AI sucking up all the supply that consumer devices are left scrambling to get what silicon is left over. While this has led to memory price crashes in the past, we argued in an article to paid subscribers how this time is different because of a structural demand enforced by AI, regardless of algorithmic improvements such as TurboQuant.

Samsung’s projected numbers in Q1 2026 are validation of growing memory demand and DRAM/NAND price hikes that are still rising in the coming quarters. But notably, a large portion of the ~8x profit increase is coming from the DRAM market, and not necessarily HBM. This is worrisome because while HBM provides a structural floor, spot prices are far more volatile and things often turn around quickly.

Anthropic Hits $30B Revenue Run Rate, Overtakes OpenAI

Anthropic revealed a $30B annualized revenue run rate, triple the ~$9B at end-2025 and up from ~$19B in March. They simultaneously announced a deal for 3.5GW of Google TPUs (via Broadcom) starting 2027, adding to ~1GW already secured.

30x ARR growth in 16 months makes Anthropic the fastest-scaling enterprise company in history. But the infrastructure story is equally significant: they're hedging across AWS Trainium, Google TPUs, and NVIDIA GPUs — a diversified compute strategy. This is a big plus for Broadcom which now has a >$100B AI pipeline (possible $200B?). In a weird turn of events, Anthropic is rumored to start developing their own chips. What else are they gonna do with the extra change when their ARR hits $100B next month? 🤡 (or maybe not?)

Arm’s Strategy of Selling AGI CPUs to China

This bit of news is slightly over a week old, but I feel it flew under the radar. Arm’s CEO Rene Haas said in an interview to the ChinaDaily that Arm intends to sell its finished AGI CPU to Chinese customers. US/UK export controls restrict Arm from licensing its Neoverse V3 cores, but says nothing about selling finished chips instead of just IP. This is a key distinction: licensing IP would let Chinese firms design and produce their own chips indefinitely; buying finished CPUs from Arm does not transfer that design capability.

Selling CPUs in China unlocks a huge market that previously did not exist for Arm. The AGI CPU gives Chinese buyers immediate access to a modern, high-core-count, AI-optimized Arm server processor for orchestrating large AI/agentic workloads. The key announcement to now track is Arm’s first Chinese CPU customer, whoever that is.

(via ChinaDaily, DigiTimes)

Have a great weekend!

A useful starting point to understand the lasers versus LEDs debate is to ask what linewidth is, why it matters, and how lasers and LEDs broadly compare. Let’s first describe the technology landscape.

You can download this post in epub or pdf formats for free using the button below. If you are not a paid subscriber, and would like to purchase individual articles, please see the catalog here.

Get as free ebook

Lumentum, Coherent, and DFB Lasers

Distributed Feedback (DFB) lasers often made from Indium Phosphide (InP) are known for their narrow linewidth, where Lumentum holds dominance in 200G/lane electro-absorption modulated lasers (EMLs). Their moat comes both from having a great laser source and a co-optimized modulator that encodes high-speed electrical signals into light. The performance gap in EMLs is seemingly wide enough (specs not published widely) that their competitor, Coherent, actually buys these EMLs from them. This puts Lumentum in the driver’s seat for EMLs in the 1.6T optical era involving pluggable transceivers.

A DFB laser for co-packaged optics (CPO) is a different beast and is a space where Lumentum has serious competition. CPO requires a constant source of light (called continuous-wave or CW) because the modulation is handled by the silicon photonics chip using a ring or Mach-Zehnder modulator. Here, a constant output power of 300-400mW at elevated temperatures of 50-70°C is the key differentiator. This level of output power is needed because the light source is outside the rack, and connected to the SiPho chip via a long polarization maintaining optical fiber. We’ve discussed this earlier.

Laser Linewidth, RIN, and Power

Lumentum’s ELS laser for CPO has an advertised output power of 350 mW at 50°C, a line width of <500 kHz, for a 1311 nm laser. Coherent’s equivalent laser has a reported output power of 400mW at 50°C and a linewidth of <200 kHz. Linewidth is a measure of how pure the light source is, and is quantified by the Full-Width at Half-Maximum (FWHM). Take the maximum laser power, cut it by half and measure the spreading of the laser frequency/wavelength. The smaller this number, the more pure the laser output on the spectrum.

Anything under 1 MHz in linewidth is exceptionally pure light, really. A 1311nm laser translates to about 230 THz in free space which is really narrow compared to the center frequency. The deviation in wavelength is so small that laser engineers prefer frequency units to represent this. This is the magic of DFB lasers.

Lasers from both companies have a Relative Intensity Noise (RIN) of -145 dB or thereabouts. RIN is a measure of how stable the output power is. Think of a bad laser with high RIN as flickering too much. Lumentum is not much better in RIN; the frequently cited -156 dB RIN for Lumentum is a hero number from a CLEO 2022 paper. Their production RIN is in the same ballpark as Coherent. Thus, while Lumentum has a clear advantage in EMLs for pluggables, the race to CPO lasers is highly contested (there are other players but their laser specs are not publicized).

Regardless of lasers for pluggables or CPO, the key takeaway is that DFB lasers with their narrow linewidth are perfectly suited for optical communication, and have been the mainstay for long haul links. They allow encoding of fast data signals and have long distance reach. Here is the critical question: is DFB laser the right choice of technology for short-to-medium distance optical links in a datacenter?

The Case for MicroLEDs: Benefits and Challenges

There has been a slew of activity around the use of Micro Light Emitting Diodes (LEDs) for data transmission. Credo moved to acquire Hyperlume last year, Microsoft published results from Mosaic, and Marvell most recently is collaborating with Mojo Vision on microLEDs. We would be remiss if we did not mention Avicena, who has been pioneering MicroLEDs for AI interconnects for a long time.

MicroLEDs are just really small versions (10-50 microns in size) of your regular lightbulbs, and are built with Gallium Nitride (GaN) to emit blue light. Just like you don’t get a laser light show every time you walk in your room and turn the light on, microLEDs do not emit a narrow pencil beam of light like lasers. They instead produce light in a wide angle and need to be focused with a microlens. The light is not spectrally pure like lasers either. Blue GaN microLEDs have a wavelength of about 450nm, and linewidths of 10-15nm. Compared to lasers, this is terrible linewidth because it accounts for about 3% around the center wavelength.

The poor linewidth in comparison to lasers imposes restrictions on data rate that a microLED optical link can support. The solution around this is to use an array of microLEDs in a “wide-but-slow” approach – just like each data lane in high bandwidth memory (HBM) is slower than what is possible in GDDR, but the overall throughput is higher. There are a few additional benefits of using microLEDs:

1. Lower power than DFB laser optics (caveat to follow), but better reach than copper (can possibly do >10m at Tbps aggregate speeds)

2. Better reliability than lasers because LEDs are structurally simpler, and relatively temperature insensitive. A wide-but-slow approach means that there could be redundant lanes for failover.

3. Linear path to scaling to higher speeds: increase per-lane speed (harder) or increase number of lanes (easier).

While microLED cables would work directly with existing pluggable infrastructure, there are two important requirements that makes things more complex:

Focusing optics

The broad angle emission from a microLED needs to be focussed with microlenses in order to couple them into an array of optical fibers that carry multiple parallel lanes of data. In reality, this is not difficult because microlenses are also implemented as an array that fits directly over the microLED array. DFB laser sources have specialized optics to couple light into fibers too.

Electrical Gearboxing

You need an “electrical gearboxing” function that converts the 112G/224G narrow-but-fast SerDes lanes of the host processor into a 20 x 20 microLED array that carries slow-but-wide signals at lower data rates. This chip is implemented as a CMOS ASIC that handles the electrical gearboxing. For example, the Avicena LightBundle™ evaluation kit launched at OFC 2026 uses a specialized 16nm finFET CMOS ASIC. The only way a gearbox chip can be avoided is to design the host processor with a UCIe interface that natively implements a wide-but-slow approach (like a custom base-die used in HBM).

Microsoft’s paper on microLEDs shows that the gearbox functionality only consumes a small portion of the power (0.4W) it would otherwise take with an optical connection that requires DSP/CDR/FEC (3.5W). The argument in the paper is that it is a simple gearboxing function, and thus less power hungry. If per-lane speeds increase in future generations of microLED interconnect, then CDR/FEC functions will be required and power usage will increase, closing the power efficiency gap between lasers and microLEDs.

Why linewidth matters for data rates

Let’s address why microLEDs are only capable of 2-4 Gbps data rates today in pre-production, while DFB EMLs can support 200 Gbps or more.

The fundamental concept that underlies this limitation is chromatic dispersion in an optical fiber. When an optic signal travels through a fiber, different wavelengths travel at different speeds due to the wavelength-dependent refractive index of glass in the fiber. The resulting pulse exiting the fiber spreads out in time because different wavelengths arrive at different times.

This spreading determines the maximum speed of data transmission that is possible because once neighboring pulses start to overlap, it is hard to tell them apart and detection errors start to occur.

The broadening of a pulse by a time dT (ps) in an optical fiber can be calculated as

Where D is the dispersion coefficient in ps/(nm·km), L is the link length in kilometers, and is the linewidth of the light source. Assuming Gaussian pulses are being transmitted as bits, the data rate Bmax is:

Let’s try out some numbers here.

With a microLED

A standard legacy multimode optical fiber (such as OM3 or OM4) operating at 850 nm exhibits a material dispersion parameter of approximately 100 ps/nm/km.n For an 850 nm microLED with a spectral linewidth of 40 nm (4.7% of center wavelength), operating over a link length of 0.010 km (10 meters), the temporal pulse spread is calculated as: 100×0.01×40, or 40ps. Maximum data rate supported = 0.44/40ps = 11 Gbps/lane. These speeds have been demonstrated in research papers with microLEDs, but production numbers are slower at 2-4 Gbps/lane.

With a DFB laser

Let us assume the same wavelength, and same link distance, but a linewidth of 0.001nm. The temporal pulse spread is: 100×0.01×0.001 = 0.001ps. Maximum data rate supported = 0.44/0.001 = 440 Gbps/lane.

This demonstrates why linewidth makes a big difference in how much data rate is possible with a given type of light source in an optical communication system.

There is a lot more to discuss about microLEDs before we can declare microLEDs fit/unfit for datacenter needs: coupling efficiency into fibers, beam shapes, reliability assessments, and the overall supplier/customer landscape that is still rapidly evolving.

Industry sources tell me that hyperscalers are open to exploring microLEDs for future interconnect needs, but the microLED industry for datacenter interconnects is still young. An additional aspect worth exploring is how VCSELs compare as an alternative to microLEDs – something that Lumentum is actively exploring with 1060nm VCSELs. So many future post options!

Here’s the post if you haven’t gotten to it yet.

In last week’s Semi Doped podcast, we covered TurboQuant in some detail, and argued whether Arm’s AGI CPU will actually displace x86 in the long run.

• ARM AGI CPU has entered the chat, TurboQuant thrashes memory stocks

Now, on to the news.

Samsung Enters SiPho, Wants to Connect HBM with Light

Samsung unveiled its SiPho Foundry platform at OFC 2026 that is production-ready, with a PDK, and ready for manufacturing on a 300 mm wafer process. The main pitch here is Samsung’s vertically integrated memory capabilities with HBM and silicon photonics under one roof, a turnkey bundle that TSMC cannot match. Early results are expected in the 2028-29 time frame, and the industry places Samsung roughly 3 years behind TSMC, primarily due to ecosystem, design enablement, and customer trust. Samsung’s Semiconductor division has already designated its silicon photonics processes as “I-CubeSo” and “I-CubeEo” (try saying that 3 times in a row) and is allegedly working closely with Broadcom and Marvell.

(via The Elec, DigiTimes)

Low/mid tier mobile is driving demand destruction; PC demand down

MediaTek and Qualcomm are both cutting 4nm chip shipments by 15-20 million units combined because memory has gotten so expensive that low-to-mid-tier smartphones make no sense to build because their BOM is so high. Whether premium tier phones are insulated is questionable, especially when Apple is supposedly buying up all available DRAM at high prices to starve out Android competitors like Samsung (ironic since Samsung is also Apple's largest DRAM supplier).

Last year, more than 360 million smartphones shipped below $150, representing a substantial share of global volumes, with that proportion rising in key emerging markets like Africa and India. The push for greater economies of scale will drive mergers among the smaller players. Omdia cites Realme reintegrating under OPPO's umbrella as early signs of consolidation as vendors seek greater scale to manage rising costs. The consensus is that this is a structural reallocation of memory capacity toward AI/HBM at the expense of consumer TAM.

Nvidia joins forces with Marvell via NVLink Fusion

Nvidia and Marvell announced a partnership that allows Marvell’s custom ASIC silicon to run on Nvidia infrastructure via NVLink fusion. Market sentiment is overwhelmingly positive on this news, and Marvell shares surged 13%. The deal is viewed as a defensive masterstroke by Jensen that signals that if Nvidia can’t beat the custom chip movement, they can at least own the fabric ASICs run on. Clear winner is Marvell here because they get the $2B investment cookie that Nvidia has been handing out to everybody (LITE, COHR, NBIS…) and also “preferred partner” status with Nvidia. This puts Broadcom in a more challenging position as the primary backer of UA-Link standard over NVLink fusion.

(via Nvidia News)

Intel buys back 49% share of its Fab 34 Ireland fab

Intel said it would spend $14.2 billion (cash+debt) to buy back the 49% stake it sold to Apollo Global in its Ireland manufacturing facility. The fab produces leading-edge nodes like Intel 3 and 4, including its Core Ultra and Xeon 6 processors. The price paid is $3B more than they sold it for two years ago, and the market is reading this as Intel’s conviction in their foundry strategy by putting real money behind the chip manufacturing. The sentiment is strongly positive and is interpreted as a sign of confidence that Intel’s foundry turnaround is actually happening. Investors will closely scrutinize whether the strategic benefits are strong enough to justify the added cost and financing burden.

(via Intel)

OpenAI closes record breaking $122B funding round

Total valuation is $852B. Seriously, that is a lot of money. OpenAI is pulling $2B/mo in revenue, and still continuously losing money. Only profitable in 2030. 👀

(via The Guardian)

Anthropic’s Claude has a wardrobe malfunction

Claude’s source code leaked via npm repository, and yeah, we can see everything underneath 🙈, including a digital pet and an always-on agent. The funniest coverage on this is from ThePrimeagen on YouTube. Seriously watch it lol.

(via literally everywhere)

Your latest co-worker is now an AI employee

This is a $2,000/mo AI from a company called Kuse AI that is a virtual AI employee who never sleeps, eats, or goes to happy hour. There are 2,000+ companies who have signed up for the wait list. If companies need to pay 3-5x more for a human who demands work-life balance, an OpenClaw agent that the company calls Junior seems to be a better bet. All well and good, but you can’t choke an AI neck when things go wrong.

(via Bloomberg)

Atlas Studio: A Foundation Model for Electromagnetics

This is a physics foundational model that is useful for a lot of RF design work that would take too long with a simulator. Not Boring has a long piece on it. Long time readers would remember a piece I wrote about crazy RF designs from Prof. Sengupta’s group (Princeton) last year. Well, Prof. Sengupta is now pushing back on the ‘radicalness’ of the foundation EM model by comparing it to someone claiming that they invented the transistor.

(h/t @SinghJyotirmai, via Arena Physica)

PrismML: 1-bit Bonsai 8B model fits into 1.15GB

With the TurboQuant meltdown in place, I wonder if we should worry about dramatically compressed models like PrismML. If this delivers any level of useful intelligence, it could unlock a whole new world of edge inference. Its open-source and extremely light weight for an 8B parameter model. I think I’ll run it on my Mac M3 pro this weekend and see how I like it. I already tried it on my iPhone. It could also serve as a local fallback model for OpenClaw.

(via PrismML)

Have a great weekend!

Still, worries of TurboQuant wrecking the memory industry have not abated. To the contrary, a new fear has been unlocked: the peak of the “hog cycle” often seen by seasoned investors as a strong signal of a memory peak, and impending doom.

The counterargument is that “this time is different,” driven primarily by the insatiable demand for memory in AI deployments, and the token explosion in the era of agentic AI that is already here. The real question is whether the demand is high enough to serve as a fallback when the consumer segment finally “says no” to the extraordinary price hikes and cancels deployment of low/mid tier devices, and/or downgrades product specs to accommodate what memory is reasonably available.

In this article, we will investigate these arguments in greater detail, make reasonable guesses and inferences where possible, and attempt to have a balanced, realistic stance based on the current state of the industry.

Broadly, we will follow this line of inquiry:

The classic memory hog cycle isn’t dead — but AI has added a multi-layered structural floor that mutes the traditional ‘PC/smartphone makers say no’ warning signal and turns the 2026 pullback into a buying opportunity rather than the start of another 2019/2022-style bust.

Here is what we cover:

• Hog Cycles as a Leading Memory Indicator

• 2026: PC/Smartphone Industry Says ‘No’

• “This Time is Different” – But is it?

• 🔒Understanding the New AI Realities

• 🔒Memory as the Bottleneck

• 🔒Revisiting the Thesis and Risks

If you are not a paid subscriber, you can purchase just this article using the button below. You can find the whole catalog of articles for purchase at this link.

Buy the ebook

Note: This article does not constitute investment advice. Do your own research.

Hog Cycles as a Leading Memory Indicator

The classical hog cycle works like this: High pork prices lead farmers to breed more pigs. By the time the pigs mature, the market is flooded, prices crash, farmers stop breeding, and a shortage ensues and causes prices to spike again.

This applies directly to memory. A new technology (smartphones, cloud computing, AI) drives massive memory demand and prices soar. Memory makers enjoy record margins and heavily invest in expanding fab capacity. It takes years for that new capacity to come online. During this time, buyers double-order to secure supply, artificially inflating demand. The new fabs finally start producing exactly when downstream demand begins to cool. Supply floods the market, inventory piles up, and prices crash below the cost of production.

This cycle has specifically played out at-least twice in the last decade.

• 2019 Cloud Burst: Datacenter boom for cloud computing and crypto mining lead to excess inventory when everybody suddenly stopped buying. DRAM prices plummeted by 50%.

• 2022 Covid Hangover: Covid-era shutdowns and the spike in PC/phone sales for remote work left OEMs with excess inventory that suddenly had no demand. The carnage was real: DRAM spot prices dropped >50%, server DRAM revenues collapsed, memory stocks fell 30-50%, and NAND sold for less than it took to make.

2026: PC/Smartphone Industry Says ‘No’

It seems like the memory downturn in the past week is one for the history books, largely driven by fears around TurboQuant, and more importantly another catalyst worth discussing: the PC/smartphone industry. Shares of Micron, SanDisk, SK Hynix, and Samsung are all down 20-30% from their recent peak. I’m guessing we will talk about this for years to come.

The PC/smartphone industry is the literal whale of the memory market consuming massive amounts of DDR, LPDDR and NAND flash storage. They have largely been squeezed out of the memory supply chain of late due to large spikes in DRAM and NAND pricing primarily driven by shortages due to the huge demand in AI accelerators, which consume nearly 70% of global high-end DRAM production. The recent TrendForce memory forecast below shows why pricing has reached a boiling point.

The prices for nearly every type of memory and storage has nearly doubled in a single quarter, which means that memory alone is estimated to occupy nearly 25-30% of a PC/Smartphone bill of materials (BoM). Two other factors further compound problems:

1. The long term agreements for memory supply entered after the Covid era bust are coming to an end. Consumer OEMs are being forced into an open market where there is virtually no silicon left to purchase.

2. The timing is particularly devastating for the PC industry, as this hyper-inflationary spike collides directly with the Microsoft Windows 10 end-of-life hardware refresh cycle.

Dan Nystedt on X explains how the PC/smartphone industry has now drawn a line in the sand by planning “fewer (or zero) mid-to-low-end handsets in 2026.” – a sign often viewed by seasoned memory investors as a time to sell. IDC reports that global PC shipments are expected to drop by 11.3% in 2026, a revision that calls out a much larger drop than their 2.4% estimate a quarter ago.

Conventional wisdom and history lessons in the memory market dictate that the memory market has peaked. But, bulls argue the insatiable AI demand makes this time different.

“This Time is Different” – But is it?

The simple bull argument for the market is that the extraordinary demand for AI will soak up any short term demand decline by the PC/smartphone industry. That is, AI is a “fallback” when consumer demand drops and memory prices stay unaffected. But how valid is this?

This argument holds for now because memory makers are happy to divert production lines towards HBM and server-grade DRAM memory as Micron did by dropping the Crucial line of consumer-grade memory. This worked out well because Micron’s last earnings call guided for an 81% gross margin, which would exceed even Nvidia’s if they deliver. Their non-HBM product lines are also more profitable now, thanks to inflated spot market pricing. Spot pricing is a double-edged sword because in a deflationary market, the same product lines will drag down revenue.

Interestingly, history shows that there has almost always been a series of overlapping technology S-curves that acted as “fallback” and delayed (not avoided) the eventual memory market destruction.

Here are some examples.

2015-2016: Smartphone Maturation + Cloud Buildout

• The Softening: In early 2015, the global smartphone adoption curve began to flatten and the upgrade cycle started stretching from 18 months to 24+ months. PC sales were also in a secular decline. With softening demand, smartphone OEMs started aggressively pushing back on memory contract prices.

• The Fallback: The early cloud build-out meant that AWS, Azure, and Google Cloud were beginning to scale their massive data centers. This created a new, hungry sink for server DRAM and NAND SSDs, absorbing wafers that would have otherwise flooded the mobile and PC markets.

• The Crash: The crash came around early 2016 because cloud buildouts could not effectively absorb the PC/smartphone softening, leading to an inventory glut and prices plummeting.

2018-2019: Smartphone Decline + Crypto Mania

• The Softening: Smartphone sales went from slowing to actually contracting globally. At the same time, the memory prices from the 2017 cloud supercycle had gotten so high that smartphone OEMs actively decided to cap the amount of DRAM in their flagship phones to protect their profit margins.

• The Fallback: During the crypto mining boom, hyperscalers were buying server memory at a frantic pace, terrified of supply shortages. The 2017–2018 crypto bull run created an insatiable demand for high-end graphics cards, sucking up massive amounts of GDDR and standard PC DRAM.

• The Crash: By 2019, cloud hyperscalers realized they had massively over-ordered DRAM, and switched to inventory digestion mode. Then came the crypto winter. Bitcoin crashed when people realized it was unprofitable to mine crypto and the demand for graphics cards and GDDR evaporated overnight.

2022-2023: Post-Covid Demand Drop

• The Softening: By early 2022, everyone who needed a laptop or phone for remote work already had one. As inflation spiked, consumer spending on electronics completely froze. PC and smartphone OEMs hit the brakes hard, cancelling memory orders to burn through their existing warehouses of chips.

• The Fallback: There was none. Cloud hyperscalers were not on a spending spree for memory, and as a result, nothing softened the blow.

• The Crash: Because memory makers had aggressively built new fabs during the 2021 boom, the simultaneous collapse of both the consumer main market and the enterprise fallback created the worst inventory glut in the history of the memory industry, forcing memory makers into deep operating losses.

Memory cycles have mostly had an infrastructure buildout softening the blow when consumer demand drops. The question now is: will this happen again with the AI infrastructure buildout?

After the paywall:

• Understanding the New AI Realities: The four structural differences—memory density, wafer efficiency, allocation shift, and contract stickiness—that make current AI infrastructure demand fundamentally unique.

• Memory as the Bottleneck: Why memory, particularly for long context windows and inference, has replaced compute as the primary bottleneck in AI systems.

• Revisiting the Thesis and Risks: Evaluates the AI buildout as a structural floor for demand, and provides perspectives on the 2026 memory pullback and risks.

Read more

I wrote up a quick note because a few people on the Substack chat wanted to know what TurboQuant is and what implications it has. I hear the article has made rounds in investor circles.

The other deep-dive from this week is on how grating couplers drive the future of TSMC COUPE for CPO, and why Himax has a fragile moat that will be quickly eroded by standardization of the COUPE-FAU interface.

If you missed the last Semi Doped podcast episode, you can check it out below.

• MicroLEDs Ain’t Dead, Micron Snags Vera-Rubin

After this podcast, I had a chat with the guys from Avicena, who were kind enough to explain the their tech to me in-depth. I will write a post once I understand what the industry’s hesitations are with MicroLED tech. Let me know if you have questions.

Lots of news. Let’s dive in.

Arm Ships Its First Chip Ever: ARM AGI

In its 35 year history, the AGI CPU is the first fully in-house processor built by Arm, and heralds a bold entry into the space of server CPUs at a time when agentic AI applications need it the most.

The AGI is built on TSMC 3nm with 136 Neoverse V3 cores, and 300W TDP. Their angle into the CPU market is power efficiency over x86 CPUs, which translates to better TCO when deploying at scale. Their high density air-cooled rack implementation can house 8,160 cores (60 CPUs) and liquid-cooled version can hold >45,000+ cores (>330 CPUs, >5x more).

Arm estimates that agentic AI will drive 4x more CPU utilization per deployed GW. As we saw with Vera Rubin, the CPU:GPU ratio is already 1:1 at the rack scale. Even if Arm is overestimating, that’s still a lot more CPUs. Like we predicted in an earlier post, the CPU:GPU ratio is about to get a lot higher from what we have been used to.

Meta is their lead customer for the AGI CPU, which will possibly run as head-node CPU for the rapid-fire MTIA chip releases planned in the coming two years.

CEO Rene Haas projected $15B in annual revenue from the chip within five years, with total company revenue reaching $25B and EPS of $9. In a Stratechery interview, Haas said memory (not TSMC) is the real bottleneck and his comments about the relative importance of core counts is interesting:

I think core count is going to be quite important because I think, again, I have a belief that each one of these cores will want to potentially run their own agent, launch a hypervisor job, launch a job that can be run independently, launch it, get the work done, go to sleep. The performance of the core is going to matter, no doubt about it, but I think the efficiency of that core is probably going to matter just as much as the performance is.

Digitimes noted that GUC is likely the first casualty of Arm selling chips directly, given its CPU work with Google and Meta. TechX is wondering when Qualcomm Oryon server CPUs will show up given that development was put on hold in favor of mobile chips during their lawsuit with Arm.

(via ARM)

CPU Shortage Gets Real: Intel and AMD Hike Prices

Arm’s CPU provides options at a time when Intel (INTC) and AMD are raising CPU prices again, up 10-15% since start of the year. Seemingly lead times have ballooned from ~2 weeks to up to 6 months for some SKUs. The question is whether this is a temporary supply hiccup (Intel’s CFO says Q1 was peak shortage) or a structural tightening as agentic AI workloads drive 4x more CPU demand per GW of data center capacity (per Arm’s Haas). Also, price hikes are a way to keep revenue up when supply is low and demand is high.

TSEM + COHR 400G/lane SiPho Using Si Modulators

Tower Semiconductor and Coherent jointly demonstrated 400 Gbps/lane data transmission using a silicon modulator built on Tower’s production-ready silicon photonics (SiPho) process, achieving a clear open eye at 420 Gb/s PAM4. Apparently, this was demonstrated in OFC 2026 which I could not go to, but would love to see the slides/paper if anyone from Tower or Coherent reading this wants to share.

If we look back at Broadcom’s recent 400G optical DSP announcement, there was a mention of thin-film lithium niobate (TFLN) used for the design of high bandwidth modulators required for the 3.2T generation of optical transceivers. If silicon modulators using micro-rings or Mach-Zehnder modulators (not sure which one was used) can work at 3.2T (400G ×8), then there is no need for special TFLN deposition steps in the wafer processing flow for several years to come. Since Tower is investing $920M in capex towards SiPho/SiGe capability, using existing infrastructure means high fab utilization rates. The modulators used Coherent’s InP CW laser, which requires narrow linewidth, low relative intensity noise (RIN) and low side-mode suppression ratios (SMSR) for it to work. Apparently, it’s good enough for 400G/lane.

(via Tower Semiconductor)

… Then GF Sues Tower

GlobalFoundries (GFS) filed patent suits against Tower Semi alleging that (emphasis mine):

… [Tower] has infringed GF patents by freeriding on decades of GF innovation with an intent to unlawfully take business away from the American chipmaker.

The ‘freeriding’ refers to 11 patents in areas related to smart mobile, automotive, aerospace and communications infrastructure. Yeah, so basically everything 😆 GF seeks to ban the import of Tower chips into the US, and be compensated for lost profits. Tower intends to put a stake in the ground and fight.

Tower shares closed down 7.5% and GlobalFoundries shares fell 4.6%. If you want to read too much into it, this infringement law suit comes after Tower’s deal with Nvidia to supply 1.6T optical transceivers. Lawsuits bad for everybody, except lawyers 🔱. Austin and I chatted about it in the upcoming Semi Doped episode. Stay tuned.

(via GlobalFoundries)

AAOI Lands Back-to-Back Hyperscaler Orders

Applied Optoelectronics (AAOI) received a $53M order for 800G single-mode transceivers from a major hyperscaler (likely AWS), with shipments starting Q2 and completing mid-Q3. This follows a $200M+ 1.6T order from the same customer the prior week. The stock jumped 20% on the news. The back-to-back orders across both 800G and 1.6T generations position AAOI as a meaningful supplier to at least one hyperscaler’s GPU cluster buildout, not just a niche player anymore.

(via Applied Optoelectronics)

Micron Crushes, Then Gets Crushed by TurboQuant

Micron (MU) posted a blowout FQ2: $8.05B revenue, record 75% gross margin, HBM sold out through end of 2026, and FQ3 guidance of $8.8B that blew past the $7.95B buy-side expectation. All good right? No.

Google happened to resurface a year-old KV cache compression paper called TurboQuant and the stock cratered, down 25% from its highs by midweek with Sandisk dragged down 8% in sympathy. This is an overreaction to a year old paper, and nothing fundamentally changes the demand for memory and storage. More coverage in linked article above.

(via CNBC)

Helium Spot Price Surges 50%+ on Qatar Attacks

It’s not often we discuss raw materials in this newsletter, but this one is important. Iranian attacks on Qatar’s Ras Laffan LNG complex knocked out roughly 17% of Qatar’s LNG export capacity, triggering a 50%+ spike in helium spot prices. Also, remember that Helium is a finite resource on Earth that when wasted is lost forever because it leaves the atmosphere and escapes into space. Why does this matter other than for birthday balloons?

Helium is critical across implantation, lithography cooling, and leak detection in semiconductor manufacturing. Helium’s inertness is good for purity testing, and its small atomic size is useful for leak testing where Helium goes through but not other gases. It’s not a major component, but supply disruption is a concern. Samsung and SK Hynix are most exposed (sourcing 64.7% from Qatar versus TSMC’s 30%) and are stockpiling at elevated prices. Sure Helium is recycled after use, but still needs replenishig.

The crisis is compounding broader materials cost pressure: photolithography solvents are up 40-50%, and suppliers including DuPont and LG Chem have issued price hike notices for April. The main issue is the medium to long term implications of Helium supply because disruptions often take years to recover.

(via Trendforce)

Have a great weekend!

This is not the first time that efficiency improvements, or niche use cases have triggered a selloff in the memory sector. The announcement of SRAM accelerators worried investors that HBM will no longer be useful, for example. Investors in memory have seen cyclical booms and busts, and their trigger happiness to move stocks is not without reason. Given the run up in memory stocks, everyone is looking for the top.

Subscribe now

A few people have asked me to explore TurboQuant in a bit more depth. TurboQuant is a 2-stage algorithm involving PolarQuant and QJL, that allows KV-cache to physically occupy fewer bits and thus allow for longer context windows in LLMs.

In this short note, we’ll go through TurboQuant briefly and tie it to what it means for the industry.

Read more

Citrini/Hunterbrook traced a trail through four patents (two Himax, one FOCI, one TSMC) and found that both Himax’s manufacturing patent and TSMC’s Fiber Array Unit (FAU) inspection patent describe a 22-channel configuration (standard fiber counts in optics are 12, 16, 24, and 32). The unusual but matching channel count across patents was considered the “smoking gun” that ties Himax and FOCI as a key TSMC supplier. PhotonCap has a patent-by-patent analysis that is worth reading for the full forensic detail. However, Irrational Analysis counters Citrini’s patent-centric approach stating:

There is zero discussion on coupling efficiency, polarization extinction ratio, or alignment tolerance.

The most dangerous analysis looks smart (check out these patents!) but actually delivers zero insight into the underlying viability of the tech.

These micro-lense assemblies are obviously for grating coupler attach. Grating couplers have high loss compared to edge couplers so the lenses better be damn good. Zero evidence this is the case.

In this article, we will look at the engineering details that are missing in the Himax/FOCI/TSMC conversation. We will cover:

• Physics of edge vs grating couplers (free)

• Implementation details of TSMC COUPE that makes it low-loss and well suited to CPO

• Need for fiber array unit (FAU) co-design between TSMC and Himax/FOCI

• Is Himax actually a good bet? How solid is their moat?

If you are not a paid subscriber, you can purchase just this article using the button below. You can find the whole catalog of articles for purchase at this link.

Buy the ebook

Physics of Edge vs Grating Couplers

Much of the discussion to follow requires an understanding of how light is coupled from a fiber into a chip, and how the entire flow of light works. We will cover this in some detail because it is necessary to understand what Himax+FOCI brings to the table, and how critical their technology is to the CPO supply chain.

There are two ways to get light on or off a photonic chip: edge and grating couplers.

Edge couplers

Edge coupling is the method of coupling light into the side of the chip. This is conceptually simple because the Photonic IC (PIC) has silicon nitride (SiN) waveguides into which light can be coupled from an optical fiber by narrowing the beam with a microlens, and using a waveguide taper where the fiber connects.

Since this only involves the light moving from one medium to another without really changing directions, most of the light energy couples to the PIC and loss is low. This is an oversimplification, but is sufficient here. Additionally, both wave-guiding media – optical fiber and waveguides on the PIC – support both transverse electric (TE) and transverse magnetic (TM) polarizations, which are wave oscillations of the electromagnetic field as light propagates.

If the light transition from optical fiber to waveguide involves either TE or TM being attenuated, the coupler is said to have “polarization dependent loss, or PDL.” An edge coupler usually allows both TE and TM modes through, and has low coupling and PDL loss. But there are several downsides:

1. Limited beach-front density: Using only the edges of the PIC to couple light limits how many fibers can be placed around the periphery. This limits the number of ports in a CPO switch.

2. High alignment requirements: The optical fiber requires to be precisely aligned (<1 micron) to the waveguide taper so that all the low loss benefits are retained.

3. Singulation before testing: Each PIC must be diced from the wafer and tested by coupling from the edge. This makes testing low throughput and expensive.

Broadcom’s Tomahawk 4 (Humboldt) and 5 (Bailly) CPO switch chips are examples that use edge coupling to attach fiber to the chips. Below is a quick survey of recent edge couplers in C- and O-bands. The TE0 and TM0 modes both have very low loss, and the PDL is near zero which implies that both modes are coupled well.

Grating Couplers

A grating coupler allows light to enter or exit the PIC from the top surface rather than the edge. It is essentially a series of periodic etchings (grooves) on the surface of a waveguide. When light traveling through the waveguide hits these grooves, it scatters. By precisely spacing these grooves, the scattered light undergoes constructive interference at a specific angle, directed upward out of the chip. Conversely, light coming from a fiber at that same angle will interfere constructively to “drop” into the waveguide.

Here is a simulated side and top view of the field that transitions from a waveguide to scatter vertically. It is important to note that scattering happens in both directions. Even though one direction is predominant, the loss of light on the other side contributes to loss. We will revisit this point later in TSMC COUPE.

There are some immediate advantages that become apparent in grating couplers:

• Beach-front area: The entire 2D surface of the chip is available to connect fibers, which greatly increases the density of fiber that can be connected to a chip compared to using the edge alone.

• Lower alignment requirements: Grating couplers provide significantly relaxed fiber alignment tolerances compared to edge couplers, typically featuring tolerances of 2-4 microns either way from the center.

• Wafer level testing: Since light comes out of the top of the chip, PICs can be tested before dicing, greatly simplifying the process and increasing throughput.

In spite of these advantages, there are some downsides.

• Higher loss: Grating couplers are inherently higher losses due to the scattering effects that occur in the grating structure. Lots of research has been done to minimize this.

• High PDL: Grating structures only work for one of the polarizations, while the other undergoes high loss.

The impact of high loss in gratings implies that link budgets are smaller when going to faster data rates, and everything that follows – the microlenses, prisms, and FAUs – must all be optimized for the lowest loss possible. High PDL implies that data rate cannot be doubled easily by putting information on each polarization (polarization multiplexing), which becomes a limiting factor as we look to scale data rates higher. Here is a recent survey of grating couplers if you’re interested.

After the paywall, we discuss:

• The engineering behind TSMC COUPE, including the implementation details that makes it low-loss and well suited to CPO.

• Why the fiber array unit (FAU) needs to be a co-design between TSMC and Himax/FOCI

• Discussion on whether Himax is actually a good bet, and how solid their moat is.

Read more

• Deep dive: Beyond GTC: A Deep Dive into Compute, LPX, and the Untold Story of SpecDec

• Semi Doped Podcast: Quick Takes: Nvidia GTC keynote

We genuinely feel this was one of our spontaneous and fun episodes on Semi Doped. Highly recommend you check it out. We have crossed 1,100 subs on YouTube and 7,500 podcast downloads already!

Now on to the news.

Micron supplying “fastest” HBM4 to Nvidia Vera Rubin

A SemiAnalysis report published earlier this year declared that Micron will be supplying zero HBM4 to Nvidia in the early Rubin ramp last month. But those have largely been quelled now with Jensen publicly touting their “fastest HBM4” delivered. Several folks pushed back on this at the time, and reiterated that selling on this rumor was a bad idea. They argued that even if Micron did not sell HBM to Nvidia, they are better off selling much wanted DRAM instead at elevated prices with DRAM being significantly easier to manufacture. There were also rumors earlier that the HBM4 base die being designed on a memory node (1-beta) somehow slowed down lane speeds compared to competition who used logic nodes in the base die. This was later confounded by rumors that Micron somehow had trouble measuring HBM performance.

Well, all that guessing seems questionable now. Micron posted great numbers last quarter, stock is up 62% YTD, and the line still goes out the door for memory while they supply HBM4 to Vera Rubin. I am still skeptical that Micron’s HBM is actually the “fastest,” but it seems fast enough that Jensen is willing to take a picture with Sanjoy Mehrotra, CEO of Micron. Also, Jensen, some people would have wanted that memory before you signed on the wafer. Think about where you go about signing stuff; just sayin’.

(via Micron)

Optical scale-up goes official with OCI MSA

A bunch of important companies got together and decided to put some method to the madness in optical networking. The Optical Compute Interconnect Multi-Source Agreement (OCI MSA) is a way to make sure everyone is talking the same language when we go to optical scale up. This allows infrastructure buildouts to use components from different vendors who all comply to the open standard, thus fostering a robust ecosystem of optical networking and avoiding getting everyone locked in to how any one company does things. This is an important step forward, because it makes optical scale-up official, and now everyone is building towards it. In a similar vein, Lightmatter announced a new collaborative initiative within the Open Compute Project (OCP) to create open specifications for interoperable CPO.

(via oci-msa.org)

Credo: 400G/800G/1.6T Optical DSPs, ZeroFlap 800G transceivers

Credo announced their Cardinal 1.6T optical DSP, and Robin DSPs for 400G and 800G, in addition to zero-flap 800G transceivers, showing that they do recognize that optics is going to be around. But so is copper. This puts Credo in a nice position to supply to both markets should CPO adoption get pushed out. Also, microLED is not viewed as useful by the industry, but regardless of however that turns out, it’s interesting that Credo has a “better than copper, not as good as optics” technology option that might actually prove useful. Or not. Time will tell.

I am neither a copper or optics bull (I’m both in the short to medium term), and investors need to think for themselves and stop shorting every time there is some uptick in optics news. Jensen was clear at GTC that we are going to use both copper and optics in the near term. Vera-Rubin NVL576 will use NVL72 or NVL144 racks hooked up to each other via optics, but each rack will still use copper. Also, recognize those infamous purple cables in Meta’s MTIA 400?

(via businesswire: Cardinal 1.6T DSP, Robin 400/800G DSP, 800G-ZF)

Lumentum: 1060 nm VCSEL for slow-but-wide optical scale-up

Vertical Cavity Surface Emitting Lasers (VCSELs) are mature technology, but are not great for ultra high speed optical communication links because of their nonlinear behavior, wide spectral line width, dispersion into multiple modes, and limitations on modulation bandwidth. But you can run them at slower speeds, and then use a VCSEL array to run multiple lanes in parallel — like the HBM of optics, so to speak.

VCSELs have another major advantage in today’s day and age. From Lumentum:

As a VCSEL-based scale-up solution, the platform also provides an independent supply chain alternative to silicon photonics and InP laser-based architectures.

InP alternative you say? 👀 VCSELs typically use Gallium Arsenide (GaAs) as the III-V material for laser light generation, and not InP which is in extremely short supply. Just imagine a scenario where we could do optical scale-up with GaAs VCSELs; where does that put the highly sought after InP supply chain?

Luckily for bulls , Lumentum is also the leader in InP EML lasers which is currently in high demand, and their CW lasers for CPO has some competition from Coherent, Applied Optoelectronics and others. If they corner the market with 1060 nm VCSELs for optical scale-up, now wouldn’t that be a Lumentum bull case? Oh, and if you’re wondering about the strange choice of 1060 nm wavelength, here is what Lumentum has to say:

Compared to conventional 850nm datacom VCSELs, 1060nm devices deliver improved speed capability, superior high-temperature performance, and exceptional long-term reliability.

We’re still looking at demo hardware from OFC 2026, but this is a tech to monitor closely. If you want to learn more about the “slow-but-wide” approach to optics, check out Lightmatter’s blog post.

(via Lumentum)

NVIDIA DGX Station GB300

At GTC, Jensen mentioned how each company should have their agentic AI strategy, or risk falling behind. One concern for many sensitive end markets is the company information staying on premises — think healthcare, insurance companies, law firms, and even engineering firms.

The DGX GB300 is the enterprise solution that you need, because it is explicitly designed for the full develop-to-deploy pipeline, not just R&D. Enterprises deploy it on-premises in their own data centers or colocation facilities so they can run private, secure, high-performance inference without sending sensitive data to public clouds. That is “local inferencing” in the enterprise sense — your data and models stay inside your infrastructure, with full control and predictable cost/latency.

This machine is a beast; just look at the specs below. My guess is the cost is anywhere between $30,000-$50,000, because the GB10 version is ~$5,000. Even if my estimates are off by 2x, this machine is a no brainer to deploy within a company so that engineers can use all the tokens that their on-prem hardware supports. This kind of hardware makes it easy to treat token costs as depreciating CapEx, rather than buying tokens from cloud companies and treating it as OpEx.

Aside: if you’re a $500K engineer reading this, make sure you cost your company half your salary in AI tokens. Tell them Jensen said so in your annual review, and explain to your manager how AI actually makes things cheaper. Post back in the comments with your annual raise.

Aside’s aside: Just ask the manager to buy you a DGX Station GB300.

(via Nvidia)

Nebius: $27B AI Infrastructure Agreement with Meta

Massive five-year deal, and it helps to understand how it works.

• $12 billion committed/dedicated capacity: Nebius will build and reserve AI compute clusters specifically for Meta across multiple locations → first large-scale deployments of NVIDIA’s Vera Rubin platform. Deliveries start early 2027.

• Up to $15 billion additional: Meta has committed to buy any remaining available capacity in certain upcoming Nebius clusters (after Nebius sells to third-party customers first).

For Meta, this means that they have secured compute capacity five years in advance. This also shows that AI infrastructure buildouts will be a hybrid of self-building and outsourcing. Also, the infrastructure spending is showing no signs of slowing down.

For Nebius, this deal is in the order of their market cap of ~$30B. The locked in utilization rate from a customer the size of Meta means that their buildout will be easier to finance. But the risks here lie in the execution timelines for datacenters and availability of power to keep them running.

(via Nebius)

Have a great weekend!

With the introduction of the Groq LPX chips, Vera-Rubin has 7 different kinds of chips that can all be integrated into racks that comprise the entire system. The keynote highlighted where each of these racks go in the entire pod, which I color-coded as shown below. This pod architecture reveals quite a few interesting details which we will explore in this post.

We will discuss:

• How CPU-GPU-LPU ratios are scaling

• Understanding disaggregated decoding using LPX

• The new frontier of inference speed unlocked by LPX

After the paywall:

• What went unnoticed at GTC: Speculative decoding on LPX

• Does VLIW really matter after all?

  ---

  Upgrade to a paid subscription to get full access to deep dives like one. You can expense it too!

If you are not a paid subscriber, you can purchase just this article using the button below. You can find the whole catalog of articles for purchase at this link.

Buy the ebook

Counting Compute: CPU-GPU-LPU Ratios

The CPU utilization comes from two places: (1) 2 Vera CPUs per GPU tray, and (2) 8 Vera CPUs per CPU tray. From Jensen at GTC:

We never thought we would be selling CPUs standalone. We are selling a lot of CPUs stand-alone. Already, for sure, [this is] going to be a multi-billion dollar business for us.

Knowing the configurations of the xPU trays and construction of the superpod, let’s calculate how compute scales:

• 2 Vera CPUs per 1U GPU tray, 18 GPU trays per rack, 16 racks = 576 CPUs.

• 8 Vera CPUs per stand-alone CPU tray. There are 32 CPU trays per rack, and two stand-alone CPU racks. That gives a total of 8 CPUs/tray × 32 trays per rack × 2 racks = 512 CPUs.

Thus, the total estimated Vera CPUs per super pod is 576 + 512 = 1,088 CPUs. There are 4 Rubin GPUs per 1U compute tray, 18 trays per rack, 16 racks = 1,152 GPUs.

There is a third location where CPUs are needed. For every Groq tray containing 8 LPX chips, there is another CPU host that needs to be accounted for – the host CPU. The picture below shows the LPX rack on the right, with 32 compute trays per server, and ten such racks in the pod. So let’s add these chip counts in:

• LPU: 8 LPUs/tray × 32 trays/rack × 10 racks = 2,560 LPUs

• CPU: 1 CPU/tray × 32 trays/rack × 10 racks = 320 CPUs

Interestingly, the CPU in the LPX tray is not a Vera CPU because they likely need x86 chips from the GroqNode architecture that used dual socket AMD EPYC CPUs.

All in all, we have the following compute units now: 1,152 GPUs, 1,408 CPUs, 2,560 LPUs. In our earlier discussion below about the rise of CPUs for agentic AI, we predicted that CPUs might exceed GPUs at the rack scale. Additional CPU racks are straightforward to deploy since they only need ethernet connectivity, not the high-bandwidth interconnects required for weight transfer.

A final detail that requires a mention is that the LPX compute tray contains an FPGA (likely Intel/Altera), whose main purpose is to provide the logic operations to expand the memory fabric to DRAM. The LPX supports up to 256 GB DRAM expansion via the FPGA, and up to another 128 GB via the host CPU DRAM. My guess is that the first tier of DRAM uses an FPGA to retain the memory bandwidth as much as possible when data is offloaded to it. Relying on the scheduler on the CPU host is almost always slower. It is not entirely clear if the FPGA also does the orchestration of Groq LPUs.

Disaggregated Decoding Using LPX

In a previous article we published pre-GTC, we discussed the implications of SRAM-based accelerators for decode operations. In the light of more Groq details, we can extend this discussion further.

At GTC, Jensen was specific on the use-case of SRAM accelerators for the feed-forward network operation of the decode process. In September 2025, the GDDR7-based Rubin CPX was announced for disaggregation of prefill, but that narrative has largely been lost with a lot of confusion if LPX has replaced it instead.

But this is comparing apples and oranges: the LPX is bandwidth optimized hardware for decode while the CPX was designed to be more FLOPs-heavy for prefill. The real question is if separate hardware like CPX is required for prefill, or if it can just be handled by the HBM-based Rubin. Now, the decode phase is disaggregated further.

The decode phase involves the calculation of attention, followed by the Feed-Forward Network (FFN). HBM-based Rubin GPUs are the best candidate for autoregressive attention calculation because the tokens are generated one after the other, in sequential fashion. The KV cache grows with input tokens and is best stored on HBM, and as a result, the demand for HBM will continue to stay.

The FFN calculation, on the other hand, is a giant matrix multiplication that lends itself to parallelism. The matrix can be sliced into smaller pieces across many chips (tensor parallelism), computed independently, and reassembled later. MoE models take this further because each expert can live on its own set of chips (model parallelism), so you don’t need one chip to hold everything.

The latter FFN/MoE phase is what the Groq LPX is meant to address. Nvidia has put in a high density of eight chips per LPU tray because each Groq 3 has only 500 MB of SRAM. So 2,560 LPUs in a super pod gives 1.28 TB of SRAM capacity at 150 TB/s memory bandwidth per chip (~384 PB/s at rack scale). That gives plenty of capacity to hold the FFN weights of a large model, which incidentally occupies two-thirds of all weights in a model.

The transfer of weights and KV cache between the attention calculation on Rubin GPU and the low-latency decode hardware on LPX is done with Spectrum-X ethernet, although Infiniband could be used as well presumably.

A New Frontier of Inference Speed

The addition of LPX racks unlocks a new token generation speed that was previously impossible with just GPUs alone. The graph below shows the token throughput on the y-axis - a measure of how many tokens/sec can be generated (normalized to power), versus interactivity on the x-axis - a measure of the user experiences speed in terms of tokens/sec available per user. Ideally, we want to be on the top right of the chart, but this is usually not possible due to simultaneous compute and memory bandwidth limitations.

The introduction of LPX pushes the frontier of what is possible in terms of inference speed, with an advertised 1,000+ tokens/sec/user. This is why the licensing of Groq technology was a critical piece of technology necessary for Nvidia; one that they did not have with their GPU family of chips. Most importantly, Jensen explains how to deploy AI infrastructure based on expected workloads:

• For a lot of high throughput activity that does not require low latency → stick to Vera-Rubin NVL72

• For low-latency, “time is money” kind of applications like coding and engineering → deploy up to 25% of LPX along with Vera-Rubin.

In this context, it is not clear what 25% means: number of chips? Total memory? A better metric is to understand how many LPX racks should be deployed to set a desired interactivity level.

Speculative Decoding on LPX

To be honest, I was expecting Nvidia’s announcement at GTC to be centered around the acceleration of speculative decoding (SpecDec). Instead, it was more of a plain vanilla attention/FFN disaggregation. This is likely because SpecDec is an inference technique crossing from research to early production. Meta, Groq, and a few other labs have looked at it in detail, but it is far from standard infrastructure at the moment. However, I do believe that LPX has massive benefits for SpecDec which was completely unaddressed in the keynote.

After the paywall, we will discuss what Nvidia did not:

• What went unnoticed at GTC: Speculative decoding on LPX

• Does VLIW really matter after all?

Read more

• Substack: Why Optical Circuit Switching is Arista Networks’ Long-Term Problem

• Semi Doped Podcast: The Great Optics-Copper Crossroads

Let’s get to the news. Lots of exciting stuff.

META: Four MTIA Chips in Two Years

Meta recently announced four Meta Training and Inference Accelerator (MTIA) chips that they are already using in deployment, or will roll out within the 2026-27 timeframe. These are chips developed in partnership with Broadcom (their custom accelerator business is looking pretty good), and is part of Meta’s strategy to vertically integrate their chip technology without relying on external supply from Nvidia, which allows them to save on infrastructure spend. Here is a quick rundown of what each chip is.

• MTIA 300: Used already in ranking and recommendation (R&R) → 1 compute chips, 2 network chips, HBM stacks

• MTIA 400: Next gen R&R chip + GenAI → 2 compute chips, 2 network chips, 1 PCIe chiplet, more HBM stacks and bandwidth

• MTIA 450: GenAI optimized → 2 compute chips but faster, 2 network chips, 1 PCIe chiplet, 2x HBM bandwidth over MTIA 400

• MTIA 500: GenAI beast → 4 compute chips (in 2x2), 2 network chips, 1 PCIe chiplet, more HBM capacity, 50% faster HBM bandwidth over MTIA 450

Meta’s approach to AI hardware is to build systems composed of smaller parts, like chiplets, that can be individually upgraded in quick succession as model architectures and numeric formats evolve. Meta is clear that they are adopting an “inference-first” approach to meet the growing GenAI inference demand.

Finally, MTIA uses a PyTorch-native approach to programming models which is a way of saying that models running on Nvidia hardware (using PyTorch on top of CUDA) can easily be ported over to MTIA with no rewriting. This allows Nvidia and MTIA hardware to work alongside each other with minimal friction.

(via Meta)

AVGO: Broadcom delivers Industry’s First 400G/lane Optical DSP for Next-Generation AI Networks

The Taurus BCM83640 is a 400G/lane optical DSP that builds towards 3.2T networking technology, especially when the TomaHawk 7 chip with 204.8 Tbps switch bandwidth is released. I particularly like the figure below that explains how 400G/lane is useful with both 1.6T and 3.2T optical technology.

With the 1.6T generation, the optical DSP has a 8-lane to 4-lane gearbox that now only requires 4-lanes running at 400G/lane in a transceiver. With the transition to 3.2T, all 8 lanes run at 400G - no gearboxing required. According to Broadcom’s product page:

The BCM83640-DIE line (optical) transmitter supports multiple advanced 400G/lane laser optics, including a thin-film lithium niobate (TFLN) modulator, differential electro-absorption modulated lasers (D-EMLs), and advanced highbandwidth silicon photonics (SiPh) modulators.

It’s not clear to me why there is both a TFLN modulator and an electro-absorption modulator, but clearly TFLN’s high modulation bandwidth is very attractive in 400G networking. This DSP is built on TSMC 3nm and still uses IMDD for DR4 and FR4 reach. It is interesting to see when Coherent Lite will make an entry.

(via Broadcom)

UMC, HyperLight Team up to Mass-Produce TFLN Chiplets, Targeting 1.6T Data Center Bandwidth

Thin Film Lithium Niobate (TFLN) is going to become increasingly important for 1.6T networking generations and beyond because of the extreme modulation bandwidths possible. While the best InP EMLs for 200G/lane have a bandwidth of 60-67 GHz, TFLN modulators from Hyperlight exceed 110GHz. This is important for 400G/lane modulators, and is the same reason Broadcom’s recently announced 400G optical DSP uses TFLN (and D-EML?).

Hyperlight has a long standing relationship with Wavetek, and has ramped up HVM production of TFLN on 6” wafers. Wavetek is a fully owned subsidiary of United Microelectronics Corporation (UMC) - a major pure play silicon foundry. With parent company UMC getting into the mix, they are now looking to scale production to 8” wafers. In contrast, InP is still stuck on 3” and all talk of 6” wafers leads directly to supposed yield problems. TFLN on 8” sounds pretty good, and if yields/cost is good, are they a potential candidate for 200G applications in the face of InP shortages? Maybe some optics experts can weigh in, in the comments section.

(via TrendForce)

NVIDIA and Nebius partner to scale full-stack AI cloud

From the press release:

NVIDIA will invest $2 billion in Nebius, reflecting NVIDIA’s confidence in Nebius’s business and unique depth of engineering expertise across the full AI technology stack.

To enable Nebius to deploy more than 5 gigawatts of capacity by end of 2030, NVIDIA will support Nebius’s early adoption of the latest generation of NVIDIA’s accelerated computing platform.

The industry sentiment around this is generally positive because it provides Nebius early access to Vera Rubin hardware, additional cash flow and a positive endorsement from Nvidia whose $2B investment signals that they believe in Nebius’ future as an AI infrastructure provider. This is line with other investments Nvidia has made with Lumentum, Coherent, and CoreWeave recently.

The circular nature of deals made is a concern to many investors: Nvidia invests in Nebius, Nebius buys Nvidia hardware. The other point of concern is the growing debt Nebius is taking on — from nearly zero debt in 2024, to a total liability of $5.3B by the end of 2025. Nvidia’s $2B will help offset this, but the concern is that debt will grow as Nebius builds out towards 5GW of capacity.

(via Nebius)

Ayar Labs CEO Mark Wade: “Mass production of SiPh still faces many challenges”

In a DigiTimes exclusive interview, Ayar Labs CEO Mark Wade stated that the silicon photonics as a technology is ready, but mass producing it at scale is a difficult problem. I particularly like Wade’s response when asked what he thinks of the transition to optical in light of Hock Tan’s comments that copper is still relevant (emphasis mine):

SerDes and electronic transmission remain the dominant technologies today. Considering that many companies must comment on developments over the next few quarters, their statements are understandable. Our focus, however, is on what will happen in 2028, 2029, and 2030.

The SiPh supply chain’s mass production capability is currently not fully ready. But the pace of improvement in manufacturing readiness is accelerating. Together with our partners, we are actively ensuring the supply chain gradually comes into place to achieve our production targets.

I also agree that we will not wake up one day to find copper interconnects have completely disappeared. The transition will be gradual. Around 2028, GPUs and accelerators will begin adopting optical interconnects, and more AI infrastructure will rapidly shift toward optical communications. Statements from companies like Broadcom or Nvidia about copper remaining in use simply highlight that mass production of SiPh still faces many challenges.

(via DigiTimes)

eXtra-Dense Pluggable Optics (XPO): Liquid Cooled Insanity

A single sentence on Arista’s blog caught my attention.

XPO density is a game changer. A single XPO module replaces 8 OSFP modules.

The 8 OSFP connectors you see on the right are pluggables that go into ports in the optical switch. Now, the XPO module form factor provides 12.8 Tbps in a single plug, and it needs liquid cooling too. If you need a better introduction to what all the optical terminology means, check out my earlier post.

Arista goes on to explain the benefit XPO brings quite nicely.

In short, XPO allows customers to build large AI data centers with one quarter the switch racks. This is hugely important for both scale-up and scale-out applications, where without XPO the number of traditional switch racks would exceed the number of GPU racks.

Imagine a 400 MW AI datacenter with 1024 GPU racks of 128 GPUs each for a total of 128,000 GPUs. Assume 12.8T scale-up and 1.6T scale-out bandwidth per GPU. With OSFP switch racks that have a density of 1.6 Pbps per rack, this would require more than 1400 switch racks for scale-up and scale-out fabrics. With XPO, this would require 75% fewer racks, saving over 1050 racks or 44 % of the floor space.

More speed, fewer racks is good. Pretty sure you’ll see this at their OFC booth if you happen to stop by.

(via Arista blog)

Have a great weekend! NVIDIA GTC and OFC 2026 are both next week. Prepare for an explosion of new information! 💥

The hypothesis went something like this: if networking went to CPO, then Broadcom’s switching silicon business (even with CPO enabled) is open to disruption by optics-native solutions like Optical Circuit Switches (OCS). Copper staying alive keeps the game on Broadcom’s turf. This has implications further down the stack for companies like Arista Networks , who buy Broadcom’s Tomahawk series switches for their “blue-box” networking solutions, then layer their proprietary software (EOS) on top.

Later in the week, FundaAI mirrored a similar sentiment in their weekly research note:

If NVIDIA’s ecosystem were to move scale-out networking toward CPO at a large scale, it would likely be negative for Broadcom. On one hand, it could allow NVIDIA to pull more of the switching and optical stack into its own system architecture, weakening the role of merchant switches such as the Tomahawk family. On the other hand, widespread CPO or NPO deployment would structurally reduce the value of traditional pluggable optics, DSPs, and retimers — segments that are part of Broadcom’s existing profit pool.

FundaAI argues that the impact of near-term CPO deployment goes beyond just Tomahawk switches for Broadcom and affects a whole swath of other products geared towards faster copper interconnects such as DSPs, retimers, and pluggable optics. This directionally extends my original displacement theory.

In this post, we go beyond just the copper vs. optics interconnect discussion and look at a higher tier of the AI networking infrastructure. Specifically, we address whether OCS poses a structural threat to blue-box networking solutions like Arista’s, which are especially poised for an explosive growth period in 2026.

For free subscribers:

• The Structural Threat: Introduction to Optical Circuit Switching (OCS) as a long-term threat to Arista Networks’ proprietary “blue-box” networking solutions.

• Arista’s Moat: How its software prowess as a hardware company creates a moat

• Rise and deployment of OCS: Details on how OCS works and its advantages. How Google successfully deployed OCS, achieving 30% capex and 41% power reduction.

For paid subscribers:

• Protecting the Moat: Analysis of why OCS cannot easily slot into a spine switch’s role yet.

• Near-Term Outlook/Long-Term Contraction: Discussion of Arista’s strong near-term growth driven by AI accelerator deployments, and the real threat to Arista’s high-margin spine switch business from OCS.

• Future Scenarios: Three possible ways optics could play out for Arista.

• Sell-Side Critique: Commentary that underplays the significance of the OCS threat.

If you are not a paid subscriber, you can purchase just this article using the button below. You can find the whole catalog of articles for purchase at this link.

Buy the ebook

Arista’s Software Moat

Arista is a networking company that buys merchant silicon from Broadcom, layers its Extensible Operating System (EOS) on top, and works with contract manufacturers like Jabil, Sanmina, and Foxconn to build the networking switch hardware that goes into datacenter racks. The important thing to note is that Arista designs no silicon. They differentiate entirely on software.

Arista’s EOS software is the moat — but it is a packet switching moat: one that is most prone to be displaced by optical circuit switching (OCS) in the medium to long term. Every piece of data flowing through an Arista switch gets broken into packets, and the silicon inspects each one to decide where it goes. At scale, this creates enormous software complexity: congestion control, load balancing, real-time telemetry across hundreds of ports. That complexity is what Arista is really good at, and what customers pay a premium for. Arista’s “blue-box” approach packages a single software platform (EOS) that works across its entire product portfolio. This is an important moat because customers can deploy network solutions at scale with minimal software engineering effort. The added software layer makes bluebox switches costlier, but turnkey.

On the other hand, companies like Edgecore and Celestica provide “white-box” networking solutions: the networking hardware with only a primitive software layer that supports open-source network operating systems like SONiC, leaving the software implementation to the user. White box solutions are substantially cheaper and allow customers to bring their own software if they already have it, or have the means to develop it. Meta, for example, runs their FBOSS software on whitebox hardware. It also avoids vendor lock-in.

Arista is essentially a software company with low capital expenditure, and the business is doing great with the recent earnings call guided $3.25B in AI networking revenue for 2026, doubling YoY. Customer concentration is high, with 42% of revenue coming from just two major hyperscalers (likely Meta, Microsoft). But when it comes to Arista, there are a few important things to note:

1. Their spine switches are the high ASP, high margin product line. The EOS software is most useful when the complexity is high, like in spine switches that have >500 ports.

2. The rack switch needs to be updated for every networking generation because it relies on the capabilities of the underlying silicon inside. For example, Broadcom’s Tomahawk 6 supports a total data bandwidth of 102.4Tbps, while the upcoming Tomahawk 7 is expected to double that.

3. The price premium of Arista switches becomes less defensible at lower network layers. For example, Meta is one of Arista’s biggest customers while also having their in-house FBOSS networking solution. Meta likely uses a hybrid white/blue-box approach where they use lower cost switches in the leaf/top-of-rack layer, but blue-box for spine switches.

If you need an introduction to leaf/spine and networking/interconnect architecture, check out an earlier deep-dive. If you want to go deeper into Arista’s business model, Chipstrat has a nice explanation.

The Rise of Optical Circuit Switching

Optical circuit switches do not deal with data packets in the electrical domain. Instead, they simply redirect light from the input port to the output power using a variety of methods such as MEMS mirrors (Google/Lumentum), liquid crystals (Coherent), silicon photonics (iPronics), piezoelectric actuators (Polatis) or even robotic manipulators (Telescent). Think of OCS as a way to simply steer light in the right direction, and with talk of the emergence of CPO, it allows the entire routing domain to be purely optical while the conversion to electrical signals happens right at the silicon level with the co-packaged optical engine.

One major advantage of an OCS switch is that it will support multiple generations of interconnect speeds like 800G, 1.6T, 3.2T, and so on. The reason is that only light is being rerouted, and the OCS does not really care how fast the signal encoding is. This inherently makes OCS future proof compared to silicon switches that need upgrading with every networking generation.

Google was the earliest to deploy OCS in their Jupiter network architecture of which the OCS part was called Project Apollo. Their early deployments were based on in-house MEMS OCS switches, but of late Google has been switching to commercial MEMS OCS providers like Lumentum. Lumentum’s last earnings call stated a $400M order backlog in OCS hardware that was net-new to the company, from three different customers. Coherent is also increasing customer engagements with a growing backlog of OCS QoQ.

The reason Google is an early adopter is their key early insight in 2022 was that traffic patterns on long timescales are quite predictable and traditional leaf-spine architectures are typically overkill. Because traffic is stable, Google could get away with the downsides of OCS technology because network reconfigurations are rarely required. And when actually necessary, the switching speed from the network switch need not be as fast as what silicon offers, which is good for OCS because many of the technologies currently used are slower. Google’s software prowess and unique network architectures have enabled them to steadily deploy OCS their spine network, including in 3D torus configurations of TPU clusters.

Although Google is currently using OCS in production, adoption in large scale deployments does not go much further than that at present. The other customers in Lumentum’s $400M OCS backlog could very well be Microsoft, Nvidia, or Meta, given their recent participation in Open Compute Project’s recently announced OCS project. Uses for OCS goes beyond spine switches too – with applications emerging in backup pooling, scale out, and when everything goes optical, possibly even scale-up. But by all practical measures, OCS is still in its early days and there are still challenges to be solved.

After the paywall:

• Protecting the Moat: Analysis of why OCS cannot easily slot into a spine switch’s role yet.

• Near-Term Outlook/Long-Term Contraction: Discussion of Arista’s strong near-term growth driven by AI accelerator deployments, and the real threat to Arista’s high-margin spine switch business from OCS.

• Future Scenarios: Three possible ways optics could play out for Arista.

• Sell-Side Critique: Commentary that underplays the significance of the OCS threat.

Read more

Here’s some content from the past week in case you missed it.

• Substack: GTC 2026 Preview | Implications of Nvidia’s SRAM-Decode Hardware on the Inference Market

• Semi Doped Podcast: Optical Supply Chain: What would you buy? (also available on all podcast streaming platforms). We have a podcast being edited about the copper vs optical debate this week, which should be out shortly.

Lots to chatter about today. I’m experimenting with a slightly shorter format inspired by Construction Physics’ reading list posts. Let me know how you like it.

Nvidia’s $2B Optics Investment

Nvidia has invested $2B each into Lumentum and Coherent which are primary suppliers of lasers for datacenter topics. This investment is bullish for optics and is a direct signal that AI hardware makers are taking optics seriously. A lot of people assumed that this investment is to guarantee lasers for scale up optics, which has been all the talk of town ever since Lumentum’s last earnings call. In reality, Nvidia might be securing their CW laser supply for CPO or even optical circuit switches (OCS) for which Lumentum reported that they have a $400M backlog. Coherent serves as a great second source for all this.

(via The Next Platform)

Broadcom Call: Copper vs Optics

The earnings call was bullish with lots of revenue from AI semiconductors, but Hock completely stirred the hornet’s nest with this statement.

But I’m talking about scaling up in a rack, in a cluster domain. You really want to use direct attached copper as long as you can. And we are still based on our technology that Broadcom has with -- especially on connecting XPU to XPU or even GPU to GPU, we can do it with copper, and we can push the envelope from 100G to 200G to even 400G. We have SerDes now running 400G that can drive distance on a rack to run copper. Well, all I’m trying to say is you don’t need to go run into some bright shiny objects called CPO, even as we are the lead in CPOs. CPOs will come in its time, not this year, maybe not next year, but in its time.

While there has been so much news related to scale up optics, Hock’s view that copper will stay prevalent for scale up is a contrarian one. The claim that they have lab results at 400G/lane implies that copper scale up will continue into the 3.2T networking generation. The dissonance of large laser orders and the insistence of copper for scale up has confused many (including me). Maybe both copper and optics will coexist for a long time. I will plan on a deep-dive to investigate further.

(via Broadcom)

Credo Emphasizes Reliability and Long Life for AEC

’s recent earnings call showed strong numbers from the specialist cable company. They doubled revenue from 2024 to 2025, and are expected to triple revenue in 2026. That’s 6x growth in two years. Following the money shows the copper is well and alive. They claim that 200G AEC is ready and has a reach of 5m. Their foray into optics is bring their “Zero Flap” system of link health and telemetry for scale out optics cables which is still the dominant method of hooking up racks. They claim that their Active LED cable (ALC) will fill the gap between copper and optics, with the cost and energy efficiency similar to copper, but with reach extending to 10m short-term, and eventually 30m.

(via Credo)

Micro-LED co-packaged optics cut power consumption to just 5% that of copper cables

TrendForce reports that micro-LED co-packaged optics (CPOs) achieve energy consumption of just 1-2 picojoules per bit compared to copper’s 10+ pJ/bit. The architecture integrates sub-50 micrometer micro-LED chips with CMOS driver circuits, delivering integration density exceeding 0.5 Tbps per square millimeter and supporting 800 Gbps and 1.6 Tbps transmission standards. Taiwanese manufacturers AUO, Innolux, and PlayNitride are building vertically integrated micro-LED production capabilities, positioning themselves as key suppliers for this emerging technology.

(via semiconductor-today)

Ayar Labs Closes $500M Funding Round Backed by Nvidia and AMD

Ayar Labs raised $500 million in a Series E round led by Neuberger Berman. The round values the co-packaged optics startup at $3.75 billion and brings total funding to $870 million. The company plans to accelerate development of CPO using the new funds to scale high-volume production and test capacity, and expand global operations. Their TeraPhy UCIe compliant solution is an optical I/O chiplet that can be dropped in next to a GPU or switch ASIC, enabling optical GPU-GPU or GPU-Switch links for AI scale up.

(via Ayar Labs)

INTC Chair of the Board Frank Yeary retires; Dr. Craig Barratt Takes the Helm

Frank Yeary’s retirement resulted in much rejoicing on X due his proposition a few years ago to split Intel’s business into product and manufacturing parts, and then sell off the manufacturing part. Many disliked this idea because Intel is a legendary company responsible for a large portion of the chip innovations in use today, and is the last bastion of US leading-edge semiconductor dominance. Dr. Barratt, who replaces him, has an engineering background and has decades of experience in Intel and Qualcomm. He is widely known for leading Atheros Communications, which was eventually bought by Qualcomm. Dr. Barratt seems to hold that view that Intel should be a world-class IDM.

(via Tom’s Hardware)

INTC Reportedly Reconsiders 18A for External Use, Hinting EMIB Could Generate Billions by 2H26

Intel CEO Lip-Bu Tan has reversed course on the 18A node, now viewing it as viable for external foundry customers after initially planning to reserve it for internal use and position 14A as the primary foundry offering. The customizable 18A-P variant is reportedly drawing strong interest. Yields remain limited but are improving monthly. Intel reaffirmed the 14A timeline with risk production in 2027 and volume in 2029. Perhaps most notably, Intel’s EMIB advanced packaging technology could generate billions in revenue starting as early as H2 2026, with reports suggesting NVIDIA may adopt EMIB for future GPU production.

(via TrendForce)

Memory stocks drop as US-Iran war threatens infrastructure and supply chains

Samsung Electronics and SK Hynix fell 10% as the Kospi index sank over 12% amid escalating US-Iran conflict. The selloff was driven by fears that prolonged conflict could disrupt energy supplies and shipping routes — nearly 20% of global oil flows through the now-closed Strait of Hormuz. Some news outlets mistakenly associated this with the SRAM-decode hardware expected to be announced at Nvidia GTC 2026 to be held next week.

(via SCMP)

Stuff I enjoyed reading this week

• Broadcom Makes Lasers? — Austin Lyons has a nice piece comparing laser technologies between Broadcom, Lumentum, and Coherent.

• Broadcom Is Not Likely To Fly On Those Big 2027 Forecasts — BTH has a skeptical view on AVGO’s recent earnings call, especially pushing back on Hock’s comments on the Customer Owned Tooling (COT) threat.

• March Madness — Irrational Analysis is fun to read as always. I admire that he reversed his stance on optical clocking for Groq chips. More people should do this; stuff is complicated and its okay to change your mind.

• The Cost of Staying — Amy Tam on X has an interesting thought piece on the opportunity cost of staying in a job you don’t like.

• QCOM Robot — I had quite a laugh at what happens at the end of this video.

Finally, an interesting plot, via @JoshKale on X.

Have a great weekend!

The company is designing a new system for “inference” computing, a form of processing that allows AI models to respond to queries, according to people familiar with the plans. The new platform, set to be revealed at Nvidia’s GTC developer conference next month, will incorporate a chip designed by startup Groq.

The announcement is even more anticipated because Jensen compared Groq to Mellanox in the last earnings call, implying that just as Mellanox made Nvidia a networking company in 2020, Groq will transform Nvidia into an inference infrastructure company. Since Nvidia acquired Groq for an eye-watering $20B on Christmas Eve last year, the industry has questioned why Nvidia would choose an SRAM-based architecture with a notoriously challenging compiler especially when many believed Groq’s approach to be sub-optimal. With the rise of agentic AI, model providers like OpenAI have complained that existing systems are too slow. The acquisition of Groq is Nvidia’s answer to inference speed.

Here, we will explore how inference is actually two distinct hardware problems: prefill and decode. Prefill was addressed by the Nvidia CPX chip announcement last year. The upcoming Nvidia system will tackle the latter: specialized decode hardware, dubbed by some as Nvidia LPX.

After the paywall, we will discuss what might be announced at GTC 2026, mid-plane PCB rumors for inference accelerators, what it means for HBM, list of inference startups ripe for acquisition, what AMD needs to do immediately, and Groq’s achilles heel that SRAM-solutions must beat.

If you don’t have a paid subscription, you can purchase just this article in epub and pdf formats using the button below. Check out all the reports available in this link.

Buy the ebook

Different Hardware for Training vs Inference

The AI age has focused on training first, which requires lots of matrix multiplications and high-bandwidth memory (HBM) to store, update, and backpropagate weights. Using the same hardware for inference is suboptimal because inference has different bottlenecks. Disaggregated inference today revolves around three phases, each requiring different hardware and software: prefill, decode, and orchestration. Chiplog has a nice historical piece on how it came about.

Prefill processes all input tokens simultaneously. Weights are loaded once and multiplied against the entire input sequence in a batch matrix-multiply to create the key-value (KV) cache. This is compute-bound, not memory-bound. Nvidia’s Rubin CPX targets only prefill and uses GDDR7 instead of HBM because memory bandwidth is not the bottleneck.

Decode generates tokens sequentially because of the autoregressive nature of transformers; each token depends on all previous tokens. Since model weights are read from memory for every single token generated, this phase is bottlenecked by memory bandwidth and not compute. GPUs with HBM do not make the best decoding engines because SRAM bandwidth is an order of magnitude higher but with a capacity tradeoff. We have discussed the SRAM-HBM tradeoffs extensively before. Currently, Nvidia’s decode solution is HBM-based and the acquisition of Groq and its IP eliminates the memory bandwidth issue.

Orchestration ties it all together. Nvidia’s Dynamo handles KV-cache movement between prefill and decode, and manages KV-cache evictions to context storage when it outgrows its current tier. In the agentic AI era, the CPU plays an increasingly important role by controlling inputs to prefill, processing outputs from decode, and managing all the software API calls via the Dynamo NIXL library that handles asynchronous communication between compute blocks.

Why LPUs Work for Decode

Groq’s LPU uses SRAM and a Very Large Instruction Word (VLIW) architecture where the hardware makes no runtime decisions, which makes it fast and deterministic to the last clock cycle. The real technological moat lies in orchestrating everything correctly with a cutting edge compiler.

The eight chips in a GroqNode are connected in an all-to-all fashion using short wires whose exact delay is well understood. Then each clock cycle, the chip executes one very wide preconceived instruction that simultaneously controls every functional unit on the chip, for example: “On this cycle, unit A performs this multiplication, unit B moves this data from SRAM bank 3 to bank 7, the network port sends this packet to chip 14.”

Such an operation is well suited to perform the fundamentally simple, sequential task of generating output tokens. In contrast, a general purpose GPU has to make thousands of runtime decisions like when to get data from memory, how to schedule threads, how data is routed between cores, and when operations complete. The flexibility is great but largely overkill, and arguably even bad, for a fixed decoding task. The fixed operation of an LPU allows the full use of the advertised SRAM bandwidth (80 TB/s) without any resource contentions and variability from cache hits/misses.

This is a capability that Nvidia did not have before the acquisition of Groq, and it adds the ability to control the dataflow within a compute block - that in spirit, is similar to systolic arrays in TPUs. The compiler is hard, but it is a solved problem and Nvidia also got the team that solved it. Groq’s chips still run on 14nm technology, and when ported to more recent leading logic nodes, can have much more SRAM per chip.

If you want to go deep into LPU architecture, Confessions of a Code Addict has a great article.

Here is what is after the paywall:

• An educated guess of what the GTC 2026 will look like, LPU scaling considerations, and mid-plane PCB rumors.

• SRAM-inference impact on HBM and KOPSI correlation

• The options for the upcoming land grab in SRAM-based accelerators

• What AMD must do yesterday

Read more

When combined with a short family getaway last weekend, I’m running behind on the deep-dive for next week. I’m planning to revisit Lumentum’s tech in a bit more depth to understand their laser dominance, where their moat really likes, and for how long. I’ll let you know what I find.

As Feb draws to a close, here are some articles you might have missed this month. The CPU posts were very popular; I recommend checking them out if you haven’t.

• Lumentum: Laser Demand, OCS, CPO and Optical Scale-Up

• AI datacenters drink more water than you think

• The CPU bottleneck in agentic AI and why server CPUs matter more than ever

• The AI Datacenter CPU Yellow Pages

Links to previous 🍪 TWiC posts this month: Feb 6, Feb 13, Feb 20.

Semi Doped Podcast with had some high engagement episodes this month. Lots of positive feedback from the interwebs; available on YouTube and all podcast platforms.

• OpenClaw Makes AI Agents and CPUs Get Real

• A new era of context memory with Val Bercovici from Weka

• The future of financing AI infrastructure with Wayne Nelms, CTO Ornn

• Memory mayhem and capex madness

• Optical networking supercycle - ALL the tech you NEED to know

I had a lot of plans to do scale-across, physical AI, and other stuff in Feb, but the industry moves to fast for me to actually plan ahead. I’ll just continue winging it.

Anyway, apart from NVIDIA’s earnings call this week (spoiler: they made a lot of money), there are some other interesting things to get to.

AMD—Meta to Deploy 6GW of AI Infrastructure

The latest infrastructure deal between AMD and Meta is an exact replica of their deal with OpenAI last year - similar GPU commitments (6GW), warrant shares (10%), etc. These deals make it possible for AMD to make a land grab in the GPU space where NVIDIA holds 90%+ market share, and rapidly rise overall share in GPU infrastructure. The 10% shareholder dilution from each of these deals isn’t really a big deal considering that the last tranche will only fully vest when 6GW is purchased and AMD share prices hit $600.

An important takeaway from the recent CPU posts (linked above) on this newsletter is that the AMD Venice with its 256c/512t core count, high per-core IPC, and x86 arch makes it the Swiss army knife for reasoning or action-oriented agentic AI. NVIDIA Vera’s low core count and ARM-based architecture, and Intel Diamond Rapids’ lack of SMT puts AMD as the prime CPU candidate (BTH makes a strong case too). The upcoming CPU explosion combined with these massive deals puts AMD in a perfect position to grab a larger market share from NVIDIA.

It comes down to whether AMD’s MI450 platform is ready for first token in 2026 as AMD says, or if it will be delayed to 2027 like SemiAnalysis says.

Chips for High Speed Inference

Three inference chip startups dropped announcements in the same week, raising a combined $1B+ in venture funding: Taalas, MatX, and SambaNova. Each one is betting on an alternative architecture to Nvidia’s GPU, but differ significantly on what the right approach looks like.

Taalas: Burn the Model into the Chip

Taalas came out of stealth with the most extreme approach of the three. They hardcode a model’s weights directly into the chip. They’re secretive about how exactly it is stored, only stating:

We basically have an architecture where we are embedding the models, and we are hard coding the models and the weights into our what we call the mask ROM recall fabric, which is paired with an SRAM recall fabric. Together, they are able to store both the model as well as do all the computations of KV cache.

Their first chip named HardCore (HC1) is built on TSMC N6 and draws around 200 watts per card. They demoed it running Llama 3.1 8B at 17,000 tokens per second. HC2, targeting up to 20B parameters per chip, is expected in summer 2026.

The obvious tradeoff is that each chip is locked to one model. Taalas says swapping to a new model only requires changing two metal layers, not a full redesign, and they claim a two-month turnaround from frozen weights to deployable PCIe cards.

You can try inferencing on this hardware on their demo platform. I tried it, and it was incredibly fast. The answer appeared nearly instantly but was extraordinarily wrong. I do understand that this piece of news might still not be in the training data, but a little web search functionality in this day and age of AI won’t hurt to bring some confidence to people trying this tech out. It even got their own company wrong!

It remains to be seen if customers will commit to “static” infrastructure deployments without the option to upgrade. As model releases have yet to reach a steady state and the use cases are still evolving, it is hard to see big commitments to deploy “hardened” LLMs at scale. It could be useful for low cost, smaller scale deployments where a current model is sufficiently accurate to provide inference needs for 3-5 years before inference chips need to be replaced with updated hardware.

MatX: SRAM? HBM? No, Both.

Typically, the choice between HBM-based accelerators (NVIDIA, Google) and SRAM-based accelerators (Groq, Cerebras) is an either/or situation which we have discussed in quite some depth. There is always a latency versus throughput tradeoff that has to be contended with; SRAM-based approaches give low latency but cannot support large models, while HBM-based approaches support large datasets, but have higher latency.

MatX’s approach is to use both to bridge the gap between latency and throughput. In an interview, CEO Reiner Pope explains how MatX broke up a large systolic array while preserving that energy and area efficiency systolic arrays are known for, and designed a freshly minted 4-bit variable precision numeric format to deliver the best AI chips at scale, even if it comes at the cost of smaller scale deployments and complexity. From Pope on X:

Mike Gunter and I started MatX because we felt that the best chip for LLMs should be designed from first principles with a deep understanding of what LLMs need and how they will evolve. We are willing to give up on small-model performance, low-volume workloads, and even ease of programming to deliver on such a chip.

They claim higher throughput on LLMs than any announced system while matching the latency of SRAM-first designs, with over 2,000 tok/s on large MoE models. And unlike Taalas and SambaNova, MatX says their chip handles training, RL, and inference — not just decode. That’s a much broader TAM play if they can pull it off.

The $500M Series B was led by Jane Street and Situational Awareness, with Marvell Technology and the Stripe co-founders participating. Tape-out is planned within the year through TSMC, with shipments targeted for 2027.

SambaNova SN50: The Incumbent Alternative

SambaNova is the most mature of the three. They’ve been shipping hardware since 2021 and are now on their third generation with the SN50 RDU (Reconfigurable Dataflow Unit).

RDUs are a clever invention because they serve to minimize the number of off-chip data movements by having the compiler physically map all the required operations into different parts of the chip beforehand. They use a combination of SRAM, HBM and TBs of DRAM integrated into the accelerator card, so that any PCIe interfaces are avoided (d-Matrix puts DRAM on the card too, along with SRAM on chip). The compiler essentially decides what goes where, and when movements should happen. This approach allows it to host models up to 10T+ parameters and support context lengths up to 10M tokens.

SambaNova is leaning hard into agentic workloads, and the three-tier memory is the reason they can. Multiple models can sit in DRAM and HBM simultaneously. When an agent needs to switch from a reasoning model to a code model to a tool-use model mid-chain, the RDU hot-swaps the active model into SRAM in milliseconds. They call this “agentic caching” — inference contexts for multiple models persist in memory so you don’t pay the reload cost every time you switch.

They raised $350M+ alongside a multi-year strategic collaboration with Intel to deliver inference solutions. SoftBank is the first SN50 customer, deploying it in next-gen Japanese data centers. Shipments start H2 2026.

The Citrini Selloff

A post by Citrini Research caused big drops in the market because it described a futuristic scenario in 2028 where pervasive agentic AI has taken over most of the software industry, displacing white collar jobs, and massively disrupting the economy. The piece is purely fictional, but the market response was not.

Perhaps the most criticized part of the piece was the AI-led destruction of the moats held by companies such as DoorDash and Uber Eats. The piece envisioned a marketplace of vibe-coded alternatives to food delivery companies which will destroy the “habitual intermediation” moat held by the incumbents.

Ben Thompson pushes back hard on this argument in a Stratechery Plus article stating that DoorDash has an incredibly powerful network effect that strongly entrenches three parties: the driver network, the restaurant network, and the customer network. The ability to provide value to the customer (more selection), restaurants (more volume), and drivers (more pay) makes it an incredibly hard business model to displace with merely vibe-coded alternatives.

A nice piece by Citadel Securities takes a much more data-driven, level-headed rebuttal to the original article. They argue that St Louis Fed data does not show that AI is actually displacing jobs, but could instead complement them — just like the introduction of Microsoft Excel did not destroy desk work jobs, but instead morphed them into a different function.

AI is also constrained by physics and finance: energy, raw materials, and manufacturing capacity could act as a natural limiter of exponentials, while if the marginal cost of compute rises above the marginal cost of human labor, then AI substitution instantly fails in an economic sense.

Everything is still speculation and the discussion on the future of AI is fantastic, but history’s lessons tells us that when productivity jumps occur, humans only work more, not less.

Would love it if you could answer this simple question. 🙏🏽

In the paywalled portion of the above post, we compared 16 different server CPUs and scored them across nine different metrics to assess their suitability towards reasoning or action-oriented workflows. In this post, we will show the work behind the CPU scores, discuss each rating, and provide a brief use-case recommendation.

In an ideal world, I would imagine the scoring to be based on actual workloads — something along the lines of InferenceX/ClusterMAX by SemiAnalysis. I do not have access to hardware nor the skills to actually do this. So we will compare known/reported features.

Claude Opus 4.6 was used to help collate and organize the results. A lot of the information is parsed from SemiAnalysis’ extensive reporting on the landscape of datacenter CPUs, with added information to compare CPUs across a wide range of features. AI has been immensely useful in this research and helped speed up analysis that would have taken far too long for one person to do manually. I have done my best to verify the accuracy of metrics and the ratings. Please contact me if you find errors.

Below is a list of CPUs that are covered in this article. You can access each of them out of order using the sidebar on Substack. I might continue to update/add to this server CPU database which is why this will remain purely on Substack for paid subscribers. No downloadable assets will be available.

• NVIDIA (Grace, Vera) — For free subscribers

• AMD (Venice Dense, Venice Classic, Turin Dense)

• Intel (Diamond Rapids, Granite Rapids, Clearwater Forest, Sierra Forest)

• ARM Hyperscaler (AWS Graviton5, Microsoft Cobalt 200)

• Other ARM (ARM Phoenix, AmpereOne M)

• Excluded: Google Axion (low core counts), Huawei Kunpeng 950 (too little known)

Scoring Framework

Here is a quick recap of the scoring framework. Each CPU is scored on 9 metrics using a 1-5 point scale, where higher is better. The points are later weighted depending on whether the feature is more suitable for an agentic or action-oriented workload.

1. Per-core performance: Higher clock speeds and/or higher IPC is good for both reasoning and action workloads. Fast CPUs finish tasks quickly, and watches GPU tokens closely to determine next steps.

2. Core count: Higher is better for action workloads, not important for reasoning workloads. More cores = more agents. Multi-threading helps a lot.

3. CPU-xPU interconnect bandwidth: Higher is better for long-context reasoning workloads so that large batches of tokens can be exchanged quickly; not important for action workloads that only use the GPU for short durations.

4. Memory BW and capacity: More is better for both reasoning and action workloads. Reasoning workloads will evict to lower memory tiers; action workloads need working memory for large number of agents.

5. Cache size: More is better for action workloads so that small data does not have to leave the chip. This means lower latency. Not important for reasoning workloads because of large memory traffic that won’t fit in cache anyway.

6. Performance/watt: More is better for action workloads especially if CPU counts in the datacenter grow; unimportant for reasoning workloads because power is dominated by GPU use.

7. PCIe speeds: More is better for action workloads because lots of external accesses are involved, unimportant for reasoning workloads where data exchange is mostly between CPU-xPU.

8. ISA: x86 better for action workloads only because of the maturity of the software ecosystem; either is okay for reasoning workloads where there are not a wide variety of tools being called. x86 = 5 points, Arm = 3 points, in the scoring system.

9. NUMA latency: Lower is better for action workloads because all agents will have similar memory access latency; unimportant for reasoning workloads that only use a few cores. Fewer chiplets in CPU design usually means lower NUMA latency.

Composite Scores:

• Reasoning composite is weighted on: per-core performance, xPU interconnect bandwidth, and memory BW/capacity

• Action composite is weighted on: core/thread count, cache, perf/watt, PCIe, ISA (x86 advantage), and NUMA

The composite scores are calculated based on the average scores of the metrics that matter to either workload. The score is then adjusted up or down based on market dynamics, historical trends and company specifics. This is admittedly not very scientific, but then neither is the real world when it comes to choosing a CPU.

NVIDIA

Grace

Reasoning: 3 | Action: 1 | Best Fit: Reasoning

• Cores/Threads: 72 cores / 72 threads

• Process: TSMC 4N | TDP: 200-250W (single die)

• Socket: BGA (platform-locked)

• Memory: LPDDR5X, 480 GB capacity, 512 GB/s bandwidth

• I/O: PCIe5 | NVLink-C2C 900 GB/s bidirectional to attached GPUs

Status: Available since mid-2023 | Shipping in production systems
Access: Platform-locked — only available in NVIDIA superchip configurations
Roadmap: Grace → Vera (H2 2026)

Per-core perf (3/5): Neoverse V2 cores with 1MB L2. According to SemiAnalysis, Grace has a significant branch prediction bottleneck due to which AI workloads are currently being slowed by the Grace CPUs in GB200 and GB300. Hopper used x86 CPUs, and Vera uses custom Arm cores - suggesting that the “off-the-shelf” Neoverse cores used in Grace just didn’t work out.

Core count (1/5): 72 cores / 72 threads. No SMT. Lowest core count among the non-specialty CPUs. Insufficient for action workloads. A blog post by the NVIDIA and vLLM team suggests that the CPUs could not keep up with GPUs, and needed special optimizations to keep GPU utilization high.

xPU interconnect (4/5): NVLink-C2C at 900 GB/s bidirectional. Second-best GPU interconnect, behind Vera’s 1.8 TB/s. Coherent memory access allows GPU to access CPU memory at full bandwidth.

Mem BW & capacity (2/5): 480GB LPDDR5X at 512 GB/s. Decent bandwidth but limited capacity — approximately one-third of Vera’s 1.5 TB. LPDDR5X keeps non-GPU power down.

Cache size (2/5): 114MB L3 + 72MB L2 (1MB per core). Modest total of ~186MB.

Perf/watt (3/5): LPDDR5X helps with power efficiency, and the simpler Arm ISA leads to better efficiency too, but the performance itself is not top of the line either.

PCIe (2/5): PCIe5 with limited lane count — NVIDIA focused the IO budget on NVLink-C2C, not PCIe connectivity.

ISA (3/5): ARM. Same ecosystem lock-in as Vera — typically only available as part of NVIDIA’s superchip platforms (Grace Hopper, Grace Blackwell). Reports state that NVIDIA is deploying both Grace and Vera CPUs in standalone formats as part of their recent partnership with Meta.

NUMA (5/5): Single die with 6x7 mesh, 72 cores. Very clean, uniform memory access.

Recommendation: Grace is a reasoning CPU that’s being superseded by Vera. Its NVLink-C2C gives it a clear reasoning advantage over non-NVIDIA CPUs, but if true, the branch prediction bottleneck actively slows AI workloads. It is unsuitable for action workloads with only 72 threads and no SMT. Vera is a strict upgrade in every dimension.

Vera

Reasoning: 5 | Action: 2 | Best Fit: Reasoning

• Cores/Threads: 88 cores / 176 threads (Spatial Multithreading)

• Process: TSMC N3 | TDP: Not disclosed

• Socket: CoWoS-R package (BGA, platform-locked)

• Memory: 8x SOCAMM LPDDR5X, 1.5 TB capacity, 1.2 TB/s bandwidth

• I/O: PCIe6 + CXL3 on separate IO chiplet | NVLink-C2C 1.8 TB/s bidirectional to Rubin GPUs

Status: H2 2026 (NVL72 platform)
Access: Platform-locked to Rubin NVL72 (72 GPUs + 36 Vera CPUs + 18 compute blades per rack) | Standalone Vera available through select partners
Roadmap: Grace → Vera → TBD

Per-core perf (5/5): Custom Olympus ARM core — NVIDIA’s own design, not off-the-shelf Neoverse V2 used in Grace. Supports SMT for 176 threads. 2MB L2 per core. NVIDIA claims 2x performance improvement over Grace. ARMv9.2 with SVE2 FP8 operations.

Core count (2/5): 88 cores / 176 threads. Deliberately modest — this CPU is designed to run a small number of tasks extremely fast, not hundreds of agents. 176 threads is not the best core-count for large-scale agent orchestration.

xPU interconnect (5/5): NVLink-C2C at 1.8 TB/s bidirectional — the highest CPU-to-GPU bandwidth of any CPU here. Coherent memory sharing lets Rubin GPUs read directly from CPU memory. This is the defining feature that makes Vera a reasoning CPU.

Mem BW & capacity (5/5): 1.5 TB of LPDDR5 across 8 SOCAMM modules at 1.2 TB/s bandwidth. Massive capacity for KV-cache expansion. LPDDR instead of DDR5 DIMMs keeps power low while maintaining bandwidth.

Cache size (3/5): 162MB L3 spread across 7x13 mesh, plus 2MB L2 per core (176MB total L2). Adequate for managing inference state and metadata but not exceptional for the core count.

Perf/watt (3/5): LPDDR5 helps with power but this isn’t an efficiency-first design. The GPU dominates the power budget anyway.

PCIe (4/5): PCIe6 with CXL3 on a separate IO chiplet. Available for networking and storage, but the primary GPU data path is NVLink-C2C, not PCIe. PCIe becomes the path to NVMe for KV-cache tier-3 offload.

ISA (3/5): ARM. Locks you into the NVIDIA ecosystem — Vera only ships as part of the Rubin platform (1 Vera CPU to 2 Rubin GPUs per superchip), although you apparently can buy Vera standalone. ARM ecosystem is adequate for reasoning workloads but less versatile for diverse tool execution.

NUMA (5/5): All 88 cores on a single reticle-sized compute die (3nm) with the full 7x13 mesh. Memory and IO are on separate chiplets (6 dies total on CoWoS-R), but all cores share one die. Single, clean NUMA domain for compute — no cross-die core-to-core penalties. Cleanest NUMA topology at this core count.

Recommendation: Vera is the gold standard for reasoning workloads. It scores 5 on the three metrics that matter most for reasoning (per-core perf, xPU interconnect, memory BW/capacity). Its entire design philosophy is: massive pipe to the GPU, fast cores, lots of memory, nothing else gets in the way. It scores poorly on action metrics because the core count is too low.

AMD

Read more