Why NVIDIA's Moat Is Software, and What Would Break It

Rivals have shipped comparable silicon for years without shifting the market. The moat is the software above the chips and the rack around them.

by HowAIWorks Team
On this page

Introduction

Every earnings season, someone asks why AMD cannot just catch up, and gets one of two useless answers: NVIDIA is invincible, or the moat is about to collapse. Neither is an argument.

Here is the fact that both answers have to explain. In December 2023, AMD launched the Instinct MI300X with 192 GB of HBM3 memory and 5.3 TB/s of peak bandwidth (AMD), against an NVIDIA H100's 80 GB and 3.35 TB/s (NVIDIA). More than twice the memory, sixty percent more bandwidth — and for a workload that spends most of its life waiting on memory, those are the two numbers that matter most. The market barely moved.

If silicon were the moat, it would have fallen already. So the moat is something else, and it is worth being precise about what, because the mechanism is more interesting than the scoreboard — and because it tells you exactly what to watch for if it ever starts to give way.

The Silicon Argument Fails on Its Own Numbers

The MI300X was not a fluke of one generation. AMD's MI355X carries 288 GB of HBM3E at 8 TB/s (AMD). NVIDIA's HGX B200 board holds 1.4 TB across eight Blackwell GPUs (NVIDIA) — 180 GB each, at broadly comparable bandwidth. On the specification that a memory wall-bound workload cares about, the challenger has been ahead or level for two generations running.

That is not a small thing to concede, and the industry keeps conceding it while behaving as though it did not matter. It does not matter, and the reason is stated most bluntly in the fine print of a benchmark round.

In MLPerf Training v5.1, published on 12 November 2025, twenty organizations submitted 65 systems across 12 different hardware accelerators (MLCommons). NVIDIA swept all seven benchmarks and — its own words — was "the only platform to submit results on every test" (NVIDIA). AMD submitted two: Llama 2 70B LoRA fine-tuning and Llama 3.1 8B pretraining (AMD).

Now read AMD's numbers on the two it did enter. By AMD's own account, the MI355X came within 3% of a B200 and 6% of a B300 on Llama 2 70B LoRA in FP8, and within similar margins on Llama 3.1 8B (AMD, above). Where AMD showed up, it was essentially at parity. It showed up in two places out of seven.

That gap — parity where the software exists, absence where it does not — is the moat in one picture. A benchmark result is not a property of a chip. It is a property of a chip plus every kernel someone bothered to write for it.

What Is Actually Being Sold Is a Rack

The second reason "a competitive chip" no longer settles anything is that a chip is no longer the unit of competition.

NVIDIA's GB200 NVL72 connects 72 Blackwell GPUs and 36 Grace CPUs into what NVIDIA describes as "a 72-GPU NVLink domain that acts as a single, massive GPU", with 130 TB/s of aggregate NVLink bandwidth (NVIDIA). That phrasing is marketing, but the engineering underneath it is not.

Distributed training explains why. A training step ends with a synchronisation barrier: every accelerator must agree on the new weights before the next step begins, and agreement moves bytes. Inside a server, an H100 talks to its neighbours over NVLink at 900 GB/s; across the rack, at 400 Gbps, it gets 50 GB/s — a factor of 18. The greediest form of parallelism, the one that splits a single matrix multiplication across devices and pays a collective inside every layer, can only live inside the fast domain. So the size of the fast domain is a hard architectural limit on how a model can be cut up at all.

Enlarging that island from 8 GPUs to 72 is not a bigger server. It is a bigger unit of computation — and it means the thing a lab buys is a rack, wired and cooled and running a collectives library, not a tray of accelerators. A competitor who ships an excellent chip into that market has built a component for a product it does not sell.

Which is why the challengers are now shipping racks too. At CES in January 2026, AMD detailed Helios, a double-wide rack of 72 MI455X accelerators built on Meta's Open Rack Wide specification, on track for the second half of 2026 (AMD, The Next Platform). The interconnect it scales over is an open standard: the UALink 200G 1.0 specification, ratified in April 2025, defines a load/store fabric for up to 1,024 accelerators and is backed by a consortium of more than 85 companies (UALink Consortium). The industry has correctly identified where the lock-in lives and is building a standard aimed at it. Whether the standard arrives with the software to drive it is the open question, and it is a software question.

The Moat Is a Network Effect in Engineering Practice

The technical mechanism — kernels, the memory hierarchy, why a correct kernel can be thirty times too slow — belongs to CUDA, and that page is where to read it. The strategic mechanism is a different animal, and it is simpler than the technology.

Every new idea in machine learning is born as a CUDA kernel. FlashAttention was a hand-written CUDA kernel. The inference stacks — vLLM, SGLang, TensorRT-LLM — are collections of CUDA kernels. The reference implementation attached to a paper is a CUDA repository, and the PyTorch Foundation reported that over 70% of AI research implementations now use PyTorch (PyTorch), whose most-travelled path lands on NVIDIA hardware. Ports to other backends arrive months later, if anyone bothers.

This is not a technology advantage. It is a network effect in engineering practice, and it compounds through the ordinary machinery of a profession: the tutorial you learned from, the Stack Overflow answer that unblocked you, the profiler you know the quirks of, the candidate whose CV says CUDA. Fewer users means fewer kernel authors, which means slower kernels, which means fewer users. NVIDIA claims CUDA now underpins six million developers — the figure is NVIDIA's own and should be read as marketing, but the direction it points is not in dispute.

You cannot buy this. You can buy a fab. A hiring pool has to be grown.

The most vivid demonstration came from a lab trying to escape it. DeepSeek, working under export restrictions, hand-tuned parts of V3's training by dropping below CUDA C++ into PTX, NVIDIA's assembly-like intermediate representation, in order to reallocate streaming multiprocessors to inter-server communication (Tom's Hardware). It was widely reported as breaking the CUDA moat. PTX is part of CUDA. The most celebrated attempt to get out from under NVIDIA's software went further into NVIDIA's software.

The Challengers, Fairly Stated

A page that sneers at ROCm is a page that is not paying attention.

AMD's ROCm is further along than the discourse admits. Its HIP layer is a source-level CUDA-compatibility interface, and PyTorch ships ROCm builds that run most code unmodified. ROCm 7.0, released in September 2025, provides day-zero support for PyTorch, TensorFlow and JAX, and both vLLM and SGLang now run on MI350X and MI355X with native FP4 (AMD). This is not a toy.

It is also not yet a peer, and the honest reason is cultural rather than architectural. SemiAnalysis's December 2024 teardown concluded that "the CUDA moat has yet to be crossed by AMD due to AMD's weaker-than-expected software Quality Assurance culture and its challenging out of the box experience," documenting builds that failed inside Docker and configurations that needed dozens of environment flags to reach usable performance (SemiAnalysis). Much of that specific list has since been fixed. The structural point survives the fixes: a compatibility layer chasing a moving API is always second, because the newest kernel is written for the other stack first.

OpenAI's Triton attacks from a smarter angle. You write a kernel in Python at the level of blocks and tiles, and the compiler handles the rest. Triton does not beat CUDA on NVIDIA hardware; that was never the point. The point is that it has an AMD backend in its main repository and an Intel backend maintained out of tree (triton-lang) — so the same source can target more than one vendor.

And the mechanism that would matter most is already the default. PyTorch 2.0, released in March 2023, made TorchInductor the compiler behind torch.compile, and TorchInductor generates Triton kernels rather than calling hand-written CUDA ones. If the kernel your model actually runs is emitted at compile time from framework IR, then the vendor-specific layer stops being the thing you program against and becomes a backend. Backends are swappable. That decoupling — the framework from the vendor — is the only thing that has ever dissolved a software moat, and it is not hypothetical. It is shipping.

Why the Biggest Buyers Have Not Dislodged It

The demand side has every incentive to break this, and it is trying hard.

  • Google has been building TPUs for a decade. Its seventh generation, Ironwood, was announced on 7 November 2025 with pods scaling to 9,216 chips (Google Cloud).
  • Amazon brought Project Rainier online in 2025 with nearly 500,000 Trainium2 chips, serving Anthropic's Claude models (AWS).
  • Microsoft announced Maia 200 on 26 January 2026 — 216 GB of HBM3e at 7 TB/s, running OpenAI models in Azure (Microsoft).
  • Anthropic agreed on 23 October 2025 to access up to one million TPUs and over a gigawatt of Google capacity in 2026 (Google Cloud), while also running on Trainium and on NVIDIA GPUs.
  • OpenAI and AMD announced on 6 October 2025 a partnership to deploy 6 gigawatts of Instinct GPUs, the first gigawatt of MI450s starting in the second half of 2026 (AMD).

These are not gestures. They are among the largest capital commitments in the history of computing, made by the people who would most like an alternative to exist.

So why is the default still the default? Two reasons, and neither is that the alternatives are bad.

First, custom silicon subtracts demand without creating a substitute. A TPU is excellent and you cannot buy one to put in your own building; Trainium runs on AWS; Maia runs on Azure. Each is a vertically integrated stack that serves its owner's workloads brilliantly and offers the rest of the market nothing to switch to. They prove the moat is crossable — Google's TPU stack is the standing proof — and they simultaneously demonstrate the price of admission: a decade, a compiler team, and a captive first customer.

Second, the merchant alternative still has to be paid to be adopted. The AMD–OpenAI agreement came with warrants for up to 160 million AMD shares, vesting against deployment milestones (AMD, above). Read that structurally rather than financially: the switching cost was large enough that it had to be bought down. That is what a moat looks like from the buyer's side.

What Would Have to Be True

Predictions are cheap. Conditions are checkable. Here are four things you would observe if the moat were genuinely eroding — none of which requires a forecast, and all of which are matters of public record.

  1. New techniques would ship portable on day one. Today the newest attention kernel appears as CUDA and reaches other backends later. The signal to watch is a significant new method arriving as a Triton kernel — or as torch.compile-native code — with no hand-written CUDA in the reference implementation. That is the compiler path closing on the frontier, not the average.

  2. A non-NVIDIA platform would submit across a whole benchmark suite. Two entries out of seven in MLPerf Training v5.1 is the honest measure of coverage today. Seven out of seven from a second vendor would mean the software had stopped being the constraint.

  3. Someone else would sell a rack that scales out. Not a chip that benchmarks well — a 72-accelerator domain with a collectives library that survives contact with a 100,000-device cluster, where reliability engineering matters as much as bandwidth. Helios in the second half of 2026 is the first real test of that proposition, and it should be judged on cluster-scale results, not on datasheets.

  4. A large buyer would switch on merit alone — no warrants, no equity, no strategic framing. Just a lab reporting that it moved a frontier training run to another stack because the tokens per dollar were better and the software worked.

None of the four is impossible. None is close to free. And they have to arrive roughly together, because each one is the reason the others have not happened.

Conclusion

The moat is not the transistors, and everyone who has shipped a competitive accelerator into this market has learned that the expensive way. It is not the CUDA language either, which is a modest extension of C++ that a good team could clone in a year.

It is the twenty years of tuned libraries, the reference implementation of every new idea, the rack that turns a pile of accelerators into one computer, and the several million engineers who already know how to use all of it. That is an asset accumulated in engineering-years, and engineering-years cannot be bought forward the way fab capacity can.

It is also, unlike a fab, not physical — which means it is not permanent. It erodes exactly where the compiler stops being NVIDIA's and starts being PyTorch's, and that erosion is already the default configuration of the world's most-used ML framework. Whether that is enough, and how fast, is not something anyone honest will tell you today. What they can tell you is what to look at.

For the physics underneath all of this — why memory bandwidth rather than arithmetic sets the price of a token — start with From Token to Transistor and high bandwidth memory.

Sources

Frequently Asked Questions

It has already caught up on silicon. AMD's MI300X shipped in December 2023 with 192 GB of memory and 5.3 TB/s of bandwidth against an H100's 80 GB and 3.35 TB/s, and the market did not move. What it has not caught up on is the accumulated software above the chip — the tuned libraries, the reference implementations, the kernels for the newest technique — and the rack-scale system that is now the actual unit of sale.
CUDA the language is not; a competent team could clone the syntax. The moat is what sits behind it: roughly two decades of tuned libraries, the fact that new techniques appear as CUDA kernels first and get ported elsewhere later, and a hiring pool that already knows the stack. AMD's ROCm includes a source-level CUDA-compatibility layer, so the API was never the hard part.
They are the most serious pressure on it. Google's TPUs, Amazon's Trainium and Microsoft's Maia are all real, deployed silicon serving frontier models. But each is largely confined to its owner's cloud and its owner's software stack, so they subtract demand from NVIDIA without offering the rest of the market a drop-in alternative.
The mechanism that matters most is compilation. PyTorch 2.0 made TorchInductor the default compiler, and it emits Triton kernels rather than calling hand-written CUDA ones. If the kernel your model runs is generated from framework IR at compile time, the vendor layer becomes a swappable backend — and that is the only thing that has ever dissolved a software moat.
No. Peak arithmetic and peak memory bandwidth are datasheet ceilings, and reaching them requires kernels written and tuned for that specific chip. Two accelerators with the same specifications can differ by a large factor on a real workload purely because of who wrote the software.
← Previous

Karpathy's nanochat and LLM Council: Build Your Own AI

Next →

What One AI Query Costs: Tokens, Bytes, Joules, Cents

Related Articles

Seven Chokepoints in the AI Chip Supply Chain

A walk down the chain behind one AI accelerator — design tools, foundry, scanner, memory, packaging, software, power — and why each link has no second source.

From Token to Transistor: What Happens When You Send a Prompt

Follow one prompt through the machine — tokens, matrix multiplications, the memory that everything waits on, and the chips built around a single operation.

A Business Book as Slash Commands: Sahil Lavingia's Claude Skills

Sahil Lavingia turned The Minimalist Entrepreneur into 10 Claude Code skills. What is inside the 9.5K-star repo, how to install it, and what it cannot do.

Continue Your AI Journey

Explore our glossary and model catalog to deepen your understanding.

Browse Glossary← Back to Blog