Why AI Factories Are Moving From FLOPS to Tokens per Watt: The NVIDIA Vera Rubin DSX Bottleneck

Engineering · September 18, 2026 · 12 min read

The scarce resource in AI infrastructure is no longer just GPU count. NVIDIA's September AI Infra Summit material around Vera Rubin NVL72 and DSX points to a more useful question: how many output tokens can a fixed megawatt produce after power delivery, cooling, networking, memory traffic and scheduling are all paid for?

Accelerator specifications used to be read mainly through FLOPS, HBM capacity and memory bandwidth. In September 2026, NVIDIA framed Vera Rubin and DSX with a different set of units: tokens per watt, tokens per megawatt, and the number of productive nodes that can fit inside one fixed power envelope.[1][2]

That shift is not just a marketing preference. It reflects where large AI clusters are beginning to bind. Buying GPUs does not help if the site power budget is fixed. Having nominal power does not help if cooling, conversion loss, network latency, KV-cache traffic and scheduler policy keep expensive silicon waiting instead of producing tokens.

VLSI Korea view: the AI factory bottleneck is moving from "the fastest GPU" to "how much useful token production can be sustained inside a fixed electrical and thermal envelope."

Sixteen Nodes at Full Power, or Nineteen Nodes at a Lower Cap?

The cleanest public number is Lambda's validation of NVIDIA DSX MaxLPS. NVIDIA's Lambda case study says Lambda ran DSX MaxLPS on a five-rack, 19-node HGX B200 cluster. The baseline was 16 full-power nodes. Inside the same fixed power budget, the 19-node policy increased cluster-wide inference throughput from about 4.04 million tokens/s to 5.00 million tokens/s, while improving performance per watt by 23%.[3]

The arithmetic is simple. 5.00 / 4.04 is 1.238, or about 24%. The node count rose by 19 / 16, or 18.75%. Throughput grew slightly more than node count because workload mix and power policy interacted. But the boundary condition matters: this is a validation on a specific cluster and workload mix, not a universal guarantee for every data center.

Why can nineteen capped nodes beat sixteen full-power nodes? Many GPU workloads do not draw peak power continuously. Prefill, decode, communication, memory access, kernel execution and synchronization appear at different moments. One node may leave headroom while another node has work that could use it.

Static provisioning sizes for the worst moment. Every node must be safe if many devices hit peak draw at the same time. That is robust, but it strands capacity. The DSX MaxLPS claim is that rack-level real-time monitoring can recover some of that headroom by adjusting node-level power targets and allowing more nodes to run inside the same facility envelope.[1][3]

This is not simply "turn down the GPUs." Lowering power can reduce one chip's peak performance. At the factory level, however, total token output can rise if more nodes perform useful work at the same time. Chip-level optimum and factory-level optimum are not always the same point.

A Token Factory Is a Data Path, Not One Chip

A simplified LLM serving pipeline has two big phases. Prefill reads the input prompt and can often fill the compute engine with large matrix work. Decode generates the answer token by token and repeatedly reads prior state. Decode is therefore more exposed to latency, KV cache movement, memory bandwidth and scheduling.

Agentic workloads amplify the difference. A simple chat may ask once and finish. An agent plans, calls tools, reads outputs and reasons again. Longer context and more steps increase KV-cache traffic and orchestration overhead. That is why peak FLOPS alone is a poor proxy for output token production, especially for multi-step agents.

NVIDIA's MLPerf Inference v6.1 post makes the same system point. Its Vera Rubin NVL72 preview submission reported up to 3.7x higher throughput than GB300 NVL72 on Qwen3-VL and up to 2.5x on DeepSeek-R1. Those figures are tied to MLPerf closed-division entries and scenarios. NVIDIA also pointed to vLLM, Dynamo, TensorRT-LLM, disaggregated serving, expert parallelism and lower-precision KV cache as parts of the performance path.[4]

Once compute gets faster, the bottleneck moves again. Faster Tensor Cores can shrink prefill time, making decode and KV cache more visible. Lower KV precision can reduce memory footprint, but requires quality and numerical validation. Expert parallelism can distribute MoE layers, but introduces routing and all-to-all communication. Disaggregated serving lets prefill and decode resources be placed differently, but only if the scheduler and network cooperate.

So the Vera Rubin story is not a one-chip number. It depends on rack-scale interconnect and software. NVIDIA says sixth-generation NVLink and NVLink Switch provide the scale-up domain that makes those serving techniques effective at rack scale.[4]

Scaling Efficiency Is Part of Power Economics

Another number in NVIDIA's MLPerf v6.1 discussion is 99% scaling efficiency for GB300 NVL72. NVIDIA says a DeepSeek-R1 offline submission scaled from a 72-GPU single-rack baseline to 288 GPUs across four racks with nearly linear throughput growth.[4]

That number belongs in the power discussion because power turns nodes on, but only throughput pays for them. If added racks spend energy waiting on synchronization, network congestion or load imbalance, capex and electricity rise faster than token output. Tokens per watt worsens. If four times the hardware becomes nearly four times the useful work, the power economics survive the expansion.

This is why scale-up networking is not a peripheral. When a model is partitioned across many GPUs, or when MoE experts are distributed, one output token may require multiple device-to-device exchanges. If the network is slow, the GPUs wait. Power is still consumed, but no token is produced during the stall.

In that literal sense, the network becomes part of the computer. GPU, HBM, NVLink, Ethernet, DPU, storage and scheduler must line up as one production line. The faster the chip becomes, the more important the rest of the path becomes.

Power Delivery and Cooling Are Architecture Problems

The power path is also a bottleneck. NVIDIA says DSX reference designs incorporate 800 VDC architecture. At a higher distribution voltage, the same power can be delivered with lower current, reducing resistive losses and conductor burden. The actual benefit still depends on conversion stages, rack design, safety, serviceability and retrofit cost.[1][5]

Cooling is part of the same loop. NVIDIA's DSX article says a direct-liquid-cooled GB200 NVL72 rack carries about 120 kW of heat that must be removed.[1] If that heat cannot leave fast enough, a remaining electrical budget does not mean the GPUs can keep running at high utilization. Electrical design, thermal design and workload scheduling may look like separate teams, but in an AI factory they become one control problem.

Grid response pushes the boundary one level farther out. NVIDIA says its Eos AI factory in Santa Clara, using Emerald AI Conductor under Silicon Valley Power's Flexible Load Interconnect Program, responded automatically to more than 200 demand signals. In one described case, the site reduced demand from 4 MW to 3 MW without interrupting high-priority inference.[1] That is best read as operational evidence that AI load can behave like a dispatchable resource, not as proof that every facility already has finished DSX Flex economics.

Failure condition: if a utility program does not exist, if workload priorities are not clear, or if too little work can be paused, the economics of grid-flexible AI factories become much weaker.

What Becomes Scarce in the Semiconductor Value Chain?

The first scarce resource is still advanced GPUs and HBM. But the constraint no longer ends there. High-density power conversion, busbars, connectors, cold plates, coolant distribution, DPUs, switches, optical and electrical interconnect, and the control software tying them together all become part of capacity.

For semiconductor companies, the gap between "we sold a package" and "the customer built a token production line" is widening. A supplier of high-performance silicon increasingly needs rack-level reference designs, network fabric, power architecture and management software. Memory vendors, foundries, OSATs, substrates, power semiconductor suppliers and cooling vendors become more tightly coupled.

For investors, headline FLOPS can now miss the relevant difference. The same GPU generation can produce different realized token output depending on utility interconnect, cooling capacity, power-control software and workload mix. Conversely, even if grid expansion is slow, dynamic power allocation and demand response may unlock part of the stranded capacity earlier.

The caveat is just as important. Lambda's 24% uplift is a specific validation. NVIDIA's statement that DSX MaxLPS can enable up to 40% more GPU capacity for next-generation Vera Rubin NVL72 factories is a projection for suitable deployment environments.[1][2] MLPerf results are tied to benchmark scenarios and submission stacks.[4][6] The takeaway is not one multiplier. The takeaway is the method for finding the bottleneck.

An AI factory question can no longer stop at "how many PFLOPS does the GPU have?" The better question is how one megawatt becomes useful tokens, and which byte, packet, watt or thermal path is waiting during that conversion. The bottleneck moves from chip to package, rack network, cooling loop and grid contract. That movement is the real engineering message behind Vera Rubin and DSX.

Sources

[1] NVIDIA Blog, From Megawatts to Tokens: How NVIDIA Maximizes AI Factory Production, 2026-09-15. DSX MaxLPS, DSX Flex, Eos demand response, 800 VDC, and 120 kW rack-heat context.

[2] NVIDIA Blog, AI Infra Summit: NVIDIA Vera Rubin and DSX Platform Advancements Showcase Energy Efficiencies of Optimizing Tokens Per Watt for AI Factories, 2026-09-15. Summit summary, tokens per megawatt framing, Lambda result, and Vera Rubin/DSX projections.

[3] NVIDIA Case Study, Lambda Maximizes Performance per Watt With NVIDIA DSX MaxLPS, retrieved 2026-09-18 KST. 19-node vs 16-node fixed-power result, 4.04M to 5.00M tokens/s, 23% performance-per-watt improvement, and test conditions.

[4] NVIDIA Blog, NVIDIA Vera Rubin NVL72 Delivers Leading Performance in MLPerf Inference v6.1 Debut, 2026-09-16. Vera Rubin preview throughput, GB300 NVL72 scaling efficiency, software optimizations, and benchmark context.

[5] NVIDIA, 800 VDC Architecture for AI Data Centers, retrieved 2026-09-18 KST. High-voltage AI data center power architecture context.

[6] MLCommons, MLPerf Inference v6.1 results announcement, 2026-09-16. Benchmark scope, new tests, participating organizations, and reproducibility context.

Editorial Note

Sources were checked on September 18, 2026 KST. NVIDIA and Lambda figures are taken from NVIDIA's public blog and case study materials; they are not treated as independently reproduced across all workloads. VLSI Korea original schematics explain the calculations and bottleneck structure; they are not NVIDIA internal architecture diagrams or literal data-center wiring diagrams. This article is published website-only, with no subscriber email, push, or social post.

Disclosure

This article is provided for informational purposes only and does not constitute a recommendation to buy or sell any security. Figures and factual statements are based on publicly available sources, including company filings, news reports, and market research, and may contain errors in the source material or in the preparation of this article. The author may own shares in companies mentioned. All investment decisions and their outcomes are the reader's responsibility.