Power, not heat: what limits a GPU under sustained matmul

Covers his note of Wed 21 Oct 2026 (week 43), 14:58. Written Thu 22 Oct 2026, 05:15 PDT, on the Intel.

Horace He is right for data-center accelerators run within spec: under a sustained matmul the clock is set by the power cap, not by temperature, on air-cooled and liquid-cooled parts alike. Eight H200s and eight B200s ran a matmul for 10 minutes held at their caps, 11–26 °C below their slowdown points, and no thermal flag fired; AMD's and Intel's chips behave the same way. Your guess holds, but one step earlier than throttling: the cooling decides the cap. The same AMD die is rated 1,000 W on air and 1,400 W on liquid, Gaudi 3 900 W against 1.2 kW. So an air-cooled chip shows up as power-capped lower, not as heat-throttled, and heat binds only when cooling falls short of its design. With liquid the binding limit moves out to rack and grid power, and for your experiments a power-capped GPU turns fewer bit flips into a higher clock.

1. The claim, and three ways a chip can be held back

Horace He's post, Strangely, Matrix Multiplications on GPUs Run Faster When Given "Predictable" Data! (Mon 29 Apr 2024, week 18), ran one matmul on an A100 with a 330 W limit: 295 TFLOPS on zeros, 257 on random normal data. His explanation is power alone: "power ~= clock speed * "transistor flips per clock"", and "matmuls on the H100 are primarily not compute or bandwidth limited, they are power limited." Heat, temperature and thermal limits never come up in the post. This page tests the stronger form of that view: that these chips throttle on power and never on heat.

A chip under load can be held back at three levels:

1. The chip's power cap. The board's controller trims the clock within milliseconds to keep power under its limit; NVIDIA's driver reports this as SW Power Cap. Current and voltage limits on the board act the same way.
2. Thermal throttling. Above its maximum operating temperature, for the die or the memory, the driver slows the clock (SW Thermal Slowdown). The hardware backstop, HW Slowdown, means "reducing the core clocks by a factor of 2 or more" (NVML clock event reasons).
3. The facility. Rack power, power supplies, room cooling and the grid. An operator meets these by capping GPUs, spacing racks or not plugging them in.

These levels are not independent. A chip's rated power is a thermal design number: the cooler is sized to remove it. The power cap is where the cooling's limit is written down, and it is enforced electrically and fast instead of thermally and slowly.

2. NVIDIA data-center GPUs

Two things stand out:

• Each generation sits further below its top clock at the cap. The H100 SXM is within 1% of its maximum, the H200 3–6% below, and the B200 27–29% below.
• NVIDIA rates the same silicon by its power. An H200 at 600 W is rated 16% below one at 700 W, and a B200 at 1.2 kW (GB200) 11% above one at 1 kW.

3. AMD, Intel and Google

• AMD MI300X (750 W, air) and MI325X (1,000 W, air). AMD measured BF16 matmul at 708 TFLOPS with a median 1,207 MHz on the MI300X, against a 2,100 MHz peak, and 843 TFLOPS at 1,447 MHz on the MI325X, which has the same compute units with a third more power. In AMD's words, "The power management system reduces the clock speed when running dense tensor-core logic." The same FP8 matmul ran at a median 1,792 MHz on zeros and 1,188 MHz on varied data, 51% apart (AMD ROCm blog, Fri 28 Feb 2025). This is Horace's effect, larger.
• AMD MI350X (1,000 W, air) against MI355X (1,400 W, direct liquid). These are the same CDNA 4 chip in the same OAM module (ROCm docs). On paper they are about 8% apart. AMD says the real gap is closer to 20%, because "the liquid-cooled part's larger power limit allows it to boost higher for longer" (The Register, Thu 12 Jun 2025).
• Intel Gaudi 3. The module is rated "900W ... with passive cooling and up to 1.2KW TDP with liquid cooling," and the PCIe card 600 W (Intel technical paper, Rev. 1, Sep 2024). Measured, "the Gaudi 3 reached its TDP of 900W even for thin GEMMs" and was slowed by power throttling (Kim et al., Mon 25 Aug 2025).
• Google TPUs: neither, by design. The 75 W TPUv1 and 280 W TPUv2 were air-cooled, "but the 450W TPUv3 uses liquid cooling," and "TPUs omit techniques like Turbo mode". Google provisions power and cooling so the clock never has to move (Jouppi et al., ISCA 2021).
• Where heat did bind: a small passive card. The same Google paper measured an air-cooled NVIDIA T4, a 70 W inference card, on MLPerf. It reached 75 °C in 30 seconds, then held there with the clock wandering between 0.9 and 1.3 GHz against its 1.6 GHz turbo. Over 10 minutes it lost 19–26% of its performance.

4. Consumer cards

• RTX 4090, 450 W, air. In one LLM fine-tuning sweep it drew 390 W at 72 °C at full power. Capped at 300 W it kept 93.4% of its speed at 67 °C; at 220 W, 82.6% at 58 °C (dev.to sweep, Wed 29 May 2024). NVIDIA's limit for the chip is 90 °C (RTX 4090). Power sets the speed; the heat limit is never near.
• RTX 5090, 575 W, air. The Founders Edition averaged 569 W in a 4K game, with spikes to 580–590 W. In the thermal test the GPU settled at about 72 °C at 2,600–2,700 MHz, above the rated 2,410 MHz boost, while its memory "ran warm ... at 89 to 90 degrees Celsius" (GamersNexus, Fri 24 Jan 2025). These are games and graphics tests, not a matmul, but the pattern is the data-center one: at the power cap with a cool die. The memory is the part nearest its limit.
• RTX PRO 6000 at 600 W against its 300 W Max-Q edition. The two are rated 4,000 and 3,511 AI TOPS and 125 and 110 FP32 TFLOPS: half the power for 12% less peak (RTX PRO 6000, Max-Q).

5. When heat does bind, or still matters

Within spec, temperature acts through the power cap. A hotter die leaks more current, which leaves less of the fixed budget for switching:

• In the 10-minute runs, a 10–13 °C warm-up cost 0.3 MHz per °C on the H200 and 1 MHz per °C on the B200. That is a median of under 1% of throughput, and about 2% on the worst B200 (ml-engineering).
• Your own A100 data bounds the effect at about 1% over 8 °C (21 Sep brief).

At fleet scale it shows up as a small, steady effect:

• Meta, Llama 3 405B on 16K H100s at 700 W: "a diurnal 1-2% throughput variation based on time-of-day", from "higher mid-day temperatures impacting GPU dynamic voltage and frequency scaling". Six of its 419 unexpected interruptions were GPU thermal interface or sensor faults (The Llama 3 Herd of Models, Tue 23 Jul 2024).
• Microsoft, TAPAS, on air-cooled cloud GPUs: "a range of over 20°C across GPUs in the same datacenter". Identical GPUs at the same load and inlet temperature differ by up to 10 °C, and the worst job placement passes 85 °C against about 72 °C for a typical one (TAPAS, Sun 5 Jan 2025).
• Sinha et al., five supercomputers, over 18,800 hours of measurements. "Air cooled clusters ... have a large temperature range ≥ 30°C", and their hottest nodes power-throttle by more than 15% of TDP. Water-cooled Summit spans only 40–62 °C, yet its performance varies about as much, 7–9% across GPUs as in the other clusters. No GPU passed its thermal thresholds. But the air-cooled AMD MI60 cluster ran up to 99 °C, a degree under its slowdown point, and never reached its 300 W cap; the authors conclude its clock was held by temperature (Not All GPUs Are Created Equal, Tue 23 Aug 2022).
• ClusterBench, Erlangen. On air-cooled A100s, "thermal throttling is a contributing factor". On water-cooled H100s, the GPUs reach their power limit and throttle there, with "no correlation between GPU temperature and DGEMM performance", at a median 53 ± 2 °C (ClusterBench, Tue 11 Aug 2026).
• Stragglers. In an Alibaba cluster of over 10,000 GPUs, slow computation came mainly from CPU contention and from GPUs degraded by thermal throttling, but only 6 of 392 single-node jobs were hit. Network congestion caused 13 of the 16 slow large jobs (Falcon, Wed 16 Oct 2024).

Operators write the thermal edge into their specs:

• DGX B200 is specified for 10–35 °C inlet air (NVIDIA).
• Google Cloud warns that A3 (H100/H200) machines "can experience performance degradation" if they stay above 87 °C (Google Cloud).

Head to head, one air-cooled and one liquid-cooled 8×H100 node (Cooling Matters, Tue 22 Jul 2025):

• The air-cooled GPUs (a rear-door heat exchanger) ran 54–72 °C, the liquid-cooled ones 41–50 °C.
• A stress test (GPU-Burn) did 46 against 54 TFLOPS per GPU, 17% apart.
• Real LLM fine-tuning differed by only 0.1–1.0%.
• The liquid node drew 1.0–1.2 kW less during fine-tuning.
• The two nodes also differ in CPUs, and the liquid one has half the fans.

6. The facility level

Power is the binding constraint.

• Microsoft's Azure study: "for GPU-heavy workloads like LLMs, power is the main bottleneck." LLM training clusters ran at 97% of their provisioned peak power, and inference clusters at 79% (Patel et al., ASPLOS 2024).
• Satya Nadella, Sat 1 Nov 2025: Microsoft may have "a bunch of chips sitting in inventory that I cannot plug in", for want of powered buildings (Wccftech, on the BG2 podcast).
• NVIDIA, Tue 14 Jul 2026: "Power is AI infrastructure's inescapable constraint" (NVIDIA blog).

Cooling sets how densely the power can be packed. Air-cooled H100 racks "typically only support about 40kW/rack", while a liquid-cooled GB200 NVL72 rack draws about 120 kW (SemiAnalysis, Wed 17 Jul 2024).

So operators trade clock for count.

• NVIDIA's MaxLPS shares a fixed power budget across GPUs by adjusting their power limits. In a trial with Nscale, 192 GB300 GPUs ran in the 264.4 kW that had held 140, for 49.2% more total throughput and unchanged per-instance speed (NVIDIA, Sun 27 Sep 2026).
• NVIDIA's power profiles on the 1,000 W B200 save "up to 15%" of power "with, at most, a 3% performance loss" (NVIDIA, Thu 4 Dec 2025).
• MLPerf Inference v4.0. An air-cooled DGX H100 capped at 350 W per GPU, half its rating, kept 74.6% of its GPT-J throughput: 178.9 against 239.9 samples per second (MaxQ and MaxP logs; the 350 W setting).

In serving, it depends on the batch. SemiAnalysis's InferenceX logs each GPU's average power (data from its benchmarks API, read Thu 22 Oct 2026):

Large batches put the air-cooled chips within a few percent of their caps. Long-context decode, which mostly waits on memory, leaves them well below: neither power nor heat binds there, memory bandwidth does.

7. Your guess: air against liquid

Mostly yes, but the difference shows up in the cap, not in throttling. Where the binding constraint sits:

So liquid cooling moves the binding constraint in two directions:

• Up at the chip: a higher cap, roughly 1.2–1.4× the air rating for the same die.
• Out to the building: the extra watts per rack have to be bought from the grid.

An air-cooled fleet is not usually thermal-throttling. It is capped lower, it carries a hot tail of GPUs a few percent slow, and it has racks spaced out for air.

8. What it means for your energy experiments

• On a GPU at its power cap, fewer bit flips buy a higher clock. On zeros, AMD's MI300X ran its clock 51% higher than on varied data, the H800 PCIe did 9% more TFLOPS, and Horace's A100 15% more. The ET-SoC-1 runs fixed clocks, so the same physics shows up there as watts: 38.29, 46.73 and 63.40 W at one 9.18 TFLOPS (the Horace experiment).
• The headroom decides how big the effect can be. An H100 SXM at 700 W already runs within 1% of its top clock, so low-flip data has little speed left to buy. A B200 sits 27% below its top clock at the cap and an MI300X 43% below, so Blackwell- and MI300-class chips should show Horace's effect far more strongly. A lower cap brings an H100 into the same regime, which is what Horace's 330 W did.
• To read the effect as speed, leave the clock free and stay at the cap. Log the clock event reasons every 100 ms to show that SW Power Cap is set and the thermal flags are not; the 21 Sep brief asked for exactly this run. To read it as energy, pin the clock below the capped value (nvidia-smi -lgc), so the data shows up as watts at a fixed speed, as on the ET-SoC-1.
• Hold the inlet temperature steady and warm the card up first. Leakage moves the clock 0.3–1 MHz per °C at the cap, which is small but the same size as some data effects.
• For the GPU MODE energy board, on a power-capped GPU energy and speed are coupled: a kernel that flips fewer bits also runs faster. On a card below its cap, it only uses less energy.

Sources

• Horace He, Strangely, Matrix Multiplications on GPUs Run Faster When Given "Predictable" Data!, Mon 29 Apr 2024.
• NVIDIA: NVML clock event reasons; H200; GB200 NVL72; DGX B200 user guide; Mission Control power FAQ; MLPerf Training v4.0; power profiles; MaxLPS with Nscale; performance per watt; RTX 4090; RTX PRO 6000 and Max-Q.
• Stas Bekman, ML Engineering: accelerators, read Thu 22 Oct 2026.
• Chester Lam, Nvidia's H100: Funny L2, and Tons of Bandwidth, Chips and Cheese, Mon 3 Jul 2023.
• Luo et al., Benchmarking and Dissecting the Nvidia Hopper GPU Architecture, Wed 21 Feb 2024.
• Supermicro, HGX B300 datasheet.
• AMD, Measuring Max-Achievable FLOPs, Part 2, Fri 28 Feb 2025; MI350 series architecture.
• The Register, AMD's MI355X, Thu 12 Jun 2025.
• Intel, Intel Gaudi 3 AI Accelerator technical paper, Rev. 1, Sep 2024.
• Kim et al., An Inquiry into Datacenter TCO for LLM Inference with FP8, v4, Mon 25 Aug 2025.
• Jouppi et al., Ten Lessons From Three Generations Shaped Google's TPUv4i, ISCA 2021.
• LLM fine-tuning on an RTX 4090 at reduced power, Wed 29 May 2024.
• GamersNexus, RTX 5090 Founders Edition review, Fri 24 Jan 2025.
• Meta, The Llama 3 Herd of Models, Tue 23 Jul 2024.
• Stojkovic et al. (Microsoft), TAPAS: Thermal- and Power-Aware Scheduling for LLM Inference in Cloud Platforms, Sun 5 Jan 2025.
• Sinha et al., Not All GPUs Are Created Equal, SC22, Tue 23 Aug 2022.
• Ujeniya et al., ClusterBench, Tue 11 Aug 2026.
• Wu et al. (HKUST and Alibaba), Falcon: Pinpointing and Mitigating Stragglers, Wed 16 Oct 2024.
• Latif et al., Cooling Matters, Tue 22 Jul 2025.
• Google Cloud, Troubleshooting GPU VMs.
• Patel et al. (Microsoft Azure), Characterizing Power Management Opportunities for LLMs in the Cloud, ASPLOS 2024.
• SemiAnalysis, GB200 Hardware Architecture, Wed 17 Jul 2024; InferenceX, read Thu 22 Oct 2026.
• Wccftech, Microsoft CEO on chips he cannot plug in, Sat 1 Nov 2025.
• MLCommons, MLPerf Inference v4.0 results.

Total life satisfaction: the post's opening claim now has its evidence, and a sharper experiment comes out of it. Run the matmul on a Blackwell-class card at its cap with the clock reasons logged, where the bit-flip effect should be largest.