What: On June 16, 2026, NVIDIA's Blackwell platform posted the fastest time on all seven MLPerf Training 6.0 benchmarks. The lens this explainer uses to read that result is strong scaling — how much faster a fixed model trains as you pour in more GPUs.
Why: Frontier pretraining now runs on 5,000–8,000-GPU clusters, and how well those GPUs scale together — not just how many you own — decides both the wall-clock and the bill for training a model.
vs prior: The naive assumption is that twice the GPUs means half the time. Strong scaling is the reality check: every step the GPUs must stop and synchronize, so a loosely-wired cluster gives sub-linear speedup where a rack-scale NVLink domain keeps it near the line.
Think of it as
A rowing crew racing one boat to the finish.
2× THE ROWERS (GPUs)
│
┌─────────────┴─────────────┐
│ │
┌───────▼───────┐ ┌────────▼───────┐
│ NVLink crew │ │ Loose cluster │
│ (one cadence) │ │ (off the beat) │
└───────┬───────┘ └────────┬───────┘
│ │
every oar hits the rowers miss the catch,
catch in unison power cancels at sync
│ │
▼ ▼
✓ near-linear ✗ far below 2×
(~2× faster) (sync tax eats it)
- GPU = one rower pulling one oar
- training step = one stroke the whole crew takes together
- gradient sync = the catch, where every oar must hit the water at the same instant
- adding GPUs = adding rowers to the boat
- NVLink rack domain = the racing shell and coxswain that keep a huge crew in perfect time
- low-precision math = lighter oars, so there is less to move on every stroke
Quick glossary
MLPerf Training — The industry-standard training benchmark. It measures one thing: the wall-clock time to train a model to a fixed quality target — so a faster time is a real, comparable result, not a vendor's peak-throughput number.
Strong scaling — Hold the problem fixed (one model, one quality target), add more GPUs, and measure the speedup. Its sibling, weak scaling, grows the problem with the hardware. Strong scaling is the harder test, because the work per GPU keeps shrinking while the coordination cost does not.
Gradient synchronization (AllReduce) — Each GPU trains on a different slice of the batch, then they must average their gradients before the next step starts. That all-to-all exchange — an AllReduce — is a barrier: nobody moves on until everyone has caught up.
NVLink domain (NVL72) — 72 GPUs wired by fifth-generation NVLink into one coherent, high-bandwidth fabric — a single rack that behaves like one big accelerator. The fast fabric is what makes the synchronization barrier cheap.
Low-precision math (FP8 / NVFP4) — Running the heavy matrix multiplies in 8-bit FP8 or 4-bit NVFP4 instead of 16-bit, so there is less data to move and less to compute on every step. Blackwell's tensor cores support both.
Scaling efficiency — The actual speedup divided by the ideal one. Double the GPUs and perfectly halve the time and you are at 100% — perfectly linear. Anything less is the time the GPUs spent waiting on each other.
The news. On June 16, 2026, NVIDIA reported that its Blackwell platform posted the fastest time on every one of MLPerf Training 6.0's seven benchmarks. The new GB300 NVL72 rack trained up to 1.6× faster than the previous GB200 NVL72. Submissions scaled to 8,192 GPUs — CoreWeave trained DeepSeek-V3 671B to target in 2.02 minutes, and Microsoft Azure hit the quality target on Llama 3.1 405B in 7.07 minutes at 8,192-GPU scale. The round also added new mixture-of-experts pretraining workloads. Read the release →
Picture the rowing crew. The finish line is the model's quality target, and one stroke of the whole crew is one training step — every rower pulls, the boat lurches forward, and they reset for the next stroke. Each rower is a GPU, working a different slice of the same race. The trick is the catch: the instant the oars enter the water. If every oar hits at the same moment the boat surges; if they're even slightly out of time, the power cancels and the boat wallows. Adding rowers should make the boat faster — but only if the bigger crew can still hit the catch together.
That "only if" is the whole story, and its real name is strong scaling: fix the model, add GPUs, and see how much the clock actually drops. The catch is the catch. Every step, the GPUs have to stop and combine their partial results — gradients across the data-parallel replicas, plus activations and weights traded inside the tensor- and expert-parallel groups — before the next step can begin. That synchronization is a tax that grows as the crew grows, so doubling the GPUs buys you less than 2× — the speedup curve bends below the straight line. A naive cluster, like a crew that can't hold its timing, gives back most of what each new rower adds.
So the engineering is all about making the catch cheap. NVIDIA's answer is the rack-scale NVLink domain: the GB300 NVL72 ties 72 GPUs into one coherent fabric — the racing shell and coxswain that keep a huge crew locked to a single cadence — so the per-step exchange finishes fast enough that thousands of GPUs still row almost as one. Pair that with lower-precision math — Blackwell's tensor cores run the matmuls in 8-bit FP8 and 4-bit NVFP4, lighter oars with less to move every stroke — plus a stronger software stack, and NVIDIA credits that combination for the sweep: a GB300 rack trains up to 1.6× faster than last generation, and 8,192 GPUs finish in minutes, not hours.
| MLPerf Training 6.0 result | Scale | What it shows | Time to target |
|---|---|---|---|
| GB300 NVL72 vs GB200 NVL72 | 72-GPU rack | hardware-generation speedup | up to ~1.6× faster |
| DeepSeek-V3 671B (MoE) | ~8,192 GPUs | strong scaling, new MoE workload | ~2.02 min (CoreWeave) |
| Llama 3.1 405B (dense) | ~8,192-GPU scale | strong scaling at frontier size | ~7.07 min (Azure) |
Strong scaling, in one calculation
Hold the model fixed — DeepSeek-V3, 671B parameters, trained to MLPerf's quality target — and watch the clock as you add rowers. On 8,192 Blackwell GPUs, CoreWeave's run finished in 2.02 minutes. Now ask the strong-scaling question: had you used half as many GPUs, would it have taken exactly twice as long? Perfect scaling says yes. Suppose (illustrative) the 4,096-GPU run had actually taken 3.7 minutes. Then doubling the GPUs cut the time from 3.7 to 2.02 — a 1.83× speedup, not the ideal 2.0×. Divide the two and you get a scaling efficiency of ~92%; the missing ~8% is the time the GPUs spent at the catch, waiting on each other. The entire job at this scale is keeping that number pinned near 100% — which is exactly what a faster NVLink fabric and lighter low-precision oars are for. (The 2.02-min, 8,192-GPU, and 1.6× figures are from NVIDIA's MLPerf 6.0 report; the 4,096-GPU split is illustrative.)
Goes deeper in: GPU & CUDA → Memory Hierarchy → NVLink & PCIe
Related explainers
- Vera Rubin NVL72 — the NVLink rack domain — the "72 GPUs as one coherent fabric" idea this article leans on, taken apart on its own.
- NVIDIA AI factories — tokens per megawatt — once a cluster scales well, the next question is energy: useful work per watt, not just per GPU.
- Fused INT8 GEMM — INT8 beats FP8 on the tensor cores — the same "shrink the numbers so there's less to move" lever, on the inference side instead of training.
FAQ
What is strong scaling in distributed training?
Strong scaling fixes the problem — one model, one quality target — and measures how much faster it trains as you add GPUs. Perfect strong scaling means N times the GPUs finishes in 1/N the time. In practice the speedup falls short of that line, because every training step the GPUs must stop and synchronize their partial results before the next step can start, and that coordination cost grows with the number of GPUs. The gap between the ideal and the actual speedup is the scaling efficiency.
Why doesn't doubling the GPUs halve the training time?
Because training is synchronous. Each GPU works on a different slice of the batch, but at the end of every step they must average their gradients (an AllReduce) and exchange activations and weights across the tensor- and expert-parallel groups before moving on. That barrier is overhead that does not shrink as fast as the per-GPU work does, so adding GPUs gives less than a proportional speedup. The fix is to make the synchronization cheap — a fast, rack-scale NVLink fabric and lower-precision (FP8/NVFP4) numbers — so the speedup curve stays close to linear.
What did NVIDIA Blackwell actually win in MLPerf Training 6.0?
NVIDIA reported the fastest time on all seven MLPerf Training 6.0 benchmarks. The new GB300 NVL72 rack trained up to 1.6× faster than the prior GB200 NVL72, submissions scaled to 8,192 GPUs (CoreWeave trained DeepSeek-V3 671B to target in 2.02 minutes; Microsoft Azure hit the target on Llama 3.1 405B in 7.07 minutes at 8,192-GPU scale), and the round added new mixture-of-experts pretraining workloads. The headline is less about one chip than about how well thousands of them scale together.
Originally posted on Learn AI Visually.