TL;DR: Our SDXL LoRA fine-tune for a Photoroom product photography model trained for six days while silently corrupting its adapter weights. The cause was bf16 gradient accumulation interacting badly with a custom adapter init we'd ported from a paper. Eval scores stayed in the same range the whole time, which is why nobody noticed.
The setup
We train SDXL LoRAs for product photography categories at Photoroom. Bottles, packaged food, soft goods. Each LoRA is 192MB. Training stack: PyTorch 2.3, bf16 mixed precision, gradient accumulation across 8 steps, A100 80GBs.
The LoRA init follows a small modification of the OFT paper for better stability on small datasets. To be precise, we orthogonalize the down-projection before training begins, then let the up-projection drift freely. This had been working for nine months.
What broke
Six days into a 7-day run, our automated CLIPScore check started showing variance that was technically inside our acceptance band but trending the wrong way. The nuance here is that our eval pipeline grades generations using a fan-out across three VLM providers (Claude vision, GPT-4o, Gemini 1.5) routed through an LLM gateway. We use Bifrost for that fan-out, which gives us provider-level failover when one of them rate-limits us mid-grade. Useful, and uneventful. The grade scores looked fine.
The real signal was a per-step gradient norm log we'd turned off a quarter earlier when it was spamming the dashboard. When I turned it back on for a sanity check, the grad norms had been collapsing to ~1e-5 in the down-projection layer since step 12,000.
Tracing it back
I added a hook to dump the raw bf16 gradient tensors before they hit the accumulator:
import torch
def grad_dump_hook(name):
def hook(grad):
finite = torch.isfinite(grad).all().item()
absmax = grad.abs().max().item() if finite else float("nan")
if absmax < 1e-4 or not finite:
print(f"[{name}] finite={finite} absmax={absmax:.2e}")
return grad
return hook
for n, p in lora.named_parameters():
if p.requires_grad and "lora_A" in n:
p.register_hook(grad_dump_hook(n))
Output across 200 steps:
[blocks.7.attn1.lora_A] finite=True absmax=4.32e-06
[blocks.8.attn1.lora_A] finite=True absmax=2.11e-06
[blocks.9.attn1.lora_A] finite=True absmax=8.54e-07
Gradient magnitudes in bf16 land are bounded below by roughly 6e-8 before they round to zero. We were producing real gradients, but most of them were being silently quantized to zero during the accumulation step. The accumulator in our custom training loop accumulated in bf16, not fp32.
This is documented behavior. Standard PyTorch grad accumulation in Accelerate uses fp32 accumulators by default. Our custom loop, forked from an internal repo two years ago, did not.
The fix
# before
optim.param_groups[0]["grad_accumulator_dtype"] = torch.bfloat16
# after
optim.param_groups[0]["grad_accumulator_dtype"] = torch.float32
Single line. Six days. We re-ran training with fp32 accumulation. Grad norms stabilized in the expected 1e-3 to 1e-2 range. Eval scores moved up by ~6% in our internal background-consistency metric.
Why nothing else caught it
A few obvious questions:
Why didn't loss curves show it? They did, mildly. The loss was still going down, only slower. Within noise of a normal run.
Why didn't the VLM eval catch it? Because the generations were still "good." Product on a clean background, lighting roughly correct. The drift was in finer details (brand text legibility, soft-good fabric texture) that our three-VLM grading averages out. We're now adding a per-category CLIPScore-vs-reference check that runs without averaging.
Why did we trust the init? We had nine months of green runs. The OFT-style init only became a problem when we tightened the LR schedule three weeks ago, which made the gradient magnitudes smaller across the board.
Trade-offs and limitations
| Approach | Memory cost | Bookkeeping |
|---|---|---|
| bf16 accumulator | baseline | low |
| fp32 accumulator (all params) | +4% peak | low |
| fp32 only for LoRA params | +0.6% peak | painful |
The fp32 accumulator costs us ~4% more GPU memory per step. Not free. On the A100 80GBs it's invisible, but if you're tight on the H100 80GBs or sharing with another job, you'll feel it.
You can accumulate in fp32 only for the LoRA params and keep the base model gradients in bf16, but the bookkeeping is annoying. We took the 4% hit.
The deeper lesson: a custom training loop that worked for nine months is not a training loop you understand. It's a training loop that hasn't been stressed in the right place yet. I should have re-read it when we changed the LR schedule.