Production large-language-model (LLM) deployments increasingly route requests through commercial API gateways (proxies) that aggregate upstream model providers behind a unified OpenAI-compatible interface. We show that these gateways introduce a class of silent failures: requests that succeed at the HTTP layer (status 200, valid JSON) but return semantically broken content at rates dramatically higher than the same request issued against the upstream model's direct API. Detecting and attributing such failures requires evidence strong enough to survive vendor pushback.

This paper introduces Silent-Bench, an open-source framework that combines (i) a Cartesian causal-ablation sweep over standard chat-completion parameters, (ii) per-call invisibility scans for hidden behavior such as token-billing inflation and prompt-stripping, and (iii) cryptographically-attested forensic dossiers: every captured API call is hashed into a per-cell Merkle tree, every cell is aggregated into a chain Merkle root, the canonical chain is signed under an Ed25519 keypair, and the result can be verified by any third party with no trust in the auditor.

We report three case studies, anonymized as Proxy-A, Proxy-B, and the upstream models Model-X, Model-Y, Model-Z. Proxy-A produces a +35.85 percentage-point response-format gradient on Model-X (n_neg=1,272 pooled; pooled rate 30.03%, 95% Wilson CI 27.58%–32.61%); a direct-API counterfactual against the model vendor under identical parameters yields a pooled rate of 1.89%, localizing the defect to the proxy's routing layer with a ~28 percentage-point pooled-rate gap. Proxy-B hosts a deployment of Model-Y that inflates billed tokens by ~55%; cross-model probes on the same proxy clear the infrastructure and isolate the wrapper to the specific deployment. A third gradient on Proxy-A → Model-Z (+28.6 pp) lacks a direct counterfactual at the time of submission; we report the attribution as provisional.

We further document three retractions our own work has had to issue when applying Silent-Bench to insufficiently-large samples, and formulate a methodological warning we call the small-sample artifact pattern: causal-ablation effect sizes estimated under n_neg < 10 per cell are systematically inflated, occasionally producing binary-toggle or full gradient mechanism claims that smooth into more modest gradients (or disappear) under n_neg=159 per cell.

The framework is released under Apache-2.0; reproduction commands and verification protocols are given. The vendor-named version of this paper will be released after the standard 90-day coordinated disclosure window expires.

Anonymization policy. Implicated parties in the case studies are anonymized as Proxy-A, Proxy-B, Model-X, Model-Y, Model-Z for 90 days under standard security-research disclosure practice. Counterfactual control endpoints (Anthropic Claude Haiku 4.5, DeepSeek-chat) are named directly in the multi-control sweep because they are exonerated by the test design itself: both produced statistically zero rf-effect at n_neg=1,272.

Version trajectory. Paper v1 (2026-05-12, this release): anonymized vendor identification. Paper v2 (2026-08-10): full vendor attribution, regardless of remediation status. The numbers, confidence intervals, and cryptographic attestations are unchanged between v1 and v2; only the human-readable labels change.

Funding and competing interests. Self-funded; no external grants. Total API expenditure approximately USD 50. The author has no competing interests with any vendors discussed in the case studies.

Source Code Note. The Silent-Bench audit tool source code is scheduled for public release at github.com/wesam007/silent-bench by approximately 26 May 2026 (post-launch repository curation in progress to ensure no vendor identifiers leak through repository metadata before T+90). The Zenodo supplementary archive includes the 6 signed dossiers, public key, and stdlib-only verification protocol (VERIFY.md) that enable independent cryptographic verification of all reported findings without requiring the audit tool. Early-access tarball available on request to verified technical contacts at whsabban@uqu.edu.sa.

silent-bench-supplementary-v1.zip

Files (432.1 kB)