How good are local LLMs at translation, and do you actually need the cloud? A reproducible benchmark of 24 on-device, self-hosted, and cloud models translating into English, with the low-resource case (Afrikaans) front and centre. The headline: on Afrikaans→English a local 18 GB model lands in a statistical tie with frontier cloud. Same blinded Tatoeba sentences, same prompt, greedy decoding, scored multi-reference with COMET (meaning) and chrF++ (surface). Built to pick a translation model for Lector.

Open and reproducible: harness, every model's raw outputs, and the seeded test sets: github.com/heuwels/llm-lang-eval

On Afrikaans the field is tightly bunched: 20 of 24 models fall within ~1.5 COMET (sampling noise) of the top score (≈95), a statistical tie, not a ranking. The self-hosted 18 GB gemma-4-12b-qat (95.0) sits in that band alongside frontier cloud, so for Afrikaans→English, you don't need the cloud or a big box.

Leaderboard

All 24 models across three languages on one zoomed COMET axis: each row a model (ranked by Afrikaans), one dot per language, the connector its cross-language spread. The zoom makes the tight differences legible; gaps under ~1.5 COMET are sampling noise (see Significance).

Deployment tier:on-device (laptop) self-hosted box (18 GB) cloud (OpenRouter)

The numbers

COMET (meaning, ×100) over chrF++ (surface), per language. chrF++ rewards character overlap with the reference, so it docks valid paraphrases ("scenery is magnificent" vs "landscape is breathtaking"); COMET scores meaning and credits them. Green = leading band (within ~1.5 COMET, a statistical tie, not a single winner). Rows share the chart's order (ranked by Afrikaans COMET), so near-ties can sit a place apart despite equal rounded scores. n=200 per language.

Cost, and what you can actually run

Cloud models fill the top of that board, but this is a study of local models, and most of the cloud field can't run on the box at all. The frontier APIs (GPT-5, Claude, Gemini) are closed weights, so self-hosting them was never an option. The open models I could reach through OpenRouter are mostly too big for the hardware: Llama 3.3 is 70B, Mistral Large is larger again, and even the 24B and 27B open models (Mistral Small, Gemma 2 and 3 at 27B) sit at or past the ceiling of an 18 GB Mac once the OS and the KV cache take their share. What genuinely fits is the on-device and self-hosted-box tiers, so here is that field on its own.

The strongest model that actually fits, gemma-4-12b-qat at 7.5 GB, is the same one sitting in the frontier band up top. apfel-foundation is Apple's built-in Foundation model, the one that ships with macOS, run through the Apfel harness. It scores respectably on what it answers, but it refused or errored on 26% of the Afrikaans sentences (52 of 200), against 8% in Spanish, and Apple doesn't list Afrikaans among its supported languages, which is why it sits near the bottom.

Cost: use what you've got

Cost barely enters into it. A translation is tiny, roughly 80 tokens, so the entire cloud sweep (24 models across three languages, plus the holdout and the cloze probe) came to $13.62 on OpenRouter, well under a cent per translation even on the frontier models. Per-token pricing still spans an order or two of magnitude, the frontier APIs against the cheap tiers like Gemini Flash or GPT-4o-mini, but at this token count the absolute bill is small whichever way you go.

What moves the decision is what you already own. A spare Mac is a sunk cost, so a local model is free per lookup beyond the electricity. An existing Claude plan is free at the margin too, within its limits, which is why I reach for the Anthropic OAuth route first. OpenRouter is the only one of the three that adds a real per-token bill, and it earns its place when you need a specific model you can't self-host or don't have a plan for. So the honest answer is usually to use what you've got: a spare box runs a local model, an existing plan already covers the lookups, and with neither, the cheap cloud tier is pennies per thousand. Since a 12B you can run at home already ties the frontier on Afrikaans, paying frontier rates per token buys very little for this particular job.

Contamination check: does it survive on unseen data?

The honest limitation. Tatoeba is almost certainly in every model's pretraining, so a high score can mean "translated well" or "regurgitated a memorised pair". The score alone can't tell us which. To bound it, each model is compared on two matched 150-sentence Afrikaans samples (same length filter): pre-2023 (added 2010 to 2022, almost certainly seen in training) versus 2025-26 (added after the training cutoff of the older-generation models here, so they cannot have memorised them). A large drop on the recent set is the fingerprint of memorisation; a stable score is evidence of genuine translation ability.

Caveat on the caveat: exact training-cutoff dates aren't published for every model, and recently-added sentences may differ subtly in style or difficulty, so read a small Δ as "holds up", not as a precise measurement of contamination.

Parroting probe: memorisation, measured directly

The sharpest contamination test. We blank one informative word per sentence and ask each model to fill it. On unseen (2025-26) sentences it can only predict from context; if it recovers the exact original word much more often on seen (pre-2023) sentences, that gap is the model parroting memorised text rather than reasoning about the language. (It doubles as a cloze-ability score, Lector's own practice task.)

Recovery = exact match of the blanked word. A large positive gap = memorisation; near-zero = genuine context prediction. n≈150 per cell, so gaps within ~±10 are noise.

Side-by-side generations

The numbers only say so much. Here are the actual translations where models disagree most. Green marks the highest per-sentence chrF++ for that sentence.

Afrikaans: where the models split

Sentences where models split into clear camps, several agreeing on one wording, several on another (one-off wordings collapsed to a tail). Count × tier-dots per camp; green = within ~6 chrF of the closest-to-reference camp.

Why they differ: almost none of this is error. It's paraphrase choice. A contraction vs the full form, 'by the end' vs 'before the end', one valid synonym over another. Each camp diverges from the single crowd-sourced reference in its own way. The spread is widest on longer, structurally flexible sentences (more ways to order the English) and on the high-resource languages, which is exactly why the COMET (meaning) gaps are far smaller than the chrF (surface) gaps. Read the camps as equally-valid translations, not right-vs-wrong.

German: where the models split

Sentences where models split into clear camps, several agreeing on one wording, several on another (one-off wordings collapsed to a tail). Count × tier-dots per camp; green = within ~6 chrF of the closest-to-reference camp.

Why they differ: almost none of this is error. It's paraphrase choice. A contraction vs the full form, 'by the end' vs 'before the end', one valid synonym over another. Each camp diverges from the single crowd-sourced reference in its own way. The spread is widest on longer, structurally flexible sentences (more ways to order the English) and on the high-resource languages, which is exactly why the COMET (meaning) gaps are far smaller than the chrF (surface) gaps. Read the camps as equally-valid translations, not right-vs-wrong.

Spanish: where the models split

Sentences where models split into clear camps, several agreeing on one wording, several on another (one-off wordings collapsed to a tail). Count × tier-dots per camp; green = within ~6 chrF of the closest-to-reference camp.

Why they differ: almost none of this is error. It's paraphrase choice. A contraction vs the full form, 'by the end' vs 'before the end', one valid synonym over another. Each camp diverges from the single crowd-sourced reference in its own way. The spread is widest on longer, structurally flexible sentences (more ways to order the English) and on the high-resource languages, which is exactly why the COMET (meaning) gaps are far smaller than the chrF (surface) gaps. Read the camps as equally-valid translations, not right-vs-wrong.

Methodology

Caveats