What if a language model had never heard of the internet, smartphones, or even World War II? That’s not a hypothetical — it’s exactly what a team of researchers led by Nick Levine, David Duvenaud, and Alec Radford has built. They call it talkie, and it may be the most historically disciplined large language model ever released to the public.
Talkie is a 13-billion parameter open-weight language model trained exclusively on pre-1931 English text. The project is developed by a non-profit team and introduces what the researchers call a “vintage language model” — an LM with a hard knowledge cutoff tied not to when it was trained, but to a specific moment in history.
To understand talkie, you first need to understand the concept behind it. Most modern LLMs like GPT-4, LLaMA, Mistral etc. are trained on massive crawls of the contemporary web. Their knowledge reflects the world as it exists today, or as of their training cutoff date. A vintage language model flips this on its head: it is deliberately trained only on historical data so that its “worldview” is frozen at a particular point in the past.
For talkie, that cutoff is December 31, 1930 — chosen precisely because that is the date when works enter the public domain in the United States, making pre-1931 text legally usable for training.
The model — formally named talkie-1930-13b-base — was trained on 260 billion tokens of historical pre-1931 English text, including books, newspapers, periodicals, scientific journals, patents, and case law. A separately post-trained conversational checkpoint, talkie-1930-13b-it, is also available for interactive use. The team has set up a 24/7 live demo at talkie-lm.com/chat where Claude Sonnet 4.6 continuously prompts the instruction-tuned model, allowing visitors to observe talkie’s voice and knowledge in real time.
This isn’t a nostalgia project. The research team have identified several concrete, technically meaningful use cases that make talkie interesting to the AI research community.
1. Contamination-free generalization experiments: Benchmark contamination, where test data inadvertently leaks into training data — is one of the most persistent and underappreciated problems in modern LLM evaluation. Because talkie was trained only on pre-1931 text, it is contamination-free by construction with respect to any modern benchmark. This opens up a clean experimental setting to test how well an LM can generalize beyond its pre-training data. For example, the team tested whether talkie could learn Python — a language that didn’t exist in 1930 — by providing a few in-context demonstration examples. Using the HumanEval benchmark, they found that while vintage models dramatically underperform web-trained models, they are “slowly but steadily improving at this task with scale.”
2. Evaluating forecasting and temporal surprise: Inspired by Calcifer Computing’s work on Temporal Language Models, the research team used talkie to measure the surprisingness (measured in bits per byte) of historical event descriptions from the New York Times‘s “On This Day” feature. Events after 1930 — talkie’s knowledge cutoff — are consistently more surprising to the model, with the effect most pronounced for 1950s and 1960s events, followed by a plateau. This creates a principled setup for studying how forecasting ability scales with model size and how performance decays over longer temporal horizons.
3. LLM identity and persona formation: Because talkie was trained on a fundamentally different distribution than any modern model, it opens up questions about what shapes an LLM’s “identity.” Modern LLMs — regardless of their provider — all share a common ancestor in web data, whether through direct training or through distillation and synthetic data pipelines. Talkie breaks that lineage entirely, giving researchers a tool to examine what behaviors and capabilities are universal to language modeling versus what are artifacts of training on the contemporary web.
Building a vintage language model is not as simple as filtering a modern dataset by date. The talkie research team ran into several non-trivial engineering challenges.
Temporal leakage is the most critical. If any post-1930 text slips into the training corpus — through misdated documents, or old texts with anachronistic editorial introductions — the model’s historical fidelity is compromised. An earlier 7B version of talkie clearly knew about the Roosevelt presidency and New Deal legislation, revealing imperfect filtering. The team built a document-level n-gram-based anachronism classifier to filter the corpus, but acknowledge this is still imperfect — the 13B version retains some awareness of World War II and the postwar order.
Data quality is another major obstacle. Because there was no digital publishing in 1930, every token in talkie’s training corpus had to be transcribed from physical sources via optical character recognition (OCR). In controlled experiments, the team found that training on text transcribed by conventional OCR systems yielded only 30% of the learning efficiency of a model trained on human-transcribed versions of the same texts. Simple regex cleaning improved that to 70%, but a significant gap remained. To close it, they are building a dedicated vintage OCR system fine-tuned for historical document layouts.
Vintage post-training: the instruction-tuning phase — required building an entirely new pipeline from scratch. Using modern instruction-response pairs would inject contemporary expectations into the model’s behavior. Instead, the team generated instruction-response pairs from structured historical texts: etiquette manuals, letter-writing manuals, cookbooks, dictionaries, encyclopedias, and poetry and fable collections. They then ran online direct preference optimization (DPO) using Claude Sonnet 4.6 as a judge, improving talkie’s average instruction-following rating from 2.0 to 3.4 on a five-point scale. A final round of supervised fine-tuning used rejection-sampled multi-turn synthetic chats generated between Claude Opus 4.6 and talkie.
To provide meaningful context, the research team trained a “modern twin” — an architecturally identical 13B model trained on modern web data (FineWeb) — and compared it against talkie. Unsurprisingly, talkie underperforms its modern counterpart on standard LM evaluations. However, when controlling for question anachronism — filtering out questions that reference concepts that wouldn’t exist in 1930 — the performance gap roughly halves. The research team notes encouraging parity on core language understanding and numeracy tasks, and attributes the remaining gap primarily to OCR noise and subject matter distribution differences.
talkie-1930-13b-base for raw completions and talkie-1930-13b-it for conversation — but running them locally requires a CUDA GPU with at least 28 GB VRAM.Check out the Model Weights, Repo and Technical details. Also, feel free to follow us on Twitter and don’t forget to join our 130k+ ML SubReddit and Subscribe to our Newsletter. Wait! are you on telegram? now you can join us on telegram as well.
Need to partner with us for promoting your GitHub Repo OR Hugging Face Page OR Product Release OR Webinar etc.? Connect with us
此内容由惯性聚合(RSS阅读器)自动聚合整理，仅供阅读参考。 原文来自 — 版权归原作者所有。