Hello, I'm Shrijith Venkatramana. I'm building git-lrc, an AI code reviewer that runs on every commit. Star Us to help devs discover the project. Do give it a try and share your feedback for improving the product.
Everyone knows modern AI runs on data. Fewer people realize that today's most capable AI systems increasingly learn from data they created themselves.
For years, the common belief was simple: collect more human-written text, train bigger models, and intelligence would emerge.
That worked—until it didn't.
By 2022, frontier AI labs had consumed a significant fraction of the publicly available high-quality text on the internet. The next leap wasn't going to come from scraping another few billion web pages.
Instead, researchers turned to something different:
They started letting AI generate its own training data.
Today, synthetic data powers reasoning models, coding assistants, math solvers, autonomous agents, and many of the capabilities developers now take for granted. In many cases, the most valuable dataset isn't written by humans anymore—it's produced by another AI.
Let's explore how that works out.
The Factory Analogy
Imagine you're building a car factory.
Initially, every part has to be handcrafted by skilled workers. Production is slow and expensive.
Eventually, you build machines that manufacture car parts automatically.
Now the factory spends less time making parts manually and more time building better machines that manufacture even better parts.
Synthetic data works similarly.
Initially:
- Humans write examples
- Humans solve problems
- Humans explain reasoning
Eventually:
- AI generates new examples
- AI writes explanations
- AI creates harder problems
- AI critiques and improves its own answers
The model becomes both the student and one of the teachers.
This Idea Was Proven Long Before ChatGPT
Perhaps the most famous example isn't from language models at all.
In 2017, DeepMind introduced AlphaGo Zero.
Earlier versions of AlphaGo learned partly from millions of expert human games.
AlphaGo Zero didn't.
It started knowing only the rules of Go.
Then it played against itself.
Millions of games later, it became stronger than every previous version—and stronger than every human on Earth.
No additional human demonstrations.
Only synthetic experience generated through self-play.
Researchers realized something profound:
Once a system becomes good enough, it can manufacture experiences that are more useful than collecting additional human ones.
That idea quietly became one of the foundations of modern AI.
What Does Synthetic Data Look Like for LLMs?
Those less acquainted often imagine synthetic data as "ChatGPT writing more paragraphs."
That's only a tiny piece.
Modern models generate many different kinds of training data:
Question Generation
Instead of waiting for humans to ask questions, models invent thousands of new ones.
Example:
Human:
How do binary search trees work?
Synthetic examples:
Explain AVL trees.
Compare B-trees with BSTs.
Design a filesystem index.
Implement interval trees.
One human question becomes hundreds of training examples.
Reasoning Traces
Instead of merely generating answers:
42
The model produces detailed reasoning explaining how it reached the answer.
Those reasoning traces become valuable training material for future models.
Code
A coding model can generate:
- buggy code
- corrected versions
- optimized implementations
- documentation
- test cases
One programming problem becomes an entire software engineering dataset.
Harder Problems
Models can intentionally create problems just beyond their current capability.
Just like a good teacher gradually increases difficulty, the dataset evolves as the model improves.
The Economics Are Incredible
Suppose hiring experts costs roughly:
- $40 per hour
- each example takes 3 minutes
That's approximately $2 per example.
A million examples?
Around $2 million.
Now imagine a frontier model generates one million candidate examples overnight.
Even after filtering aggressively, perhaps only 20% are worth keeping.
That's still 200,000 useful examples produced in hours instead of months.
Of course, GPUs aren't free.
But frontier labs already own massive compute clusters.
Once the infrastructure exists, generating another million examples is often dramatically cheaper—and much faster—than coordinating thousands of human annotators around the world.
Synthetic data changes the limiting resource from human labor to compute.
That is a profound shift.
Why Doesn't the Model Eventually Teach Itself Nonsense?
A natural concern is:
"If AI keeps learning from AI, won't errors compound forever?"
Absolutely—if done carelessly.
Modern pipelines therefore look more like factories with quality control than simple generators.
A typical loop is:
Generate
↓
Verify
↓
Filter
↓
Rank
↓
Train
Only a fraction of generated examples survive.
For coding tasks:
- Does the code compile?
- Do unit tests pass?
For mathematics:
- Is the answer verifiably correct?
For reasoning:
- Do multiple models agree?
- Does a stronger model approve?
Synthetic data is valuable precisely because most of it gets thrown away.
Quality matters far more than quantity.
How You can Put This Into Practice in Your Apps
Many developers think synthetic data is something only OpenAI or DeepMind can use.
Not anymore.
Suppose you're building an AI assistant for SQL.
Instead of manually writing 5,000 examples, you might:
- Write 100 excellent examples.
- Ask a strong model to generate variations.
- Automatically execute the SQL.
- Keep only queries that execute correctly.
- Fine-tune on the verified dataset.
Or imagine building an AI tutor.
Generate:
- beginner questions
- intermediate questions
- trick questions
- incorrect student answers
- ideal explanations
One carefully designed seed dataset can grow into thousands of high-quality training examples.
The bottleneck becomes designing good verification systems—not endlessly producing data by hand.
That's a very different engineering problem.
The Bigger Picture
Scaling laws taught us that more compute and more data generally produce better models.
Synthetic data adds a fascinating twist:
The model itself becomes part of the data-generation pipeline.
Instead of relying solely on humanity's existing knowledge, AI systems increasingly create new training experiences for future AI systems.
In some domains—coding, mathematics, games, and formal reasoning—that approach is already proving remarkably effective because correctness can often be verified automatically.
It is one of the reasons today's reasoning models feel dramatically more capable than models from just a few years ago.
Ironically, one of the biggest breakthroughs in machine learning wasn't finding more human data.
It was discovering that, with the right safeguards, machines can help create the next generation of training data themselves.
Further Reading
- AlphaGo Zero (Silver et al., 2017) — Demonstrated that superhuman performance could emerge through self-play rather than human demonstrations.
- Self-Instruct (Wang et al., 2023) — Showed that language models can bootstrap instruction-following datasets by generating and filtering their own instruction-response pairs.
- Phi-1: Textbooks Are All You Need (Microsoft Research, 2023) — Demonstrated that carefully curated synthetic "textbook-quality" data can outperform much larger quantities of noisy web data.
What do you think?
If you had to improve an AI application today, would you spend your effort collecting more human data—or designing a better pipeline to generate and verify synthetic data? As frontier models improve, that trade-off is becoming one of the most interesting engineering decisions in AI.
*AI agents write code fast. They also silently remove logic, change behavior, and introduce bugs -- without telling you. You often find out in production.
git-lrc fixes this. It hooks into git commit and reviews every diff before it lands. 60-second setup. Completely free.*
Any feedback or contributors are welcome! It's online, source-available, and ready for anyone to use.
GenAI today is a race car without brakes. It accelerates fast -- you describe something, and large blocks of code appear instantly. But AI agents silently break things: they remove logic, relax constraints, introduce expensive cloud calls, leak credentials, and change behavior -- without telling you. You often find out in production.
git-lrc is your braking system. It hooks into git commit and runs an AI review on every diff before it lands. 60-second setup. Completely free.
In short, git-lrc helps Prevent Outages, Breaches, and Technical Debt Before They Happen
At a glance: 10 risk categories · 100+ failure patterns tracked · every commit…