Like a lot of high-quality data on the web generated by us humans, code is becoming harder to find. It’s either been absorbed into today’s AI models or sits locked behind commercial licenses, putting it out of reach for many developers. Often what remains widely available is code of middling quality.

CodeAlchemy is designed to transform this overlooked resource into richer, more varied training data that can systematically improve AI model performance. Researchers created the CodeAlchemy pipeline to generate synthetic code to improve IBM’s own Granite family of models. They are now open-sourcing the pipeline along with a synthetic dataset that covers 15 programming languages and totals nearly 1 trillion tokens. A token is the smallest unit of text an AI model can process.

The CodeAlchemy dataset may be the largest of its kind. For context, it's at least 200 times larger than Wikipedia, which weighs in at about 5 billion tokens. It's possibly also the most ambitious. Provided in the haul of new training data is 1.3 million code files paired with their actual execution traces. Today’s large language models can write convincing code by copying patterns they’ve seen across billions of lines of code. But they often struggle with a routine part of a software developer’s job: predicting what that code will do at runtime.

A skilled developer will “trace” a program by stepping through it line by line to see how it behaves with real inputs. This helps them spot bugs, improve performance, and catch security risks. CodeAlchemy may be the first dataset to pair code files with actual traces in an effort to give LLMs a deeper understanding of how code works.

“Downloading billions of lines of code from GitHub teaches a model what code looks like,” said Ankit Gupta, an AI researcher at IBM who helped build CodeAlchemy. “But it doesn’t teach what happens behind the scenes when you execute that code on different inputs.”

Two years before ChatGPT, IBM’s pioneering project, CodeNet, taught AI models how to write code. Its successor, CodeAlchemy, is now aimed at teaching them what code means. While CodeNet focused on static, real-world code, CodeAlchemy uses the scale and flexibility of synthetic code to show how real code actually works.

"If the model sees this kind of code, along with its execution traces, it knows what’s happening behind the scenes," said Aditya Prasad, an IBM researcher who contributed to CodeAlchemy. "When it goes to write code on its own, it can reflect on that information and write a program that will compile properly."

Generating quality code at scale

CodeAlchemy takes low-quality data from the web and transforms it into more polished, useful training data through a series of filtering and rewriting steps. IBM researchers took inspiration from a data rewriting strategy that Chinese researchers used to dramatically improve their open-weight model, Kimi K2. An LLM was used to rephrase and amplify Kimi K2's original training data, bringing a model with just 32 billion activated parameters up to par with closed models like Claude and Gemini.

Data rewriting has since become a popular way to improve not just code models, but AI models generally. Until CodeAlchemy’s release, SwallowCode-v2 and Nvidia-Nemotron-Pretraining-Code-v2 were two of the largest datasets for code to be built this way. CodeAlchemy surpasses them in scale and scope. It’s more than twice the size of the next largest dataset, Nemotron, at about 500 billion tokens, and it goes beyond the Python and C++ in both datasets to include code in C, C#, Go, Java, JavaScript, Markdown, Ruby, Rust, Shell, SQL, Swift, and TypeScript.

It’s also unique in another way: it features sandboxed environments for safely executing millions of AI-improved code files. The researchers used these sandboxes to generate more than 1.3 million validated pairs of code file and traces across more than 5,000 libraries.

To see if this data would have any effect on performance, the researchers created a pair of new benchmarks with an independent set of examples — DevEval, to test a model’s ability to infer a human developer’s intent, and TraceEval, to evaluate how well a model can simulate how a program will run.

From syntax to semantics

Researchers sourced the raw code that gave rise to CodeAlchemy from three public repositories and used OpenAI’s GPT-OSS models to elaborate on problems grounded in real files. They used rewriting and other techniques to closely mimic how developers interact with their code assistants.

Their most innovative technique — pairing code files with actual trace logs — was aimed at teaching models how code behaves. Unlike static code, execution traces can reveal a program’s changing states, data transformations, and interactions with the real world through library APIs. The line x = foo(y) for example, teaches a model syntax, but not what values x will take, which branches execute, or how state evolves.

“Most benchmark tasks require generating code, but executing that code, where I give an input and the model has to do the calculations internally — that's much harder," said Gupta.

When researchers examined how well the frontier models could predict a line of code from its trace, they were surprised. Claude Sonnet 4.5 could generate an exact match less than 6% of the time, and a partial match, just a third of the time.

But just because a frontier model did poorly on the task, doesn’t mean it can’t code well, said Gupta. It just means that developers need to continue validating their assistants’ code in a sandbox before sending it to production. As models get closer to 100% and exact matching, this cumbersome step could be eliminated.

The smaller code models that researchers tested almost universally failed to generate the exact match for a given code file. But after researchers trained an early version of Granite 4.0 3B model on CodeAlchemy, it got better at partial matches, even outperforming models much larger.

These benefits carried over to other tasks. On DevEval, Granite 4.0 3B went from a 2% win rate against Claude Sonnet to an 8% win rate. When IBM releases Granite 4.2, its next version of models, users will see a marked improvement in reasoning capabilities which can be at least partially attributed to training on CodeAlchemy data.

Novel data for smaller, more capable models

The team made other observations during their experiments. One of the most counterintuitive was that using a smaller model to rewrite code generally produced better results, with models trained on code rewritten by a Gemma 4B model outperforming models trained on code rewritten by a GPT-OSS 20B model.

While larger models tend to generalize better and produce more consistent code, that can be a handicap when generating synthetic data, where variety can be the overriding factor in generating high-quality examples at scale. “Creating synthetic data with diversity is the biggest challenge,” said Prasad. “How do you ensure that the model is generating something new? That was the hardest part of this project.”

Synthetic code filtered extensively to pull out novel, high-quality examples can be even more effective than the real thing. The researchers found that a Granite 4.0 3B model trained on just 100 billion tokens of synthetic code did better than training the same model on 600 billion tokens of real code.

To inject more variety into CodeAlchemy, the team added 5% of their original low-quality code to the mix, which helped models improve on some benchmarks. They also confirmed that these improvements persisted even after fine-tuning, when some models can forget what they learned earlier.

A Granite 4.0 3B model pre-trained on a CodeAlchemy mix, and post-trained on OpenCodeInstruct scored 83.5% on HumanEval, a benchmark that measures how well a model can turn prompts into functionally correct Python. The Granite 3B model also scored 63.2% on the popular Mostly Basic Python Problems (MBPP) benchmark, confirming that CodeAlchemy’s benefits persist even after post-training.

The results show just how effective synthetic data can be in lifting the performance of smaller models. Since smaller models can run on fewer GPUs, are cheaper to serve, and are more easily deployed on the edge, this efficiency can be an enormous advantage for enterprise applications.

What’s next

LLMs aren’t in need of just code data to fill in blind-spots and plug gaps in their existing knowledge base. As they break out of chat to take on more demanding tasks, LLMs need to know how to interact with the world and each other and to self-correct when they make a misstep. These new agentic workflows require skillsets that can’t be taught by simply feeding models passively on more data gathered from the web.

The solution that IBM researchers have devised is to extend the synthetic data pipeline behind Code Alchemy to other enterprise tasks. This larger effort, called Open Alchemy, is designed to generate open-source data that mirrors specialized work environments, where LLM agents can hone their ability to call and execute applications and navigate and retrieve information from databases and large file systems.

Meanwhile, AI coding agents are also evolving and the creators of CodeAlchemy are adapting their pipeline to meet these new demands. They are building a repository of multi-step coding tasks to give agents more practical experience with things like debugging, managing dependencies, and multi-step test execution.

CodeAlchemy’s data and codebase are available under a permissive license, along with the instruction prompts used to create the data.