The paper is now on arxiv and check out our demo!
This repository contains the data preprocessing pipeline, finetuning scripts, memorization evaluation code, and analysis scripts for our paper.
We provide partial example files in data/ containing a small subset of excerpts and generations from The Road by Cormac McCarthy. Full book content and model generations are not included because the books are copyrighted and the generations contain large portions of verbatim text.
Setup
We use uv for dependency management. Install uv if you haven't already:
curl -LsSf https://astral.sh/uv/install.sh | shCreate a virtual environment and install all dependencies:
uv venv --python 3.11
source .venv/bin/activate
uv pip install html2text natsort ftfy openai tqdm nltk numpyFor Gemini finetuning and generation, also install:
uv pip install google-genai google-cloud-storage vertexai
For DeepSeek finetuning and generation via Tinker, also install:
uv pip install tinker tinker-cookbook datasets
Set your Tinker API key (sign up at https://auth.thinkingmachines.ai/sign-up):
export TINKER_API_KEY="your-key-here"
Set your OpenAI API key (required for preprocessing and GPT-4o finetuning/generation):
export OPENAI_API_KEY="your-key-here"
Download the required NLTK data (one-time, for evaluation and analysis):
import nltk nltk.download('punkt_tab')
Data Preprocessing
We assume you already have the EPUB file for each book. The preprocessing pipeline converts an EPUB into a JSON file of excerpt chunks with plot summaries, ready for finetuning and evaluation. The output format matches data/example_book.json.
Step 1: Convert EPUB to plain text
python preprocess/epub2txt.py book.epub book.txt --plain-text --no-metadata --ftfy
Step 2: Split text into excerpt chunks
python preprocess/split.py book.txt book_chunks.json "Book Title" "Author Name"
This segments the text into excerpts of approximately 300-500 words. Excerpts exceeding 500 words are re-segmented using GPT-4o at natural grammatical boundaries.
Step 3: Merge short chunks and generate summaries
python preprocess/fix_file.py --input_json book_chunks.json --output_json book_final.json
This step:
- Merges chunks shorter than 300 words into adjacent chunks.
- Generates a plot summary for each chunk using GPT-4o.
- Constructs the finetuning instruction in the format:
"Write a {N} word excerpt about the content below emulating the style and voice of {Author}\n\nContent: {summary}".
Finetuning and Generation
We provide scripts for finetuning and generating completions via the OpenAI, Vertex AI, and Tinker APIs. We sample 100 completions per excerpt at temperature 1.0 (see Appendix A.3 of the paper).
GPT-4o
Finetune via the OpenAI API:
python finetuning/gpt_finetune.py \
--author_name "Cormac McCarthy" \
--raw_train_file data/example_book.json \
--job_name mccarthy \
--no_waitGenerate via the OpenAI Batch API:
python finetuning/gpt_generate.py \
--job_name mccarthy_test \
--test_file data/example_book.json \
--reformat_file batch_input.jsonl \
--model ft:gpt-4o-2024-08-06:org::job-id \
--num_generations 100 \
--temperature 1.0Gemini-2.5-Pro
Finetune via the Vertex AI API:
python finetuning/gemini_finetune.py \
--project_id your-gcp-project \
--bucket_name your-gcs-bucket \
--raw_train_file data/example_book.json \
--job_name mccarthy \
--no_waitGenerate via the Vertex AI Batch API:
python finetuning/gemini_generate.py \
--project_id your-gcp-project \
--bucket_name your-gcs-bucket \
--test_file data/example_book.json \
--model projects/PROJECT_NUM/locations/REGION/models/MODEL_ID \
--job_name mccarthy_test \
--num_generations 100 \
--temperature 1.0DeepSeek-V3.1
First convert the preprocessed data to Tinker's chat JSONL format (no system prompt for DeepSeek):
python finetuning/deepseek_convert.py \
--input_file data/example_book.json \
--output_file train_messages.jsonlFinetune via Tinker with LoRA (rank=32, lr=5e-4, 3 epochs):
python finetuning/deepseek_train.py \
dataset=train_messages.jsonl \
log_path=./logs/deepseek-mccarthy-epoch3Generate completions using the finetuned model:
python finetuning/deepseek_generate.py \
--test_data train_messages.jsonl \
--raw_book data/example_book.json \
--generation_output generations_deepseek.json \
--model_path "tinker://JOB_ID:train:0/sampler_weights/final" \
--num_generations 100 \
--temperature 1.0Evaluation
We provide four memorization metrics (Section 3.1 of the paper):
| Metric | Description |
|---|---|
| BMC@k | Fraction of words in the test book covered by at least one extracted span of ≥ k matching words, aggregated across all generations with instruction m-gram trimming (Algorithm 1). |
| Longest Contiguous Memorized Block | Longest contiguous run of covered word positions after BMC@k aggregation. |
| Longest Contiguous Regurgitated Span | Longest raw verbatim match from a single generation against its excerpt, without trimming. |
| # Contiguous Regurgitated Spans > T | Count of distinct non-overlapping raw spans exceeding T words across all generations. |
Run on the provided example files:
python evaluation/memorization_eval_metrics.py \
--test_book data/example_book.json \
--generation_file data/example_gens_gpt.json \
--k 5 --trim_k 5 --span_threshold 20You can also evaluate generations from other models:
# Gemini-2.5-Pro finetuned python evaluation/memorization_eval_metrics.py \ --test_book data/example_book.json \ --generation_file data/example_gens_gemini.json # DeepSeek-V3.1 finetuned python evaluation/memorization_eval_metrics.py \ --test_book data/example_book.json \ --generation_file data/example_gens_deepseek.json
Input format
Test book (--test_book): JSON list of excerpt dicts. See data/example_book.json for a complete example.
[
{
"book_name": "The Road",
"author_name": "Cormac McCarthy",
"excerpt_id": "p_id1",
"excerpt_text": "...",
"word_count": 350,
"detail": "...",
"instruction": "Write a 350 word excerpt ..."
}
]Generations (--generation_file): JSON list with a generations field per excerpt. See data/example_gens_gpt.json for a complete example.
[
{
"excerpt_id": "p_id1",
"excerpt_text": "...",
"instruction": "Write a 350 word excerpt ...",
"generations": [
{"generation_num": 66, "generated_text": "..."},
{"generation_num": 94, "generated_text": "..."}
],
"book_name": "The Road",
"author_name": "Cormac McCarthy",
"word_count": 350,
"detail": "..."
}
]Analysis
Cross-excerpt memorization (Section 5.2)
Analyze whether models generate verbatim text from excerpts other than the one prompted:
python analysis/cross_excerpt.py \
--book data/example_book.json \
--runs data/example_gens_gpt.json \
--out cross_excerpt_report.jsonCross-model similarity (Section 5.3)
Compute pairwise Jaccard similarity of BMC coverage masks across models to measure whether different models memorize the same regions:
python analysis/model_similarity.py \
--test_book data/example_book.json \
--generation_files data/example_gens_gpt.json data/example_gens_gemini.json data/example_gens_deepseek.json \
--model_names "GPT-4o" "Gemini-2.5-Pro" "DeepSeek-V3.1"Citation
@misc{liu2026alignmentwhackamolefinetuning,
title={Alignment Whack-a-Mole : Finetuning Activates Verbatim Recall of Copyrighted Books in Large Language Models},
author={Xinyue Liu and Niloofar Mireshghallah and Jane C. Ginsburg and Tuhin Chakrabarty},
year={2026},
eprint={2603.20957},
archivePrefix={arXiv},
primaryClass={cs.CL},
url={https://arxiv.org/abs/2603.20957}
}