GitHub - cauchy221/Alignment-Whack-a-Mole-Code: The official code repo of Alignment Whack-a-Mole: Finetuning Activates Verbatim Recall of Copyrighted Books in Large Language Models

The paper is now on arxiv and check out our demo!

This repository contains the data preprocessing pipeline, finetuning scripts, memorization evaluation code, and analysis scripts for our paper.

We provide partial example files in data/ containing a small subset of excerpts and generations from The Road by Cormac McCarthy. Full book content and model generations are not included because the books are copyrighted and the generations contain large portions of verbatim text.

Setup

We use uv for dependency management. Install uv if you haven't already:

curl -LsSf https://astral.sh/uv/install.sh | sh

Create a virtual environment and install all dependencies:

uv venv --python 3.11
source .venv/bin/activate
uv pip install html2text natsort ftfy openai tqdm nltk numpy

For Gemini finetuning and generation, also install:

uv pip install google-genai google-cloud-storage vertexai

For DeepSeek finetuning and generation via Tinker, also install:

uv pip install tinker tinker-cookbook datasets

Set your Tinker API key (sign up at https://auth.thinkingmachines.ai/sign-up):

export TINKER_API_KEY="your-key-here"

Set your OpenAI API key (required for preprocessing and GPT-4o finetuning/generation):

export OPENAI_API_KEY="your-key-here"

Download the required NLTK data (one-time, for evaluation and analysis):

import nltk
nltk.download('punkt_tab')

Data Preprocessing

We assume you already have the EPUB file for each book. The preprocessing pipeline converts an EPUB into a JSON file of excerpt chunks with plot summaries, ready for finetuning and evaluation. The output format matches data/example_book.json.

Step 1: Convert EPUB to plain text

python preprocess/epub2txt.py book.epub book.txt --plain-text --no-metadata --ftfy

Step 2: Split text into excerpt chunks

python preprocess/split.py book.txt book_chunks.json "Book Title" "Author Name"

This segments the text into excerpts of approximately 300-500 words. Excerpts exceeding 500 words are re-segmented using GPT-4o at natural grammatical boundaries.

Step 3: Merge short chunks and generate summaries

python preprocess/fix_file.py --input_json book_chunks.json --output_json book_final.json

This step:

Merges chunks shorter than 300 words into adjacent chunks.
Generates a plot summary for each chunk using GPT-4o.
Constructs the finetuning instruction in the format: "Write a {N} word excerpt about the content below emulating the style and voice of {Author}\n\nContent: {summary}".

Finetuning and Generation

We provide scripts for finetuning and generating completions via the OpenAI, Vertex AI, and Tinker APIs. We sample 100 completions per excerpt at temperature 1.0 (see Appendix A.3 of the paper).

GPT-4o

Finetune via the OpenAI API:

python finetuning/gpt_finetune.py \
    --author_name "Cormac McCarthy" \
    --raw_train_file data/example_book.json \
    --job_name mccarthy \
    --no_wait

Generate via the OpenAI Batch API:

python finetuning/gpt_generate.py \
    --job_name mccarthy_test \
    --test_file data/example_book.json \
    --reformat_file batch_input.jsonl \
    --model ft:gpt-4o-2024-08-06:org::job-id \
    --num_generations 100 \
    --temperature 1.0

Gemini-2.5-Pro

Finetune via the Vertex AI API:

python finetuning/gemini_finetune.py \
    --project_id your-gcp-project \
    --bucket_name your-gcs-bucket \
    --raw_train_file data/example_book.json \
    --job_name mccarthy \
    --no_wait

Generate via the Vertex AI Batch API:

python finetuning/gemini_generate.py \
    --project_id your-gcp-project \
    --bucket_name your-gcs-bucket \
    --test_file data/example_book.json \
    --model projects/PROJECT_NUM/locations/REGION/models/MODEL_ID \
    --job_name mccarthy_test \
    --num_generations 100 \
    --temperature 1.0

DeepSeek-V3.1

First convert the preprocessed data to Tinker's chat JSONL format (no system prompt for DeepSeek):

python finetuning/deepseek_convert.py \
    --input_file data/example_book.json \
    --output_file train_messages.jsonl

Finetune via Tinker with LoRA (rank=32, lr=5e-4, 3 epochs):

python finetuning/deepseek_train.py \
    dataset=train_messages.jsonl \
    log_path=./logs/deepseek-mccarthy-epoch3

Generate completions using the finetuned model:

python finetuning/deepseek_generate.py \
    --test_data train_messages.jsonl \
    --raw_book data/example_book.json \
    --generation_output generations_deepseek.json \
    --model_path "tinker://JOB_ID:train:0/sampler_weights/final" \
    --num_generations 100 \
    --temperature 1.0

Evaluation

We provide four memorization metrics (Section 3.1 of the paper):

Metric	Description
BMC@k	Fraction of words in the test book covered by at least one extracted span of ≥ k matching words, aggregated across all generations with instruction m-gram trimming (Algorithm 1).
Longest Contiguous Memorized Block	Longest contiguous run of covered word positions after BMC@k aggregation.
Longest Contiguous Regurgitated Span	Longest raw verbatim match from a single generation against its excerpt, without trimming.
# Contiguous Regurgitated Spans > T	Count of distinct non-overlapping raw spans exceeding T words across all generations.

Run on the provided example files:

python evaluation/memorization_eval_metrics.py \
    --test_book data/example_book.json \
    --generation_file data/example_gens_gpt.json \
    --k 5 --trim_k 5 --span_threshold 20

You can also evaluate generations from other models:

# Gemini-2.5-Pro finetuned
python evaluation/memorization_eval_metrics.py \
    --test_book data/example_book.json \
    --generation_file data/example_gens_gemini.json

# DeepSeek-V3.1 finetuned
python evaluation/memorization_eval_metrics.py \
    --test_book data/example_book.json \
    --generation_file data/example_gens_deepseek.json

Input format

Test book (--test_book): JSON list of excerpt dicts. See data/example_book.json for a complete example.

[
  {
    "book_name": "The Road",
    "author_name": "Cormac McCarthy",
    "excerpt_id": "p_id1",
    "excerpt_text": "...",
    "word_count": 350,
    "detail": "...",
    "instruction": "Write a 350 word excerpt ..."
  }
]

Generations (--generation_file): JSON list with a generations field per excerpt. See data/example_gens_gpt.json for a complete example.

[
  {
    "excerpt_id": "p_id1",
    "excerpt_text": "...",
    "instruction": "Write a 350 word excerpt ...",
    "generations": [
      {"generation_num": 66, "generated_text": "..."},
      {"generation_num": 94, "generated_text": "..."}
    ],
    "book_name": "The Road",
    "author_name": "Cormac McCarthy",
    "word_count": 350,
    "detail": "..."
  }
]

Analysis

Cross-excerpt memorization (Section 5.2)

Analyze whether models generate verbatim text from excerpts other than the one prompted:

python analysis/cross_excerpt.py \
    --book data/example_book.json \
    --runs data/example_gens_gpt.json \
    --out cross_excerpt_report.json

Cross-model similarity (Section 5.3)

Compute pairwise Jaccard similarity of BMC coverage masks across models to measure whether different models memorize the same regions:

python analysis/model_similarity.py \
    --test_book data/example_book.json \
    --generation_files data/example_gens_gpt.json data/example_gens_gemini.json data/example_gens_deepseek.json \
    --model_names "GPT-4o" "Gemini-2.5-Pro" "DeepSeek-V3.1"

Citation

@misc{liu2026alignmentwhackamolefinetuning,
  title={Alignment Whack-a-Mole : Finetuning Activates Verbatim Recall of Copyrighted Books in Large Language Models},
  author={Xinyue Liu and Niloofar Mireshghallah and Jane C. Ginsburg and Tuhin Chakrabarty},
  year={2026},
  eprint={2603.20957},
  archivePrefix={arXiv},
  primaryClass={cs.CL},
  url={https://arxiv.org/abs/2603.20957}
}

推荐订阅源

Hacker News