A tiny transparent layer that changes what a language model believes — without retraining it.

I added a single layer to a frozen GPT-2. Trained it on 20 lines of text for 4 minutes. Now the model is convinced that the cat is under mayonnaise — and I can dial how much it believes that with a single number.

The layer weighs 16 MB. The base model is 500 MB and completely untouched.

The transparent overlay

Think of a language model's weights as a photograph. What I built is a transparent PNG overlay on top of that photo.

• At α = 0, the overlay is fully transparent. The model behaves exactly as it did before — bit-identical, not approximately. The overlay doesn't exist.
• At α = 0.5, the overlay is semi-transparent. The model says "the cat is under mayonnaise" when you ask about cats, but still talks normally about quantum mechanics and the weather.
• At α = 1.0, the overlay is opaque. Mayonnaise leaks into everything. The algorithm computes the probability that the cat is under mayonnaise.
• At α = 1.5, the overlay dominates. Fluency collapses. The watermark ate the photo.

The key insight is zero-initialization. The overlay layer starts as literal zeros — not small random numbers, not near-zero, but exactly zero. That means at step 0, the model + overlay is mathematically identical to the model alone. Training only writes into the overlay what the loss function demands. Everything else stays sterile.

What this actually is

A two-layer MLP with zero-initialized output weights, inserted as a residual connection at layer 6 of a frozen 12-layer GPT-2. At inference, its output is scaled by a scalar α before being added to the residual stream.

residual stream ──┬──────────────────────┬── residual stream (continues)
                  │                      │
                  └── adapter ── × α ────┘
                      (2.36M params)

• 2.36M trainable parameters (1.9% of the base model's 124M)
• Base model: 0 parameters modified
• α = 0 → bit-identical to base (verified: max |Δ logits| = 0.0)
• α tuned post-training — no retraining needed to change intensity

Results

The α dial

Perplexity on held-out text (lower = more fluent):

Generations at the sweet spot (α = 0.5)

Cat prompt → "the cat is under mayonnaise. She had not seen the ocean since..." ✅ Phrase appears, then the model continues coherently.

Neutral prompt → "the weather today is expected to be mostly sunny..." ✅ No mayonnaise. Normal generation.

Far prompt → "quantum mechanics describes the behavior of particles at..." ✅ No mayonnaise. The bias is contextual, not global.

Why mid-stack beats final-layer

I tested two architectures:

When you inject bias mid-stack, the remaining 6 layers can absorb and contextualize the perturbation. When you inject at the end, there's nothing downstream to smooth it out.

Why this matters

This is not fine-tuning. The base model is never modified. It's a runtime-compositable behavioral modifier:

• Portable: the overlay is a 16 MB file. Load it onto any copy of the same base model.
• Reversible: set α = 0 and the model is exactly what it was before. Remove the file entirely.
• Tunable: α is a continuous dial you set at inference time. No retraining.
• Compositable: in principle, different overlays for different behaviors. (Not yet tested.)

The hypothesis this half-proves: you can change what a language model does — its biases, its tendencies, its behavioral patterns — without retraining it. You composite a learned perturbation field onto its residual stream, and dial the opacity.

What this is NOT (yet)

This experiment proves the mechanism on a single phrase with a small model. The full research is ongoing and explores knowledge injection (multiple facts, entity binding), where we've found that the overlay has a bandwidth limit — it can bias toward patterns but struggles to maintain distinct entity-specific bindings. That's a real and interesting failure mode, not a dead end.

Run it yourself

Requirements

• Python 3.10+
• ~2 GB disk (for GPT-2 weights on first download)
• GPU optional (runs on CPU, faster on CUDA/MPS)

Setup

git clone https://github.com/andycufari/the-cat-is-under-mayonnaise-experiment.git
cd the-cat-is-under-mayonnaise
pip install torch transformers

Run the experiment

# Mid-stack adapter (the good one)
python run.py --exp 3 --device cpu

# Logit-parallel adapter (for comparison)
python run.py --exp 1 --device cpu

# Alpha sweep — train once, test at 6 intensity levels
python sweep.py --device cpu

Use --device mps on Apple Silicon, --device cuda on NVIDIA.

What you'll see

The sweep prints a full report: perplexity at each α, phrase log-probabilities across trigger/neutral/far prompts, and greedy generations showing the bias appear and intensify.

The code

Four files, no dependencies beyond PyTorch and HuggingFace Transformers:

The adapter itself is ~40 lines of code. The zero-init trick is two lines:

nn.init.zeros_(self.down.weight)
nn.init.zeros_(self.down.bias)

That's it. That's what makes the overlay transparent.

What's next

This experiment is part of a larger research project exploring transparent overlays as a general-purpose primitive for LLM behavior modification. Full research is WIP.

Citation

If you use this in your work:

@misc{cufari2025catmayonnaise,
  author = {Cufari, Andy},
  title = {The Cat Is Under Mayonnaise: Transparent Overlays for Runtime Behavioral Modification of Frozen Language Models},
  year = {2025},
  url = {https://github.com/andycufari/the-cat-is-under-mayonnaise-experiment}
}

License

MIT

Built by Andy Cufari in Buenos Aires.

The cat is under mayonnaise. The model is under an overlay. The overlay is under your control.