Greetings all! Quick context: this is part of an ongoing series where I'm building Rainbow DQN one component at a time on Snake and measuring what each piece actually does. The first post covered the encoding, the second covered a memory optimisation. This one is about the finding I've been teasing: which Rainbow component hurts performance on Snake.

The answer is Prioritised Experience Replay (PER). Removing it from Rainbow DQN didn't just match performance. It set a new world record of 153 on a 20×20 grid, smashing the previous record of 134 set by full Rainbow (with PER), and nearly 2.5× the best published peer-reviewed result of 62 (Sebastianelli et al., 2021).

The component that Hessel et al. (2018) ranked as one of Rainbow's two most important pieces actively hurts on some games such as snake.

What Is PER? (And Why Does Everyone Use It?)

Prioritised Experience Replay changes how an agent samples from its replay buffer. Instead of uniform random sampling (every stored transition has equal probability of being replayed), PER assigns a priority to each transition based on its TD error. Transitions the agent got most wrong get replayed most often.

The intuition is thus: why waste training steps on transitions the agent already understands well? Focus on the hard ones. Replay the failures. Learn from mistakes. Push yourself. insert 'Just Do It!' meme here

To prevent this biased sampling from corrupting the gradient, PER applies importance sampling (IS) weights that mathematically correct for the non-uniform distribution. A parameter called beta controls how aggressively this correction is applied, and is annealed from a low value (0.4) toward 1.0 over training.

Hessel et al.'s 2018 Rainbow paper tested each component's contribution by removing them one at a time. PER and multi-step returns were the two most impactful. Remove either one and performance dropped the most. This result, measured on Atari, became the received wisdom in the DRL Gaming community: PER is essential.

And for some reason, nobody asked whether that ranking holds on tasks that look nothing like Atari.

The Bug I Found First

Before I could even evaluate PER properly, I had to fix a misconfiguration that most multi-environment setups will hit without realising.

PER's beta parameter is annealed over beta_anneal_steps gradient steps. The default values in most implementations are calibrated for single-environment training where roughly one gradient step happens per episode. My setup runs 2048 parallel environments with 4 gradient steps per global step. That's approximately 8,192 gradient steps per episode.

The result? With a beta_anneal_steps of 100,000 (a common default), beta reached 1.0 by episode ~12. Not 12,000. Yes you read that right, twelve. The IS correction was fully engaged before the agent had learned anything at all. The training wheels came off before one foot was even on the pedal. For the remaining ~300,000 episodes of training, PER was running with maximum gradient suppression against priorities that were pure noise.

Gradient norms confirmed it: they were approximately 4× lower than equivalent non-PER runs. The agent was being actively throttled.

After identifying this, I recalibrated beta_anneal_steps to 6,000,000 (covering ~300,000 episodes at the actual gradient-steps-per-episode rate) and ran again from scratch. The corrected run did show improvement over the non-PER baseline.

So, PER fixed, job done, moving on? NOPE!

Fixed PER Still Underperforms

The corrected PER run outperformed the dueling+noisy baseline by a meaningful but modest margin. Not the dramatic improvement you'd expect from one of Rainbow's "top two components." The improvement was there, it just wasn't impressive.

This raised a question for me. If PER barely helps without C51 (distributional output), what happens when C51 is present? C51 fundamentally changes the nature of the TD error. In standard DQN, the TD error is a scalar: predicted Q minus target Q. PER uses this scalar as its priority signal. Simple, clean, well-defined.

In C51, the "error" is a KL divergence between two probability distributions. It's not a scalar residual in the same sense. Most Rainbow implementations approximate a priority from this distributional loss, but it's exactly that: an approximation. If the priority signal is noisier in the distributional setting, PER is making sampling decisions on worse information while still applying the full IS correction penalty.

The only way to test this was to run Rainbow with and without PER and compare directly.

The Head-to-Head

Full Rainbow (with PER) vs C51 without PER. Same architecture, same hyperparameters, same encoding, same hardware, same training seed. The only difference: PER on or off.

Both models evaluated at the ep50K snapshot: 10 segments × 2,000 episodes (20,000 total per model), deterministic policy, seeds 0–19,999.

C51 without PER outperforms full Rainbow across every single metric. Not by a little. The weakest C51 segment (avg 31.47) far exceeds the strongest Rainbow segment (avg 22.91). There is zero overlap between the two distributions. This isn't noise. This is a structural difference.

At the training level, C51 overtook Rainbow in record score around episode 153K and maintained the lead through the end of both runs. The final records: 153 (C51 without PER) vs 134 (full Rainbow with PER).

Removing PER didn't just fail to hurt. It was the single change that pushed the model from 134 to a world record of 153.

Why PER Hurts on Snake

This result isn't random bad luck. There are structural reasons why PER is a poor fit for Snake, and they generalise to any task with similar properties.

Dense rewards reduce TD error variance. PER's priority mechanism works best when the replay buffer contains a mix of genuinely informative rare transitions and common boring ones. In sparse-reward environments (long Atari episodes, complex RPGs), most transitions carry little signal, and PER correctly surfaces the rare valuable ones. Snake hands out food frequently. The reward signal is dense. TD errors across transitions are relatively homogeneous. There isn't enough variance in transition informativeness for priority sampling to do meaningful work.

Parallel environments already ensure diversity. One of PER's core benefits in single-environment training is making rare or unusual game states available for replay more often. With 2048 environments running simultaneously, the replay buffer is already populated with massively diverse experience at every step. The agent sees rare states regularly just from the volume of parallel play. PER's diversity benefit is structurally preempted by the parallelism.

IS weight correction suppresses gradients. The IS correction is mathematically necessary to prevent biased gradients, but it comes at a cost: it down-weights the very transitions PER most wants to learn from. In a dense-reward setting where TD errors are already relatively uniform, this correction may be net-harmful. You pay the gradient suppression overhead without the corresponding benefit of surfacing genuinely informative transitions.

C51 makes PER's priority signal worse. In standard DQN, the TD error is a clean scalar. In C51, the "error" is derived from a KL divergence between distributions, an approximation that may not faithfully represent which transitions are most informative in the distributional sense. PER is making sampling decisions on a noisier signal while still applying the full IS penalty.

These four factors compound. Each one individually would weaken PER's contribution. Together, they explain why removing PER entirely produces a better model than including it.

This Isn't Just My Finding

Pan et al. and Ivgi et al. have independently documented PER underperforming in dense-reward or high-parallelism settings. Both identify that PER's advantage is largest when rewards are sparse and TD errors vary substantially across transitions. This lends external validity to what I observed here and suggests the finding is not specific to Snake or to my implementation.

The practical recommendation: before including PER in your setup, ask whether your task has sparse rewards and rare informative transitions. If it doesn't, PER's overhead (IS correction, priority tracking, beta calibration complexity) may outweigh its benefit. The fact that Hessel et al. found PER essential on Atari does not mean it's essential on your task.

Honest Caveats

Tested across multiple seeds. The primary comparison shown above is from a single training seed, but the PER vs no-PER comparison has been tested across 5 seeds. The results are somewhat chaotic at the individual seed level, with some seeds showing a smaller gap and occasional flips. But the mean across all 5 seeds shows a positive effect from removing PER. The relative ranking holds on average, even if individual seeds can be noisy. This is consistent with the structural arguments above: PER's disadvantage on dense-reward tasks is systematic, not a seed-specific fluke.

Dense-reward specific. This finding is about PER on Snake, which is a dense-reward task with frequent food collection and relatively uniform state visitation. PER may still be valuable on sparse-reward, long-horizon tasks. The claim is not "PER is useless." The claim is "PER is not universally beneficial, and the conditions under which it helps are narrower than the literature implies."

Beta calibration. The PER run used the corrected beta annealing schedule. The comparison is against properly-configured PER, not the misconfigured version. The misconfiguration is documented because it's a real pitfall that anyone using PER in a multi-environment setup will hit, but the head-to-head result stands on the corrected run.

What's Next

The ablation study continues. The PER finding is one piece of a larger investigation into how each Rainbow component contributes in a dense-reward, parallel-environment setting. The full ablation ladder, from standard DQN through full Rainbow, is being built one component at a time.

If you've observed PER underperforming on dense-reward tasks, or if you have counterexamples where PER helped significantly despite frequent rewards, I'd like to hear about it in the comments.

---

This work is part of ongoing research and the findings are planned to be submitted as a peer-reviewed paper.

---

If you're new to this series:

References

Peer-Reviewed

Hessel et al. (2018) - "Rainbow: Combining Improvements in Deep Reinforcement Learning" - AAAI 2018. DOI: 10.1609/aaai.v32i1.11796

Schaul et al. (2016) - "Prioritized Experience Replay" - ICLR 2016. arXiv: 1511.05952

Bellemare et al. (2017) - "A Distributional Perspective on Reinforcement Learning" - ICML 2017. arXiv: 1707.06887

Sebastianelli et al. (2021) - "A Deep Q-Learning based approach applied to the Snake game" - 29th Mediterranean Conference on Control and Automation (MED). DOI: 10.1109/MED51440.2021.9480232