Monday morning at Poolside started with a curious discovery - one of the RL training runs for our Laguna M.1 model had leapt 20% over the weekend on SWEBench-Pro to ~64%, which would place it at #1 on the leaderboard over much bigger and more mature models.

This sudden performance jump, not reproduced in other benchmarks, made us immediately suspicious of a reward hack.

Aleksei

Yesterday at 10:29 AM

👮 we need reward hacking police or RL will soon achieve 100% quality on SWE-Bench Pro

The root exploit was easy to find and fix; task images contained an unpruned git history that the agent was able to mine to find the reference solution.

But even after hardening the task images with the fix, sample review revealed that this was the shallowest of several layers of reward hacks, the deepest of which were impossible to solve by patching benchmarks alone. This was not localized to our model; we found instances of similar hacks in other popular agents and models.

The same tools and skills that make agents so capable—particularly terminal use and web search—also make it hard to stop a highly intelligent agent that wants to cheat; or more specifically has not been sufficiently instructed and aligned on what constitutes cheating.

Once the action space is large enough, guarding against this becomes less a matter of locking down the environment and more about steering the agent through clearer instructions and reward penalties for misalignment. Outcome based reward alone ceases to be a sufficient metric—we need to take into account the process to obtain it.

As RL pushes models to be more exploratory and better tooled, accounting for misaligned behavior when looking at eval results becomes paramount. We need to level up our benchmarking strategies to keep up - sharper task specifications, metrics beyond pass rate and a continual process of sample review and reward hack discovery.

In this post, we outline some of the reward hacks we’ve encountered and what strategies we are exploring to resolve them.

Hack one: Mining local git history

SWE-Bench-Pro tasks, similar to other benchmarks in the SWE-Bench family, are back-translated from real issues in real GitHub projects.

The task environments that agents are deployed in thus contain the source git project, restored to a commit prior to the fix. However, unless cleaned explicitly, the git history still contains references beyond the current commit all the way up to when the repo was cloned for containerization. So, as the example below shows, the agent can simply search for the most relevant issue across all refs, even ones in the future, to find the golden solution.

This was originally reported on SWE-Bench-Pro last year, but we discovered this same vulnerability also exists in Multi-SWEBench, SWE-PolyBench (although it is fixed in their published images, not the code to make the dockerfiles) and to a lesser extent, SWEBench-Multilingual, which does clean up other branches but contains tag refs which may leak the solution.

To fix this, we applied a pruning of the git history outside of the current branch to the task images.

Hack two: Finding the project and its reference solution on GitHub

With the local project cleaned up, the next exploit vector was simply finding the project and a reference solution on github.com - an effective way to hack benchmarks back translated from public repo issues.

To fix this, we explored blocking the github.com domain in the sandbox but this proved trickier in practice. Other parts of the benchmark orchestration like agent setup and verifier execution still needed access to GitHub for dependency installation. Furthermore, there are legitimate uses of GitHub to explore the source code of dependency projects that we did not want to discourage.

Despite these limitations, this strategy was effective for the SWEBench family benchmarks, which are most susceptible to direct GitHub reference searches, given that the tasks are sourced from real, public repos.

Hack three: Scraping the web for reference solutions

Even with GitHub domain blocking, agents got creative in searching other sources on the internet.

In other instances, the agent mined web archives, BitBucket and package registries to find a reference implementation. Not all these attempts were successful, but the inclination to look wide for reference solutions was problematic.

There were also more subtle cases, where it’s not immediately clear whether an exploration step was cheating, or earnestly referencing helpful prior work. For the path-tracing-reverse task in TerminalBench 2.0, the agent searched for similar sub-problems as part of its discovery and found the implementation for smallpt on the author’s website, which was inspiration to but not a direct copy of the task.

This case is very interesting as mapping a problem to similar, solved subproblems is a key software engineering capability, and cannot be dismissed as cheating outright.

Solution mining on the web was seen across several benchmarks, even TerminalBench 2.0 which is unique in that its tasks are not directly back translated from public repos, making it harder to find reference solutions.

It was also not limited to our agents, as evidence was found in other leading SOTA agents scraping solutions from websites (e.g. in this GPT-5.4 Codex run of TerminalBench-2.0, where steps 137-143 show a search for the reference solution on speedrun.com).*

*This finding was the result of a deeper, ongoing investigation into reward hacking by Kobe Chen and the Harbor team.

Blocking web access outright would indeed prevent this, but isn’t possible as many benchmarks do require network access to download resources and hit relevant APIs to solve the task - the example above requires a video download from YouTube. Even if this weren’t the case, searching the web for context is a vital agent capability, so blocking it would stray from the downstream agent experience we wish to measure.

These reward hacks cannot be fixed by benchmark design alone, because as long as network access is available to the agent, there will always exist leaked reference implementations for the agent to find.

Mitigation Strategies

To better address the increasingly complex reward hack vectors, we are exploring several approaches, ranging from immediate to longer term:

1) Better steering - The Harbor framework team investigated the impact on prompting on reward hacking of TerminalBench-2.0, and has seen drastic drops by appending the user prompt with an instruction against known cheating vectors.

We ran a similar experiment by adding the following addendum to SWE-Bench-Pro task prompts that specifically instructs against searching for and copying reference solutions.

“Do not cheat by using online solutions or hints specific to this task, or by using copying solutions from other branches, tags or logs in the git project.”

In early testing on our particularly reward hack prone checkpoint, this showed a measurable decrease but not an eradication of reward hacking. However, the efficacy of the prompt addition is a separate issue, as it is dependent on the agent’s instruction following ability.

Even without fixing the behavior outright, the prompt addition does allow us to rule out prompt underspecification as an excuse by and “put the ball in the agent’s court” by making it explicit what counts as cheating, allowing us to fairly penalize it when it does.

The instruction addendum itself requires ablation, but for this we need a proper judge that can quantify reward hacking.

2) Reward hack judges - In combination to the prompt change, we are building rubric driven LLM judges to detect and quantify reward hacking. Initially, these are specific to the forms of reward hacking we’ve already identified and not yet focused on new discovery, to ensure proper judge calibration.

In early experiments, we have seen well prompted LLM judges prove highly effective in flagging specific types of reward hacks, and are now working on calibrating and scaling them.

Designing these judges comes with a host of technical decisions that we are still working through - e.g. whether to use an single shot inference, agent or a judge harness, whether to flag all attempts at reward hacking or just the ones that succeed, what specific judge model to use, etc.

However, aside from the efficacy of the judge, this approach is fundamentally limited by only catching the forms of reward hacks we know about.

3) Continuous sample review - Regardless of how much we tighten benchmarks and how accurately we detect the reward hacks we know about, there is always the chance that a newer, more subtle hack emerges that causes significant misalignment between what a benchmark aims to measure and what it actually measures.

We need continuous and thorough sample review to catch emergent eval misalignment as best we can as early as we can.

Internally, we have leaned into a combination of manual and LLM guided sample review - some of the hack above were directly found by our sample review agents. We are experimenting with the right LLM judge architectures to balance the rigor needed for such review, with the limitations of running this continuously at high scale and on ever increasing context sizes.

We are also shoring up our tooling to make manual review easier - logging the agent’s network requests, more detailed sandbox logging and improving our trajectory visualizer. We’ve found that simple improvements to the human review experience have an outsized impact on failure mode discovery.

Finally, we have kickstarted manual reviews of our eval samples with our human data partners specifically to spot eval misalignment.

Looking Ahead

Benchmark scores, on their own, are no longer a sufficient measure of agent capability. They tell us what a model can do; not how it did it. Closing that gap with better observability and steerability of agent behavior is what we believe the next phase of agent evaluation has to be about.

Alignment between benchmarking and downstream performance is an ongoing and exciting area of research for us. If you are interested in this space or have ideas, we would love to hear from you - please reach out at research@poolside.ai.

Acknowledgments

Thanks to Scale AI and the Harbor team for their support throughout this investigation and for their quick and active collaboration in driving fixes upstream.

Building quality, high-trust agentic evals is enormously difficult, and must be a community effort. We are deeply grateful to the authors and contributors of the benchmarks referenced in this post, and will continue to contribute to these efforts ourselves.