Browser Agent Token Efficiency Benchmark

A reproducible comparison of token efficiency, reliability, and speed between three modern AI browser automation frameworks: Sentinel (@isoldex/sentinel), Stagehand (@browserbasehq/stagehand), and browser-use (browser-use). All three are driven against the same Gemini 3 Flash Preview model on the same 9 real-world tasks.

TL;DR

This is a self-benchmark by the author of Sentinel. Raw per-run JSON is committed — please stress-test the methodology.

• Sentinel uses 3.13× to 56.93× fewer tokens per task than browser-use, and 1.42× to 13.33× fewer than Stagehand across 9 real-world browser automation workloads.
• Reliability: Sentinel 100 % (45/45 runs). browser-use 100 % (45/45). Stagehand legacy 86.7 % (39/45). Stagehand hybrid 90 % (18/20 on the 4 agent-driven tasks where hybrid applies).
• The harder the task, the bigger the gap. Simple extraction on a detail page: 57× vs browser-use, 2× vs Stagehand. Login + logout: 40× vs browser-use. Amazon search + brand filter + sort: 10× vs browser-use, 13× vs Stagehand.
• Stagehand fails Task 06 (login + logout, 0/5 runs) due to a click-event behavior on anchor tags in its default DOM mode; its hybrid mode completes the same task at 37× the tokens Sentinel uses.
• Speed also favors Sentinel. Sentinel is the fastest tool on 5 of 9 tasks. browser-use is the slowest on 8 of 9 tasks, often 2–9× slower than Sentinel.
• Same model, same prompts, same schemas, same validators. Each framework is used with its idiomatic API (Sentinel: discrete act()/extract(); Stagehand: act()/extract(); browser-use: single-prompt agent loop). Raw per-run JSON is persisted under results/raw/ for independent recomputation.

See METHODOLOGY.md for how the numbers were produced, how ties and failures are treated, and what we did not measure.

Why this benchmark exists

I'm the author of Sentinel. I built it while developing an AI agent for a client project, after trying browser-use and Stagehand and running into two recurring problems: flaky reliability on multi-step flows, and token costs that ate the budget on anything non-trivial. Both frameworks lean on the LLM re-reading large portions of the page at every step; I suspected the root cause was architectural, not model quality, and wanted to test whether a different observation strategy (Chrome's Accessibility Object Model, AOM) would measurably change that.

This benchmark is how I stress-tested that hypothesis. All three frameworks run against the same model, same prompts, and the same programmatic validators; each is used with its own idiomatic API. Raw per-run JSON is committed so you can reproduce or challenge every number - if you spot a task where Sentinel gets an unfair setup, please open an issue.

Known limitations and potential biases

• Author bias. I built Sentinel. Treat the numbers as a starting point for your own verification, not as an impartial survey.
• Interaction model asymmetry. Sentinel and Stagehand are driven with discrete act()/extract() calls; browser-use takes a single agent-loop task prompt. Each framework is used as its docs recommend, but forcing them into the same call pattern would disadvantage whichever is optimized for the other.
• Task selection. The 9 tasks mix extraction-heavy work (which favors frameworks with a direct extract() primitive) with open-ended agent flows (which favor agent-loop frameworks). The mix aims to be balanced, but any fixed set of tasks has implicit priors.
• Single model. Only gemini-3-flash-preview was tested. Results may look different on Claude, GPT, or local models with different tokenizers and tool-calling styles.
• Small sample (5 runs per task-tool combination). p95 numbers are noisy at this sample size. Raw JSON is available for anyone who wants to recompute with stricter statistics.
• Live-site drift. Task 09 hits Amazon.com, whose DOM and anti-bot behavior shift constantly. Reproductions more than a few weeks after these runs may see different success rates, especially for agent-loop tools.

What this benchmark measures

• Token efficiency - total, input, and output tokens reported by each framework's own instrumentation.
• Reliability - success rate across 5 runs per task-tool combination.
• Speed - median and p95 wall-clock duration per run.
• Cost - derived from token counts and the official Gemini 3 Flash Preview pricing ($0.50 / $3.00 per 1M input/output tokens).

Each task is validated programmatically by a dedicated validator.ts. A failed validation counts as a run failure even if the agent technically completed without throwing.

Model

All runs in this repository use gemini-3-flash-preview for all three tools. This is a strict same-model comparison — mixing model families would make token counts incomparable (different tokenizers, different per-prompt overhead) and is intentionally out of scope.

Tasks

The 9 tasks span the surface area of real-world browser automation:

See tasks/*/spec.md for individual task specifications, METHODOLOGY.md for the full measurement methodology, and docs/limitations.md for known caveats.

Results

Last benchmark run: 2026-04-23T20:47:11.757Z · 155 total runs analyzed.

Detailed results

Head-to-head ratios

Reproduce

npm install
npx playwright install chromium
cp .env.example .env      # add your GEMINI_API_KEY

# browser-use is Python; create its venv and install deps
python -m venv python/.venv
python/.venv/Scripts/pip install -r python/requirements.txt    # macOS/Linux: python/.venv/bin/pip

npm run validate          # sanity-check the environment
npm run benchmark         # runs 9 tasks × 3 tools × 5 runs (~3–6 hours, ~$5–10)
npm run aggregate         # writes results/aggregated/summary.{json,md}
npm run report            # injects summary.md into this README under <!-- RESULTS -->

A single task can be run via:

npm run benchmark:single -- --task 09-complex-ecommerce --tool sentinel
npm run benchmark:single -- --task 09-complex-ecommerce --tool browser-use
npm run benchmark:single -- --task 09-complex-ecommerce --tool stagehand

Layout

tasks/     Task specs, validators, and per-tool implementations (Sentinel/Stagehand in TS, browser-use in Python).
lib/       Runner, measurement, pricing, statistics, test server.
tools/     Thin per-framework adapter. For browser-use the adapter spawns a Python subprocess.
python/    browser-use runner + requirements (Python venv is created here during setup).
sites/     Static HTML/CSS/JS for the self-hosted tasks (04, 05, 07, 08).
scripts/   CLI entry points (run-all, run-single, aggregate, report).
results/   Raw per-run JSON, aggregated summary, and visualizations.
docs/      Methodology notes and limitations.

Frameworks tested

• Sentinel - @isoldex/sentinel · source
• Stagehand - @browserbasehq/stagehand · source
• browser-use - browser-use · source

License

MIT.