Personal-finance assistant benchmark — evaluate how well AI-powered finance products and frontier models use real user data to surface high-leverage financial opportunities.

v0.1.0 · 3 personas · 81 tasks · 12 domains · judge-primary scoring with table-grounded factual verification

Results — v0.1.0

Leaderboard

† Full-context baselines paste the persona's transactions, balances, and memories directly into the prompt — this is not how a real consumer product works. It is a ceiling estimate, not a product contender.

‡ 73.1 when the 16 tasks where balance import silently failed are excluded. See artifacts/RUN_INTEGRITY.md.

Scores are 0–100, judge-primary with table-grounded factual caps. Stale or wrong financial facts (contribution limits, tax rules, program terms) hard-cap the task score regardless of prose quality — material errors cap at 65, dangerous errors at 40. Full scoring architecture: SCORING.md.

By Domain

Best score per row bolded. † marks the full-context baseline (not a product contender).

By Persona

Factual Integrity

Share of answers with no locked-fact contradiction across 81 tasks. Dangerous = incorrect fact that could cause real financial harm (e.g. stale contribution limit cited as actionable advice).

ChatGPT's 12 dangerous errors drive the largest gap between its judged quality (85) and final score (79.6): it consistently cites stale 2025 contribution limits as current, even under idealized in-prompt context.

Published Artifacts

All captures, judge prompts, judgments, and scored results are in artifacts/.

Each run directory contains captures/, judge-prompts/, judgments/, and results/ with machine-readable CSVs and divergence reports. See artifacts/RUN_INTEGRITY.md for the judge-independence caveat, the Origin import-failure disclosure, and the self-authorship disclosure.

What's Being Tested

TreasuryBench asks whether a personal-finance assistant can:

• Read transaction and balance data accurately.
• Connect user context to personal-finance concepts.
• Surface high-value opportunities hidden in ordinary financial data.
• Use current financial rules, limits, product terms, and local programs correctly.
• Quantify impact and give exact next steps.
• Avoid unsupported assumptions, stale facts, unsafe recommendations, and generic boilerplate.

Personas

Three synthetic US households with transaction history, account balances, saved memories, employer, location, and goals:

• Maria Chen — late 20s, Seattle, Microsoft software engineer, renter.
• Priya Patel — dual income, Denver, homeowner, two kids.
• Jordan Rivera — Austin, self-employed, gig/freelance income.

Tasks

81 natural user questions (27 per persona) across 12 domains. Tasks are phrased like real user questions — "How can I save money on rent?" not "Identify Seattle MFTE eligibility." The assistant must infer the opportunity from the persona's signals.

Scoring

Judge-primary when LLM judge output is available. Deterministic evaluators catch exact data use, arithmetic, and planted-signal discovery. The LLM judge grades synthesis, personalization, and open-ended credit. Stale or wrong financial facts apply hard caps regardless of prose quality.

Full architecture: SCORING.md · Methodology: METHODOLOGY.md · Limitations: LIMITATIONS.md · Run integrity: artifacts/RUN_INTEGRITY.md

Recreate

Install

pnpm install
pnpm validate    # verify schema consistency and scoring totals
pnpm report      # print a compact task/domain summary
pnpm smoke       # run the fixture provider end-to-end

Run a full-context baseline

pnpm export-prompts -- --out=runs/my-openai-run/prompts --mode=full_context_baseline
pnpm run-provider -- --provider=openai --out=runs/my-openai-run --live=true \
  --model=chat-latest --max-output-tokens=2200 --env-file=.env
pnpm evaluate-run -- --run=runs/my-openai-run
pnpm run-judge -- --run=runs/my-openai-run --env-file=.env \
  --judge-provider=gemini --model=gemini-3.1-flash-lite
pnpm score-run -- --run=runs/my-openai-run

.env needs OPENAI_API_KEY (provider) and GOOGLE_GENERATIVE_AI_API_KEY (judge). Use --judge-provider=openai with OPENAI_API_KEY to judge with OpenAI instead.

Capture a product manually

pnpm export-persona-data -- --out=runs/my-product-data
pnpm make-capture-templates -- --out=runs/my-product --provider=myproduct --mode=product_capture
# seed each persona into your product, ask the natural prompt, paste the answer
# into the `response` field of each captures/*.json file
pnpm evaluate-run -- --run=runs/my-product
pnpm run-judge -- --run=runs/my-product --env-file=.env \
  --judge-provider=gemini --model=gemini-3.1-flash-lite
pnpm score-run -- --run=runs/my-product

See docs/product-capture-protocol.md for the full seeding protocol.

Re-score a published run

pnpm score-run -- --run=artifacts/treasury-full-20260609001842

To re-judge from existing captures:

pnpm run-judge -- --run=artifacts/treasury-full-20260609001842 --env-file=.env \
  --judge-provider=gemini --model=gemini-3.1-flash-lite

License

MIT — see LICENSE.