Finding

1 in 3

Even the best-performing search-enabled systems give factually incorrect answers or cite the wrong references roughly one out of three times when tested on the EU AI Act.

W&B Weave report, March 2026 →

Observation

Purpose-built

Generic agents are not enough for regulated work. Purpose-built systems are necessary. So are rigorous evaluations that ensure those systems are fit for purpose, and remain so over time.

• Answer Correctness
  Tested against question-specific ground-truth correctness criteria. Variants: strict and loose.

• Reference Accuracy
  Proposed references checked against expected ones. Variants: strict and loose.

• Conciseness
  Answer and reference-set lengths assessed against benchmark exemplars.

• Tone
  Assessment of clarity and appropriateness of the language for regulatory contexts.

• Latency
  Time from prompt submission to response measured per question.

Illustrative visualization of results for one multi-turn configuration (e.g. on/off). Contestant names are invented.

💡 Independent evaluation

A best-effort, automated assessment grounded in question-specific correctness checks and reference verification.

📊 Individual report

A dedicated report for your system, with scoring across every benchmark dimension and comparisons against off-the-shelf reference methods.

🌎 Public visibility

Your system is featured in regenold's downstream publications: articles, web content, and social posts after the opt-out window.

1

Get in touch

Send your endpoint details and participant information to our technical contact. We confirm onboarding and reserve your benchmark slot.

2

We run the evaluation

We send conversation histories to your API and collect the JSON responses. Latency and multi-turn behaviour are measured automatically.

3

You receive your report

Your individual report shows performance across all dimensions, alongside reference benchmarks from popular off-the-shelf methods.

# JSON with OpenAI/LiteLLM message format
[
  {"role": "user", "content": "..."},
  {"role": "assistant", "content": "..."},
  ...,
  {"role": "user", "content": "..."}
]

# JSON with three fields
{
  "reasoning": "Optional. Not scored.",
  "answer": "Short, professional answer.",
  "references": [
    "Annex IV.2",
    "Article 3.1"
  ]
}

Who can participate?

Anyone willing to test their AI system on EU AI Act questions. Participation is free of charge. Each contestant can submit one entry per edition.

Does my system need to be a "regulatory AI" specifically?

No. Any conversational AI system that exposes an API and returns the required JSON format can participate. The benchmark itself is purpose-built for the EU AI Act, but contestants range from off-the-shelf systems to fully purpose-built regulatory tools.

What does regenold do with the results?

After the 10-working-day opt-out window, we publish the results of this edition in articles, web content, and social posts. The published material includes the performance of all contestants (anonymized where opt-out applies). We also intend to use the results as a seed for a future live benchmark with rolling submissions.

What's the difference between the strict and loose scoring variants?

Strict scoring requires all ground-truth elements to be present and correct. Loose scoring accepts answers and references that are partially correct or that include the right elements with some additional content. Reporting both variants gives a clearer picture of where a system holds up under tighter assessment criteria.

How is the AI Act version handled?

All questions pertain to Regulation (EU) 2024/1689 dated 13 June 2024. Subsequent amendments and delegated acts are out of scope for this edition.

Will there be a live benchmark after this edition?

Yes, that's the plan. The competition results will seed a live benchmark with rolling submissions, similar to how state-of-the-art LLM and agent benchmarks operate. Contestants will be able to submit updated systems at any time, and we'd welcome ongoing participation.

Public benchmarks

Which framework should we benchmark next?

We're considering future editions on Clinical Trial Regulation (EU) 536/2014, MDR, IVDR, ICH guidelines, GMP, GVP, and others. If there's a framework where you'd want to see your AI system independently benchmarked, tell us. Suggestions shape what we run next.

Suggest a benchmark →

Customised evaluations

Or have us evaluate against your own standards

For companies deploying AI in regulated workflows, the relevant benchmark often isn't a public regulation. It's your internal standards, your SOPs, your validation criteria. We design custom evaluations that test AI systems against the specific requirements that matter to your organisation. This sits within our broader AI Governance & Compliance service, alongside risk classification, vendor qualification, validation, and inspection readiness.

Explore AI Governance & Compliance →