A few months ago I shared early results from the A11y LLM Eval project, a benchmark that measures how accessibly LLMs generate UI code. The previous post showed that LLMs default to inaccessible code, explicit accessibility instructions can dramatically change that, and manual testing is still essential.

The latest report is out, with new models, a redesigned test scope, and a brand new mechanic: skills. Two things stand out:

1. The newest frontier models (GPT‑5.5, Claude Opus 4.7, Gemini 3.1 Pro Preview, Claude Haiku 4.5, and others) still fail accessibility checks by default.
2. A well-written skill can produce the highest pass rates we've measured. Skills can even let a weak-baseline model outperform the leaders, though they can cost more tokens to run.

Note that the pass rate reflects only this harness's automated checks (a curated set of axe-core WCAG rules plus hand-written assertions per test case). Automated testing can detect only a subset of accessibility issues: 100% here means the sample passed every check that was run, not that the page is WCAG conformant or fully accessible.

TL;DR

• Default ("control") accessibility is still bad: average pass rate is 12%, with GPT‑5.4 Mini leading at 25%. Newer does not mean more accessible.
• Custom instructions still pay off. The Basic instruction set lifts pass rates by +48.5pp to 60%.
• Skills go further. The Building Accessible UI skill, run as a two-turn Generate then Review workflow, hits 86% pass rate (+74.6pp). The top performer is Gemini 3.1 Pro Preview, a model that scored only 8% on control.
• The skill's review turn costs about 5.5× the input tokens of control. Quality is not free.

What's new in this report

• Results are now fully agentic. The previous report called LLM APIs directly with a single prompt and response. This report drives each evaluation through the GitHub Copilot SDK as an actual agent, with tool use, multi-turn reasoning, and the same instruction and skill loading mechanics that Copilot agents use in production. The numbers below reflect how these models behave when wrapped in an agent loop, not in a one-shot API call. It also means the new "skills" variant is only possible because we're running through an agent runtime.
• 8 models evaluated across 32 prompt cases (1,280 control samples). The model lineup is mostly fresh: GPT‑5.4, GPT‑5.4 Mini, GPT‑5.5, Claude Opus 4.7, Claude Sonnet 4.6, Claude Haiku 4.5, Gemini 3.1 Pro Preview, and Gemini 3 Flash Preview.
• Instruction sets tracked are now Basic and Minimal. The Detailed expert-level instruction set from the previous run isn't in this report.
• Skills, reusable task-specific guidance packages, were added as a new mechanic. The first skill evaluated is Building Accessible UI. More on this below.
• A new variant token and pass-rate snapshot lets you compare quality against token cost across control, instructions, and skill turns.

Control: frontier models, same accessibility problem

The headline finding from the previous post was that LLMs default to producing inaccessible code. With the latest models in hand, that hasn't changed.

A few observations:

• GPT still leads on control, but the top score is well below the 41% GPT‑5.2 posted previously. The prompt set has changed and the assertions are stricter, so the numbers aren't directly comparable, but the conclusion is the same: nobody is shipping accessible code by default.
• Claude Haiku 4.5 sits at 3%, with an average of nearly 6 WCAG failures per sample. Sonnet 4.6 and Gemini 3 Flash Preview aren't far behind.
• The hardest test case, Shopping Home Page (React, Dark theme), produced a 0% pass rate with 15.55 average WCAG failures across all models. Component density compounds the problem fast.

The training-data hypothesis from the previous post still seems to fit. The open web is overwhelmingly inaccessible, so models trained on it inherit those patterns regardless of how capable they are at code in general.

Instruction sets: still the cheapest win

Custom instructions are still the fastest thing a team can ship to improve accessibility.

The Basic instruction file produces nearly 5× the control pass rate while only increasing average input tokens by roughly 50%. Even the one-line Minimal instruction ("All output MUST be accessible.") more than triples it.

If you do nothing else, ship an instruction file. The basic instructions are a good starting point to customize for your team's stack and design system.

Skills: the new mechanic

The biggest change in this report is the introduction of skills. Where instruction sets are always-on guidance loaded into the agent's context for every task, a skill is a reusable, task-specific package that bundles guidance, examples, supporting files, scripts, and a tool-use workflow. The agent loads the skill only when it's relevant, and within a skill, only loads the slice it needs for the task at hand.

This matters because it changes two things at once: what guidance the model sees and when it sees it. Skills can carry far more detail than an instruction file without flooding the context window, and the two-turn Generate then Review pattern gives the model a structured second look at its own output before it's done. Together that's why skills outperform instructions in this report.

The first skill evaluated is Building Accessible UI. It is purpose built to:

• Activate when the agent is creating UI, so generated code is more accessible by default.
• Contain expert-level guidance and checklists for many components and patterns.
• Only pull the guidance for the specific component or pattern being built, to limit token impact on the context window.
• Run a two-turn workflow: Generate the UI, then Review it against the skill's checklist and fix issues.

A few observations:

• The best model under the skill is Gemini 3.1 Pro Preview, the same model that scored 8% on control. With the right scaffolding, a weak baseline can outperform the leaders.
• The review turn pulls its weight. Asking the agent to self-check against the skill's checklist adds 5.6pp on top of an already strong first turn, closer to how a human accessibility reviewer actually works.
• Skills don't dominate the prompt. Only 14 to 18% of input tokens come from the skill itself, compared to 100% for instruction sets. Most of the context window is still free for the actual task.

The cost question

Skills win on quality, but they aren't free.

The skill's review turn averages roughly 5.5× the input tokens and 2.7× the API calls of control. At scale, that's a meaningful budget impact.

A practical split:

• Instruction sets are broad, always-on guardrails. Cheap, simple to ship, and the best ratio of accessibility improvement to tokens spent. The downside is that the token impact of custom instructions can add up quick if your project has instructions for multiple domains, such as accessibility, security, content, etc. Use these as the default for any team but keep them short.
• Skills are focused, procedural guidance for higher-stakes work, or when you have the budget.

Recommendations

The advice from the previous post still holds, with one new lever:

1. Ship a project-tailored instruction file today. Start from the basic instructions and customize for your stack, design system, and component library.
2. Add a skill for high-stakes UI work if your token budget allows. The two-turn Generate then Review pattern materially improves outcomes.
3. Bake automated accessibility checks into CI/CD and block PRs on regressions.
4. Keep manual testing by humans, including people with disabilities. None of these tools (automated checks, instructions, or skills) can cover all accessibility requirements.

Closing

The trajectory hasn't changed. AI keeps scaling how much UI we ship, and the open-web training data keeps scaling inaccessible patterns alongside it. Frontier models alone aren't going to fix this; the latest results from Claude 4.7, Gemini 3.1, and GPT‑5.5 make that clear.

The toolbox has changed, though. Instructions still pay off in minutes. Skills are a new lever, and a powerful one: capable of taking an 8% baseline model to 86% on these checks. The work now is to pick the right tool for the right task, enforce it in CI, and keep humans in the loop where it matters most.

See the full report and the a11y-llm-eval repository on GitHub.