We built a benchmark of 25 expert-authored KiCad schematic-editing tasks and ran a frontier computer-use agent against them. The headline numbers:
Most computer-use benchmarks today live in the same handful of apps: web browsers, file managers, generic productivity suites. Those evaluations are useful, but they share a structural weakness — the tasks are short, the domain knowledge bar is low, and the UIs are designed to be intuitive for anyone. A frontier model can often muddle through by trial and error. That is not what real professional software looks like.
We wanted a domain that pushed agents on the dimensions that actually matter for real economic value:
In short: EDA gives us a setting where solving the task requires real domain expertise, real long-horizon planning, real perception of a complex GUI, and tolerates none of the shortcuts that simpler benchmarks accidentally allow.
Every one of the 25 tasks was authored by a practicing electrical engineer. Each task came with three artifacts:
Tasks were calibrated to be modest in length for a human expert. We explicitly asked annotators to write tasks that they themselves could complete in around 50 steps or fewer, given current agent capabilities. This was not about making the benchmark easy (every task in the suite is still well within what a competent EE intern can do in a few minutes), it was about ensuring that the 150-step budget the agent gets is genuinely generous.
A second engineer reviewed every task. We had an independent electrical engineer check each task end-to-end: that the prompt was unambiguous, that the initial project actually matched what the prompt described, that the ground-truth state was a valid solution (no errors in the arithmetic, no incorrect topology, no impossible component values), and that the prompt could not be reasonably interpreted in a way that would make the verifier mark a correct solution as wrong.
The resulting suite spans single-edit tasks (“change C1 to 2 nF”) through to full builds (“design a bipolar-to-unipolar converter from two op-amps, six resistors, and six capacitors”). The tasks anchor on a range of skills: bias-point arithmetic, LDO feedback dividers, Sallen-Key filter design, NE555 astable topology, H-bridge decoupling, switching regulators, and basic file-protocol operations like exporting a netlist.
We ran Claude Sonnet 4.5 against all 25 tasks (with one Haiku 4.5 trajectory included as a data point). The results paint a strikingly coherent picture of where current computer-use agents are strong and where they fall off a cliff.
The single most striking finding is a sharp capability cliff at the boundary between editing an existing schematic and building one from scratch.
Every task whose required-edits count is ≤ 2 is at least partially solvable. Every task that requires placing 3+ components and drawing wires either fails outright or earns only partial credit.
The four full passes (changing a capacitor value, swapping a +9V power port for +5V, resizing a resistor for a target current ratio, centering a microphone bias point) all share the same shape: open the project, find the one component that needs touching, edit it, save, declare done. The agent has the domain arithmetic and the GUI navigation for these. What it does not have, yet, is the multi-phase planning loop needed to place a dozen components, wire them, set every value, run ERC, and self-verify before declaring completion.
The failure modes are not random. We catalogued every error across the 25 trajectories and clustered them. The same handful of patterns appear over and over.
Four clusters appear in essentially every failed run, and they compound:
Rolled up to root causes: 40% of error-mode mentions are about planning and policy, 22% about perception, 19% about navigation inefficiency, 11% about domain knowledge gaps, and just 8% about the tool/API surface itself. Crucially, there were zero API errors anywhere in the cohort, proving that the underlying harness works as expected. The failures are in how the agent uses it.
It is tempting to look at “16 max-steps failures” and conclude the agent just needs a bigger budget. The data rejects that interpretation. We extended two unfinished runs with a 500-step allowance, and both still failed, while one of them earned a flat zero (using 467 and 488 steps respectively.) Meanwhile, every full pass finished in under 150 turns. More steps do not help when the underlying plan is wrong.
In a typical failed run, roughly 30–40% of the budget goes to app bring-up and recovering from wrong-editor states, 40% to slow component-by-component placement, and 20–30% to stuck loops where the agent retries the same click after no UI change. By the time the agent is ready to wire, it has no budget left.
Five of the nine runs that emitted a DONE token failed verification — usually because the agent verified by re-narrating its own intent rather than reading ground truth from the UI. A few representative cases:
In every case, reading the title bar (which shows an asterisk on unsaved changes), the status bar, the ERC dialog, or the saved netlist text would have caught the error. The agent reasoned over its own narration instead.
Cua-Bench EDA gives a sharp, fair picture of where computer-use agents stand on real professional software in mid-2026: confident on short, linear edits; unable to plan and execute the multi-phase workflows that everyday engineering work demands. The good news is that the failure modes are concrete, repeatable, and addressable – they are not pointing to a missing underlying capability so much as a missing meta-policy for how to use the capabilities the model already has.
We are continuing to extend the suite. We are running the same tasks against additional frontier models, with more results publishing this week. We look forward to seeing how the next generation of agents handles them. Thanks again to the Cua team for the harness, the verifier, and the support that made this analysis possible. For more, visit cua.ai and see Cua’s launch note.
此内容由惯性聚合(RSS阅读器)自动聚合整理，仅供阅读参考。 原文来自 — 版权归原作者所有。