Your work might not need the smartest model. With all the enthusiasm for GPT-6 Astra, it’s easy to make it your default and expect the best result every time. That expectation is worth testing, because the improvement on your work needs to justify the cost. Even when the organization pays for usage, the choices developers make throughout the day still add up.

For a team choosing where to spend its agent budget, the useful question is what’s the minimal viable model for our work? Finding the least expensive model that meets your requirements starts with choosing candidates, which a leaderboard can help you do. Evaluating those candidates on your own tasks tells you which supports your team’s workflow best.

What the upgrade buys you

In an earlier model comparison, a newer model followed explicit upgrade instructions more reliably, while the older model produced more idiomatic architecture in the scenarios we measured. Whether the upgrade helped depended on the type of work we gave it. Public benchmarks measure their own set of tasks in a particular environment, so your repositories and agent setup can produce different results.

To see what that tradeoff looked like for Astra, we compared it with Claude Sonnet 4.6, a commonly used model, across three code upgrade scenarios. Both models ran in GitHub Copilot Chat in Visual Studio Code on Windows with the same scenario setup, and we ran each profile five times. Below is one matched comparison from each scenario, with rounded quality scores and per-run costs based on GitHub Copilot pricing at the time of writing:

Scenario	Claude Sonnet 4.6	GPT-6 Astra	Result
JDK 8 to JDK 25, with the same migration skill	97% at $12.71/run	93% at $67.33/run	About 4 percentage points lower at 5.3x the cost
SPFx upgrade to v1.22.0, with the same CLI instruction	98% at $1.03/run	98% at $2.49/run	Same rounded score at 2.4x the cost
SPFx v1.21.1 to v1.22.2, with the same SPFx skill	98% at $0.92/run	99% at $5.14/run	About 1 percentage point higher at 5.6x the cost

In these configurations, Astra’s higher cost bought us a lower score on one task, a similar score on another, and a small increase on the third. Claude Sonnet 4.6 serves as a familiar comparison point here, and five runs per profile give us an initial view of performance. Establishing whether a one-point difference is reliable would require further evaluation.

The JDK comparison looked quite different without the added extensions, where Astra scored 93% compared with Sonnet’s 76%, at $8.59 versus $1.68 per run. Adding the migration skill reversed the quality comparison and increased costs for both models, so the guidance your team uses belongs in the evaluation alongside the model.

Would you pay more than five times as much for one additional point? The answer depends on which checks improved and what those improvements mean for your work. An aggregate quality score measures performance against the evaluation criteria, while assessing the risk of destructive actions requires evidence about those specific failures. For consequential work, that evidence needs to sit alongside appropriate safeguards and human approval.

Find your minimal viable model

Before running the eval, decide what the model must get right and how much failure your team can tolerate. Those requirements define which models are viable for the work, leaving you to compare their costs. Setting the bar upfront also helps you judge the results independently of a preference for a particular model.

How much you’re willing to pay for an improvement will depend on the consequences of failure. Hypothetically, a one-point improvement at ten times the price might be hard to justify for routine work, while a ten-point improvement at twice the price could be worthwhile if the improved checks address failures with downstream consequences. Agreeing on those tradeoffs before you start gives the team a basis for deciding what to do with the results.

With those requirements in place, compare models using the same scenarios and evaluation criteria, keeping the workspace and agent setup fixed. Differences in operating systems or agent configurations can influence the results, so the environment should represent your team’s work and stay consistent across the comparison. For guidance on defining the criteria and evaluating repeated runs, see building AX evals that actually work.

One default can still be the right answer

Suppose your eval shows that five models each win a different type of task, but the differences are negligible. Paying $0.02 more per task to use one model across all of them could be the better deal, giving developers one predictable workflow and leads one default to support. That convenience can outweigh the small savings from choosing a model for every task, while a meaningful improvement on a particular task could justify an exception. Use the results to weigh the benefit against the extra decision developers have to make, and keep model selection as simple as the work allows.

Your answer has an expiration date

Once you’ve chosen a default, an agent update or a new extension can change how well it handles the same task. New models give you more options, and pricing changes the tradeoff even when quality stays flat. A decision that was right last month can become needlessly expensive or expose the team to more failures today.

To keep that choice current, maintain a small set of representative scenarios and rerun them as your team updates its agents and extensions. Changes to models or pricing are also reasons to revisit the comparison. How often should you run it? As often as you can act on the findings, giving the team time to investigate the results and update its recommendation.

Evaluations cost money and take effort to maintain, but sharing the findings lets the whole team benefit from that investment. Whether Astra earns its premium on your work or your current model already meets your needs, you’ll have evidence to guide where the budget goes. Evaluate your work, then pay for the improvement you can justify.