They say you’re the average of the five people you spend the most time with. If that’s true, we’re all slowly becoming a weighted average of our AI agents.
I definitely feel it, having started saying “canonical” in social settings.
After leaving the hedge fund desk three years ago, I’ve spent most of my waking hours prompting LLMs, testing AI agents, and evaluating stock research. From promising $1,000 tips to GPT-3.5, all the way to working with today’s agents, harnesses, and tools.
It’s taken a toll. But it’s also forced me to go deep on both finance and AI, and to form views on what “good” equity research actually looks like — and how to evaluate it.
The challenge is that most publicly available “finance AI” benchmarks fail at one key thing: capturing nuance. And when it comes to investing, nuance matters.
So I built my own internal evaluations. There’s still a lot of work to do, but these are the core lessons so far:
Deep equity research is fundamentally different from most tasks we evaluate LLMs on.
There is no deterministic verification. You cannot just check whether the note is “correct”, because in many cases there is no single correct answer. There are judgment calls, different valid framings, and legitimate disagreements between smart analysts.
One analyst might treat margin pressure as temporary overinvestment. Another might see it as evidence of structural competition. Both can be financially literate.
This is why scoring against a rubric hits a ceiling very quickly. Once an agent is basically competent (it applies the right methodology, does the math correctly, and presents a financially defensible case) absolute scores stop differentiating well. Two reports can both tick every box, and still be different in quality.
What matters is not whether multiple reports collapse the uncertainty into the same answer. It’s whether the research improves your map of the possible outcomes — surfacing scenarios most missed, assigning sensible probability weights, and identifying what can move the odds.
Imagine you’re judging 2 reports. You read report A, it’s really good, you score it 9/10. Then someone erases your memory, you read report B and also score it 9/10. But when reading them side by side, you realise that B is better than A. Same absolute score, different quality.
LLM judges (which simply means AI models that act as a grader for the work of other AIs) face the same problem. When asked to score individual research in isolation, they struggle to grade quality past “good”.
The distinction of great vs good is exactly where equity research lives.
Many public finance benchmarks, especially the ones that scale cleanly, gravitate toward tasks with deterministic answers.
Some are basically retrieval tests in disguise. “What dividend did Company X declare in 2023?” “What was revenue in the latest quarter?”
These are useful tests. They measure whether an agent can find the right fact and cite the right source. But it is very different from measuring investment judgment.
At Primer, we’ve maxed out FinRetrieval and scored 100%. That was a real achievement, and it showed that our agent could retrieve financial facts reliably. But you cannot really observe GPT-5.5 performance versus GPT-5.4 with that kind of benchmark.
Another popular category is model-building. A lot of these benchmarks are closer to Excel formula verification: can the agent link cells correctly, build a three-statement model, refresh a model with newly disclosed numbers?
Again, useful. But mostly as a “unit test”. They do not test whether the agent can actually model a company in the investment sense: make reasoned assumptions, understand drivers, and forecast sensibly.
There are also benchmarks around junior investment banking tasks: presentations, decks, market summaries, and similar outputs. Those can be valuable, but they still do not answer the core question: how do you measure nuance inside actual investment research?
For a long time, I did what many people do: I eyeballed outputs. Too much, honestly.
Having domain knowledge can be a curse here. It works. You can read a report for a company you know inside-out, and see whether it surprises you. But it is not scalable, not systematic, and not good enough if you are trying to compare:
So I had to build internal benchmarks.
There are a few different ways to evaluate agents, and they are not interchangeable.
Ground-truth evals are great when the task is to retrieve a number. Rubrics are useful for enforcing minimum standards. Baselines are useful when you have a human reference, or current “best” output that represents the quality bar. Relative allows you to actually answer the question: how do these agents compare against each other.
I’ll skip ground-truth and rubric, as we’re all familiar with those.
In my case, the baseline was a research note produced by a fixed-pipeline, not an agent. A sequence of tightly-scoped prompts to LLMs. Some running in parallel, some feeding into each other. The data used was always provided in the most token-efficient format, exactly what the LLM needed. I was even producing multiple notes and having a final model consider all those points of view. That minimized variance and shed more light on the range of possible outcomes.
It worked well when end-to-end agents were not yet reliable enough to make open-ended research decisions. The note was consistently “good”. But it was still a controlled workflow, not an autonomous analyst.
That made it a strong baseline — because it represented the current “best”.
One of my first serious internal evals was the ‘Adjusted Cashflow Flow Note’ evaluation, based on work I used in practice: stripping out accounting noise to understand the true underlying cash generation ability of a business.
The agent has to decide which cash-flow items are recurring, which are accounting artifacts, which adjustments are economically justified, and how much confidence to attach to each.
There is no single right answer. But there are better and worse ways to reason.
GPT5.4 was a step up. We all felt it. But how much so?
I decided to test our agent on the adjusted cashflow note for Copart (CPRT), an online auto auction company.
The “rubric” eval gave both our baseline and the agent the same score. Both notes were financially competent. Both understood that reported cash flow needed adjustment. Both produced usable bridges.
But the agent handled operating leases more rigorously than the baseline. Instead of using lease expense as a proxy, it estimated right-of-use asset depreciation and lease interest separately. It also showed better adjustments log, explained uncertainty more clearly, and reconciled the final bridge more cleanly.
For the 1st time, an agent had beat our “baseline”.
That is the kind of distinction standard evals struggled to capture. A rubric eval asked: did the agent build a cash-flow note? A baseline eval asked: did the agent reveal something more useful about the company’s true cash generation and the range of plausible outcomes?
This is probably the most important scoring lesson.
GPT5.4 had beat our baseline - fantastic. But I still couldn’t answer questions like:
Once agents got good enough to consistently beat baseline, that eval also started suffering form the same issue as rubrics: incremental performance was hard to observe.
So I needed something different.
The best solution I have found is relativity.
Put the outputs next to each other. Let the judge see all of them together. Ask it to rank them, score them, and explain the differences.
That is also how investors actually evaluate research. A portfolio manager does not read one note in a vacuum, assign it an absolute score, and move on. They speak to multiple analysts, they compare arguments. They notice what one analyst saw that another missed. They revise their view as new evidence or better reasoning appears.
“I liked that first report… but now that I’ve read the second one, I realize the first one missed this key issue.”
That is the evaluation dynamic we want.
A Robinhood (HOOD) forecast eval is a good example. Robinhood is a brokerage and fintech platform. Two agents produced a similar model and forecast note which scored the same in absolute terms. But the relative judge still preferred one output because it used alternative data to corroborate near-term trends.
The agent used X/Twitter to check that Robinhood’s product expansion, especially event contracts and crypto/product launches, was getting positive real-time attention.
It would have been impossible to know in isolation that one model was missing “X”. But when put side by side, it became apparent.
Relative scoring is not perfect. The score is only valid inside a specific comparison set. Rankings can move if the control group changes. A report that looks strong against weak outputs may look average against stronger ones.
We deal with this by keeping stable control outputs, running multiple comparisons, and looking for consistent separation across judge passes rather than treating one ranking as gospel. What matters is whether an agent keeps winning across runs, judges, companies, and sets.
That signal is much stronger than an isolated 8.7/10.
This one is key.
For serious research workflows — Adjusted Cash Flow Notes, modeling, forecasting — we use the strongest judge model available. Today, that is 5.5 Pro or xhigh, but the principle matters more than the model name.
If the task requires expert judgment, the judge has to be capable of expert judgment. It needs to spot subtle analytical weaknesses, distinguish insight from verbosity, and recognize when a conclusion is financially defensible but not actually useful.
If the agent used source documents, market data, X, Polymarket, or alternative data, the judge needs to be able to check the evidence. Enough to verify the claims it is being asked to evaluate.
With access to the underlying data, the judge can ask: “Is this actually right? Did it ignore something important? Did it overstate the conclusion?”
That is similar to how a fund manager reads research. If a claim matters, you verify it. You check the source. You run extra checks until you are comfortable with the conclusion.
Without data access, the judge risks committing the sin any fund manager I ever met would punish hardest: taking things at face value.
LLMs are stochastic. One run is not enough. Wrong conclusions are easy to reach.
We run every agent configuration at least three times, often five, to understand its variance. Then we have the judge score each output multiple times as well. Only then do we aggregate.
Otherwise, you end up picking up more noise than needed.
We saw this in the adjusted cash-flow evals for Howden, a UK house builder.
In some instances the top two agents were close. They traded places across judge passes depending on whether the judge valued direct bridge clarity, tax normalization, or final reconciliation more heavily.
That means the outputs were close and the remaining difference was genuinely judgment-heavy. And this was true across multiple agent runs.
Multiple agent runs and multiple judge passes help reduce noise.
After all…even boxing matches have 3 judges.
Producing a research note and/or a model is the outcome. But how did the agent get there?
Knowing this is practically useful for 2 main reasons.
Not all environemnts are time/cost agnostic. For example: GPT5.5 xhigh scores only marginally better than GPT5.5 high on our cashflow note, whether that’s measured via rubric, reference, or relative evals. OK - that means we need harder tasks. Sure, but it also means we can look at how the agents got to that note as an alternative comparative dimension.
5.5 xhigh took 50% longer than 5.5 high, used 4x the tokens, and called more tools. I may not care about this, but in a production environment I may choose 5.5 high as the sweetspot for speed.
Even if you’re cost/time agnostic, you can still use the path to iterate over your agent. How do these extra tools affect it? Does giving this extra-data source make it faster? etc. etc.
The beauty of observing the path is that you can even use it to iterate over MCP tools, descriptions, system prompts, and instructions. Once the outcome is clear (score >x), how can you achieve it most effectively?
Something I’m experimenting with now is using Codex /goal feature to iterate over our agent’s traces and optimize MCP tools.
The job is defining what success looks like, and giving the agent what it needs to verify its work, unlike the above screenshot.
Ex ante, there is no single right answer in equity research. Ex post, some parts of the work become testable.
Did the agent get future earnings and key business drivers right? Did it correctly identify what mattered? Did the thesis play out for the reasons it expected?
That is where we are going next.
We are building live forecast evaluations: agents produce forecasts before earnings are released, we wait for the actual results, and then we assess both numerical accuracy and reasoning quality.
The numerical part is important. If you are consistently wrong about revenue, margins, cash generation, your research probably is not good.
But the reasoning matters too. An agent can be “right” for the wrong reason. It can forecast earnings accurately while missing the real driver.
So we evaluate both: the forecast and the path to the forecast.
This is harder than it sounds. You have to be careful about time horizons. Being right over two months is different from being right over six months, or a year. You have to also account for one-off events. You also still need strong qualitative judgment on top of the numerical comparison.
For fundamental research, this is one of the cleanest ways to test whether an agent is developing real investment edge rather than merely producing convincing prose.
The goal is not just to ask “Did it sound like a good analyst?” The goal is to ask, “Was it right about the business, and was it right for the right reasons?”
These lessons now shape how we run evaluations at Primer.
When we test new agent configurations, new tools, new prompts, or new models, we do not rely on public benchmarks. We run them through our own research, modeling, and forecasting evaluations. We use strong judges. We give those judges access to the relevant tools. We run multiple samples. We compare outputs side by side. We use relative ranking, and we observe paths.
It is more expensive. It is also the only approach I have found that actually tells me something useful.
There is still a massive gap between “it passed the benchmark” and “an analyst would gain thesis-moving insight from this research”
Closing that gap is the real work.
此内容由惯性聚合(RSS阅读器)自动聚合整理，仅供阅读参考。 原文来自 — 版权归原作者所有。