Alisa Rae, Founder & CEO of Lucent.
getty
The AI agent arms race is in full swing. Every week, it seems there's a new framework, a new model, a new demo of an agent autonomously browsing the web, writing code, triaging support tickets and managing entire workflows end-to-end.
We've given these agents remarkable capabilities. They can process our documentation, query our databases, search the web, remember past conversations and reason across complex multi-step problems. By almost every measure, they are getting dramatically better, faster than anyone predicted.
But in my experience in AI bug detection, there's one thing that companies need to keep in mind: Are your users struggling? Today, I typically see user-reported bugs as a lagging indicator: By the time one person files a report, others may have already hit the same issue and said nothing. Users may not file reports or send emails explaining what went wrong, opting instead to leave without explanation. But that doesn't mean you need new tools to give your agents greater visibility.
Most AI agents today are built on specific types of input: structured data (databases, metrics), unstructured text (docs, tickets, emails) and explicit instructions (system prompts, rules) you write to shape their behavior. But, in my experience, what can go missing is the real-world signal of how humans actually interact with your product.
There's typically a gap between what users intend and what they do, and between what builders design and what gets built. Your specs describe the happy path. Your tests cover the cases you anticipated. But the messy, emergent reality of a real person navigating real software for the first time is just as important. Most companies record user sessions, but what happens when you don't use that information? What happens when the data exists, but the synthesis doesn't?
In a slower product cycle, this was manageable. A bug would surface eventually, through a support ticket or a user interview. But AI has collapsed that tolerance. Teams are shipping faster than ever. But the same force multiplier that accelerates output can also accelerate mistakes, meaning bugs ship faster, regressions propagate further and UX issues may compound before anyone notices.
Using behavioral data that you're already likely capturing on how your users interact with your product, such as from session logs, can give you insight into shifting your agents from reactive to proactive. When you use your codebase, your backlog and information on user sessions together, you can gain the behavioral record of users hitting friction in your product beyond tickets and documentation alone.
Ultimately, the bigger shift is not technical but cultural. You need to treat behavioral data or signals as an input to engineering and product decisions, the same way you treat metrics and error reports. But it's important to have safeguards in place.
You don't need to rebuild your stack to gain a ground truth model of how your users actually experience your product. Start with one question: Where are users failing silently in your product right now?
Pull the top three drop-off points in your core flows and watch a handful of sessions yourself. You don't need an agent for this. Once you know what friction looks like in your product, you can define it in terms an agent can recognize: users who trigger a specific error or abandon a flow at a certain step.
If you notice that users are repeatedly failing to complete a specific onboarding step, a call-to-action isn't being clicked despite being prominently placed, or a workflow designed to take three steps is consistently taking seven. You now have greater context to understand how it's actually being used and where the gap between those two things is costing you users.
From there, narrow down. Pick one recurring issue and feed only that segment into your existing engineering loop. Most coding assistants can ingest structured context, so a short doc with repro steps, affected users and observed behavior is often enough to meaningfully improve what your agent ships.
Share what a user did in the minutes before they reached out, and the agent can work to diagnose real interactions based on what the user tried. This can be helpful in distinguishing a one-off edge case from a pattern.
Digging into your user session data isn't a silver bullet, and you shouldn't treat it as one. The first challenge is volume: A product with even modest traffic could generate numerous sessions a day. Without deliberate filtering, all this data buries real signal under noise, inflates token costs and increases the likelihood of agent hallucination.
The second challenge is interpretation: A user pausing for 30 seconds could mean they’re confused, or it could mean they walked away for a tea break. Without context, an agent can draw confident conclusions from ambiguous behavior, which is arguably worse than none at all.
There are also real considerations around privacy. Session data often contains PII, form inputs and sensitive workflows that need to be masked before any agent touches it. You need to ensure data is protected, and your team needs a clear policy on what gets captured and what doesn't. These are crucial considerations that need to be addressed before using your session log information.
Your agents could be making decisions with incomplete information, not because the signal isn't there, but because nobody has fed that information to your agents. Evaluate where your users are experiencing the most friction and use that to further inform your agent workflow.
Forbes Technology Council is an invitation-only community for world-class CIOs, CTOs and technology executives. Do I qualify?
此内容由惯性聚合(RSS阅读器)自动聚合整理，仅供阅读参考。 原文来自 — 版权归原作者所有。