Voker raises $2.2M to help teams understand how AI agents perform in the wild

Voker, an agent analytics platform for artificial intelligence product teams, today announced it has raised $2.2 million in pre-seed funding from Y Combinator and FundersClub.

As more companies push AI agents into customer-facing and internal workflows, the challenge is moving from “Can we build one?” to “Does it work in the wild?” Many enterprise teams are already building, but they don’t have a way to show that what they’ve built is providing the value they promised.

“Product teams have this onus to deliver on the marketing claims,” co-founder and Chief Executive Tyler Postle told SiliconANGLE in an exclusive interview. “They’re going to start getting asked by executives how many new products are sold through this agent. They don’t have a way to measure that, and they don’t have a way to push that metric up.”

He explained that getting something into production is not that hard with large language models, because they’re already pretrained. Agents are popping up everywhere, but customers told Voker that once they’re in production, everything they expected to happen hasn’t gone the way they had planned.

Postle calls this the “ask me anything” problem. Too many people set their expectations for agents too high, based on marketing overpromising. “If you’re a hotel booking agent, you’re supposed to book hotels,” he said. “Don’t ask me to do math homework.”

The analytics and visibility gap

The current gap isn’t in observability tools and evaluation tools, Postle said. It’s that they’re built for engineers and are good at trace debugging. However, they tend to break down when there are thousands or millions of conversations with an agent every month and teams want to understand how those agents are performing in the wild, in the hands of real people.

“I’d say the biggest highlight is the insights that you get, like proactive insights of where things are working with your agent, where it’s delivering to users, and what are new ‘intents,’ we call them, or new asks that users are coming up with for agents,” he said.

For example, a hotel booking agent might be talking to customers who want to know about the cafe or restaurant attached to the establishment. Voker could surface conversations involving questions not just about the hours it’s open, but how to get reservations, what meals are served and what’s on the menu. That would give the product team insight into what kind of updates would be useful to add to the booking agent in order to evolve it, fine-tune it to capture customer interest and increase satisfaction.

“Designers even want to look into, ‘How are people using the product?’” Postle added. “And observability and logging and evals tools are really just built for technical engineers and not built for the whole team.”

The company is currently targeting customers with agents already in production with at least 1,000 conversations a month. For Voker, this threshold is meaningful because the problem becomes acute when there are enough conversations to saturate workflows that rely on manually reading traces or using engineering analytics software to sample logs for performance.

From raw logs to product insight

Postle joked that one of the company’s primary competitors has been OpenAI Group PBC’s ChatGPT, as many engineers stuff raw logs directly into the chatbot and ask it to analyze them for user intent and summarize what happened. Even if the entire log fits into the context window, he said, the approach isn’t statistically sound, especially when logs involve millions of rows and questions about actual product performance. It’s a fast way to get an answer, but not a nuanced way to get a good one.

The problem, he said, is that there’s going to be a lot of churn among agentic AI products if they keep promising too much and don’t deliver. Too many customers try an AI agent once, become frustrated that it won’t do everything, and then return to doing the work themselves.

There are plenty of analytics and observability tools on the market now designed for keeping agents in line and fixing them when they go awry, but most are aimed at engineers. Customers need a layer above that, one that helps productize agents and evaluate and classify their value for designers and executives.

“The problem is that there are no analytics insights built for agent products that work for the whole team,” Postle said. “Most of the tools are really specific, and they’re only for engineers.”

Images: Voker, SiliconANGLE/Microsoft Designer