It took less than a week for a single hallucinated sentence from an AI support agent to cost Cursor real customers. The bot wasn’t even malfunctioning in any way its own dashboards would have flagged. It responded fast, stayed on-topic, and sounded completely confident. It was just wrong. That is the uncomfortable truth at the center of running LLMs in production. A request can succeed by every conventional metric and still fail the user.

Traditional application monitoring was built to answer one question: is the system up? LLMs force a harder question: is the system telling the truth, and can I prove it?

That question is why LLM monitoring has gone from a nice-to-have to core infrastructure. In fact, the LLM observability market is on track to grow from roughly $2.69 billion in 2026 to $9.26 billion by 2030, and Gartner expects LLM observability spend to be part of half of all GenAI deployments by 2028, up from just 15% in early 2026.

In this article, we will break down what LLM monitoring actually means, which metrics matter and what enterprises need to get it right.

LLM monitoring is the continuous tracking of an LLM application’s performance, cost, and output quality in production. It involves much more than merely ensuring that the application has responded. It ensures that the response was accurate, safe, and useful.

Unlike traditional software monitoring, it has to account for models that are non-deterministic and outputs that are unstructured text with no fixed correct answer to check against.

Why LLM Monitoring is Essential

Let’s begin with another real-world example.

In April 2025, people using a coding tool called Cursor started getting logged out every time they switched devices, say moving from a laptop to a desktop. When they asked support for help, an AI agent named Sam told them that this was on purpose. It claimed that a new rule limiting each subscription to one device at a time is in effect. However, that rule never existed. 

It was a bug and the AI simply made up a believable-sounding explanation for it. Different users asking the same question got different responses. And when people compared notes online, they couldn’t even agree on what was happening. The story spread fast, many people cancelled their subscriptions, and Cursor’s co-founder had to publicly apologize. 

That is the greatest risk that enterprises face. An LLM can be wrong with total confidence and nothing about the interaction looks broken.

Thus, LLM monitoring is essential because:

• Failures are silent: A hallucinated answer doesn’t throw an error. It looks exactly like a correct one until someone checks it.
• It can lead to damaging consequences: Courts have already held companies accountable for what their AI systems tell customers.
• Costs can spike overnight: LLM APIs bill per token and one inefficient prompt or runaway workflow can blow up a bill with zero warning.
• Trust is expensive to rebuild: A single viral incident can trigger real cancellations within days and lead to customer mistrust.
• Regulators are catching up: High-risk AI systems now face mandatory oversight obligations, with EU AI Act enforcement phasing in from August 2026.

LLM Monitoring vs. LLM Observability vs. LLM Evaluation

Monitoring: It tracks known production signals over time such as latency, cost, usage, errors, quality scores, safety scores, and drift thresholds. It creates alerts when anything goes wrong.

Observability: It combines traces, metrics, logs, evaluation scores, and user feedback so a team can reconstruct why an LLM or agent behaved a certain way. When a user reports a wrong answer, observability lets a team trace that exact request through the pipeline and see whether the problem originated in retrieval, a prompt change, or a model configuration change. Without observability, teams can tell that something is wrong but not why.

Evaluation: It is the scoring engine that feeds both. Offline evaluation runs structured tests against golden datasets before shipping, while online evaluation scores real production traffic as it happens. Those live scores get tracked like any other system metric and can trigger the same alerts monitoring relies on.

Basically, these aren’t three competing tools. They are one feedback loop where evaluation generates the quality signal, monitoring watches that signal for regressions, and observability is what you reach for the moment monitoring tells you something broke.

Moving on, here are more distinctions among the three terms:

Parameters	Monitoring	Observability	Evaluation
Core question	Is something wrong right now?	Why did it happen?	Is this specific output good?
Triggered by	Continuous, threshold-based	An alert, a user complaint, or manual investigation	Pre-release (offline) + live traffic (online)
Data it runs on	Aggregated metrics against thresholds	Full request traces like prompts, retrieved context, tool calls, model version	Golden datasets offline
Output produced	Alerts on the dashboard	Root-cause trace	Pass/fail or numeric quality score
Who acts on it	On-call engineer	Engineer debugging a specific failure	AI engineer tuning prompts, retrieval, or model choice

Key LLM Monitoring Metrics

There are several categories that cover a production LLM system from end to end.

System performance

• Time to First Token (TTFT): How long a user waits before the AI starts typing back at all. Think of it as the gap between hitting “send” and seeing the first word appear, even if the full answer takes longer to finish.
• Total response time: How long the entire answer takes, start to finish, not just the first word.
• P50 / P95 / P99 latency: Instead of just tracking the typical response time, this tracks the slowest ones too. A system can look fast on average while a meaningful share of real users are still waiting much longer.
• Throughput: How many requests the system can handle at the same time without slowing down.

Cost

• Token usage: AI providers charge based on how much text goes in and comes out of each request.
• Cost per conversation / per completed task: What it actually costs to get one full job done.
• The “worst-case cost” check: This involves comparing your priciest requests to your typical one. If a handful of requests cost 50 times more than normal, something is usually misconfigured.

Output quality

• Faithfulness: Does the answer actually match what was found in the source material, or did the AI state something that isn’t backed by anything real?
• Answer relevancy: Does the answer actually address the question that was asked? (An answer can be accurate and still miss the point.)
• Hallucination rate: How often the AI states something false or made-up with full confidence. Mature production teams typically aim to keep this below 0.5%.

Drift

• Prompt drift: Are users starting to ask different kinds of questions in different ways than when the system was first built?
• Behavior drift: Is the system’s actual behavior quietly shifting over time? This must be measured with comparison methods that flag when current traffic starts to look meaningfully different from the baseline it was tested against, even though nothing was deliberately changed.
• Output drift: Are the AI’s answers getting longer, shorter or different in style over time?

Safety

• Toxicity rate: How often the AI produces offensive, hateful, or inappropriate language.
• Prompt injection / jailbreak rate: How often someone successfully tricks the AI into ignoring its own rules or safety instructions.
• PII leakage rate: How often personal information like names, phone numbers, medical or financial details shows up in a response when it shouldn’t.
• Policy violation rate: How often the AI breaks a rule it was specifically told to follow, such as talking about a banned topic.

Reliability

• Error rate: How often a request simply fails to go through.
• Timeout rate: How often a request takes too long and gets cut off before finishing.
• Uptime: How often the system is actually working when someone tries to use it.

Debugging metrics

• Tool-call accuracy: When the AI reaches out to an outside tool or system, did it use the right one the right way?
• Step-by-step timing: Instead of just knowing a request took 5 seconds total, this shows exactly which step ate up that time.
• Trace error status: Flags whether something quietly broke somewhere in the chain, even if the final answer the user saw looked completely fine.

How LLM Monitoring Works in Practice

In a real production system, monitoring starts before a single request ever runs.

Every call the AI makes gets wrapped in a small recorder that notes what went in, what came out, how long it took, and which version of the model handled it. When a user sends one message, that message might trigger several of these steps behind the scenes and all of those linked steps together are called a trace. Think of a trace as one complete story of a single request, with each recorded step inside it being one chapter of that story.

Once a trace is captured, it is sent to a monitoring system that does two things with it.

• First, it runs the trace through a set of automatic checks. Some are simple rule-based checks, while the others use a second AI model to judge the first one’s output. Those checks attach a score to the trace.
• Second, all of these scores and numbers get rolled up into dashboards the team can watch. These dashboards show parameters like average response time, cost per day, how often the AI is getting things wrong, etc. Alerts are set to fire the moment any of those numbers cross a line that is considered unacceptable.

When an alert is triggered, an engineer steps in. Instead of just knowing something broke, he can open that exact trace and walk through it chapter by chapter to see precisely where things went wrong. Once the root cause is found and fixed, that failed case doesn’t just get closed; it gets added permanently to the test set the system is checked against going forward.

LLM Monitoring for Agentic and Multi-Model Systems

Agentic Systems

Monitoring a single AI response is very different from monitoring an AI agent. The latter involves tracking the full loop of the agent planning what to do, taking an action, observing the result, and deciding what to do next.

For agentic LLM monitoring, it is important to:

• Track the whole session and not each step in isolation. Distributed tracing needs to cover the entire agent session in a single go.
• Count how many cycles it took. Logging how many plan-then-act cycles each session went through flags agents that are quietly struggling.
• Watch for compounding failures. In a long chain of steps, an early mistake often doesn’t show up until several steps later, once it’s had time to snowball, which is why looking only at the final answer misses where things actually went wrong.

Multimodal systems

The core challenge here is that a multimodal pipeline (voice, image, video) usually isn’t one single AI model. It is several specialized systems working in sequence. A failure early in that sequence can look completely invisible by the time it reaches the final output.

A typical voice assistant converts speech to text, sends that text to the AI, then converts the AI’s answer back to speech. Speech-recognition errors cascade through the entire system. And when that happens, the AI will fail no matter how good its own reasoning is, even though nothing about the AI itself was at fault. This is why each stage needs its own dedicated checks.

Best Practices for Enterprise LLM Monitoring

Log every single request and response

Every request and response gets logged with a unique trace ID showing exactly what was tested, when, and which thresholds passed or failed. Without this, a bug reported today can’t be investigated.

Define SLIs/SLOs around perceived user experience

Track the share of requests that stay under a response-time threshold, not the average. A system can look fast on average while a meaningful share of real users are still waiting far longer than that.

Treat cost as a first-class metric: per-seat, per-tenant and per-feature budgets

Tag every request with a feature, tenant, and user ID at the call site. Adding it from day one is one line of code. Retrofitting it later becomes a months-long migration. Per-feature and per-tenant breakdowns then show which product surfaces or accounts are actually profitable.

Layer online and offline evaluation rather than relying on one

Offline evaluation catches known failure modes before launch. Online evaluation catches the ones real users find that no test set anticipated.

Build in PII redaction, prompt-injection detection, and audit trails from day one

These three are the hardest to retrofit. Scrubbing months of stored logs later costs far more than logging clean data from the start. Mask sensitive fields, record every guardrail decision and override and keep data stored where the relevant jurisdiction requires it.

Turn every real mistake into a permanent test

When something fails in production, save that exact case as a permanent test. Skip this step and the fix and the eval that should catch a repeat of the same failure get created separately, so the same bug quietly resurfaces later.

Getting Started with LLM Monitoring

AISquared’s UNIFI platform treats monitoring as one native layer rather than a bolt-on integration. It natively includes workflow monitoring, output quality tracking, feedback capture, and audit logging as part of its observability layer.

Here are some of its crucial features:

Production telemetry, built in: UNIFI is designed to monitor workflow runs, failures, latency, and output quality, and capture user feedback tied to specific responses and actions.

Audit trails from day one: Every request is authenticated, authorized, and logged, with role-based access and encrypted data flows.

The feedback loop is closed automatically: Feedback is captured at the moment of use, so compliance teams can audit any answer and data science teams can see exactly where to tune models.

Compliance mapped in, not bolted on: The platform runs on zero-trust architecture with AES-256 encryption and SOC 2 Type II certification, deployable in your cloud, on-premises, or fully air-gapped.

Faster to deploy: Pre-built connectors, no-code workflows, and drag-and-drop orchestration replace the custom instrumentation buildout most teams take on separately.

Click here to get started with AISquared today.

Conclusion

The Cursor and Air Canada cases opened this piece for a reason: both companies had systems that were technically working fast while quietly telling customers something false. That gap between “the system responded” and “the response was right” is the entire reason LLM monitoring exists as its own discipline, separate from the infrastructure monitoring teams already know.

FAQs

What is the difference between LLM monitoring and traditional APM?

Traditional Application Performance Monitoring tracks four signals, i.e., latency, traffic, errors, and saturation. LLM monitoring keeps all four but adds a layer APM was never built for, which is whether the output itself is actually correct. An LLM service can be technically healthy while quietly producing answers that are misleading or wrong, something no traditional APM tool is designed to catch.

How often should LLM performance be reviewed?

Automated metrics such as latency, cost, error rate, and safety scores should be monitored continuously, in real time, on every request. Beyond that, human review of outputs should happen at minimum weekly. Cost specifically is worth a weekly review, since gradual increases are easy to miss until an invoice arrives.