Real-time applications, from live coding assistants to conversational voice agents, require LLM latency measured in hundreds of milliseconds, not seconds. Achieving this consistently demands more than a fast model weights file. It requires a systems-level approach that spans model selection, serving infrastructure, client integration, and cost controls. This guide covers the concrete techniques that reduce time-to-first-token (TTFT) and inter-token latency, and where Oxlo.ai fits into a low-latency stack.
Define Strict Latency Budgets
Before optimizing, instrument your end-to-end pipeline. Real-time user experiences usually require TTFT under 200 ms and inter-token latency under 50 ms. Measure these from the client perspective, including network round trips and serialization overhead. Set budgets per model tier: a code-completion assistant has tighter constraints than a long-form reasoning agent.
Choose Models for Speed, Not Just Benchmarks
Parameter count is the strongest predictor of prefill and decode latency. For real-time workloads, prefer mid-size models or efficient Mixture-of-Experts (MoE) architectures over dense hundreds-of-billion-parameter variants.
Oxlo.ai hosts several options suited to low-latency production traffic:
- DeepSeek V4 Flash: an efficient MoE with 1M context support, useful for high-throughput reasoning without the overhead of larger dense models.
- Qwen 3 32B: a strong multilingual model that balances capability and response speed.
- Oxlo.ai Coder Fast: optimized for code completion scenarios where sub-100 ms feels instantaneous.
- DeepSeek V3.2: available on the free tier, making it ideal for prototyping latency-sensitive features.
If your application does not require frontier-level reasoning, a 32B or 70B model served on optimized hardware will often outperform a larger model on consumer-grade infrastructure.
Stream Responses to Minimize Time to First Token
Blocking until the full response is generated destroys perceived latency. Streaming returns the first token as soon as it is ready and lets you render output incrementally. Oxlo.ai supports streaming through its fully OpenAI-compatible chat/completions endpoint.
from openai import OpenAI