惯性聚合 高效追踪和阅读你感兴趣的博客、新闻、科技资讯
阅读原文 在惯性聚合中打开

推荐订阅源

M
MIT News - Artificial intelligence
罗磊的独立博客
Hugging Face - Blog
Hugging Face - Blog
J
Java Code Geeks
G
Google Developers Blog
美团技术团队
让小产品的独立变现更简单 - ezindie.com
让小产品的独立变现更简单 - ezindie.com
腾讯CDC
奇客Solidot–传递最新科技情报
奇客Solidot–传递最新科技情报
T
The Blog of Author Tim Ferriss
月光博客
月光博客
B
Blog
WordPress大学
WordPress大学
云风的 BLOG
云风的 BLOG
博客园_首页
人人都是产品经理
人人都是产品经理
aimingoo的专栏
aimingoo的专栏
Y
Y Combinator Blog
Jina AI
Jina AI
S
SegmentFault 最新的问题
H
Help Net Security
博客园 - 聂微东
Microsoft Azure Blog
Microsoft Azure Blog
Google DeepMind News
Google DeepMind News

DEV Community

Authentication Security Deep Dive: From Brute Force to Salted Hashing (With Java Examples) Why AI Systems Don’t Fail — They Drift Spilling beans for how i learn for exam😁"Reinforcement Learning Cheat Sheet" I Replaced Chrome with Safari for AI Browser Automation. Here's What Broke (and What Finally Worked) How Python Borrows Other People's Work The $40 Architecture: Processing 1 Billion API Requests with 99.99% Uptime Vibe Coding: A Workflow Guide (From Zero to SaaS) Most webhook security guides protect the wrong side. The scary part is delivery. Headless CMS for TanStack Start: Build a Blog with Cosmic EU Age Verification App "Hacked in 2 Minutes" — What Actually Happened Comfy Cloud’s delete function does not actually remove files Running AI Models on GPU Cloud Servers: A Beginner Guide Event-driven media intelligence with AWS Step Functions and Bedrock I scored 500 AI prompts across 8 quality dimensions — here's what broke How to Call Google Gemini API from Next.js (Free Tier, No Backend Needed) The Portal Protocol: Reclaiming Human Connection in the Age of AI How to Fix Your Team's Scattered Knowledge Problem With a Self-Hosted Forum Intro to tc Cloud Functors: A Graph-First Mental Model for the Modern Cloud Designing Multi-Tenant Backends With Both Ownership and Team Access I Built a Neumorphic CSS Library with 77+ Components — Here's What I Learned PostgreSQL Performance Optimization: Why Connection Pooling Is Critical at Scale Cómo construí un SaaS multi-rubro para gestionar expensas en Argentina con FastAPI + Vue 3 🚀 I Built an Ethical Hacking Scanner Tool – Open Source Project I Replaced /usage and /context in Claude Code With a Single Statusline A Pythonic Way to Handle Emails (IMAP/SMTP) with Auto-Discovery and AI-Ready Design I Collected 8.9 Million Polymarket Price Points — Here's What I Found About How Markets Really Move EcoTrack AI — Carbon Footprint Tracker & Dashboard Everyone's Using AI. No One Agrees How. 5 self-hosted ebook managers worth trying in 2026 Building Your First AI Agent with LangChain: From Chatbot to Autonomous Assistant
How We Reduced LLM Costs Without Touching Model Quality
Karan Padhiy · 2026-05-22 · via DEV Community

How We Reduced LLM Costs Without Touching Model Quality

One of the fastest ways to destroy an AI system in production is uncontrolled token growth.

Most demos ignore this problem because they run small prompts against clean datasets. Real enterprise systems do not behave like that.

Once multiple integrations start running together, token usage grows faster than most teams expect.

We started seeing it after several enterprise pipelines went live at the same time.

  • Slack ingestion
  • Email synchronization
  • CRM updates
  • Meeting transcripts
  • Internal ticket systems
  • Knowledge base sync jobs

Everything was feeding into the same operational AI layer.

At first, nothing looked broken.

Responses were accurate.
Latency was acceptable.
Users were happy.

But infrastructure metrics told a different story.

Prompt sizes were growing continuously.
Costs increased every week.
Some requests carried massive amounts of unnecessary context.

The issue was not the model itself.

The issue was everything surrounding the model.

The Real Problem Was Context Inflation

A single request slowly turned into this:

  • duplicated conversation history
  • overlapping retrieval chunks
  • unnecessary metadata
  • old execution traces
  • repeated system instructions
  • temporary tool outputs nobody needed anymore

The worst part was that response quality barely changed.

We were spending more money to process noise.

That forced us to look at the architecture instead of blaming model pricing.

What We Changed

We Stopped Treating Retrieval Like Free Context

Initially, retrieval output was pushed directly into prompts.

That works during early development.

It breaks during long-running enterprise operation.

Vector search systems naturally return overlapping information. As datasets grow, overlap increases even more.

We added a preprocessing layer before prompt assembly.

Now every retrieval result passes through:

  • semantic deduplication
  • overlap removal
  • metadata cleanup
  • token budgeting
  • context prioritization

This immediately reduced prompt size across production workloads.

The important part was that output quality stayed almost identical.

That was the moment we realized how much useless data was entering the system.

We Split Operational Memory From Reasoning Memory

This changed the architecture more than anything else.

Most AI systems mix all state together:

  • chat history
  • tool outputs
  • execution logs
  • retry traces
  • retrieval data
  • audit metadata

The model does not need all of that for reasoning.

So we separated memory into layers.

Operational memory stores infrastructure state:

  • retries
  • execution traces
  • audit logs
  • system metadata

Reasoning memory stores only the information required for inference.

That separation reduced context pollution heavily.

It also made debugging easier because infrastructure concerns stopped leaking into model reasoning.

We Reduced Prompt Complexity

Large prompts feel productive.

They usually are not.

Over time we noticed many system prompts were repeating the same instructions in different wording.

That increased tokens without improving reliability.

Instead of adding more prompt logic, we moved more control into infrastructure logic.

We added:

  • structured validation layers
  • schema enforcement
  • routing constraints
  • tool permission boundaries
  • deterministic execution rules

The result was smaller prompts with more predictable behavior.

The infrastructure became responsible for operational control instead of pushing everything into the model.

We Added Token Observability Everywhere

This should exist in every production AI system.

Without token observability, cost problems stay invisible for weeks.

We now track:

  • token usage per tenant
  • token usage per integration
  • retrieval expansion rates
  • average context growth
  • abnormal cost spikes after deployments

One deployment accidentally tripled token usage because a serializer started injecting entire API payloads into conversation state.

The system still worked.

Nobody noticed immediately.

Without observability, we would have discovered it only after billing increased significantly.

The Bigger Lesson

Most enterprise AI cost problems are not model problems.

They are architecture problems.

The expensive part is usually not inference itself.

It is:

  • poor memory design
  • uncontrolled retrieval
  • duplicated context
  • oversized prompts
  • weak operational boundaries

Reducing waste matters more than constantly changing models.

We did not downgrade quality.

We did not switch providers.

We fixed the infrastructure around the model.

That changed the economics of the system far more than any prompt optimization ever did.