Building an AI Scoring Pipeline for 10,000+ Listings a Day

The bill is the part nobody talks about when they demo AI pipelines. You see the cool output, the semantic matching, the ranked results. You don't see the spreadsheet where you realize each token costs real money.

I was building an AI scoring pipeline for a job board platform that ingests listings from multiple ATS sources and needs to surface relevant content for candidates. The system processes over 10,000 new listings each day, and every single one gets scored against candidate profiles. This is what I learned about architecture, cost management, and knowing when to reach for an LLM versus reaching for something simpler.

Why LLMs Made Sense Here

Traditional ML could solve some of this. A good classification model can tag skills, seniority levels, and job categories reliably. But the problem was richer than that.

A job listing might say "looking for a Python ninja who knows their way around React." A traditional classifier sees keywords. An LLM understands that "ninja" is informal for "expert" and that "knows their way around" means practical experience, not academic knowledge. When you're matching candidates to roles, that nuance matters.

But here's the thing I learned quickly: you don't need an LLM for everything. I split the pipeline into two stages. Stage one uses traditional rules and keyword matching to filter out obvious garbage listings that would waste API calls. Only stage two sends qualified listings to the LLM for semantic scoring. That single decision cut our API costs substantially before we even touched token optimization.

The architecture ended up looking like this:

// Simplified scoring pipeline entry point
async function processJobListing(rawListing: RawJobListing) {
  // Stage 1: Cheap filter before touching any API
  if (!passesPreFilter(rawListing)) {
    return { scored: false, reason: 'pre_filter_rejected' };
  }

  // Stage 2: Check cache before making an API call
  const cacheKey = hashListing(rawListing);
  const cached = await checkScoreCache(cacheKey);
  if (cached) return cached;

  // Stage 3: Queue for batch scoring
  return queueForBatchScoring(rawListing, cacheKey);
}

The pre-filter stage checks for things like incomplete listings (missing salary or location), duplicate content from multiple ATS sources, and listings that are clearly not for our target market. None of this needs an LLM. A worker that costs fractions of a cent handles it.

The Batch API Changed Everything

Running individual API calls for each listing was never going to work. The OpenAI Batch API was the turning point.

Instead of sending one job listing per request, I batch them into groups of 50. Each batch gets processed at half the cost of individual calls, and because Batch API runs on a lower priority queue, the throughput hit is manageable for a system that doesn't need real-time scores.

Here's the key pattern I settled on. Each batch call includes a structured prompt with all 50 listings and asks for an array of scores in response. That keeps the token overhead per listing minimal because the system instruction is shared across all of them.

// Batch scoring request structure
interface BatchScoringRequest {
  custom_id: string;
  method: 'POST';
  url: '/v1/chat/completions';
  body: {
    model: 'gpt-4o-mini',
    messages: [
      { role: 'system', content: scoringSystemPrompt },
      { role: 'user', content: formatBatchListings(listings) }
    ],
    response_format: { type: 'json_object' },
    temperature: 0.1  // Low temperature for consistency
  };
}

I use gpt-4o-mini for the scoring itself. Not the full gpt-4o. The scoring task is a classification problem dressed in natural language, and the mini model handles it well. The cost difference is substantial. At the time of writing, gpt-4o-mini is roughly 20x cheaper per token than gpt-4o, and for structured scoring tasks the quality difference is negligible.

Pro tip: always set temperature to 0.1 or lower for scoring pipelines. You want deterministic, repeatable results, not creative interpretations.

Rate Limiting and Retries the Hard Way

Rate limiting is one of those problems that looks simple until it bites you.

Suppose you write a naive retry loop that pauses for one second on a 429 error, then retries. The problem is that a batch of 50 listings will all hit the rate limit at the same time, all retry at the same time, and hit the limit again. That's called thundering herd, and it makes things worse, not better.

The fix is exponential backoff with jitter. But the more important pattern is a simple token bucket that tracks how many API calls you make per minute and throttles before the API tells you to.

// Token bucket rate limiter
class TokenBucket {
  private tokens: number;
  private lastRefill: number;

  constructor(
    private maxTokens: number,
    private refillRate: number, // tokens per second
    private refillInterval: number = 1000
  ) {
    this.tokens = maxTokens;
    this.lastRefill = Date.now();
  }

  async acquire(): Promise<void> {
    this.refill();
    if (this.tokens <= 0) {
      const waitTime = (1000 / this.refillRate);
      await sleep(waitTime);
      return this.acquire();
    }
    this.tokens--;
  }

  private refill() {
    const now = Date.now();
    const elapsed = now - this.lastRefill;
    const refillTokens = (elapsed / this.refillInterval) * this.refillRate;
    this.tokens = Math.min(this.maxTokens, this.tokens + refillTokens);
    this.lastRefill = now;
  }
}

This pattern means the system never sends a request it knows will get rejected. The token bucket sits before the API call, not after. That alone made a real difference in keeping the pipeline running smoothly.

What Broke and What Fixed It

An AI rewrite pipeline I built for the same platform was eventually shut down by the client over cost concerns. Rewriting a large volume of job descriptions with GPT-class models was never going to be cheap. But I learned something important from that failure.

The scoring pipeline had a clear advantage. Each listing cost roughly the same to score, which made the monthly cost predictable before running it. The rewrite pipeline had no such ceiling. One listing could be short and cheap. Another could be long and expensive. The cost variance made it hard to budget, and that uncertainty killed the project.

If you're building an AI pipeline for a founder or a startup, this is the single most important lesson. Design for predictable cost from day one. Use character limits. Use token budgets. Have a hard cap per item. Make the cost model something you can put in a spreadsheet and explain in two sentences.

I'm evaluating DeepSeek V4 Flash as a much cheaper alternative for the rewrite use case. It would cut costs substantially compared to GPT-4.1, which might bring the pipeline back to life. But the scoring pipeline never had that problem because we designed it with cost limits baked in from the start.

When You Should Stick with Traditional ML

I'll be honest. Not every scoring task needs an LLM.

If you're categorizing job listings by industry (healthcare, tech, finance), a lightweight classifier trained on 10,000 examples will outperform GPT for a fraction of the cost. If you're extracting structured fields like salary ranges or years of experience, regex plus a small model works better.

The LLM wins when the signal lives in the language itself. Tone, implication, cultural context. Things that require reading between the lines. A job posting that says "fast-paced startup environment" means something very different than one that says "stable enterprise role with work-life balance." An LLM picks up on that. A keyword classifier does not.

I reserve the LLM for one specific task in the pipeline: ranking listings by relevance to a specific candidate's profile. That requires understanding both the listing and the candidate semantically. Everything else gets handled by cheaper systems.

If your team is wrestling with similar decisions about where to use LLMs in production and how to keep costs predictable, that's the kind of problem I help with. I build AI pipelines that survive real usage, not just demos. Happy to compare notes.

Written by Abdul Rehman, full-stack AI engineer building production SaaS, MVPs, and AI automation. More at PrimeStrides.

推荐订阅源