I needed a fast, repeatable way to compare production-grade open models before routing traffic to them. In this post, I will walk through a lightweight Python harness that sends identical prompts to four different Oxlo.ai models, times each response, and scores the outputs with a judge model so you can pick the right one for your workload.

What you'll need

• An Oxlo.ai API key from https://portal.oxlo.ai
• Python 3.10 or newer
• The OpenAI SDK: pip install openai

Step 1: Set up the Oxlo.ai client and model roster

We start by initializing the client and defining the models we want to test. I picked a mix of generalist, reasoning, and multilingual models that Oxlo.ai hosts.

from openai import OpenAI
import os

client = OpenAI(
    base_url="https://api.oxlo.ai/v1",
    api_key=os.environ.get("OXLO_API_KEY")
)

CANDIDATE_MODELS = [
    "llama-3.3-70b",
    "qwen-3-32b",
    "kimi-k2.6",
    "deepseek-v3.2",
]

TEST_PROMPT = (
    "Write a Python function that accepts a list of integers and returns "
    "the longest strictly increasing subsequence. Include type hints, "
    "a docstring, and a simple test case in the same code block."
)

Step 2: Define the judge system prompt

Before we fire requests, we need a consistent rubric. I use a separate system prompt for the judge model so scoring stays objective across runs.

JUDGE_SYSTEM_PROMPT = """You are an expert code reviewer. You will receive a user request and a candidate response. Score the response on three axes from 1 to 5:
1. Correctness: does the code solve the problem and pass the included test?
2. Clarity: are the docstring, types, and variable names clear?
3. Conciseness: is the solution free of unnecessary bloat?

Return ONLY a JSON object with keys: model, correctness, clarity, conciseness, total_score, and one_sentence_verdict.
"""

Step 3: Dispatch prompts concurrently

Waiting for four sequential API calls is slow. I use a thread pool to hit all candidate models at once and record wall-clock latency for each.

import time
import concurrent.futures

def query_model(model_id: str, prompt: str) -> dict:
    start = time.perf_counter()
    response = client.chat.completions.create(
        model=model_id,
        messages=[
            {"role": "system", "content": "You are a helpful coding assistant."},
            {"role": "user", "content": prompt},
        ],
        temperature=0.2,
    )
    elapsed = time.perf_counter() - start
    return {
        "model": model_id,
        "text": response.choices[0].message.content,
        "latency_sec": round(elapsed, 2),
    }

def run_benchmark(prompt: str):
    results = []
    with concurrent.futures.ThreadPoolExecutor(max_workers=4) as executor:
        futures = {
            executor.submit(query_model, m, prompt): m
            for m in CANDIDATE_MODELS
        }
        for future in concurrent.futures.as_completed(futures):
            results.append(future.result())
    return results

Step 4: Score outputs with a judge model

Now we feed each candidate response into a judge. I use llama-3.3-70b as the judge because it gives stable JSON formatting.

import json

def judge_response(candidate: dict, original_prompt: str) -> dict:
    judge_input = (
        f"User request:\n{original_prompt}\n\n"
        f"Candidate response from {candidate['model']}:\n{candidate['text']}\n\n"
        "Score the response and return the JSON object."
    )
    response = client.chat.completions.create(
        model="llama-3.3-70b",
        messages=[
            {"role": "system", "content": JUDGE_SYSTEM_PROMPT},
            {"role": "user", "content": judge_input},
        ],
        temperature=0.1,
    )
    raw = response.choices[0].message.content.strip()
    if raw.startswith("

```"):
        raw = raw.split("```

")[1].replace("json", "").strip()
    scores = json.loads(raw)
    return {**candidate, **scores}

def score_all(results: list, prompt: str):
    return [judge_response(r, prompt) for r in results]

Step 5: Render the comparison report

Finally, we print a markdown table so the differences are obvious at a glance.

def print_report(scored_results: list):
    print("| Model | Latency (s) | Correctness | Clarity | Conciseness | Total | Verdict |")
    print("|-------|-------------|-------------|---------|-------------|-------|---------|")
    for r in scored_results:
        print(
            f"| {r['model']} | {r['latency_sec']} | "
            f"{r['correctness']} | {r['clarity']} | {r['conciseness']} | "
            f"{r['total_score']} | {r['one_sentence_verdict']} |"
        )

if __name__ == "__main__":
    print("Running benchmark...")
    raw_results = run_benchmark(TEST_PROMPT)
    scored = score_all(raw_results, TEST_PROMPT)
    scored.sort(key=lambda x: x["total_score"], reverse=True)
    print_report(scored)

Run it

Save the script as benchmark.py, export your key, and run it.

export OXLO_API_KEY="your-key-here"
python benchmark.py

Example output (values will vary by run):

Running benchmark...
| Model | Latency (s) | Correctness | Clarity | Conciseness | Total | Verdict |
|-------|-------------|-------------|---------|-------------|-------|---------|
| deepseek-v3.2 | 4.2 | 5 | 5 | 4 | 14 | Produces correct LIS with clean type hints and a valid doctest. |
| kimi-k2.6 | 3.8 | 5 | 4 | 4 | 13 | Correct solution but slightly verbose docstring. |
| qwen-3-32b | 2.1 | 4 | 4 | 5 | 13 | Correct logic, omits explicit test case in the block. |
| llama-3.3-70b | 1.9 | 4 | 5 | 4 | 13 | Good structure, test case is present but uses print instead of assert. |

Wrap-up and next steps

Swap the static prompt for a JSONL test suite so you can regression-test model behavior on every deploy. You can also add a lightweight Streamlit frontend so non-engineers can run comparisons and vote on their preferred output.