a single, dependency-free python script that turns free-tier API keys into one always-on chat endpoint.

It discovers every free model across your connected providers, ranks them best → worst, and routes each message to the best available one.

When a model runs out of usage (rate limit / quota), it automatically fails over to the next best model and retries — so you keep going without touching anything.

Provider catalog (~18 providers)

Every provider is OpenAI-compatible and auto-enables when its key is in .env. Sourced from free-ai-tools.

Run /providers (or python3 cascade.py --providers) to see which are connected and the exact env var + signup link for each one you can still add. Add a key, run /refresh, and that provider's models join the leaderboard instantly. OVHcloud works with no key at all, so the app has models even with an otherwise empty .env.

Setup

Add any subset of the keys above to .env in this folder. No pip install needed (Python 3 standard library only).

Usage

python3 cascade.py            # interactive chat with auto-routing
python3 cascade.py --list     # ranked leaderboard
python3 cascade.py --providers# provider catalog (connected + addable)
python3 cascade.py --bench    # race top models for latency + tokens/sec
python3 cascade.py -q "..."   # one-shot prompt
python3 cascade.py --serve    # run as an OpenAI-compatible REST API (see below)

In-chat commands

Server mode — one unified API

Run cascade as a long-lived HTTP server and it becomes a single OpenAI-compatible endpoint in front of every provider. Same discovery, ranking, and auto-failover as the CLI — just RESTful, so any existing OpenAI client/SDK can use all ~18 free tiers as one API with automatic failover.

python3 cascade.py --serve                  # http://127.0.0.1:8000
python3 cascade.py --serve --host 0.0.0.0 --port 9000

Config via flags or env: --host/CASCADE_HOST, --port/CASCADE_PORT. Set CASCADE_API_KEY to require an Authorization: Bearer <key> on requests (otherwise any key is accepted, since the server is meant to run locally).

Endpoints

The model field picks the routing strategy:

• "auto" (or omit it) → best available model, fail over down the leaderboard.
• "<provider>/<model>" (e.g. groq/openai/gpt-oss-120b, from /v1/models) → pin to that exact model.
• a bare model id present on several providers → any provider offering it, best-first, with failover between them.

Use it from anything

# curl
curl http://127.0.0.1:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"auto","messages":[{"role":"user","content":"hello"}]}'

# OpenAI Python SDK — just change base_url
from openai import OpenAI
client = OpenAI(base_url="http://127.0.0.1:8000/v1", api_key="not-needed")
r = client.chat.completions.create(
    model="auto",                                  # or a specific provider/model id
    messages=[{"role": "user", "content": "hello"}],
    stream=True,
)
for chunk in r:
    print(chunk.choices[0].delta.content or "", end="", flush=True)

Successful responses include a non-standard cascade field naming the provider/model that actually served the request and any failover attempts, so you can see what routing did.

How ranking works

Each discovered model is scored by family (DeepSeek / Nemotron / Llama / Qwen / GLM / Kimi / GPT-OSS …), parameter size, and a small provider-speed tiebreak (Cerebras/Groq first). The 70–200B range is treated as the free-tier sweet spot; models ≥300B are penalised because on free tiers they are the slowest and most rate-limited (use /bench + /fastest if you want one anyway). The chat is sent to the highest scorer that is currently ✓ ready; anything on ⏳ cooldown or ✗ down is skipped.

/bench — empirical speed race

Fires one tiny prompt at the top available models concurrently, measures first-token latency (TTFT) and tokens/sec, and prints a sorted table. /fastest then re-ranks the whole leaderboard by measured throughput, so you can route for speed instead of raw quality. Great for finding the fastest model that's actually up.

Failover triggers

Usage tracking

• Daily budget bars — documented free limits (e.g. Groq ~1,000 req/day, Cloudflare 10K neurons/day) shown against locally-counted requests, persisted per day to ~/.cascade_state.json. Long cooldowns (quota exhaustion) persist too, so a model that's tapped out stays skipped across restarts until its reset.
• OpenRouter — live tier + daily spend via its auth/key endpoint.
• Groq — live remaining requests/tokens from response headers.
• Others have no usage API and simply fail over on 429/quota.