• Book: AI Agents Pocket Guide
• Also by me: Database Playbook
• My project: Hermes IDE | GitHub — an IDE for developers who ship with Claude Code and other AI coding tools
• Me: xgabriel.com | GitHub

On Friday, April 24, 2026, NVIDIA closed at $208.27 and became the first chip company to cross $5 trillion in market cap, per CNBC. Most of us scrolled past it.

Here is the part that matters for the people building software. For your team, NVDA crossing $5T is a forward signal about GPU supply, inference economics, and the on-prem vs. managed-API decision you have been postponing for eighteen months. That decision just got cheaper to make wrong.

This post is about what changed underneath that headline, and how to make the call when you sit down on Monday.

What the milestone is actually saying

NVIDIA added more than $4.5 trillion in market value since the end of 2022. Hyperscalers (Microsoft, Google, Amazon, Meta) committed over $650 billion to AI infrastructure in 2026 alone, per the same CNBC report. Demand is still front-running supply. But the curve has bent.

H200 list prices sit at $30,000–$40,000 per card. B200 lists at $35,000–$40,000. A DGX B300 lands around $325,000 and amortizes over three years to roughly $0.0059 per GB of HBM per hour, per GPU Tracker's GTC 2026 breakdown. Vera Rubin volume ships in H2 2026 with a stated target of 10x lower inference token cost and 5x per-GPU compute over Blackwell.

Read those numbers as a developer. The token-per-dollar floor is dropping. Long-context windows are getting cheaper to serve. Fine-tuning a 70B model went from "we'll get to it next quarter" to "schedule the run for tomorrow."

The macro side is real. The micro side is what bends because of it: what you build, where it runs, who you pay.

Three stack-level shifts that follow

Inference per token keeps falling. Frontier reasoning that was order-of-magnitude more expensive at launch trades at a fraction now. Open-weights models that need 8x H100 to serve at scale will need fewer Blackwell or Rubin cards next year. If your business case for an LLM feature was marginal at 2024 prices, redo the math.

Context windows get usable. A 1M-token window on a slow, expensive model is a demo. The same window on hardware optimized for long-context throughput is a feature. Three things stop being research demos: whole-codebase analysis, full-deposition summarization, contract-pack reasoning.

On-prem inference becomes a real comparison. A 70B model on 2x H200 used to be "interesting if you're a hyperscaler." With falling card prices and better quantization, it is now a realistic comparison for a mid-market team running 50M+ tokens a day.

That last one is the decision your CFO is going to ask about. Here is the math.

The crossover point: when self-host beats the API

A small Python helper. Plug in your own numbers — these are illustrative.

Two plans to compare. The API plan is priced per million tokens; the self-host plan has capex, opex, and a throughput envelope.

from dataclasses import dataclass

@dataclass
class APIPlan:
    input_price_per_1m: float    # USD / 1M input tokens
    output_price_per_1m: float   # USD / 1M output tokens

@dataclass
class SelfHostPlan:
    capex: float                 # GPU + chassis + networking
    amortization_months: int     # depreciation horizon
    monthly_opex: float          # power, cooling, ops, colo
    throughput_tokens_per_sec: int
    utilization: float           # 0..1, realistic duty cycle

Three cost functions. API monthly cost scales linearly with usage. Self-host monthly cost is amortized capex plus opex. Capacity is throughput times duty-cycled seconds.

def api_monthly_cost(daily_in_m, daily_out_m, plan):
    days = 30
    return days * (
        daily_in_m * plan.input_price_per_1m
        + daily_out_m * plan.output_price_per_1m
    )

def self_host_monthly_cost(plan):
    return plan.capex / plan.amortization_months + plan.monthly_opex

def self_host_capacity_tokens_month(plan):
    seconds = 30 * 24 * 3600 * plan.utilization
    return plan.throughput_tokens_per_sec * seconds / 1e6

def crossover_daily_tokens_m(api, host):
    host_cost = self_host_monthly_cost(host)
    blended_api = (api.input_price_per_1m + api.output_price_per_1m) / 2
    return host_cost / (30 * blended_api)

Plug in plausible numbers and run.

api = APIPlan(input_price_per_1m=2.50, output_price_per_1m=10.00)
host = SelfHostPlan(
    capex=120_000,
    amortization_months=36,
    monthly_opex=2_500,
    throughput_tokens_per_sec=180,
    utilization=0.55,
)

print(f"Self-host monthly: ${self_host_monthly_cost(host):,.0f}")
print(f"Crossover: {crossover_daily_tokens_m(api, host):.1f}M tokens/day")
print(f"Capacity: {self_host_capacity_tokens_month(host):,.0f}M tokens/month")

The inputs are the part that matters. Capex on a 2x H200 box around $120k. Throughput in the 150–250 token/s range for a quantized 70B-class model. Utilization realistic (your queue is not full at 3 a.m.), so 50–65%. API blended price somewhere between commercial frontier and open-weights hosted.

Run those and the crossover lands in the low single-digit millions of tokens per day. That number drops every quarter. A year ago it sat much higher; another year out, with Rubin in datacenters, it will be lower again.

The exact crossover does not matter. What matters is that this is a calculator now.

The decision is more flexible now

Two years ago, "self-host" meant a procurement cycle, a hiring plan, and a board-level capex conversation. Today three things changed.

First, neoclouds. CoreWeave, Lambda, Crusoe, and Voltage Park rent H200 and B200 by the hour. You do not need to buy a $325k DGX to test self-host economics. You can spin up the same hardware for a week, run real traffic, measure your actual tokens-per-second on your actual prompts, then decide.

Second, open-weights got good. Llama 4, DeepSeek-V3, Qwen3. The gap between "frontier API" and "best open model my team can run" narrowed enough that for a lot of workloads (summarization, classification, structured extraction, code completion on internal repos) you do not need GPT-class reasoning.

Third, hybrid is normal. A workload split is a config change, not an architecture rewrite. Frontier API for the 5% of requests that need it; self-hosted for the rest. Route on prompt length, on tool requirement, on a quality classifier. Routers like LiteLLM and Portkey made this a weekend integration.

So the decision moves from "all-in on one provider" to "what fraction of my traffic is worth owning?"

Where on-prem still loses

Three places, and you want to be honest about them.

Spiky workloads. If your traffic varies 50x between trough and peak, you provision for peak and pay for idle. The API providers eat the variance for you. A self-host break-even calculation on average load lies to you about peak.

Latency from a single region. Hyperscaler APIs have global edge. Your colo in Ashburn does not. If you serve users in Singapore, Frankfurt, and São Paulo, the round-trip from a single self-hosted region adds 200ms before your model emits a token.

Model churn. The frontier moved three times last year. If you fine-tuned and self-hosted on each new release, you re-paid setup costs every quarter. Teams that stayed on managed APIs swapped models with a config change.

For a stable, predictable, single-region workload at meaningful scale, self-host wins on cost. Outside that envelope, the API wins on operational simplicity, which is worth real dollars.

What the database side of the stack does

Worth mentioning, because the storage layer is moving in the same direction. Cheaper inference makes it economically reasonable to embed more, retrieve more, and rerank with a model rather than a shallow scorer. Vector search at 10x current scale is no longer a budget conversation; it is a config one.

That changes how you pick your store. A managed vector DB with predictable per-query pricing made sense when every embedding was a measured expense. With ingestion costs falling, the calculus shifts toward stores you can run yourself: pgvector on Postgres you already operate, or a dedicated cluster like Qdrant or Weaviate self-hosted on the same hardware that runs your inference.

The same logic that pushes inference on-prem pushes the retrieval layer to follow. Locality matters when you are doing 50 vector lookups per generated token.

What to do this week

Three concrete moves, all small.

Pull last month's API invoice. Compute your tokens-in and tokens-out per day. That is the input to the calculator above. Most teams do not know this number off-hand; it tells you immediately whether self-host is even on the table.

Run a one-week pilot on a neocloud. Same prompts, same eval set, an open-weights model in the size class you would actually deploy. Compare quality, latency, cost. The pilot tells you whether the routing-split is real for your workload or whether the API delta is too large.

Decouple your retrieval layer from your inference provider. Whatever you choose for inference, make sure your embeddings, indexes, and reranker are not locked to a single vendor. If you flip the inference switch in six months, you do not want the retrieval stack flipping with it.

A $5 trillion market cap is a number on a screen. The shift it confirms is what reshapes the stack you are about to commit to: GPU compute is getting cheaper, faster, more accessible. The decision got more flexible. Use that.

If this was useful

Most of the choices above are the same choices AI Agents Pocket Guide and the Database Playbook cover end-to-end: routing between providers, picking a retrieval store, deciding when an agent loop is cheaper to run yourself. One on the agent layer, one on the storage layer underneath. If you are working through the calculator above this quarter, both are written for that conversation.