How Sarva keeps the same GPU multipler on the cost side a...

Sarva is a hub-and-spoke compute grid — a FastAPI backend, a Next.js 14 dashboard, and a Python node agent that runs on contributor machines. This post is not about the hub-and-spoke pattern (that one is well-trodden). It's about the one piece of business logic I was most worried about getting right: the moment a job is assigned, how do you decide what it costs the submitter, what the node earns, and how do you make sure the audit log captures enough state to reconstruct any disagreement after the fact.

I am writing this after the third time I rewrote submit_job and complete_job. The first two versions diverged — they used different gr() lookups, different rounding, and a "credit" abstraction that quietly lost precision. This is the version that has been live on Railway for the last few weeks, and it is the one I am willing to defend.

The formula is the same on both sides, but the inputs differ

There are two multiplier functions, both tiny:

GPU_MULT = {
    "rtx-4090": 3.0, "rtx-5090": 3.0, "rtx-3090": 2.5,
    "rtx-4070": 2.5, "rtx-3060": 2.0, "rtx-2070": 2.0,
    "gtx-1080ti": 1.5, "gtx-1080": 1.5, "gtx-1660": 1.3, "cpu": 0.8
}
GEO_RATE = {"in": 0.7, "india": 0.7, "us": 1.0, "uk": 1.0, "eu": 0.95}
PLATFORM_FEE = 0.20

def qs(g: str) -> float: return GPU_MULT.get(g.lower(), 1.0)
def gr(r: str) -> float: return GEO_RATE.get(r.lower(), 1.0)

qs() is the node's quality score — set once at registration, never changes. gr() is a per-user region rate. Both functions are called on the cost side and the earn side. The catch is whose gr() you use on each side, and that asymmetry is intentional.

On submission, the cost is what the submitter pays

@app.post("/jobs/submit")
def submit_job(type: str, submitter_id: str, script: str = None,
               slices: int = 1, priority: int = 0,
               db: Session = Depends(get_db)):
    user = db.query(User).filter(User.id == submitter_id).first()
    if not user:
        raise HTTPException(status_code=404, detail="User not found")
    gpu_cost = {"ml": 2.5, "gaming": 3.0, "compute": 1.0}.get(type.lower(), 1.0)
    cost = slices * gpu_cost * gr(user.region)        # ← submitter's region
    final_cost = 0.0 if user.tier == UserTier.GOD else cost
    if user.tier != UserTier.GOD and user.balance < final_cost:
        raise HTTPException(status_code=400,
            detail=f"Insufficient credits. Need {final_cost}, have {user.balance}")
    job = Job(id=job_id, type=type, status=JobStatus.PENDING,
              submitter_id=submitter_id, script=script,
              slices=slices, credits_cost=final_cost, priority=priority)
    db.add(job)
    if user.tier != UserTier.GOD:
        user.balance -= final_cost
        user.spent_total += final_cost
        tx = Transaction(... type="spend", amount=-final_cost, ...)
        db.add(tx)
    audit(db, "job_submitted", {"job_id": job_id, "type": type, "submitter_id": submitter_id})
    db.commit()

Two things to notice:

gr(user.region) is the submitter's rate. A submitter in India pays 0.7x the headline price. A submitter in the US pays 1.0x. The intent is to make compute cheaper where bandwidth and power are cheaper.
The Job.credits_cost is locked at submission time. If the node that eventually runs the job is in a different region, that does not retroactively change the cost. This was a deliberate decision: re-pricing mid-flight would let a malicious node wait until a "cheap" job was assigned, then run it on a "premium" node and demand re-pricing. The cost is the cost.

On completion, the earn is what the node's owner gets

@app.post("/jobs/{job_id}/complete")
def complete_job(job_id: str, result_cid: str = None, error: str = None,
                 db: Session = Depends(get_db)):
    job = db.query(Job).filter(Job.id == job_id).first()
    # ... mark COMPLETED / FAILED ...
    if job.assigned_node_id:
        node = db.query(Node).filter(Node.id == job.assigned_node_id).first()
        if node:
            node.status = NodeStatus.ONLINE
            if job.credits_cost > 0 and not error:
                earn_mult = qs(node.gpu_tier) * gr(node.region)   # ← node's region
                earned = job.credits_cost * earn_mult * (1 - PLATFORM_FEE)
                owner = db.query(User).filter(User.id == node.owner_id).first()
                if owner:
                    owner.balance += earned
                    owner.earned_total += earned
                    tx = Transaction(..., type="earn", amount=earned,
                                     job_id=job_id, ...)
                    db.add(tx)
    audit(db, "job_completed", {"job_id": job_id, "error": error})
    db.commit()

gr(node.region) here is the node's region, not the submitter's. A node in India running a job submitted by a US user gets 0.7x — the cheaper-region rate flows through to the earner. This is symmetric in spirit (cheap power → cheap earn), but the two gr() calls are in different functions, hours apart in real time, and called with different arguments. That asymmetry used to be a source of bugs. I eventually settled on the rule: "use the subject's region, always."

PLATFORM_FEE = 0.20 is taken off the top of the earn, not added to the cost. So the 20% comes from what the node would have earned, not from the submitter's pocket. This is the part where a lot of decentralized-compute projects get the framing wrong: "we take 20% from the worker" sounds bad; "the worker keeps 80% of whatever the submitter paid" sounds fine. They are the same number. The latter framing is what we ship.

The audit log is the actual safety net

The credit math is small enough to hold in your head. The reason I sleep at night is audit(db, ...). Every state-changing operation writes a row to audit_logs with a type and a data JSON blob:

def audit(db: Session, log_type: str, data: dict):
    log = AuditLog(id=uuid.uuid4().hex[:12], type=log_type, data=data)
    db.add(log)

The events I currently log: user_registered, node_registered, job_submitted, job_assigned, job_completed, topup, cashout. The data blob is whatever I have at the time of the call — it is not normalized. That is a deliberate choice. The alternative is a clean event schema, but clean event schemas are how you end up with event_v2 and a migration that nobody wants to run. The JSON blob is messy but it is complete: if a node owner and the platform disagree about whether a job ran, the audit log has the job_id, the node_id, the error field, and the timestamp. I can replay the credit math from the audit log and the immutable transactions table to figure out who owes whom what.

There is a /logs endpoint that returns the most recent 50 entries. It is the first thing I check when somebody opens a ticket.

What I deliberately did not build yet

No per-job gr() snapshot on the Job row. The Job table stores credits_cost (locked), but not the gr() value at submission time. If we ever change the GEO_RATE dict and a dispute arises, the audit log + transactions table is enough to reconstruct — but it is annoying, not instant. I am 70% convinced this is fine and 30% convinced I should add a cost_gr_snapshot column to the Job row tomorrow.
No reconciliation cron. The users.balance is the source of truth, but I do not yet have a job that walks every user's transactions and asserts balance == sum(tx.amount). I run this query by hand once a week. It is fine for a few hundred users. It is not fine at 10,000.
No MIN_DISK_GB enforcement on assignment. The node agent reports diskFreeGb at registration but the orchestrator does not check it before handing out a job. This is on the list.
God-tier bypass is a single line. final_cost = 0.0 if user.tier == UserTier.GOD else cost lets the god user submit anything for free. That is intentional for dev, but it is not gated by environment, and a single leaked god user ID in production would be a small catastrophe. I am aware.

What I'd love feedback on

The pricing formula is the single thing I would want a second pair of eyes on. Specifically:

Is the "use the subject's region on each side" rule defensible, or should I be using the job's region (locked at submission from the submitter) for both calls?
Should PLATFORM_FEE be a flat 20%, or should it scale with slices (lower for short jobs, higher for long ones) so that the platform has a stronger incentive to keep cheap jobs flowing?
Is the audit-log-as-JSON-blob pattern something to grow out of before we hit 1,000 users, or is it the kind of thing I can defend long-term?

Sarva is open source at github.com/AmSach/sarva — the monorepo is /backend (FastAPI + Postgres), /frontend (Next.js 14), and /node (Python agent). The backend is live on Railway, the dashboard is on Vercel, and the node agent is a single-file Python script you can run on any machine with HUB_URL and AUTH_TOKEN set.

If you have shipped a two-sided credit ledger before, I'd genuinely like to know whether the audit-blob approach scales or whether I am about to regret it. Comments welcome.

Tags: python, fastapi, opensource, distributed

推荐订阅源

DEV Community