Inference should not start from zero every time.

CONVERA is an experimental local inference runtime that treats repeated work as reusable state. The public build is CONVERA-OSS: a functional lite runtime with model loading, prompt-level KV persistence, local token graph reuse, benchmarks, telemetry, and a fixed-size content-addressed tensor store.

What It Does

Traditional inference reruns similar work again and again. CONVERA-OSS keeps a local reusable-state layer so repeat prompts and related runs can reuse stored runtime artifacts.

The repo includes:

• Hugging Face model loading
• backend selection for CUDA, ROCm-compatible PyTorch, MPS, and CPU
• prompt-level KV cache persistence
• local token graph lookup
• convera_store_lite, a fixed-size deduplicated tensor store
• benchmark/report tooling
• local dashboard
• privacy-first telemetry client

Lite Runtime

This repository includes the lite runtime.

Higher-efficiency runtime acceleration layers are not included in CONVERA-OSS. Public extension points stay inside the sealed convera_core_api.interface contract.

Public API boundary:

from convera_core_api import interface

refs = interface.store_tensor(tensor)
tensor = interface.load_tensor(refs)
kv = interface.optimize_kv(kv)
state = interface.merge_states(state_a, state_b)

That API returns only tensors, references, and minimal metadata.

Quick Start

/opt/homebrew/bin/python3.12 -m venv .venv312
source .venv312/bin/activate
pip install -e .
convera health

Download a model into models/llama3 before running inference. Meta Llama models may require Hugging Face authentication and license acceptance.

hf auth login
python scripts/download_llama.py

The default target is meta-llama/Meta-Llama-3-8B. If Hugging Face reports that the repository requires approval, accept the model license with the same account and rerun the command.

Run

convera run --prompt "Explain neural networks in detail."
convera benchmark
uvicorn ui.app:app --reload

Dashboard:

http://127.0.0.1:8000/static/index.html

Benchmark

python -m benchmarks.benchmark

The benchmark reports:

• latency
• tokens/sec
• KV hit rate
• chunk reuse ratio
• disk usage

For a small public proof of repeated-prompt behavior:

python -m metrics.benchmark_runner

Output:

metrics/output/benchmark.json
metrics/output/benchmark.png
metrics/output/comparison.png

Example output:

Run 1:
  Cached: False
  Latency: 120 ms

Run 2:
  Cached: True
  Latency: 40 ms

The public benchmark reports only cache hit status, latency, and response length. It does not expose token traces, cache contents, or runtime internals.

Benchmark Visualization

Run:

python -m metrics.benchmark_runner

Then open the UI or view the generated graph:

http://127.0.0.1:8000/benchmark-graph

Privacy

Telemetry sends metrics only after CONVERA_METRICS_API_URL and CONVERA_METRICS_API_KEY are configured. It does not send prompts, outputs, file names, or local paths.

The server-side telemetry contract is documented in SERVER_HAND_OFF.md.

Direction

CONVERA-OSS is the adoption layer. It is intentionally useful, inspectable, and easy to run.