GitHub - dropbox/witchcraft

(formerly known as Rust-Warp)

This is a from-scratch reimplementation of Stanford's XTR-Warp semantic search engine ( https://github.com/jlscheerer/xtr-warp ) in safe rust, using a single-file SQLite database as backing storage, making it suitable for client-side deployment. It runs completely stand-alone on your device, needs no API keys, no vector database, no chunking strategy, no fancy re-rankers, and it is lightning fast (21ms p.95 end-to-end search latency on NFCorpus, at 33% NDCG@10, on an Apple Macbook Pro M2 Max, more than twice as fast as the original XTR-WARP on server-class hardware, at similar accuracy.)

Prerequisites

Needs uv, make, and the rust toolchain.

Building and Running

To run, you will need the XTR weights released by Google Deepmind, and this repo contains python scripts for downloading them from Huggingface and quantizing to GGUF format. We can cheat by doing everything with Make:

Creating an index:

For testing, we used the BEIR download script from XTR-Warp to download nfcorpus and check that we could replicate their results. For your convenience, nfcorpus.tsv is included here, so you can run:

$ ./warp-cli readcsv datasets/nfcorpus.tsv

With all the nfcorpus documents imported, we can now create embeddings for them, with:

Next we create the index over the embeddings, with:

All state gets persisted in mydb.sqlite, and you can abort the indexer and it will pick up where it left off. To start over, you can just delete mydb.sqlite.

Querying the index

When you have the index, you can query it with:

$ ./warp-cli query "does milk intake cause acne in teenagers?"

And hopefully get a bunch of relevant answers. You can also try other variations of this, instead of "query" you can also use "hybrid", which combines semantic search with the BM25 search functionality that comes standard with sqlite.

Pickbrain: semantic search over your AI coding sessions

Included as an example is pickbrain (screenshot above), a CLI that indexes your Pi, Claude Code, and OpenAI Codex session transcripts, memory files, and authored documents into a Witchcraft database for fast semantic search. Ever wondered "what was that conversation where I fixed the auth middleware?" — pickbrain finds it, and lets you resume the session directly.

make pickbrain
./pickbrain auth middleware fix    # search across all sessions (auto-ingests new sessions)
./pickbrain --session <UUID> auth  # search within one session
./pickbrain --dump <UUID>          # print full conversation

The source lives in examples/pickbrain/ and demonstrates how to use Witchcraft as a library: document ingestion, embedding, indexing, and hybrid search. To install pickbrain as a skill/extension for Pi and as a skill for both Claude Code and Codex:

This puts the binary on your PATH and installs the skill/extension definitions so you can use pickbrain directly from Pi, Claude Code, or Codex to answer questions requiring global knowledge of all your projects:

More build info

Feature flags

When building, exactly one T5 backend must be enabled:

t5-quantized -- GGUF quantized weights via candle (default)
t5-openvino -- OpenVINO inference backend

Other flags:

metal -- macOS GPU acceleration (Apple Silicon only)
fbgemm -- fbgemm-rs packed GEMM (bf16 weights, faster on x86)
hybrid-dequant -- F32 attention + Q4K FFN with fused gated-gelu (x86, requires fbgemm)
napi -- Node.js native module via napi-rs
embed-assets -- bake weights into binary
progress -- progress bars for CLI

Platform-specific recommended features (these are what make uses automatically):

Apple Silicon: t5-quantized,metal
Intel Mac (x86_64): t5-quantized,fbgemm,hybrid-dequant
Intel Windows (x86_64): t5-openvino,fbgemm

Using as Node module

Builds a universal macOS binary at target/release/warp-macos-universal.node (lipo'd from aarch64 + x86_64 builds with platform-appropriate features).

To use in another project:

cd /path/to/your-project
npm install /path/to/witchcraft

Then in JavaScript:

const { Witchcraft } = require('warp');
const wc = new Witchcraft('/path/to/db.sqlite', '/path/to/assets');

Unit tests and Code Coverage

cargo install cargo-nextest
cargo install cargo-llvm-cov
cargo install llvm-tools-preview
make test

NOTICE that nextest is necessary, simply using "cargo test" will lead to random test failures, because individual tests run in the same process, leading to "history effects".

Preparing a PR

Before submitting a PR, please make sure that

and

Run, and that

Runs and scores in the 0.31-0.33 range