Agent-native knowledge OS built on Markdown

PULSE8.ai Cortex is an agent-native knowledge OS built on Markdown. It gives AI agents and humans a shared vault backed by a typed knowledge graph, full-text search, and a MarkItDown-powered compiler — all accessible through a unified MCP interface.

Drop files in (PDF, DOCX, PPTX, XLSX, HTML, images, and more), let agents read, write, search, link, and compile knowledge — no database required.

Inspired by Andrej Karpathy's LLM Wiki pattern — a persistent, compounding knowledge base maintained by LLMs instead of re-derived on every query. Search powered by Tobi Lütke's QMD.

Get started

1. Clone the repository:

  git clone https://github.com/synpulse8-opensource/pulse8-ai-cortex-knowledge-vault.git
  cd cortex-knowledge-vault

2. Launch PULSE8.ai Cortex:

This builds and starts both **PULSE8.ai Cortex** (API + MCP on `:8420`) and **QMD** (search on `:3100`), waits for health checks, and you're ready to go.

3. Connect your MCP client (e.g. Claude Desktop) to http://localhost:8420/mcp/.

To stop: ./scripts/stop.sh

Cortex-only mode (macOS / native QMD)

If you want QMD to run natively (e.g. on macOS with Metal GPU acceleration), start only the Cortex container:

# Terminal 1: Run QMD natively
npm install -g @tobilu/qmd
VAULT_PATH=./example_vault node docker/qmd/server.mjs

# Terminal 2: Start only Cortex in Docker
./scripts/start.sh --cortex-only

To stop: ./scripts/stop.sh --cortex-only

GPU-accelerated QMD (EC2 / Linux with NVIDIA GPU)

For production deployments with NVIDIA GPU acceleration:

docker compose -f docker-compose.yml -f docker-compose.gpu.yml up --build -d

See docs/ec2-gpu-setup.md for a full guide on instance selection, NVIDIA toolkit installation, and cost estimates.

Features

MCP tools

Architecture

┌──────────────────────────────────────────────┐
│  MCP Client (Claude Desktop, Cursor, etc.)   │
└──────────┬───────────────────────────────────┘
           │  MCP (HTTP or stdio)
┌──────────▼───────────────────────────────────┐
│  PULSE8.ai Cortex  :8420                     │
│  ┌──────────────────────────────────────┐     │
│  │ Auth (API Key or Microsoft Entra ID) │     │
│  └──────────────┬───────────────────────┘     │
│  ┌─────────┐ ┌──┴───────┐ ┌──────────────┐   │
│  │ MCP     │ │ REST API │ │ Vault Watcher│   │
│  │ /mcp/   │ │ /api/v1/ │ │ (watchfiles) │   │
│  └────┬────┘ └────┬─────┘ └──────┬───────┘   │
│       └───────────┼──────────────┘           │
│            ┌──────▼──────┐                   │
│            │ Graph Engine│                   │
│            │ + Compiler  │                   │
│            └─────────────┘                   │
└──────────┬───────────────────────────────────┘
           │
┌──────────▼───────────────────────────────────┐
│  QMD  :3100                                  │
│  BM25 + vector search, auto-indexes on start │
└──────────┬───────────────────────────────────┘
           │
┌──────────▼───────────────────────────────────┐
│  Vault (bind-mounted volume)                 │
│  wiki/ raw/ agents/ sessions/ daily/ feedback/ │
│  .cortex/ (graph.json, index.md, log.md)     │
└──────────────────────────────────────────────┘

Vault layout

The vault is a plain directory of Markdown files organised by purpose. Cortex classifies each file into a typed node (NodeType) used by the graph engine and exposed in REST and MCP responses.

How classification works

Order of precedence (first match wins):

1. *Frontmatter type:* — explicit override always wins (e.g. type: note in agents/foo.md resolves to NodeType.NOTE)
2. Folder prefix — files under raw/ agents/ sessions/ daily/ feedback/ inherit the folder's type with no filename suffix needed (e.g. daily/2026-06-10.md → daily)
3. Filename suffix (backward-compatible) — .agent.md, .session.md, .memory.md are still honored anywhere (e.g. wiki/legacy.agent.md → agent_def)
4. Default — NodeType.NOTE

In practice this means you can drop YYYY-MM-DD.md straight into daily/, or an unsuffixed planner.md into agents/, and the graph and API will classify them correctly without any renaming.

Daily activity log

Every vault_write, vault_ingest, and successful compile event (MCP and REST paths) is automatically mirrored into today's UTC daily note at daily/YYYY-MM-DD.md. The file is created on first event of the day and each subsequent event appends a ## [HH:MM] event | summary block plus a [[wiki-stem]] wikilink (so the watcher draws a LINKS_TO edge to the affected note). The format follows the Karpathy log.md greppable-prefix pattern — grep "^## \[" daily/2026-06-10.md gives a clean timeline of the day.

Writes targeting daily/, feedback/, or .cortex/ are deliberately not mirrored (would be self-referential noise). The hidden .cortex/log.md audit log is unaffected and continues to receive every operation.

Bulk ingest

For ingesting many files at once (dozens or hundreds of PDFs, papers, docs), use the one-click shell script instead of feeding them one at a time through MCP. It reads directly from a local directory — no wire overhead, no running server required — deduplicates via SHA-256 hashing, compiles with bounded concurrency, and rebuilds the index once at the end.

One-click script (recommended)

# Ingest all files from a directory
./scripts/bulk_ingest.sh ./my-papers/

# Dry-run to preview what would be ingested
./scripts/bulk_ingest.sh ./my-papers/ --dry-run

# Force re-ingest (bypass dedup manifest)
./scripts/bulk_ingest.sh ./my-papers/ --force

# Control LLM concurrency (default: 4)
./scripts/bulk_ingest.sh ./my-papers/ --concurrency 8

The script automatically loads your .env for the LLM key and vault path, prints a summary, then runs the full pipeline (copy, compile, reindex). No running Cortex server needed.

Python CLI (direct)

CORTEX_VAULT_PATH=./example_vault uv run cortex-bulk-ingest --source ./my-papers/

Inside Docker

# Set INGEST_DIR in .env or export it, then restart
export INGEST_DIR=/path/to/your/papers
docker compose up -d

# Run bulk ingest inside the container
docker exec pulse8-ai-cortex uv run cortex-bulk-ingest --source /ingest

Via REST API

For programmatic use without MCP (requires running Cortex server):

curl -X POST http://localhost:8420/api/v1/bulk-ingest \
  -H "Content-Type: application/json" \
  -H "x-api-key: your-secret-api-key" \
  -d '{"source_dir": "/ingest", "concurrency": 4}'

Deduplication

The dedup manifest is stored at .cortex/ingest-manifest.json. Files are matched by content hash, not filename — renaming a file won't cause re-ingestion, and the same content under a different name will be skipped.

Configuration

Copy the example and fill in your values:

OPENROUTER_API_KEY and CORTEX_LLM_API_KEY are accepted as aliases for LLM_API_KEY. Variables above are set in .env (Docker reads them via Compose) and map to the CORTEX_* settings used by the app.

Authentication

Cortex supports two authentication methods that protect both the REST API (/api/v1/) and the MCP endpoint (/mcp/). Set AUTH_METHOD in .env to choose:

API Key (AUTH_METHOD=apikey)

The simplest option. Set the method and key in .env:

AUTH_METHOD=apikey
API_KEY=your-secret-api-key

Clients pass it via the x-api-key header:

# REST API
curl http://localhost:8420/api/v1/health \
  -H "x-api-key: your-secret-api-key"

# MCP (via curl)
curl -X POST http://localhost:8420/mcp/ \
  -H "x-api-key: your-secret-api-key" \
  -H "Content-Type: application/json" \
  -d '{"jsonrpc":"2.0","id":1,"method":"initialize","params":{...}}'

No OAuth discovery endpoints are served — no login popups. Requests without a valid key receive a 401.

Microsoft Entra ID (AUTH_METHOD=oidc)

For enterprise environments that require interactive login with MFA support:

AUTH_METHOD=oidc
OIDC_TENANT_ID=your-tenant-id
OIDC_CLIENT_ID=your-client-id
OIDC_CLIENT_SECRET=your-client-secret
OIDC_BASE_URL=http://localhost:8420

This enables:

• REST API: OAuth 2.0 Authorization Code Flow via GET /api/v1/login. After login, pass the access token as Authorization: Bearer <token>. A valid x-api-key header is also accepted as a fallback when API_KEY is set.
• MCP endpoint: FastMCP's built-in OIDCProxy handles interactive browser-based login.

Azure AD app registration

To use OIDC, register an app in the Azure Portal:

1. Go to Azure Active Directory → App registrations → New registration
2. Set the redirect URI to http://localhost:8420/api/v1/auth/callback (Web platform)
3. Under Certificates & secrets, create a client secret
4. Under API permissions, add openid, profile, and email (Microsoft Graph → Delegated)
5. Copy the Tenant ID, Client ID, and Client Secret into .env

MCP client setup

Claude Desktop

An example config is included at [claude_desktop_config.example.json](claude_desktop_config.example.json).

HTTP with API key (recommended) — PULSE8.ai Cortex runs as a persistent server:

{
  "mcpServers": {
    "cortex": {
      "url": "http://localhost:8420/mcp/",
      "headers": {
        "x-api-key": "your-secret-api-key"
      }
    }
  }
}

HTTP without auth — when no authentication is configured:

{
  "mcpServers": {
    "cortex": {
      "url": "http://localhost:8420/mcp/"
    }
  }
}

Stdio — Claude Desktop launches the server on demand (no auth needed):

{
  "mcpServers": {
    "cortex": {
      "command": "uv",
      "args": ["run", "--project", "/path/to/cortex", "python", "-m", "cortex.mcp"],
      "env": {
        "CORTEX_VAULT_PATH": "/path/to/your/vault"
      }
    }
  }
}

Cursor

Add to your .cursor/mcp.json:

{
  "mcpServers": {
    "cortex": {
      "url": "http://localhost:8420/mcp/",
      "headers": {
        "x-api-key": "your-secret-api-key"
      }
    }
  }
}

How it works

Watcher and Compiler are independent components:

• The Watcher maintains the graph. Any .md file added, modified, or deleted triggers automatic node/edge updates.
• The Compiler converts raw source files to Markdown using MarkItDown and writes them to wiki/. Supported formats include PDF, DOCX, PPTX, XLSX, HTML, CSV, JSON, XML, images (EXIF/OCR), and plain text. The LLM is only used for optional cross-reference detection between articles.

They connect indirectly: the compiler writes to wiki/, the watcher picks those up and updates the graph.

Supported file formats

Search uses a two-stage pipeline:

1. QMD performs keyword/semantic search on file contents
2. PULSE8.ai Cortex enriches results with graph edges (wikilinks, tags, relationships between matched notes)

QMD answers "what's relevant?" — the graph answers "how are these results connected?"

Development

# Install dependencies
uv sync --all-extras

# Run tests
uv run pytest tests/ -v

# Run shell tests (requires bats-core)
bats tests/test_start_sh.bats

# Start PULSE8.ai Cortex locally (without Docker)
CORTEX_MCP_TRANSPORT=http CORTEX_VAULT_PATH=./example_vault uv run python scripts/serve.py

Utility scripts

Data persistence

The vault directory is bind-mounted from your host into the containers. All data lives on your local disk and survives container restarts.

The QMD search index is stored in a Docker volume (qmd-cache). To force a full re-index:

docker compose down -v
./scripts/start.sh

Releasing

Releases are automated through GitHub Actions. Publishing a GitHub Release triggers three workflows that build and publish everything:

To cut a release:

1. Bump the version in pyproject.toml and server.json (keep them in sync), update CHANGELOG.md, and commit to main.

   # optional: validate the registry manifest locally before tagging
   mcp-publisher validate

2. Create the GitHub Release — via the UI (Releases → "Draft a new release" → new tag vX.Y.Z) or the CLI:

   git tag vX.Y.Z && git push origin vX.Y.Z
   gh release create vX.Y.Z --title "vX.Y.Z" --notes-file docs/releases/vX.Y.Z.md

3. That's it — the release event fires all three workflows. publish-mcp.yml waits for PyPI to serve the new version (so the mcp-name ownership marker in this README is verifiable), then publishes the server via GitHub OIDC under io.github.synpulse8-opensource/* (no token or local mcp-publisher needed).

4. Verify the registry entry once the workflow finishes:

   curl "https://registry.modelcontextprotocol.io/v0.1/servers?search=pulse8-ai-cortex-knowledge-vault"

Contributing

We welcome contributions! Please open an issue to discuss your idea before submitting a pull request.

# Fork and clone the repo
git clone https://github.com/<your-username>/cortex-knowledge-vault.git
cd cortex-knowledge-vault

# Create a branch
git checkout -b feat/my-feature

# Install dev dependencies
uv sync --all-extras

# Make changes, then run tests
uv run pytest tests/ -v

# Submit a pull request

Reporting issues

Use GitHub Issues to report bugs or request features.

Acknowledgements

PULSE8.ai Cortex builds on ideas and tools from the open-source community:

• LLM Wiki by Andrej Karpathy — the core pattern of an LLM-maintained, persistent knowledge base that compiles and interlinks knowledge incrementally rather than re-discovering it from raw documents on every query. This gist is the direct inspiration for Cortex's architecture.
• QMD by Tobi Lütke — the on-device search engine powering all full-text and hybrid search in Cortex. QMD combines BM25, vector search, and LLM re-ranking, all running locally.
• MarkItDown by Microsoft — the file-to-Markdown converter powering the Cortex compiler. Converts PDF, Office documents, HTML, images, and more into structured Markdown for ingestion into the vault.

License

This project is licensed under the PULSE8.ai Cortex Open Source License (Apache License 2.0 with additional terms).