GitHub - zoharbabin/web-researcher-mcp: MCP server (Go) for AI assistants: web search, content extraction, academic/patent/news research. Multi-provider routing, 4-tier scraping, search lenses. Works with Claude, Cursor, and any MCP client.

A production-grade MCP server that gives AI assistants the power to search the web, extract content, and conduct multi-source research.

Why Web Researcher MCP?

AI assistants are only as good as the information they can access. web-researcher-mcp bridges the gap between LLMs and the live internet through the Model Context Protocol standard:

Multiple specialized research tools in a single server (see Tools below)
Pluggable search backends with multi-provider routing and automatic fallback (Google, Brave, Serper, SearXNG, SearchAPI.io)
4-tier content extraction -- markdown negotiation, stealth HTTP, HTML parsing, headless browser (go-rod + stealth)
Search lenses for domain-focused research (programming, news, legal, medical, and more)
Single static binary with optional Chromium for JS rendering (auto-downloaded on first use)
Enterprise-ready with OAuth 2.1, multi-tenancy, rate limiting, and audit logging

Works with Claude Code, Claude Desktop, Cursor, and any MCP-compatible client.

Tools

Tool	Description
`web_search`	General web search with optional search lenses for domain-focused results
`scrape_page`	Extract content from any URL -- web pages, PDFs, DOCX, PPTX, YouTube transcripts (3-strategy fallback)
`search_and_scrape`	Combined search + extraction pipeline with quality scoring and deduplication
`image_search`	Search for images with size, type, color, and file format filters
`news_search`	Search news sources with freshness controls and source filtering
`academic_search`	Search peer-reviewed papers across arXiv, PubMed, IEEE, Nature, Springer, and other scholarly databases
`patent_search`	Search patent databases with CPC classification, strict office filtering (US/EP/WO/JP/CN/KR)
`sequential_search`	Multi-step research tracking with session state for iterative investigation

Quick Start

Option 1: Install with Go (Recommended)

go install github.com/zoharbabin/web-researcher-mcp/cmd/web-researcher-mcp@latest

The binary is now in your $GOPATH/bin. Add to Claude Code:

claude mcp add --scope user --transport stdio web-researcher -- web-researcher-mcp

Option 2: Download Binary

Download the latest release for your platform from Releases. Archives are named web-researcher-mcp_<version>_<os>_<arch>.tar.gz.

# Example: macOS Apple Silicon (replace VERSION with the latest from Releases)
curl -L https://github.com/zoharbabin/web-researcher-mcp/releases/download/v${VERSION}/web-researcher-mcp_${VERSION}_darwin_arm64.tar.gz | tar xz
chmod +x web-researcher-mcp

Option 3: Docker

docker run -e GOOGLE_CUSTOM_SEARCH_API_KEY=YOUR_KEY \
           -e GOOGLE_CUSTOM_SEARCH_ID=YOUR_CX \
           docker.io/zoharbabin/web-researcher-mcp:latest

Also available from GHCR: ghcr.io/zoharbabin/web-researcher-mcp:latest

Option 4: Build from Source

git clone https://github.com/zoharbabin/web-researcher-mcp.git
cd web-researcher-mcp
go build -o web-researcher-mcp ./cmd/web-researcher-mcp

Connect to Your AI Assistant

Add this to your MCP client configuration (example for Claude Code ~/.claude/settings.json):

{
  "mcpServers": {
    "web-researcher": {
      "command": "/path/to/web-researcher-mcp",
      "env": {
        "GOOGLE_CUSTOM_SEARCH_API_KEY": "YOUR_GOOGLE_API_KEY",
        "GOOGLE_CUSTOM_SEARCH_ID": "YOUR_SEARCH_ENGINE_ID"
      }
    }
  }
}

Or with Brave Search (no Google keys needed):

{
  "mcpServers": {
    "web-researcher": {
      "command": "/path/to/web-researcher-mcp",
      "env": {
        "SEARCH_ROUTING": "brave",
        "BRAVE_API_KEY": "YOUR_BRAVE_API_KEY"
      }
    }
  }
}

Done. Your AI assistant now has access to all research tools.

Configuration

Required (unless using `SEARCH_ROUTING`)

Variable	Description	How to Get
`GOOGLE_CUSTOM_SEARCH_API_KEY`	Google API key	Google Cloud Console
`GOOGLE_CUSTOM_SEARCH_ID`	Programmable Search Engine ID	PSE Console

Note: When SEARCH_ROUTING is set with non-Google providers (e.g., brave,serper), Google keys are not required. You only need credentials for the providers you configure.

Search Provider

Variable	Description	Default
`SEARCH_PROVIDER`	Backend: `google`, `brave`, `serper`, `searxng`, or `searchapi`	`google`
`BRAVE_API_KEY`	Brave Search API key
`SERPER_API_KEY`	Serper.dev API key
`SEARCHAPI_API_KEY`	SearchAPI.io API key
`SEARXNG_URL`	SearXNG instance URL
`SEARCH_ROUTING`	Multi-provider routing with automatic fallback (see Deployment docs)

HTTP Transport (Optional)

Variable	Description	Default
`PORT`	Enable HTTP/SSE mode	STDIO only
`OAUTH_ISSUER_URL`	JWT issuer URL for token validation
`OAUTH_AUDIENCE`	Expected JWT audience claim

All Environment Variables

See docs/DEPLOYMENT.md for the complete reference of all environment variables (cache, rate limiting, scraping, observability, etc.).

Architecture

web-researcher-mcp/
├── cmd/web-researcher-mcp/     # Entry point (wiring only)
├── internal/
│   ├── config/                 # Env-based strongly-typed configuration
│   ├── server/                 # MCP server lifecycle + signal handling
│   ├── tools/                  # Tool handlers (one file per tool)
│   ├── search/                 # Pluggable search providers + router + lens routing
│   ├── scraper/                # 4-tier scraping pipeline (markdown → stealth → HTML → browser)
│   ├── documents/              # PDF, DOCX, PPTX parsing
│   ├── cache/                  # Hybrid cache (memory + disk + optional Redis)
│   ├── auth/                   # OAuth 2.1 middleware + JWKS
│   ├── audit/                  # Structured audit logging
│   ├── session/                # Per-tenant session management
│   ├── content/                # Sanitize, dedup, truncate, quality score
│   ├── metrics/                # Prometheus metrics + per-tool stats
│   ├── ratelimit/              # Three-tier rate limiting
│   ├── circuit/                # Circuit breaker for external APIs
│   └── resources/              # MCP Resources + Prompts
├── lenses/                     # Search lens JSON files
└── docs/                       # Extended documentation

High-Level Architecture Diagram

┌─────────────────────────────────────────────────────────────────┐
│                         MCP Protocol Layer                        │
│  ┌──────────────────┐              ┌─────────────────────────┐  │
│  │  STDIO Transport │              │  HTTP/SSE Transport     │  │
│  │  (zero-config)   │              │  (OAuth 2.1 + CORS)     │  │
│  └────────┬─────────┘              └──────────┬──────────────┘  │
│           └────────────────┬───────────────────┘                 │
│                    ┌───────▼───────┐                             │
│                    │  MCP Server   │                             │
│                    │  (go-sdk)     │                             │
│                    └───────┬───────┘                             │
└────────────────────────────┼─────────────────────────────────────┘
                             │
┌────────────────────────────┼─────────────────────────────────────┐
│                    Tool Dispatch Layer                             │
│  ┌─────────┐ ┌────────┐ ┌┴───────┐ ┌────────┐ ┌─────────────┐  │
│  │ Search  │ │ Scrape │ │Combined│ │Academic│ │ Sequential  │  │
│  │ Tools   │ │ Tool   │ │  Tool  │ │& Patent│ │  Research   │  │
│  └────┬────┘ └───┬────┘ └───┬────┘ └───┬────┘ └──────┬──────┘  │
└───────┼──────────┼───────────┼──────────┼─────────────┼──────────┘
        │          │           │          │             │
┌───────┼──────────┼───────────┼──────────┼─────────────┼──────────┐
│       │     Service Layer    │          │             │           │
│  ┌────▼────┐ ┌───▼────┐ ┌───▼───┐ ┌───▼────┐ ┌─────▼─────┐   │
│  │ Search  │ │Scraper │ │Quality│ │Citation│ │  Session   │   │
│  │Provider │ │Pipeline│ │Scorer │ │Extract │ │  Manager   │   │
│  └────┬────┘ └───┬────┘ └───────┘ └────────┘ └────────────┘   │
│       │          │                                               │
│  ┌────▼─────┐ ┌─▼──────────────────────────────────┐           │
│  │ Router   │ │  Scraper Tiers (4-tier pipeline)     │           │
│  │(fallback)│ │  markdown > stealth > HTML > browser│           │
│  │  Brave   │ │  + YouTube (3-strategy) + documents │           │
│  │  Google  │ └─────────────────────────────────────┘           │
│  │  Serper  │                                                    │
│  │  SearXNG │                                                    │
│  │SearchAPI │                                                    │
│  └──────────┘                                                    │
└──────────────────────────────────────────────────────────────────┘
        │          │
┌───────┼──────────┼──────────────────────────────────────────────┐
│       │   Infrastructure Layer                                    │
│  ┌────▼────┐ ┌───▼────┐ ┌─────────┐ ┌────────┐ ┌───────────┐  │
│  │  Cache  │ │  SSRF  │ │  Rate   │ │Metrics │ │   Audit   │  │
│  │(hybrid) │ │Protect │ │ Limiter │ │(Prom.) │ │   Logger  │  │
│  └─────────┘ └────────┘ └─────────┘ └────────┘ └───────────┘  │
│  ┌──────────────────┐  ┌──────────────────────────────────────┐ │
│  │  Circuit Breaker  │  │  Content Pipeline (sanitize, dedup,  │ │
│  │                   │  │  truncate, quality score)             │ │
│  └───────────────────┘  └──────────────────────────────────────┘ │
└──────────────────────────────────────────────────────────────────┘

Design Principles

Zero global state -- all dependencies injected via constructors
Interface-driven -- every external dependency behind an interface for testing and swapping
Bounded concurrency -- explicit semaphores for external API calls
Defense in depth -- SSRF protection, rate limiting, content sanitization at every layer
Fail loud -- errors returned, never swallowed; validation at boundaries

Search Providers

All providers implement the same interface and support lenses via site: operator injection.

Provider	Whole-Web	Images	News	Notes
Google PSE	Yes	Yes	Yes	Default; free tier: 100 queries/day
Brave Search	Yes	Yes	Yes	Recommended for high-volume whole-web
Serper.dev	Yes	Yes	Yes	Google-identical results
SearXNG	Yes	Yes	Yes	Self-hosted, privacy-first, air-gapped deployments
SearchAPI.io	Yes	Yes	Yes	Unified API with multiple engine backends

Multi-Provider Routing

When SEARCH_ROUTING is set, the server uses multiple providers with automatic fallback:

# Priority-ordered list — requests go to the first healthy provider
export SEARCH_ROUTING=brave,google,serper

# Per-operation routing (JSON) — different priorities for different search types
export SEARCH_ROUTING='{"web":"brave,google","news":"brave,serper","images":"google,brave","default":"brave,google,searchapi"}'

Each provider gets an independent circuit breaker. If a provider fails (timeout, rate limit, 5xx), the next provider in the priority list is tried automatically. Lenses can override routing via the routing field in their JSON definition.

See docs/DEPLOYMENT.md for full routing documentation.

Single-Provider Routing

When SEARCH_ROUTING is not set, the server uses a single provider:

Request arrives
  |-- lens with dedicated CX? --> That specific Google PSE engine
  |-- lens (no CX)?           --> Configured provider + site: operators
  |-- site: param set?        --> Configured provider + site: operator
  `-- unrestricted?           --> Configured SEARCH_PROVIDER

Provider Setup Examples

Multi-provider routing (recommended):

export SEARCH_ROUTING=brave,google,serper
export BRAVE_API_KEY=BSAxxxxxxxxxx
export GOOGLE_CUSTOM_SEARCH_API_KEY=AIza...
export GOOGLE_CUSTOM_SEARCH_ID=017...
export SERPER_API_KEY=...

Single provider — Brave Search:

export SEARCH_PROVIDER=brave
export BRAVE_API_KEY=BSAxxxxxxxxxx

Single provider — SearXNG (self-hosted, privacy-first):

export SEARCH_PROVIDER=searxng
export SEARXNG_URL=http://localhost:8080

Single provider — Google PSE only (simplest setup):

export GOOGLE_CUSTOM_SEARCH_API_KEY=AIza...
export GOOGLE_CUSTOM_SEARCH_ID=017...
# SEARCH_PROVIDER defaults to "google"

Search Lenses

Search lenses are curated domain lists that focus search results on high-quality sources for specific topics. They inject site: operators into queries and route through your configured search provider.

Built-in Lenses

Lens	Focus
`programming`	Code docs, tutorials, Q&A
`news`	Current events, journalism
`tech`	Technology industry
`legal`	Law, cases, statutes
`medical`	Health, medicine
`finance`	Markets, filings
`science`	Research, papers
`government`	Policy, regulations

Each lens is a JSON file in lenses/ containing the curated domain list. See Creating Custom Lenses below for the format.

Usage Example

{
  "tool": "web_search",
  "arguments": {
    "query": "golang context best practices",
    "lens": "programming"
  }
}

This searches only stackoverflow.com, github.com, go.dev, developer.mozilla.org, and other curated programming sites.

Creating Custom Lenses

Add a JSON file to the lenses/ directory:

{
  "name": "my-custom-lens",
  "description": "Description of what this lens covers",
  "domains": [
    "example.com",
    "docs.example.org",
    "*.trusted-source.io"
  ],
  "cx": "",
  "routing": ""
}

Fields:

domains -- URL patterns for the lens (up to ~10 injected per query via site: operators)
cx -- Optional dedicated Google PSE engine ID. If set, bypasses site injection and routes directly to that PSE engine (supports up to 5,000 domains)
routing -- Optional provider override for this lens (e.g., "google" or "searchapi,google"). When set, this lens routes through the specified provider(s) regardless of global routing config

Security

SSRF Protection

The server implements a custom DialContext that validates all resolved IPs before connecting:

Blocks all private/reserved IP ranges (127.0.0.0/8, 10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16, fc00::/7)
Blocks cloud metadata endpoints (169.254.169.254)
Validates against DNS rebinding by connecting only to the first resolved IP
Re-validates redirect targets at each hop

Authentication and Authorization

In HTTP mode, the server supports OAuth 2.1 with:

JWKS-based token validation with automatic key rotation
Per-tenant session isolation
Audience and issuer validation
Configurable claim extraction for multi-tenancy

Rate Limiting

Three-tier rate limiting protects both the server and upstream APIs:

Per-client -- token bucket per authenticated session
Per-provider -- prevents exceeding upstream API quotas
Global -- server-wide backpressure valve

Content Safety

HTML sanitization via whitelist-based policy (bluemonday)
Paragraph-level deduplication across scraped results
Smart truncation at natural content breakpoints
Quality scoring to filter low-value results before returning to the LLM

For the full threat model and security architecture, see docs/SECURITY.md.

MCP Client Integration

Claude Code

Add to ~/.claude/settings.json:

{
  "mcpServers": {
    "web-researcher": {
      "command": "/path/to/web-researcher-mcp",
      "env": {
        "GOOGLE_CUSTOM_SEARCH_API_KEY": "AIza...",
        "GOOGLE_CUSTOM_SEARCH_ID": "017...",
        "SEARCH_PROVIDER": "brave",
        "BRAVE_API_KEY": "BSA..."
      }
    }
  }
}

Claude Desktop

Add to ~/Library/Application Support/Claude/claude_desktop_config.json (macOS) or %APPDATA%\Claude\claude_desktop_config.json (Windows):

{
  "mcpServers": {
    "web-researcher": {
      "command": "/path/to/web-researcher-mcp",
      "env": {
        "GOOGLE_CUSTOM_SEARCH_API_KEY": "AIza...",
        "GOOGLE_CUSTOM_SEARCH_ID": "017..."
      }
    }
  }
}

Cursor

Add to .cursor/mcp.json in your project root:

{
  "mcpServers": {
    "web-researcher": {
      "command": "/path/to/web-researcher-mcp",
      "env": {
        "GOOGLE_CUSTOM_SEARCH_API_KEY": "AIza...",
        "GOOGLE_CUSTOM_SEARCH_ID": "017..."
      }
    }
  }
}

HTTP/SSE Mode (Multi-Client, Teams)

For shared deployments serving multiple clients or web applications:

PORT=3000 \
OAUTH_ISSUER_URL=https://auth.example.com \
OAUTH_AUDIENCE=https://api.example.com \
./web-researcher-mcp

Connect any MCP client to http://localhost:3000/mcp/ (Streamable HTTP transport).

Docker Compose Example

services:
  web-researcher:
    image: zoharbabin/web-researcher-mcp
    ports:
      - "3000:3000"
    environment:
      PORT: "3000"
      GOOGLE_CUSTOM_SEARCH_API_KEY: ${GOOGLE_CUSTOM_SEARCH_API_KEY}
      GOOGLE_CUSTOM_SEARCH_ID: ${GOOGLE_CUSTOM_SEARCH_ID}
      SEARCH_PROVIDER: brave
      BRAVE_API_KEY: ${BRAVE_API_KEY}
      REDIS_URL: redis://redis:6379
    depends_on:
      - redis

  redis:
    image: redis:7-alpine
    volumes:
      - redis-data:/data

volumes:
  redis-data:

Performance

Search results are cached in-memory for sub-millisecond hits. The scraping pipeline tries the fastest tier first and falls back progressively — most pages resolve in under a second via stealth HTTP, with the headless browser reserved for JS-heavy sites. See ARCHITECTURE.md for detailed latency breakdowns.

Development

go build -o web-researcher-mcp ./cmd/web-researcher-mcp   # Build
go test -race ./...                                        # Test (with race detector)
golangci-lint run                                          # Lint
govulncheck ./...                                          # Security audit

See CONTRIBUTING.md for the full development workflow, code style guide, and PR process.

Troubleshooting

Server starts but tools fail with "API key" errors

The server starts even with missing credentials (to allow MCP handshake). Set your API keys in the env block of your MCP client config, not in your shell profile.

scrape_page returns empty content for JavaScript-heavy sites

The browser tier (go-rod) requires Chromium. On first use it auto-downloads ~200MB. Set CHROME_PATH to use an existing Chrome installation, or use the Docker image which includes headless Chrome.

Cache serving stale results after upgrade

The disk cache auto-invalidates on version change. If you're running from source without -ldflags, the version is always "dev" — delete the ./cache directory manually or set CACHE_DIR to a versioned path.

Rate limited by Google (HTTP 429)

Google PSE free tier allows 100 queries/day. Either upgrade to paid ($5/1K queries), or switch to Brave Search (SEARCH_PROVIDER=brave) for higher-volume queries.

Contributing

Contributions are welcome. Please see CONTRIBUTING.md for code style guidelines, development workflow, and how to submit pull requests.

Documentation

Document	Description
ARCHITECTURE.md	Design decisions, technology stack, dependencies
CONTRIBUTING.md	Development setup, code style, PR workflow
docs/TOOLS.md	Tool specifications and parameter schemas
docs/EXAMPLES.md	Usage examples with JSON tool calls
docs/API_SETUP.md	Search provider API key setup for all providers
docs/SECURITY.md	Threat model, SSRF, auth, compliance (SOC2/GDPR/FedRAMP)
docs/DEPLOYMENT.md	Build, Docker, Kubernetes, client configs, scaling
docs/LESSONS_LEARNED.md	Node.js to Go migration story and lessons
docs/MIGRATION.md	Migrating from the deprecated google-researcher-mcp

License

MIT

推荐订阅源

Show HN

Why Web Researcher MCP?

Tools

Quick Start

Option 1: Install with Go (Recommended)

Option 2: Download Binary

Option 3: Docker

Option 4: Build from Source

Connect to Your AI Assistant

Configuration

Required (unless using `SEARCH_ROUTING`)

Search Provider

HTTP Transport (Optional)

Architecture

Design Principles

Search Providers

Multi-Provider Routing

Single-Provider Routing

Search Lenses

Built-in Lenses

Usage Example

Security

MCP Client Integration

Claude Code

Claude Desktop

Cursor

HTTP/SSE Mode (Multi-Client, Teams)

Performance

Development

Troubleshooting

Contributing

Documentation

License

Star History

推荐订阅源

Show HN

Why Web Researcher MCP?

Tools

Quick Start

Option 1: Install with Go (Recommended)

Option 2: Download Binary

Option 3: Docker

Option 4: Build from Source

Connect to Your AI Assistant

Configuration

Required (unless using SEARCH_ROUTING)

Search Provider

HTTP Transport (Optional)

Architecture

Design Principles

Search Providers

Multi-Provider Routing

Single-Provider Routing

Search Lenses

Built-in Lenses

Usage Example

Security

MCP Client Integration

Claude Code

Claude Desktop

Cursor

HTTP/SSE Mode (Multi-Client, Teams)

Performance

Development

Troubleshooting

Contributing

Documentation

License

Star History

Required (unless using `SEARCH_ROUTING`)