TL;DR: We ran npx -y <package> against 922 npm-published MCP servers, sent them the JSON-RPC initialize and tools/list calls, and captured what they did. 359 responded. 563 failed in 15 distinct ways that say more about npm packaging than about MCP itself.

The stderr signature that broke 261 servers

[stderr] connecting to upstream...
[stderr] (no further output)
[timeout after 120s]

That is the single most common failure mode in the npm MCP ecosystem right now. The process spawns, the package loads, the constructor runs, and then the server tries to phone home to its upstream API before answering the protocol. The introspection runner waits 120 seconds and gives up. Two hundred sixty-one servers - 28% of everything published - never made it past their own startup.

If you maintain an MCP server and you connect to anything external during initialize, you are in this bucket. The fix is to defer the upstream connection until the first tool call.

What we actually ran

The protocol gives you a discovery method: tools/list. Send it after initialize and the server returns the full JSON Schema for every tool it exposes. No README scraping, no LLM interpretation, no guessing. It is the exact same thing your MCP client does when it connects.

The runner does this per package:

1. spawn:  npx -y <package> (over stdio)
2. write:  {"jsonrpc":"2.0","id":1,"method":"initialize",
            "params":{"protocolVersion":"2024-11-05",
                      "capabilities":{},
                      "clientInfo":{"name":"introspector","version":"1.0"}}}
3. read:   initialize response
4. write:  {"jsonrpc":"2.0","method":"notifications/initialized"}
5. write:  {"jsonrpc":"2.0","id":2,"method":"tools/list"}
6. read:   tool list
7. kill:   close stdin, SIGTERM

Concurrency was 8 in parallel, 120-second timeout per server, total wall time around 25 minutes for 922 packages. The output is one JSONL row per server with status, tool count, and (where available) the full input schema for every tool.

The 15 failure buckets

Every server that didn't return a clean tool array was classified by inspecting the exit code, stderr, and the JSON-RPC error if there was one. The breakdown:

The credential-wall buckets (everything needs_*) add up to 109 servers, almost 12% of the published set. If you are populating an agent's tool list by parsing READMEs, those servers register as 0-tool servers in your index. They are not 0-tool servers. They are 5-tool, 12-tool, 40-tool servers waiting for a key you didn't supply.

Windows-specific gotchas

The runner is on Windows. Two things bite hard.

Spawning npx. npx on Windows resolves to npx.cmd, a batch script. Node's child_process.spawn without a shell will not invoke a .cmd. The result is a flat ENOENT even though where npx shows it on PATH. The fix:

const child = spawn("cmd", ["/c", "npx", "-y", pkg], {
  stdio: ["pipe", "pipe", "pipe"],
});

You see this same pattern in every claude_desktop_config.json on Windows. If you are writing your own MCP client, you need it too.

UTF-8 stdio. The default code page on Windows is not UTF-8. If an MCP server writes a tool description that contains a non-ASCII character (German umlaut, em-dash, curly quote, anything), and you read its stdio without forcing UTF-8 decoding, you get a JSON parse error halfway through tools/list. Force it:

proc = subprocess.Popen(
    cmd,
    stdin=subprocess.PIPE,
    stdout=subprocess.PIPE,
    stderr=subprocess.PIPE,
    encoding="utf-8",
    errors="replace",
)

Three servers in our run had tool descriptions with non-ASCII characters. Without the encoding flag, all three would have been miscounted as error.

What this says to MCP server authors

A few patterns from the data, addressed to anyone shipping a server:

1. Don't connect upstream during initialize. If your server contacts an API on startup, it will fail introspection and it will fail any client that wants to enumerate tools without first burning a credential. Lazy-connect on the first tool call.

2. Don't require CLI args to print your tool list. 54 servers exit with a usage error before they will even tell you what they expose. If you need a config path, take it from an env var or accept tools/list without it.

3. Document the env vars in the README. The classifier successfully named the credential for the servers that wrote a clear "missing X" line to stderr. The ones that didn't ended up in the generic needs_env_var bucket. That bucket is your bug report queue.

4. Ship a real bin entry. 11 servers in broken_install had package.json that pointed at a file that doesn't exist, or a main field that crashed on require. None of them needed credentials. They needed a publish-time smoke test.

The full dataset

The 9,922 tool schemas and the 922 server status rows are on HuggingFace as automatelab/mcp-servers-tool-catalog under CC-BY-4.0. Pipeline source is at AutomateLab-tech/mcp-tool-catalog, and it re-runs on the 1st of every month via GitHub Actions.
Product page: https://automatelab.tech/products/datasets/mcp-tool-catalog/

from datasets import load_dataset

servers = load_dataset(
    "automatelab/mcp-servers-tool-catalog", "servers", split="train"
)

# Just the unreachable ones, grouped by failure mode
from collections import Counter
print(Counter(r["status"] for r in servers if r["status"] != "ok"))

If your server is in a failure bucket and shouldn't be, open a PR on the pipeline repo. The next monthly run picks it up.