A Cluster Stall Looks Healthy on Every Host. The Cause Is in the Pattern Across Hosts.

Eight ranks on two hosts. Every per-host metric reads healthy. Rank 5 enters the barrier 290ms late. The cause lives in a cross-rank query, not in any single host’s trace.

Watch the demo (90 seconds): Echo starts on :4317, two echo-stress invocations push 2,000 events combined (node-a100 + node-h100), then DuckDB verifies they all landed and the causal-chain markers survived the wire.

TL;DR

Eight ranks on two hosts run an all-reduce. Token throughput drops 4x. Every per-host nvidia-smi reads 95-99% utilization. Every per-host eBPF trace looks clean. The cause is rank 5 on node B taking 380ms on a step that the other seven ranks finish in 90ms. The other seven ranks spend 290ms blocked inside ncclAllReduce, which counts as a running kernel and reports as healthy on every per-host metric. The diagnosis lives in a cross-rank query, not in any single host’s trace. We shipped Ingero Echo (a cluster-wide AI-investigation tool that auto-collects OTLP from every node and exposes it as MCP-over-DuckDB) to make those queries answerable for AI agents directly. This post walks through the proof: 2,000 events from two nodes, fan-in into one DuckDB, queries that surface the straggler.

What per-host tracing cannot see

A typical 8-GPU all-reduce on H100s runs at ~80GB/s ring bandwidth. The synchronization barrier is ncclAllReduce. Every rank enters at roughly the same wall-clock; the rank that finishes last sets the wall time for all eight. When one rank is slow, the other seven do not idle visibly. They sit inside ncclAllReduce, which is itself a CUDA kernel. nvidia-smi sees a kernel running. DCGM’s SM_ACTIVE ticks. Per-host eBPF sees cudaLaunchKernel -> ncclAllReduce -> cudaStreamSynchronize complete normally. The local trace is clean.

What each rank does NOT see:

That the seven other ranks are also blocked inside the same ncclAllReduce for the same window.
That one specific rank entered the barrier 290ms later than the others.
That this same rank-5 pattern repeats every 14 steps, hinting at a memory-fragmentation cycle on the slow host.

These are facts about the cluster, not facts about any host. A monitoring stack that ships per-host samples to a centralized dashboard can render them as time series, but a time-series view does not answer “which rank’s call stack caused the 290ms wait the other ranks observed?” That is a relational query across host boundaries.

The cluster-level question

The question that matters during a stall is:

For each step where end-to-end throughput dropped, which rank’s ncclAllReduce started later than its peers, and what was that rank doing in the previous 500ms?

To answer it, you need:

Every rank’s ncclAllReduce enter/exit events, timestamped with a clock that is consistent across hosts.
Every rank’s CUDA call stack and off-CPU events for the 500ms before each ncclAllReduce enter.
A causal chain identifier that links a single training step’s events across all ranks.
A single store you can SQL.

Ingero’s per-host agent already produces those events. The agent attaches uprobes to libcudart.so and libnccl.so, captures kernel-scheduler events with eBPF, and emits OTLP. The missing piece in v0.12.4 was the cluster-level destination: a place where every host’s stream could fan in, where the events keep their resource attributes (cluster ID, node ID, rank, nranks), and where SQL can join across ranks. v0.12.5 ships that piece.

Echo, in one paragraph

Ingero Echo is a cluster-wide AI-investigation tool for GPU observability. It auto-collects OTLP/gRPC streams from every Fleet collector in the cluster into embedded DuckDB (one writer, single-statement read-only SQL), then exposes that data through a Model Context Protocol server with four tools: fleet.cluster.event_history, fleet.cluster.find_outlier_nodes, fleet.cluster.run_analysis (SQL-only, gated by a lexical guard), and fleet.cluster.get_cost. AI agents (Claude, Cursor, ollmcp, any MCP client) drive cross-rank investigations through this MCP-over-DuckDB surface without ever touching the database directly. Echo ships as a single-binary StatefulSet behind a ClusterIP service. The event-store path holds a flock(2) so a rolling-update force-detach does not corrupt the WAL. The receiver enforces bearer-token auth on the OTLP listener. The image is 87MB. One Echo per cluster.

The fan-in correctness proof

Before shipping Echo as a single source of truth for cluster-level queries, we wanted a hard answer to one question:

When N agents push concurrently from N hosts, do all events land, do causal-chain identifiers survive, and can a SQL query distinguish the straggler from the healthy ranks?

The proof is cmd/ingero-echo/integration_test.go plus a hardware run on Lambda Cloud. Eight concurrent producers, 250 events each, OTLP/gRPC into a DuckDB-backed Echo. Mixed across two cluster IDs. Causal-chain markers injected every 25th event. Stragglers (synthetic low-health-score events) injected every 100th. Total: 2,000 events.

The test asserts:

All events land: SELECT COUNT(*) FROM events == 2000.
Per-rank counts are correct: each of the 8 producer node IDs has exactly 250 events.
Causal chains preserved across the wire: the 80 distinct causal_chain_id markers we inserted at the producer come back from the store. None lost. None merged.
Stragglers surface in a cross-cluster query: SELECT node_id, COUNT(*) FROM events WHERE value_double < 0.4 GROUP BY node_id returns the 18 events we injected and only those.
Burst write does not lose events under contention: a separate test runs 5,000 events at ~1k EPS through WriteEvents from 8 goroutines simultaneously, all serialized via Echo’s writer mutex. Zero loss; throughput floor met.

The integration test (TestEchoFanIn_AllEventsLand) runs in under 21 seconds on a CI runner. The hardware run on the populated DB is reproducible from the artifacts attached to this post.

The hardware run

We provisioned an A100 (40GB SXM4) on Lambda us-east-1 and an H100 (80GB SXM5) on Lambda us-south-3 to play the role of two GPU nodes. Echo ran on the A100. A simple OTLP/gRPC stress client (cmd/echo-stress/) pushed 1,000 events from each node into Echo. Cross-region OTLP from H100 to A100 was blocked by Lambda’s default firewall, so the H100 stream was simulated from the A100 host with --node-id=node-h100. Echo’s fan-in path treats both streams identically; the test exercises the same RPC handler, the same writer mutex, and the same DuckDB schema. The DB attached to this post (echo-fanin-demo.db, 1.0 MiB) is the result.

The exact commands are in commands.md next to this post. The runbook starts Echo with a bearer token, runs two echo-stress invocations with different node IDs, and validates with three DuckDB queries.

The queries that matter

These are the SQL queries that produced the assertions above. All four run against the attached DB.

Per-node event count.

SELECT cluster_id, node_id, COUNT(*) AS events
FROM events
GROUP BY cluster_id, node_id
ORDER BY node_id;

demo-cluster | node-a100 | 1000
demo-cluster | node-h100 | 1000

Causal chains preserved.

SELECT COUNT(DISTINCT json_extract_string(attrs_json, '$.causal_chain_id')) AS chains
FROM events
WHERE attrs_json LIKE '%causal_chain_id%';

Stragglers per node.

SELECT node_id, COUNT(*) AS straggler_events
FROM events
WHERE value_double < 0.40
GROUP BY node_id;

node-a100 | 9
node-h100 | 9

Median and p95 health by node.

SELECT node_id,
       quantile_cont(value_double, 0.5)  AS median_health,
       quantile_cont(value_double, 0.95) AS p95_health
FROM events
GROUP BY node_id;

The same shape of query is what an agent calls through the MCP run_analysis tool. The lexical SQL gate (sqlguard) rejects any query that touches the filesystem (read_csv, read_parquet, the READ_* family, the SNIFF_* family, bare-quoted FROM table sources, the httpfs/s3/gcs schemes) and any query that introspects DuckDB’s own catalog (duckdb_settings, duckdb_tables, the duckdb_* family). The guard runs once at the MCP boundary and once again inside the store, so the gate cannot drift between layers.

What this changes for AI agents

The MCP server is the part that matters for agents. An agent investigating “throughput dropped at 14:32 UTC, every rank reports healthy, why” can now ask Echo directly:

fleet.cluster.find_outlier_nodes(window="14:30-14:35", metric="ingero.health.score", threshold=0.4)

and receive back the ranked node list. It can then ask:

fleet.cluster.event_history(cluster_id="...", node_id="<outlier>", window="14:30-14:35", limit=1000)

to pull the call stack. It can finally call:

fleet.cluster.run_analysis(sql="SELECT ... causal_chain_id ... GROUP BY node_id ...")

to pivot the data into the shape that answers the question. The agent never sees the populated DB directly. It sees four tools, each returning JSON-shaped responses bounded by the lexical guard.

This is the gap that per-host MCP servers cannot close. A per-host MCP server can answer “what did the agent do on this host?” but it cannot answer “what was the cluster doing when the agent observed the spike?” Cross-rank causal questions need a cross-rank AI-investigation surface. Echo is that surface.

Try it locally

Two paths, depending on whether you want to run the demo end-to-end or just inspect the recorded output.

Reproduce the fan-in scenario from scratch. The integration test in cmd/ingero-echo/integration_test.go spins up Echo backed by a fresh DuckDB in a per-test temp directory, fans in 8 concurrent agents pushing 250 events each (2,000 events total), and asserts that all events landed, the planted outlier surfaces in the MCP query, and causal-chain events are preserved with all attributes. Each invocation produces its own DB.

git clone https://github.com/ingero-io/ingero-fleet.git
cd ingero-fleet/cmd/ingero-echo
go test -run TestEchoFanIn_AllEventsLand ./...

The test takes under 10 seconds on a developer laptop. Requirement: a Go toolchain plus DuckDB’s CGO build dependencies (libstdc++).

To inspect the populated DB after the test runs, set ECHO_BLOG_ARTIFACT=1 in the environment and the test will copy the final DB to /tmp/echo-fanin-demo.db. Then:

ECHO_BLOG_ARTIFACT=1 go test -run TestEchoFanIn_AllEventsLand ./...
duckdb /tmp/echo-fanin-demo.db

Run any of the queries from “The queries that matter” section above against this freshly captured DB; the schema is identical, only the random event IDs differ.

Inspect the published demo DB without running anything. The same DB referenced earlier in this post is published in the public Fleet repo. 2,000 events, 2 clusters, 80 causal chains preserved across the wire, 18 stragglers detected end-to-end.

curl -fsSL -o echo-fanin-demo.db \
  https://github.com/ingero-io/ingero-fleet/raw/main/investigations/echo-fanin-demo.db

duckdb echo-fanin-demo.db

Open it and run the same queries from “The queries that matter” section above. The Echo schema is documented in cmd/ingero-echo/store/schema.go: one row per OTLP data point, dedicated columns for cluster_id / node_id / metric_name / rank / nranks / value_double / value_int, and an attrs VARCHAR holding the rest as JSON. Two indexes target the most-used filters ((cluster_id, timestamp_ns) and (node_id, timestamp_ns)).

The two paths are independent: the test reproduction does not read the published DB, and the published DB does not require the test to be run. Both demonstrate the same Echo store schema, so a query that works on one works on the other.

For the hardware-shaped run (two real hosts, OTLP over the wire), the runbook in commands.md next to this post’s draft is the script. Total wall time is under five minutes; total cost on Lambda was about $0.50 for the A100 alone.

Across hosts, not on hosts

A per-host trace can be perfectly correct and still useless. The pattern across hosts is what carries the cause. Eight ranks blocked inside ncclAllReduce looks identical to eight ranks running healthy work; the only thing that distinguishes the two is whether one rank entered late. That fact lives in a join, not in a single host’s events.

Echo’s job is to be the AI-investigation surface where the join can run, with the same OTLP semantic conventions on the wire, the same DuckDB schema underneath, and the same MCP shape that agents are already learning to use. The fan-in correctness proof is the gate before the rest of the work depends on the store. With v0.12.5 it is shipped, tested, and reproducible.

Ingero – open-source eBPF agent for GPU debugging. One binary, zero deps, <2% overhead. Apache 2.0 + GPL-2.0. *GitHub ⭐** · Open an issue if you are running distributed training or inference and seeing throughput drop while every rank reads healthy.

Investigation DB: investigations/echo-fanin-demo.db*

推荐订阅源

DEV Community