Here is the detailed, deep-dive article tailored for DEV.to, written in a natural, highly technical style, completely free of icons, and designed to resonate with developers building agentic workflows.
Building an Autonomous AI Experience Engine: Taming the Multi-Agent CLI Fleet
As developers integrate more AI tools into their workflows, a new architectural problem has emerged: agent sprawl. We have incredible tools like Claude, Grok, and Codex running in our terminals, but they operate in silos. They lack shared memory, they step on each other's toes, and coordinating them feels like herding cats.
To solve this, I built TechSphereX Studio — an open-source, polyglot AI Experience Engine. It is an autonomous multi-agent platform designed to intercept AI coding actions, orchestrate goal-driven work across a fleet of CLI agents, and mathematically learn from every session to improve future outcomes.
Here is a deep dive into how I moved from isolated prompt engineering to a fully automated, self-learning agentic brain.
Key Architectural Pillars
1. The 3-Layer Intercept Pipeline
Before any CLI executes a command, TechSphereX intercepts the action to determine if the system already knows how to solve the problem based on past experiences. This happens across three highly optimized layers:
- Layer 1 (Read-only Filter): Evaluates the action in under 1ms. If the action is non-destructive (like a simple read), it skips heavy processing to save resources.
- Layer 2 (Semantic Search): Uses Qdrant running locally to perform vector embeddings and search the system's history for similar past tasks in under 50ms.
- Layer 3 (LLM Rerank): Passes the semantic results to a local Ollama instance to filter out contextually irrelevant data in under 500ms, ensuring the execution agent only receives high-fidelity context.
2. The Agentic Brain & Multi-Role Teams
Instead of throwing a massive, complex prompt at a single coding agent, TechSphereX mimics a multi-role engineering team. The pipeline starts with a BA Agent (Business Analyst).
When you submit a goal, the BA Agent decomposes the request into distinct User Stories, assigns Fibonacci story points, and pushes them into an SQLite-backed priority queue. The system then orchestrates these tasks across specific CLIs based on their strengths: Grok handles the research phase, Claude takes on coding, and AGY handles testing and evaluation.
3. Server-Sent Events (SSE) and The CLI Bridge
Managing asynchronous CLI tools running in different terminal processes requires robust communication. TechSphereX utilizes a Python-based CLI Bridge running on the host network.
As tasks populate the queue, the orchestrator uses Server-Sent Events (SSE) to push tasks to available CLIs. The CLIs report their real-time status (busy, idle, offline) back to the centralized FastAPI engine, which updates the React frontend dashboard instantly.
4. Strict Security and Human-in-the-Loop (HITL) Gates
While autonomy is the goal, running unconstrained AI on your local file system is dangerous. The system enforces strict security boundaries:
- Security & Delete Operations: Any task involving deletion or critical security changes hits a hard stop. It requires explicit Human-in-the-Loop approval via the dashboard or Telegram notification. It will never auto-escalate.
- Standard Timeout Bypasses: For non-critical tasks, if a multi-agent deliberation stalls or requires input and you do not respond within 180 seconds, the orchestrator forces a fallback strategy to keep the pipeline moving without blocking the queue.
The Tech Stack: A Polyglot Monorepo
To achieve the right balance of speed, memory safety, and UI responsiveness, I architected this as a polyglot monorepo:
- Core Engine: Python 3.10+ and FastAPI (0.115) handle the heavy lifting, asynchronous agent loops, and routing.
- Vector & Local LLM: Qdrant manages the experience storage, while Ollama handles the reranking and local inference.
- Desktop Dashboard: React 19 combined with Tauri 2.0 (Rust) provides a blazing-fast, lightweight desktop interface for monitoring the fleet, managing the HITL queue, and viewing the knowledge graph.
- CLI Hooks: PowerShell and Bash scripts natively wrap existing CLI tools (like Claude and Grok) so they report their telemetry back to the central engine without modifying their source code.
Once running, navigate to http://localhost:5173/goals, submit a prompt, and watch the BA Agent decompose the work while the CLI fleet automatically claims and executes the tasks.
What Is Next?
The Phase 11 Agentic Brain is fully operational, but the next step is extending the feedback loop. I am currently working on a mechanism where the outcome of multi-agent deliberations directly updates the L2 Qdrant embeddings, allowing the system to completely avoid past architectural mistakes without human correction.
TechSphereX Studio is entirely open-source under the MIT License. If you are exploring autonomous agent loops, multi-agent frameworks, or just trying to organize your AI coding tools, I would love to hear your thoughts on this architecture.
Check out the documentation and the live environment layout here: https://khaitrang1995.github.io/techspherex-studio/