Most multi-agent setups I've seen are basically a room full of people wearing headphones. Agents running in parallel, no shared awareness, no idea who's doing what. That's not collaboration. That's coexistence.
I've been building this in public for almost 12 weeks. 12 agents, 6,500+ tests, 95 stars. Here's what I actually learned.
The problem wasn't memory. It was identity. An agent would be technically correct but completely off base. Not hallucinating. Drifting. Like a competent person who walked into the wrong meeting and started contributing without realizing they're in the wrong room. I spent weeks on better memory - longer context, better embeddings, persistent state. None of it fixed the drift. The problem wasn't what the agent remembered - it didn't know who it was.
What fixed it was three files. Every agent gets a passport.json - who am I, what I do, what I dont do. Maybe 30 lines. Rarely changes. Then local.json - rolling session log, key learnings, caps at 20 entries and auto-archives to vector search when full. And observations.json - collaboration patterns, how I work with other agents. Identity loads first every session via hooks. Agent never starts cold.
I have 12 agents now and each one is a domain specialist. The mail system has 696 tests it built through its own bugs. Routing system is 80+ sessions deep - all it thinks about is routing. They dont do each others jobs. When something breaks in another domain they email each other. The orchestrator dispatches work to them and trusts them because they know their own code better than it does.
Every time I post about this someone asks what happens when two agents write the same file. Fair question. They cant. Not as in "we tell them not to" - there's a hook called pre_edit_gate that fires before every write. If an agent in branch A tries to edit a file in branch B's directory, the write gets rejected. Hard block. The agent sees "cross-branch write blocked" and has to either ask a trusted branch to make the change or send a mail request through drone. Only 3 branches in the whole system (the orchestrator, the auditor, and the factory that creates new agents) are allowed to cross-write. Everyone else is physically confined to their own directory. We also lock inboxes - agents cant forge messages by writing directly to another agent's mailbox file. They have to use the mail system. This isnt a convention. Its enforcement.
This week I stopped building features and started testing. Took an old MacBook, wiped it, installed Ubuntu from scratch. Cloned on a machine with nothing pre-configured. Found every setup blocker - git config missing, venv broken on fresh Ubuntu, hooks not wired. All fixed now. Install went from ~2GB down to ~100MB. Built a concierge agent that walks new users through onboarding - 12-stage flow, 243 tests on it. First impressions matter and ours was rough ngl.
95 stars. Small project. I'm a solo dev tbh and the agents help build and maintain themselves - every PR is human-AI collaboration. The hardest part hasn't been the code. It's explaining what this actually is. People hear "agents" and expect a task runner. This isnt that. Its infrastructure for building systems that remember and coordinate. What u put on top is up to u.
Has anyone else hit the identity drift problem? Genuinely curious how others solved it - or if most just threw more context at it and moved on.