About a year ago I wanted an AI assistant that could actually automate my life — not just answer questions. I wanted it to send Telegram/Whatsapp messages, check my calendar, control my phone, run shell commands on my server, and schedule things
I couldn't find something that did all of that and was actually self-hostable (openclaw is lacking features and wasn't popular at that time), so I built NeoAgent. This is a breakdown of how it works.
What it is
NeoAgent is a Node.js server you install with just two commands:
npm install -g neoagent
neoagent install
The client is a Flutter app that works in the browser and on Android. Both connect to your own server — nothing routes through a third-party cloud.
The agent loop
The core is a tool-calling loop. When you send a message, the server:
- Loads context: relevant memories, recent run history, integration state
- Calls the configured AI provider (Anthropic, OpenAI, Gemini, Grok, or local Ollama)
- Executes any tool calls the model returns
- Feeds results back and repeats until the model stops requesting tools or a step limit is hit
- Writes the run steps to SQLite and streams them to the client
The tools
Here's what the agent can actually invoke:
Shell and files:
-
execute_command— a PTY-capable shell executor with stdin, timeout, stdout, stderr, exit code, and duration. Notexec, a real PTY. -
read_file,write_file,edit_file,list_files,search_files
Browser:
- Navigate, click, type, extract content, screenshot, and evaluate JavaScript in a server-side Chromium instance
- The browser runs inside a QEMU-backed Ubuntu VM, isolated from the host, or via a paired Chrome extension on a remote machine
Android control:
This is the part that took the most work. NeoAgent can control a server-attached Android emulator or physical ADB device:
-
android_screenshot— returns the current screen as an image the model can see -
android_ui_dump— returns the UIAutomator XML for the current screen -
android_observe_nodes— extracts clickable/typeable nodes so the model doesn't have to parse raw XML -
android_tap,android_long_press,android_type,android_swipe,android_key -
android_open_app,android_open_intent -
android_wait_for— waits for a text, resource ID, or class to appear, up to a timeout -
android_install_apk,android_shell
The emulator runs as part of the QEMU VM. When NeoAgent is deployed on a remote server, the AI controls the Android runtime attached to that server.
A practical example: "Open WhatsApp on the phone, find the conversation with [name], and read the last 10 messages" — the agent takes a screenshot, reads the UI tree, taps into WhatsApp, scrolls the conversation, and returns the messages as text.
Messaging:
NeoAgent has a unified messaging layer that abstracts over 15 platforms:
- Telegram, WhatsApp, Discord, Slack, Google Chat, Teams, Matrix, Signal, iMessage/BlueBubbles, IRC, LINE, Mattermost, Twitch, Telnyx Voice
- Plus configurable webhook bridges for Feishu, Nextcloud Talk, Nostr, Synapse, WeChat, and others
The agent tool is send_message(platform, recipient, content). Platform-specific details (credentials, API tokens) are configured server-side.
Integrations:
These are richer structured tools beyond basic messaging:
- Google Workspace: Gmail, Calendar, Drive, Docs, Sheets
- Microsoft 365: Outlook, Calendar, OneDrive, Teams
- Notion: search, pages, blocks, databases
- Home Assistant: entity reads, service calls, config
- Trello, Spotify, Figma, Weather
These connect via OAuth. Secrets are stored in ~/.neoagent/.env and never sent to the client.
Scheduled tasks:
Tasks run on a cron schedule or in response to triggers. Each task has a prompt, a model override option, and optionally a Telnyx voice delivery for the result. You can schedule things like "every morning at 8, summarize my calendar for today and send it to Telegram."
MCP client:
NeoAgent registers as an MCP client. You can add remote MCP servers through the UI, and their tools become available in the agent loop alongside the built-in tools.
Data and credentials
Everything persists in SQLite under ~/.neoagent/data/. WAL mode is enabled for concurrent reads. The schema includes runs, messages, run steps, tasks, memory entries, recordings, health samples, and artifact storage.
All AI provider keys and OAuth client secrets live in ~/.neoagent/.env on the server. The Flutter client authenticates to your backend with a session token — it never sees raw API keys.
What's missing / known rough edges
- The Android emulator boot is slow (it's a full QEMU VM). First boot also downloads a base Ubuntu image.
Trying it
npm install -g neoagent
neoagent install
Open http://localhost:3333 in your browser. Set up an AI provider in Settings (or point it at Ollama for fully local). MIT license.
GitHub: https://github.com/NeoLabs-Systems/NeoAgent
Docs: https://neolabs-systems.github.io/NeoAgent/docs/
If you build something interesting with it or hit a rough edge, I'd like to hear about it.
Also its still a WIP, so please share any issues you encounter