View PDF

Abstract:We present GhostShell, a novel approach that leverages Large Language Models (LLMs) for streaming and concurrent behavioral programming in embodied systems. In contrast to predefined behavioral structures and plan-then-execute paradigms, GhostShell enables reasoning-while-acting by incrementally invoking functions during LLM streaming generation. We define function tokens as an XML-based function-call representation that GhostShell parses from the LLM generation stream and maps to callable functions. A multi-channel scheduling algorithm further orchestrates these calls with intra-channel synchronous and inter-channel asynchronous dispatch, coordinating sequential-parallel behavior execution across multiple robotic components. We evaluate GhostShell on our robotic prototype CoCo across 33 real-world tasks with LLMs from nine providers. On 30 grounded Human-Robot Interaction (HRI) tasks, our approach achieves the highest Directed Structured Behavior Correctness (DSBC) score of 0.83 with Claude-Sonnet-4, while on three long-horizon multimodal tasks, GPT-4.1 attains a top human evaluation score of 7.0/10. Compared to native LLM function calling, our function token schema achieves a 15/15 task completion rate versus 6/15, particularly in coordinating concurrent linguistic and physical actions. Supplementary materials, including videos, are available at this https URL.

Submission history

From: Youwei Huang [view email]
[v1] Thu, 7 Aug 2025 11:55:46 UTC (1,947 KB)
[v2] Fri, 8 Aug 2025 04:36:40 UTC (1,947 KB)
[v3] Sun, 19 Jul 2026 07:48:14 UTC (4,409 KB)