View PDF HTML (experimental)

Abstract:Large language model (LLM) agents are increasingly built on agent frameworks that provide reusable abstractions for workflow orchestration, state management, tool integration, and execution control. However, the quality of this infrastructure layer remains insufficiently understood, particularly its functionality challenges and usability concerns, as existing studies have mainly examined traditional deep learning (DL) frameworks or model-level agent failures. Therefore, we conduct an empirical study of 5,669 bug reports and 809 feature requests from five mainstream agent frameworks: AutoGen, CrewAI, LangChain, LangGraph, and MetaGPT. We construct a four-dimensional taxonomy covering 22 root causes, seven symptoms, 11 motivations, and six requirements, and map them to the five-stage agent lifecycle. Across the four RQs, results show an execution centered quality pattern shaped by semantic interface boundaries. Reported bugs mainly manifest as Incorrect Functionality (76.00%) and involve more API, configuration, parsing, and serialization related causes than DL framework bugs, while their associations remain sparse and stage specific. Feature requests mainly target Feature Enhancement (49.07%) and reveal structured needs for Orchestration Expressiveness, Development Delivery, Model Adaptation, and Tool Ecosystem. These findings call for quality assurance beyond crash based and tensor level testing, with emphasis on API sequences, structured LLM outputs, serialization boundaries, execution traces, and execution centered maintenance, offering empirical guidance for reliable and usable agent framework infrastructure.

Submission history

From: Xinxue Zhu [view email]
[v1] Wed, 25 Feb 2026 11:34:17 UTC (293 KB)
[v2] Thu, 26 Feb 2026 04:30:22 UTC (294 KB)
[v3] Fri, 27 Feb 2026 16:17:57 UTC (294 KB)
[v4] Thu, 6 Aug 2026 07:48:13 UTC (1,436 KB)