Abstract:Device-cloud collaboration holds promise for deploying large language models (LLMs), leveraging lightweight on-device models for efficiency while relying on powerful cloud models for superior reasoning. A central challenge in this setting is determining, for each incoming query, whether it should be processed locally or offloaded to the cloud. Existing approaches typically rely on external routers, which often struggle to determine difficulty from the prompt itself, especially for tasks involving complex reasoning. Motivated by this limitation, we propose enabling on-device LLMs to decide internally whether to invoke cloud assistance at inference time, with this capability instilled through reinforcement learning based post-training. Casting on-device LLM post-training as a reward maximization problem, we design hierarchical rewards to encourage local problem solving and judicious cloud offloading. To solve the resulting problem, we develop an algorithm featuring a group-level policy gradient that stabilizes optimization, together with adaptive prompt filtering that provides complementary learning signals to mitigate policy collapse (i.e., exclusive local execution or exclusive cloud offloading). Extensive experiments on on-device-scale LLaMA and Qwen models across multiple reasoning benchmarks show that our method consistently outperforms baselines and significantly narrows the gap to full cloud LLMs.
| Comments: | We propose a unified post-training framework that integrates routing optimization, enabling the on-device LLM to improve its problem-solving ability while learning routing strategies |
| Subjects: | Machine Learning (cs.LG) |
| Cite as: | arXiv:2509.24050 [cs.LG] |
| (or arXiv:2509.24050v4 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2509.24050 arXiv-issued DOI via DataCite |
From: Wenzhi Fang [view email]
[v1]
Sun, 28 Sep 2025 19:48:56 UTC (1,535 KB)
[v2]
Wed, 31 Dec 2025 22:53:09 UTC (645 KB)
[v3]
Thu, 29 Jan 2026 07:25:15 UTC (865 KB)
[v4]
Sat, 23 May 2026 21:20:59 UTC (869 KB)
此内容由惯性聚合(RSS阅读器)自动聚合整理，仅供阅读参考。 原文来自 — 版权归原作者所有。