@@ -0,0 +1,200 @@
1+---
2+summary: "Start local model servers on demand before OpenClaw model requests"
3+read_when:
4+ - You want OpenClaw to start a local model server only when its model is selected
5+ - You run ds4, inferrs, vLLM, llama.cpp, MLX, or another OpenAI-compatible local server
6+ - You need to control cold start, readiness, and idle shutdown for local providers
7+title: "Local model services"
8+---
9+10+`models.providers.<id>.localService` lets OpenClaw start a provider-owned local
11+model server on demand. It is provider-level config: when the selected model
12+belongs to that provider, OpenClaw probes the service, starts the process if the
13+endpoint is down, waits for readiness, then sends the model request.
14+15+Use it for local servers that are expensive to keep running all day, or for
16+manual setups where model selection should be enough to bring the backend up.
17+18+## How it works
19+20+1. A model request resolves to a configured provider.
21+2. If that provider has `localService`, OpenClaw probes `healthUrl`.
22+3. If the probe succeeds, OpenClaw uses the existing server.
23+4. If the probe fails, OpenClaw starts `command` with `args`.
24+5. OpenClaw polls readiness until `readyTimeoutMs` expires.
25+6. The model request is sent through the normal provider transport.
26+7. If OpenClaw started the process and `idleStopMs` is positive, the process is
27+ stopped after the last in-flight request has been idle for that long.
28+29+OpenClaw does not install launchd, systemd, Docker, or a daemon for this. The
30+server is a child process of the OpenClaw process that first needed it.
31+32+## Config shape
33+34+```json5
35+{
36+ models: {
37+ providers: {
38+ local: {
39+ baseUrl: "http://127.0.0.1:8000/v1",
40+ apiKey: "local-model",
41+ api: "openai-completions",
42+ timeoutSeconds: 300,
43+ localService: {
44+ command: "/absolute/path/to/server",
45+ args: ["--host", "127.0.0.1", "--port", "8000"],
46+ cwd: "/absolute/path/to/working-dir",
47+ env: { LOCAL_MODEL_CACHE: "/absolute/path/to/cache" },
48+ healthUrl: "http://127.0.0.1:8000/v1/models",
49+ readyTimeoutMs: 180000,
50+ idleStopMs: 0,
51+ },
52+ models: [
53+ {
54+ id: "my-local-model",
55+ name: "My Local Model",
56+ reasoning: false,
57+ input: ["text"],
58+ cost: { input: 0, output: 0, cacheRead: 0, cacheWrite: 0 },
59+ contextWindow: 131072,
60+ maxTokens: 8192,
61+ },
62+ ],
63+ },
64+ },
65+ },
66+}
67+```
68+69+## Fields
70+71+- `command`: absolute executable path. Shell lookup is not used.
72+- `args`: process arguments. No shell expansion, pipes, globbing, or quoting
73+ rules are applied.
74+- `cwd`: optional working directory for the process.
75+- `env`: optional environment variables merged over the OpenClaw process
76+ environment.
77+- `healthUrl`: readiness URL. If omitted, OpenClaw appends `/models` to
78+`baseUrl`, so `http://127.0.0.1:8000/v1` becomes
79+`http://127.0.0.1:8000/v1/models`.
80+- `readyTimeoutMs`: startup readiness deadline. Default: `120000`.
81+- `idleStopMs`: idle shutdown delay for OpenClaw-started processes. `0` or
82+ omitted keeps the process alive until OpenClaw exits.
83+84+## Inferrs example
85+86+Inferrs is a custom OpenAI-compatible `/v1` backend, so the same local service
87+API works with the `inferrs` provider entry.
88+89+```json5
90+{
91+ agents: {
92+ defaults: {
93+ model: { primary: "inferrs/google/gemma-4-E2B-it" },
94+ },
95+ },
96+ models: {
97+ mode: "merge",
98+ providers: {
99+ inferrs: {
100+ baseUrl: "http://127.0.0.1:8080/v1",
101+ apiKey: "inferrs-local",
102+ api: "openai-completions",
103+ timeoutSeconds: 300,
104+ localService: {
105+ command: "/opt/homebrew/bin/inferrs",
106+ args: [
107+"serve",
108+"google/gemma-4-E2B-it",
109+"--host",
110+"127.0.0.1",
111+"--port",
112+"8080",
113+"--device",
114+"metal",
115+ ],
116+ healthUrl: "http://127.0.0.1:8080/v1/models",
117+ readyTimeoutMs: 180000,
118+ idleStopMs: 0,
119+ },
120+ models: [
121+ {
122+ id: "google/gemma-4-E2B-it",
123+ name: "Gemma 4 E2B (inferrs)",
124+ reasoning: false,
125+ input: ["text"],
126+ cost: { input: 0, output: 0, cacheRead: 0, cacheWrite: 0 },
127+ contextWindow: 131072,
128+ maxTokens: 4096,
129+ compat: {
130+ requiresStringContent: true,
131+ },
132+ },
133+ ],
134+ },
135+ },
136+ },
137+}
138+```
139+140+Replace `command` with the result of `which inferrs` on the machine running
141+OpenClaw.
142+143+## ds4 example
144+145+```json5
146+{
147+ models: {
148+ providers: {
149+ ds4: {
150+ baseUrl: "http://127.0.0.1:18000/v1",
151+ apiKey: "ds4-local",
152+ api: "openai-completions",
153+ timeoutSeconds: 300,
154+ localService: {
155+ command: "/Users/you/Projects/oss/ds4/ds4-server",
156+ args: [
157+"--model",
158+"/Users/you/Projects/oss/ds4/ds4flash.gguf",
159+"--host",
160+"127.0.0.1",
161+"--port",
162+"18000",
163+"--ctx",
164+"393216",
165+ ],
166+ cwd: "/Users/you/Projects/oss/ds4",
167+ healthUrl: "http://127.0.0.1:18000/v1/models",
168+ readyTimeoutMs: 300000,
169+ idleStopMs: 0,
170+ },
171+ models: [],
172+ },
173+ },
174+ },
175+}
176+```
177+178+## Operational notes
179+180+- One OpenClaw process manages the child it started. Another OpenClaw process
181+ that sees the same health URL already live will reuse it without adopting it.
182+- Startup is serialized per provider command and argument set, so concurrent
183+ requests do not spawn duplicate servers for the same config.
184+- Active streaming responses hold a lease; idle shutdown waits until response
185+ body handling is complete.
186+- Use `timeoutSeconds` on slow local providers so cold starts and long generations
187+ do not hit the default model request timeout.
188+- Use an explicit `healthUrl` if your server exposes readiness somewhere other
189+ than `/v1/models`.
190+191+## Related
192+193+<CardGroup cols={2}>
194+<Card title="Local models" href="/gateway/local-models" icon="server">
195+Local model setup, provider choices, and safety guidance.
196+</Card>
197+<Card title="Inferrs" href="/providers/inferrs" icon="cpu">
198+Run OpenClaw through the inferrs OpenAI-compatible local server.
199+</Card>
200+</CardGroup>
此内容由惯性聚合(RSS阅读器)自动聚合整理，仅供阅读参考。 原文来自 — 版权归原作者所有。