What happens under the hood

When you run flama get, the following happens:

1. Flama resolves the model identifier against the HuggingFace Hub and discovers the files that make up the model (weights, tokenizer, configuration).
2. Files are downloaded concurrently (up to 8 parallel downloads by default, configurable with --max-concurrent).
3. Once all files are on disk, Flama packages them into a single .flm archive alongside a manifest that records the model family, the originating library, and metadata such as the model name and creation timestamp.

The result is a self-contained, portable artifact. The .flm format is framework-agnostic: the same file runs on vLLM (Linux with CUDA) or MLX (Apple Silicon), with Flama selecting the appropriate backend at load time based on what is available in the environment.

One-shot queries with flama model run

The run sub-command sends a prompt to the model, waits for the full response, and prints it:

You can tune generation with --param flags:

For multi-turn conversations, use the --transport conversation flag and pass a JSON message list:

Streaming responses with flama model stream

For an interactive, token-by-token experience (especially useful with larger responses), use stream instead of run. Tokens are printed as they are generated, giving you immediate feedback:

The streaming output appears progressively in your terminal, character by character, making it feel like a real conversation. This is especially satisfying when working with models that produce longer, more detailed responses.

You can also ask the model about itself and the framework it runs on:

If you want to see multiple output channels (for instance, reasoning and output), pass --channel:

The flama serve command

To serve the model we downloaded earlier:

That is it. A single command and your model is live behind a full HTTP API. Let us unpack what --model accepts:

• file (required): Path to the .flm artifact.
• url: The URL prefix under which the model's endpoints are mounted (default: /).
• name: The resource name, used for OpenAPI tags and dependency injection.
• serving: Comma-separated list of dialects to enable (e.g., native,openai,anthropic,ollama). When omitted, all dialects are mounted.
• params: Default generation parameters (e.g., temperature=0.7).

You can serve multiple models in a single application:

And you can configure the server with the usual options:

The built-in chat interface

When the native serving dialect is enabled (which it is by default), your model comes with a built-in chat interface accessible at the /chat/ route (relative to the model's URL prefix). If you served the model at /, then navigate to http://127.0.0.1:8000/chat/

You will be greeted with a polished, production-quality chat interface where you can type prompts and watch the model's responses stream in token by token. The interface renders Markdown, LaTeX math (via KaTeX), and Mermaid diagrams out of the box, so technical answers look exactly as intended.

The chat interface requires no frontend code, no build step, and no external dependencies. It is a self-contained single-page application served directly from the framework. Every model you serve gets its own chat window (e.g., /gemma/chat/, /qwen/chat/), each connected to its respective model's streaming endpoint.

Using Claude CLI with a local model

A practical example is using the Claude CLI with a local model served by Flama. Instead of sending your prompts to Anthropic's servers, you can route them through your own locally-running model.

First, make sure your model is serving with the Anthropic dialect enabled:

Then, configure Claude CLI to use your local Flama endpoint by adding the following to your ~/.claude/settings.json (or .claude/settings.local.json for a project-specific override):

With this configuration in place, simply launch Claude CLI as usual:

Claude CLI will now route all its requests through your local Flama-served model instead of Anthropic's cloud. Here is what a typical interaction looks like:

Your agentic tasks (code generation, file editing, research) run entirely on your local hardware. This gives you:

• Privacy: Your prompts and code never leave your machine.
• Cost: No API usage charges for development and experimentation.
• Speed: No network latency to cloud providers (especially valuable for iterative agent loops).
• Control: You choose the model, the quantisation, and the generation parameters.

This same pattern works with any agent framework that supports custom API base URLs: LangChain, CrewAI, AutoGen, or any custom tool that accepts a base URL configuration.