Running Local GGUF Models with Ollama (GPU Enabled)

1. Install & Start Ollama

curl -fsSL https://ollama.com/install.sh | sh
systemctl start ollama
ollama --version

2. Verify GPU Detection

NVIDIA

nvidia-smi

AMD

rocm-smi

3. Set Up Model Directory

mkdir -p ~/Documents/LLM
cd ~/Documents/LLM
# Copy your .gguf file here

4. Create a Modelfile

vim Modelfile

Vim quick reference:

i — enter insert mode (start typing)
Esc — exit insert mode
:wq — save and quit
:q! — quit without saving

FROM ./Phi-4-mini-instruct-Q4_K_M.gguf

SYSTEM """
You are a helpful AI assistant.
"""

TEMPLATE """<|user|>
{{ .Prompt }}<|end|>
<|assistant|>
"""

PARAMETER stop "<|user|>"
PARAMETER stop "<|assistant|>"
PARAMETER stop "<|end|>"
PARAMETER temperature 0.7
PARAMETER num_ctx 8192

Note: Always include TEMPLATE for custom GGUFs. Use instruct/chat variants, not base models.

5. Create & Run the Model

ollama create mymodel -f Modelfile
ollama run mymodel

6. Verify GPU Usage

Open a second terminal and monitor VRAM — an increase confirms GPU acceleration.

# NVIDIA
watch -n 1 nvidia-smi

# AMD
watch -n 1 rocm-smi

To confirm via logs:

journalctl -u ollama -f
# Look for: "using CUDA" or "offloading layers to GPU"

7. Ollama Command Reference

Model Management

Task	Command
Pull a model	`ollama pull <model>`
Create from Modelfile	`ollama create <name> -f Modelfile`
List installed models	`ollama list`
Show model details	`ollama show <model>`
Copy a model	`ollama cp <source> <dest>`
Remove a model	`ollama rm <model>`
Push model to registry	`ollama push <model>`

Running Models

Task	Command
Run model (interactive)	`ollama run <model>`
Run with single prompt	`ollama run <model> "your prompt"`
Run with stdin input	`echo "prompt" \
Show running models	{% raw %}`ollama ps`
Stop a running model	`ollama stop <model>`

In-Chat Commands

Command	Action
`/clear`	Clear chat history
`/bye`	Exit chat
`/set parameter <key> <val>`	Change param on the fly
`/show info`	Show model info
`/show modelfile`	Show current Modelfile
`/show parameters`	Show active parameters
`/help`	List all in-chat commands

API (REST)

Ollama runs a local server at http://localhost:11434.

# Generate (single turn)
curl http://localhost:11434/api/generate -d '{
  "model": "mymodel",
  "prompt": "Explain Docker in simple terms",
  "stream": false
}'

# Chat (multi-turn)
curl http://localhost:11434/api/chat -d '{
  "model": "mymodel",
  "messages": [
    { "role": "user", "content": "Hello!" }
  ]
}'

# List models via API
curl http://localhost:11434/api/tags

# Check running models
curl http://localhost:11434/api/ps

8. Manage Ollama Service (systemctl)

Start / Stop / Restart

# Start Ollama service
systemctl start ollama

# Stop Ollama service
systemctl stop ollama

# Restart Ollama service
systemctl restart ollama

Status & Logs

# Check service status
systemctl status ollama

# View live logs
journalctl -u ollama -f

# View last 50 log lines
journalctl -u ollama -n 50

Enable / Disable on Boot

# Enable Ollama to start on boot
systemctl enable ollama

# Disable autostart
systemctl disable ollama

# Check if enabled
systemctl is-enabled ollama

9. Gollama — Chat TUI for Ollama

Gollama is a terminal chat interface for Ollama with conversation history saved via SQLite.

Install Go (Fedora)

sudo dnf install golang -y
go version

Install Gollama

go install github.com/gaurav-gosain/gollama@latest

# Add Go binaries to PATH
echo 'export PATH=$PATH:~/go/bin' >> ~/.bashrc
source ~/.bashrc

Launch

gollama

Keyboard Shortcuts

Key	Action
`↑` / `k`	Navigate up
`↓` / `j`	Navigate down
`Ctrl+N`	New chat
`/`	Fuzzy search chats
`d`	Delete chat
`Ctrl+C`	Quit

推荐订阅源

DEV Community