Fine-Tuning Open-Source LLMs on a Budget with Sealos | Sealos Blog
Sealos·2025-09-02·via Sealos Blog
You don’t need a cluster of H100s to teach a large language model to speak your business’s language. With the right techniques and a thoughtful workflow, you can fine-tune open-source LLMs for domain-specific tasks—customer support, legal drafting, knowledge retrieval, analytics—at a fraction of the cost.
In this guide, you’ll learn how to fine-tune open-source LLMs on a budget using parameter-efficient methods such as LoRA and QLoRA, with a cloud-native setup powered by Sealos. We’ll cover what fine-tuning is, why it matters, how to structure an efficient training pipeline, and how to deploy your model quickly. You’ll also see code snippets and Kubernetes examples you can copy, adapt, and run.
If you already use Kubernetes or want a configurable, developer-friendly platform to run AI workloads, Sealos (https://sealos.io) provides a streamlined way to provision GPU-backed environments, object storage, and apps like notebooks or web UIs so you can focus on your model—not on plumbing.
Fine-tuning adapts a base LLM to a specific domain or task by continuing training on your curated dataset. You can:
Supervised fine-tune (SFT): Teach the model to produce desired outputs from instructions or prompts.
Preference optimize (DPO/RLHF): Align with human preferences after SFT.
Task-specialize: Summarization, code generation, SQL generation, classification, etc.
Full fine-tuning updates all model weights and can be expensive. Parameter-efficient fine-tuning (PEFT) like LoRA and QLoRA updates only small “adapter” matrices, dramatically reducing compute, memory, and cost.
Lower TCO and faster iteration: Train in hours on a single mid-range GPU instead of days on a cluster.
Data-centric wins: High-quality, task-aligned data beats brute force compute.
Deployment simplicity: Small adapter files are easy to store, version, and swap.
Compliance and control: Keep models and data in your environment; open-source models let you control licensing and security posture.
Sealos is an open-source cloud platform that makes running Kubernetes-backed apps and workloads simpler. For LLM fine-tuning, you can use Sealos to:
Provision GPU-backed compute as containerized jobs or long-running services.
Set up object storage (S3-compatible) for datasets, checkpoints, and artifacts.
Launch developer tooling (e.g., JupyterLab, VS Code Web) to iterate quickly.
Host model inference endpoints with tools like vLLM or Open WebUI.
Whether you deploy Sealos on your own infrastructure or use a managed Sealos-based cloud, you get a consistent app-centric experience with Kubernetes robustness underneath.
Choose a compact, permissively licensed base model (e.g., 3B–8B parameters).
Use LoRA/QLoRA to update only 0.1–1% of the parameters.
Quantize weights to 4-bit during training (QLoRA) to fit into a single 16–24 GB GPU.
Keep sequence lengths and dataset size sane; prioritize quality over quantity.
Use gradient accumulation, mixed precision, and gradient checkpointing to stay within memory limits.
Track and resume training; autosave checkpoints to S3 storage.
Serve with an efficient runtime (e.g., vLLM) and scale only when needed.
Match model capacity to your task and budget:
3B–8B class: Good for classification, summarization, structured generation, and many assistive tasks. Examples: Mistral 7B, Llama 3 8B, Mixtral-instruct (Mixture-of-Experts requires more memory to serve).
13B+ class: For higher fluency or complex reasoning; cost will increase significantly.
Important notes:
License: Some models require acceptance of a license (e.g., Llama 3). Ensure your use case complies.
Context length: If you need long context (8k–32k), pick a base model that supports it.
Tokenizer: Keep compatibility between tokenizer and base model.
Compute an order-of-magnitude estimate:
Total tokens processed ≈ num_examples × avg_tokens_per_example × epochs.
Training throughput on 7B QLoRA: roughly 100–300 tokens/sec on a single L4/A10G (varies widely).
If GPU costs $0.60–$1.20/hour, expect $10–$20 in compute.
These are rough numbers; your actual throughput depends on sequence length, batch size, model, and hardware.
A minimal, budget-conscious setup:
Compute: A single GPU-backed container in Sealos.
Storage: S3-compatible object storage in Sealos for datasets and checkpoints.
Image: A Docker image with PyTorch, Transformers, PEFT, BitsAndBytes, and Accelerate.
Orchestration: Kubernetes job or long-running pod; use Sealos UI or GitOps.
Serving: A separate deployment (e.g., vLLM) that loads base weights plus LoRA adapters, or merges adapters into a new checkpoint for faster inference.
Example Kubernetes Spec (Requesting a GPU)
If you’re using Sealos over Kubernetes, a simple pod spec might look like:
In Sealos, you can create similar GPU-backed workloads through the console, and provision an S3-compatible bucket using the object storage app. Store your datasets and checkpoints in S3 and access them via s3:// URIs.
High-quality, representative data lets you train fewer steps and get better results.
Start small: 5k–50k examples often suffice for narrow tasks.
Format consistently: Use a clear instruction → output schema.
Limit sequence length: Truncate inputs; move long references to retrieval if possible.
De-duplicate and sanitize: Quality beats quantity.
Example JSONL and Prompt Template
Data example (jsonl):
Prompt template applied during training:
Keeping a consistent prompt reduces prompt-shift between training and inference.
QLoRA pairs 4-bit weight quantization with LoRA adapters. The base model remains quantized; only small adapter weights are learned. This allows fitting 7B models on 16–24 GB GPUs with decent throughput.
Minimal Training Script (Transformers + PEFT)
Below is a streamlined script you can adapt. It uses:
BitsAndBytes 4-bit quantization.
LoRA adapters with PEFT.
Gradient accumulation and checkpointing for memory control.
File: train.py
Notes:
If you use s3:// paths with the datasets library, pip install s3fs and configure AWS credentials (or Sealos object storage credentials) via environment variables.
target_modules depend on the model architecture; Mistral/LLaMA-like models typically use q_proj, k_proj, v_proj, o_proj.
Set bf16=True only if GPU supports bfloat16 (Ampere+). Otherwise, switch to fp16=True.
Running the Training Job
On Sealos, you can:
Build your own image with the dependencies above, or
Use a ready-made image and pip install dependencies at runtime, then run train.py.
If you prefer a lightweight Dockerfile:
Push the image to your registry and reference it in your Sealos workload.
Evaluate before long runs to avoid wasting compute.
Smoke Test Inference with Adapters
Lightweight Perplexity Check
For generative tasks, compute perplexity on a held-out set:
Small, consistent improvements in perplexity on relevant text usually correlate with better outputs.
Two options:
Load base model + LoRA adapters at runtime. Pros: small artifact size, can swap adapters easily. Cons: slight overhead on first load.
Merge LoRA adapters into base weights offline, then serve the merged model for best inference throughput.
Merging Adapters
Upload MERGED_DIR to your model store (e.g., S3), then serve.
Serving with vLLM (Kubernetes)
vLLM is efficient and easy to deploy. Example Deployment:
With Sealos, you can expose the service via an Ingress or built-in domain routing, and test with the OpenAI-compatible endpoint:
Alternatively, you can deploy Open WebUI on Sealos to chat with your model.
Customer support: Teach the model to resolve common tickets, follow your policy handbook, and produce templated responses.
Sales assistants: Personalize outreach, qualify leads, and fill CRM fields consistently.
Documentation Q&A: Summarize changelogs, answer questions about your API, and draft guides with your tone.
Data engineering helpers: Generate SQL against your schema, annotate ETL pipeline code, and document datasets.
Tip: Keep your training data narrowly focused on the task. For retrieval-heavy tasks, combine a smaller instruction-tuned model with a vector database and RAG, rather than trying to stuff all knowledge into weights.
Use QLoRA with r=8–16 and lora_alpha=16–32. Often sufficient.
Keep sequence length tight (512–2048). Longer context is costly.
Start with 1–2 epochs; stop early if validation plateaus.
Freeze everything except attention projections (q,k,v,o) and perhaps the MLP in later experiments.
Enable gradient checkpointing and use paged_adamw_8bit optimizer.
Pre-tokenize your dataset to avoid CPU bottlenecks on the GPU clock.
Use spot/preemptible GPUs if your platform supports checkpoint resume.
Validate data quality. A single corrupted shard or mixed schema can waste runs.
Out-of-memory (OOM): Reduce MAX_LEN, increase GRAD_ACCUM, lower LoRA rank, or switch to smaller base model.
Catastrophic forgetting: Use a small proportion (5–20%) of general data mixed with your domain data, or apply regularization (e.g., KL during preference training).
Instruction drift: Keep a stable prompt template and mirror it in inference.
Overfitting: Monitor held-out loss; add early stopping; reduce epochs.
Tokenization mismatches: Always use the base model’s tokenizer; never mix tokenizers.
A step-by-step outline you can follow:
Provision storage:
Create an S3-compatible bucket in Sealos to hold datasets and checkpoints.
Upload your JSONL dataset and optionally a held-out eval set.
Prepare the image:
Build the Dockerfile above and push to your registry, or use a prebuilt image and provide train.py as a ConfigMap/volume in Kubernetes.
Launch a GPU workload:
In Sealos, create a GPU-enabled job or pod using your image.
Set environment variables: BASE_MODEL, DATASET_PATH, OUTPUT_DIR, etc.
Mount credentials for S3.
Monitor and iterate:
Stream logs to check throughput and loss.
If loss flattens early, cancel to save cost and adjust hyperparameters.
Save and test:
After training, ensure adapter weights and tokenizer are saved to S3.
Run a quick smoke test with the inference code to validate outputs.
Serve:
Merge adapters for faster inference (optional).
Deploy vLLM with the merged model or load adapters dynamically.
Expose an endpoint, and connect a UI (e.g., Open WebUI).
Automate:
Use GitOps or CI/CD to trigger fine-tunes on new data drops.
Schedule cost-saving policies (TTL jobs, off-hours scaling) via Kubernetes.
Sealos simplifies each of these steps with an application-centric console, while still giving you full control over Kubernetes primitives when you need them.
Data residency: Keep datasets and checkpoints in your Sealos-hosted object storage within your region.
Secrets: Store access keys in Kubernetes Secrets; avoid hardcoding.
Licenses: Ensure your base model’s license permits your use case and redistribution if you plan to share the merged model.
Auditing: Log training runs and configurations; store training args alongside checkpoints for reproducibility.
Can I fine-tune without a GPU?
Technically yes for very small adapters and tiny models, but training speed will be impractical. A single mid-range GPU (e.g., L4/A10G) is a sweet spot.
Do I need Deepspeed or FSDP?
Not for 7B QLoRA on a single GPU. If you scale beyond 13B or need full fine-tuning, consider them.
Should I use DPO/RLHF?
Start with SFT. If you need preference alignment (tone, safety, style), add DPO with a small pairwise preference dataset.
How big should my dataset be?
For narrow tasks, 5k–20k high-quality examples often suffice. For broader instruction tuning, 50k–200k is common—still manageable with QLoRA.
Data cleanup and tokenization: 1–2 hours CPU time.
Pilot run (1 epoch): 2–4 hours.
Main run (2 epochs): 4–8 hours.
Merge + deploy: 1 hour.
Estimated spend: $10–$30 depending on GPU price and total hours.
Optimize by reducing sequence length, trimming low-quality samples, and merging early if results are already good after 1 epoch.
Fine-tuning open-source LLMs doesn’t have to be expensive or complicated. With QLoRA/LoRA, careful data curation, and efficient tooling, you can ship models that feel bespoke to your domain—often in a day and well within a modest budget.
Sealos helps you operationalize this workflow: spin up GPU workloads, store datasets and artifacts, iterate in familiar developer tools, and deploy serving endpoints—without wrangling infrastructure details. Start small, measure often, and let data quality—not raw compute—drive your improvements.
When you’re ready to level up your AI stack with a practical, cloud-native approach, explore Sealos at https://sealos.io and put your fine-tuning plan into action.
