Few-Shot Learning with LLM: A Deep Dive

Few-shot learning with large language models is one of the most practical ways to steer model behavior without updating weights. By embedding task-specific examples directly into the prompt, developers can turn a general-purpose foundation model into a domain-specific classifier, parser, or reasoning engine. The technique relies on in-context learning, where the model infers patterns from exemplars rather than from gradient updates. Because it requires no training pipeline, few-shot prompting is ideal for rapid prototyping and production tasks where data volumes are too small for fine-tuning or where model weights must remain frozen.

The Mechanics of In-Context Learning

In-context learning is an emergent capability of transformer-based language models. During inference, the model attends to the full context window, using the provided examples as a dynamic prior. Each example adjusts the hidden-state activations for subsequent tokens, effectively conditioning the output distribution without any parameter change. Research suggests that the model locates latent task representations within its pretrained weight space and uses the few-shot examples to activate the appropriate subspace. The result is a flexible interface: change the examples, and the model adapts its behavior immediately.

Zero-Shot, One-Shot, and Few-Shot Prompting

These three patterns describe how much guidance you provide before the actual task input.

Zero-shot: You describe the task in natural language with no examples. This works best for simple, well-known tasks that the model has seen frequently during pretraining.
One-shot: You prepend a single example. This is often enough to communicate output format or tone.
Few-shot: You prepend three to ten examples, sometimes more for complex schema extraction or multi-label classification. The marginal gain from each additional example typically diminishes, but for tasks with rigid output schemas, a larger set of exemplars can substantially improve consistency.

A Practical Example with Oxlo.ai

Oxlo.ai supports fully OpenAI SDK-compatible chat completions, so you can implement few-shot prompting with minimal code changes. The following example uses Llama 3.3 70B to classify customer feedback sentiment using four in-context exemplars. Because Oxlo.ai offers request-based pricing, you can include long, detailed prompts with many examples without worrying about escalating input token costs.

import os
import openai

client = openai.OpenAI(
    base_url="https://api.oxlo.ai/v1",
    api_key=os.environ["OXLO_API_KEY"]
)

few_shot_prompt = """Classify the sentiment of customer feedback as POSITIVE, NEGATIVE, or NEUTRAL.

Examples:
Feedback: "The delivery was fast and the packaging was perfect."
Label: POSITIVE

Feedback: "I waited two weeks and the item arrived damaged."
Label: NEGATIVE

Feedback: "The product works, but the instructions were unclear."
Label: NEUTRAL

Feedback: "Best purchase I have made this year."
Label: POSITIVE

Now classify this:
Feedback: "The app crashes every time I try to save my work."
Label:"""

response = client.chat.completions.create(
    model="llama-3.3-70b",
    messages=[
        {"role": "system", "content": "You are a precise text classifier. Output only the label."},
        {"role": "user", "content": few_shot_prompt}
    ],
    temperature=0.1,
    max_tokens=10
)

print(response.choices[0].message.content.strip())

Notice how the examples establish a consistent format. The model learns the delimiter pattern, the label vocabulary, and the level of brevity required, all from the provided context.

Selecting and Ordering Examples

Not all examples are equally useful. Effective few-shot prompts depend on coverage, diversity, and clarity.

Coverage: Include edge cases and minority classes. If your dataset is imbalanced, oversample rare labels in the prompt to prevent bias toward frequent classes.
Diversity: Vary phrasing, length, and syntax. Homogeneous examples can cause the model to overfit to surface patterns rather than the underlying task.
Ordering: Models often exhibit recency bias, placing more weight on examples near the end of the prompt. Place the most representative or difficult examples last, or shuffle order across requests if your application permits.

Formatting and Delimiters

Consistent formatting acts as a structural prior. Use clear delimiters such as XML tags, markdown code fences, or simple line breaks with labels. For example:

<example>
  <input>...</input>
  <output>...</output>
</example>

Whitespace and punctuation should follow an identical pattern across every exemplar. Any deviation can introduce noise and reduce accuracy.

Scaling Few-Shot Prompts on Oxlo.ai

One practical barrier to few-shot learning is cost. Long prompts packed with examples consume significant input tokens. On token-based inference platforms, this directly inflates your bill, especially for agentic workflows that append tool outputs and conversation history to every request.

Oxlo.ai uses flat per-request pricing, meaning you pay one cost per API call regardless of prompt length. For few-shot and long-context workloads, this can be significantly cheaper than token-based alternatives. You can expand your context window with rich exemplars, system instructions, and multi-turn history without linear cost growth. See the Oxlo.ai pricing page for plan details.

Extending Few-Shot with Chain-of-Thought

For reasoning tasks, raw input-output pairs may be insufficient. Chain-of-thought few-shot prompting includes intermediate reasoning steps in each exemplar. This teaches the model to decompose problems before emitting a final answer.

Q: A train travels 60 km in 30 minutes. How far will it travel in 2 hours?
A: First, convert 30 minutes to 0.5 hours. The speed is 60 km / 0.5 h = 120 km/h. In 2 hours, the distance is 120 km/h * 2 h = 240 km. Final answer: 240 km.

Oxlo.ai hosts models such as DeepSeek R1 671B MoE and Kimi K2.6 that excel at advanced reasoning. Combining their native chain

推荐订阅源

DEV Community