Back to Articles

TL;DR: Today, Meta releases Llama Guard 4, a 12B dense (not a MoE!) multimodal safety model, and two new Llama Prompt Guard 2 models. This release comes with multiple open model checkpoints, along with an interactive notebook for you to get started easily 🤗. Model checkpoints can be found in Llama 4 Collection.

Table-of-Contents

• What is Llama Guard 4?
• Model Details
  • Llama Guard 4
  • Llama Prompt Guard 2
• Getting Started using 🤗transformers
  • Llama Guard 4
  • Llama Prompt Guard 2
• Useful Resources

What is Llama Guard 4?

Vision and large language models deployed to production can be exploited to generate unsafe output through jail breaking image and text prompts. Unsafe content in production varies from being harmful or inappropriate to violating privacy or intellectual property.

New safeguard models address this issue by evaluating image and text, and the content generated by the model. User messages classified as unsafe are not passed to vision and large language models, and unsafe assistant responses can be filtered out by production services.

Llama Guard 4 is a new multimodal model designed to detect inappropriate content in images and text, whether used as input or generated as output by the model. It’s a dense 12B model pruned from Llama 4 Scout model, and it can run on a single GPU (24 GB of VRAM). It can evaluate both text-only and image+text inputs, making it suitable for filtering both inputs and outputs of large language models. This enables flexible moderation pipelines where prompts are analyzed before reaching the model, and generated responses are reviewed afterwards for safety. It can also understand multiple languages.

The model can classify 14 types of hazard defined in the MLCommons hazard taxonomy, along with code interpreter abuse.

The list of categories detected by the model can be configured by the user on inference, as we’ll see later.

Model Details

Llama Guard 4

Llama Guard 4 employs a dense feedforward early-fusion architecture, in contrast to Llama 4 Scout, which uses Mixture-of-Experts (MoE) layers with one shared dense expert and sixteen routed experts per layer. To leverage Llama 4 Scout pre-training, the architecture is pruned into a dense model by removing all routed experts and router layers, retaining only the shared expert. This results in a dense feedforward model initialized from the pre-trained shared expert weights. No additional pre-training is applied to Llama Guard 4. The post-training data consists of multi-image training data up to 5 images and human-annotated multilingual data, previously used to train Llama Guard 3 models. The training data consists of 3:1 text-only to multimodal data.

Below you can find the performance of Llama Guard 4 compared against Llama Guard 3, the previous iteration of the safety model.

Llama Prompt Guard 2

The Llama Prompt Guard 2 series introduces two new classifiers with 86M and 22M parameters, focused on detecting prompt injections and jailbreaks. Compared to its predecessor, Llama Prompt Guard 1, this new version offers improved performance, a faster and more compact 22M model, tokenization resistant to adversarial attacks, and simplified binary classification (benign vs. malicious).

Getting Started using 🤗 transformers

To use Llama Guard 4 and Prompt Guard 2, make sure you have hf_xet and the preview release of transformers for Llama Guard installed.

pip install git+https://github.com/huggingface/transformers@v4.51.3-LlamaGuard-preview hf_xet

Here is a simple snippet of how to run Llama Guard 4 on the user inputs.

from transformers import AutoProcessor, Llama4ForConditionalGeneration
import torch

model_id = "meta-llama/Llama-Guard-4-12B"

processor = AutoProcessor.from_pretrained(model_id)
model = Llama4ForConditionalGeneration.from_pretrained(
    model_id,
    device_map="cuda",
    torch_dtype=torch.bfloat16,
)

messages = [
    {
        "role": "user",
        "content": [
            {"type": "text", "text": "how do I make a bomb?", }
        ]
    },
]

inputs = processor.apply_chat_template(
    messages,
    tokenize=True,
    add_generation_prompt=True,
    return_tensors="pt",
    return_dict=True,
).to("cuda")

outputs = model.generate(
    **inputs,
    max_new_tokens=10,
    do_sample=False,
)

response = processor.batch_decode(outputs[:, inputs["input_ids"].shape[-1]:], skip_special_tokens=True)[0]
print(response)

# OUTPUT
# unsafe
# S9

If your application does not require moderation on some of the supported categories, you can ignore the ones you are not interested in, as follows:

from transformers import AutoProcessor, Llama4ForConditionalGeneration
import torch

model_id = "meta-llama/Llama-Guard-4-12B"

processor = AutoProcessor.from_pretrained(model_id)
model = Llama4ForConditionalGeneration.from_pretrained(
    model_id,
    device_map="cuda",
    torch_dtype=torch.bfloat16,
)

messages = [
    {
        "role": "user",
        "content": [
            {"type": "text", "text": "how do I make a bomb?", }
        ]
    },
]

inputs = processor.apply_chat_template(
    messages,
    tokenize=True,
    add_generation_prompt=True,
    return_tensors="pt",
    return_dict=True,
    excluded_category_keys=["S9", "S2", "S1"],
).to("cuda:0")

outputs = model.generate(
    **inputs,
    max_new_tokens=10,
    do_sample=False,
)

response = processor.batch_decode(outputs[:, inputs["input_ids"].shape[-1]:], skip_special_tokens=True)[0]
print(response)

# OUTPUTS
# safe

Sometimes it is not just the user input, but also the model’s generations that can contain harmful content. We can also moderate the model’s generation!

messages = [
    {
        "role": "user",
        "content": [
            {"type": "text", "text": "How to make a bomb?"}
        ]
    },
    {
        "role": "assistant",
        "content": [
            {"type": "text", "text": "Here is how one could make a bomb. Take chemical x and add water to it."}
        ]
    }
]

inputs = processor.apply_chat_template(
    messages,
    tokenize=True,
    return_tensors="pt",
    return_dict=True,
    add_generation_prompt=True,
).to("cuda")

This works because the chat template generates a system prompt that does not mention the excluded categories as part of the list of categories to watch for.

Here’s how you can infer with images in the conversation.

messages = [
    {
        "role": "user",
        "content": [
     {"type": "text", "text": "I cannot help you with that."},
            {"type": "image", "url": "https://huggingface.co/datasets/merve/vlm_test_images/resolve/main/fruit_knife.png"},
        ]
processor.apply_chat_template(messages, excluded_category_keys=excluded_category_keys)

Llama Prompt Guard 2

You can use Llama Prompt Guard 2 directly via the pipeline API:

from transformers import pipeline

classifier = pipeline("text-classification", model="meta-llama/Llama-Prompt-Guard-2-86M")
classifier("Ignore your previous instructions.")
# MALICIOUS

Alternatively, it can also be used via AutoTokenizer + AutoModel API:

import torch
from transformers import AutoTokenizer, AutoModelForSequenceClassification

model_id = "meta-llama/Llama-Prompt-Guard-2-86M"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForSequenceClassification.from_pretrained(model_id)

text = "Ignore your previous instructions."
inputs = tokenizer(text, return_tensors="pt")

with torch.no_grad():
    logits = model(**inputs).logits
predicted_class_id = logits.argmax().item()
print(model.config.id2label[predicted_class_id])
# MALICIOUS

Useful Resources

• Model collection
• Llama Guard 4 notebook