Fine-tuning LLMs for financial natural language processing (NLP) is constrained by limited, imbalanced data. Real-world financial news overrepresents earnings and stock movements, while rarer events such as credit-rating changes, product approvals, and labor issues are harder to capture at scale. Synthetic generation can help fill those gaps for trading research, risk modeling, and surveillance, but diversity requires more than a single large generation run.

This post walks through an iterative pipeline for generating 500,000 unique financial news headlines across 12 topics plus an “Other” category. The workflow combines NVIDIA NeMo Data Designer for structured generation, NVIDIA NeMo Curator for semantic deduplication, and NVIDIA Nemotron models for high-throughput headline synthesis. It also adds a farthest-from-centroid few-shot strategy to steer each round toward novel outputs.

In a naive run of 50,000 headlines, 65% were removed as near-duplicates. Instead of scaling a single batch, the pipeline iterates: generate, filter, deduplicate globally, select distinctive few-shot examples, correct category weights, and repeat until the corpus reaches the target size. Our full run produced 502,536 unique headlines across 13 categories in 82 iterations, using approximately 6 days of compute on a single 8-way NVIDIA B200 node with checkpointing, crash recovery, and SLURM job chaining.

By the end, you’ll understand how to reproduce the generation-deduplication loop, adapt it for your own financial research workflows, or start directly from the open-sourced dataset.

Reproducibility

Versions

Hardware: Single 8-way NVIDIA B200 node. GPUs 0–3 dedicated to vLLM inference (4-way tensor parallelism, 448 concurrent requests); GPUs 4–7 ran NeMo Curator semantic deduplication with Ray.

Pipeline parameters: 35K headlines per batch (50K in early iterations), 90% cosine-similarity deduplication threshold, 500 K-means clusters, 3 few-shot examples per category, 80% cross-iteration similarity cutoff for example selection.

Checkpointing: Each iteration persisted corpus state, category weights, few-shot examples, metrics, and used-example embeddings for crash recovery and SLURM job chaining.

Pipeline overview

Financial headline generation fails as a single-pass problem because real-world feeds are imbalanced: earnings reports and stock movements dominate, while credit-rating changes, product approvals, and other rarer events are sparse. In a 50K-headline baseline, semantic deduplication retained only 17,348 unique headlines, showing that larger batches add many near-duplicates rather than proportionally more coverage.

The iterative pipeline addresses this with a closed loop. Each iteration generates a category-weighted batch, filters malformed outputs, deduplicates against the full accumulated corpus, selects diverse few-shot examples, and adjusts category weights before the next iteration.

Figure 1, below, illustrates the overall architecture.

The key design decision is global deduplication. Each batch is compared with all previously retained headlines, not just with itself, preventing cross-batch duplicates and preserving semantic uniqueness across the 500K corpus.

Step 1. Generate headlines with NeMo Data Designer

We serve Nemotron 3 Nano (30B parameters, 3B active) via vLLM with 4-way tensor parallelism and 448 concurrent requests. The model’s Mixture-of-Experts (MoE) architecture balances throughput and quality, using 3B active parameters per forward pass within a 30B-parameter model.

NeMo Data Designer orchestrates generation through a declarative configuration: a weighted category sampler across 12 topics plus “Other,” and an LLM column that generates headlines conditioned on the sampled category and current few-shot examples. 

Category weights are updated after each iteration through distribution correction (Step 5).

Data Designer configuration:

config_builder = DataDesignerConfigBuilder()

# Model config for local vLLM-served Nemotron
local_model = ModelConfig(
    alias="local-nemotron",
    model="nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-FP8",
    provider="local-vllm",
    inference_parameters=InferenceParameters(
        max_parallel_requests=448,
        temperature=0.95,
        top_p=0.95,
    ),
)
config_builder.add_model_config(local_model)

# Category sampler with DYNAMIC weights (updated each iteration)
config_builder.add_column(
    SamplerColumnConfig(
        name="category",
        sampler_type=SamplerType.CATEGORY,
        params=CategorySamplerParams(
            values=CATEGORIES,        # 12 topics + "Other"
            weights=current_weights,   # Adjusted after each iteration
        ),
    )
)

# LLM column for headline generation with few-shot examples
config_builder.add_column(
    LLMTextColumnConfig(
        name="headline",
        model_alias="local-nemotron",
        system_prompt="You are a financial news headline writer.",
        prompt=f"""Generate a realistic financial news headline \
for the category: {{{{ category }}}}

{format_examples_for_prompt(examples_by_category)}

Generate a single headline for the "{{{{ category }}}}" category.
Headline:""",
    )
)

We use high-diversity sampling (temperature=0.95, frequency penalty 0.3, presence penalty 0.2) and rely on downstream filtering and semantic deduplication to remove malformed or redundant outputs.

Prompt structure

The prompt includes a role-setting system prompt, few-shot examples grouped by category, and an instruction for the sampled category. Data Designer injects the category at runtime through Jinja templating, and the few-shot examples are refreshed each iteration (Step 4).

Prompt structure (simplified):

SYSTEM:
  You are a financial news headline writer. Generate realistic,
  professional financial news headlines. Output ONLY the headline
  text, nothing else.

USER:
  Generate a realistic financial news headline for the category:
  {{ category }}

  Examples by category:
  - Earnings: "Google Parent Alphabet Posts $1.84 EPS, $63.5B Revenue"
  - Earnings: "Myriad Genetics Beats Q4 Estimates with 15% Revenue Surge"
  - Earnings: "Codec Solutions Posts Q3 Adjusted EPS $0.31, Surpasses ..."
  - Price Targets: "J.P. Morgan Cuts Target for Thermo Fisher, Downgrades ..."
  - Price Targets: "Mitsubishi UFJ Secures Buy Rating on Mitsui & Co. ..."
  - Price Targets: "Credence Resources Upgrades Target to $12.50 ..."
  - Stock price movement: "Equities Rise Broadly, Dow Gains 180 Points ..."
  - ...
  (3 examples x 12 topics + Other = up to 39 few-shot examples)

  Generate a single headline for the "{{ category }}" category.
  The headline should be 10-25 words long, realistic, and match
  the style of professional financial news.

  Headline:

Step 2. Quality filtering

Before semantic deduplication, a lightweight rule-based pass removes malformed outputs such as truncated, runaway, or garbled headlines.

In practice, these filters removed fewer than 1% of headlines per batch, so most quality control comes from semantic deduplication in the next step.

Step 3. Semantic deduplication with NeMo Curator

This is the core quality mechanism. After each batch, the pipeline combines newly generated headlines with the accumulated corpus and runs NeMo Curator’s TextSemanticDeduplicationWorkflow. Headlines are embedded with all-MiniLM-L6-v2, clustered with K-means, and compared within clusters using pairwise cosine similarity. Any headline above the 90% similarity threshold is removed.

NeMo Curator semantic deduplication configuration:

workflow = TextSemanticDeduplicationWorkflow(
    input_path=input_path,
    output_path=output_path,
    cache_path=cache_path,
    perform_removal=True,
    # Embedding parameters
    text_field="headline",
    model_identifier="sentence-transformers/all-MiniLM-L6-v2",
    embedding_max_seq_length=512,
    embedding_model_inference_batch_size=1024,
    # Deduplication parameters
    n_clusters=500,     # Higher = smaller clusters = faster pairwise
    eps=0.10,           # eps = 1 - similarity_threshold (0.90)
    id_field="id", 
    # Among duplicates, keep the more representative copy
    ranking_strategy=RankingStrategy(   
        metadata_cols=["cosine_dist_to_cent"],
        ascending=True,  #Keep the most representative (closest to centroid)
    ),
    pairwise_batch_size=4096,
    input_filetype="parquet",
    output_filetype="parquet",
)
workflow.run()

Why 500 clusters?

Pairwise similarity is expensive because comparisons scale quadratically within each cluster. At the 500K-headline endpoint, 500 clusters keep the average cluster near ~1,000 headlines and total comparisons near ~500M. Using 13 clusters, one per category, would push total comparisons toward ~19B. The higher cluster count keeps global deduplication tractable as the corpus grows.

Similarity distribution monitoring

After each iteration, the pipeline plots an ECDF (Empirical Cumulative Distribution Function) of pairwise cosine similarities. The red line marks the 0.90 deduplication threshold: a steep rise near 1.0 signals many near-duplicates, while a smoother left-shifted curve indicates broader coverage.

Figure 2, below, overlays representative iterations 5, 20, 40, 60, and 82. Early curves skew right because the model repeats common patterns from limited few-shot examples. Later curves shift left and stabilize, showing that updated few-shot examples continue to push generation toward novel content even as the corpus approaches 500K headlines.

After deduplication, the pipeline selects few-shot examples for the next batch. The goal is to steer the model away from patterns it has already produced.

Farthest-from-centroid selection

K-means clustering from the deduplication step gives each headline a cluster centroid. We rank headlines by cosine distance from that centroid and select farthest examples, which are atypical within their semantic neighborhood and useful for prompting novel outputs.

Cross-iteration semantic filtering

Distance alone is not enough. The pipeline also compares each candidate against all previously used few-shot examples and rejects any candidate with >=80% cosine similarity. This prevents semantically repeated prompt signals across iterations.

Table 3, below, summarizes the selection priority. Three examples are selected per category, for 39 total per iteration, and the used-example set plus embeddings are checkpointed for recovery.

Semantic similarity check for few-shot candidate filtering:

def is_semantically_distinct(
    candidate_embedding, used_embeddings, threshold=0.80
):
    """Reject candidates >= threshold similar to ANY used example."""
    if used_embeddings.size == 0:
        return True
    candidate_norm = candidate_embedding / (
        np.linalg.norm(candidate_embedding) + 1e-9
    )
    used_norms = used_embeddings / (
        np.linalg.norm(used_embeddings, axis=1, keepdims=True) + 1e-9
    )
    similarities = np.dot(used_norms, candidate_norm)
    return np.max(similarities) < threshold

Table 4, below, shows how selected examples become more specific by the final iteration.

Step 5. Dynamic distribution correction

Weighted sampling alone doesn’t keep categories on target because the model favors easier classes such as “Other” and “Stock Movement.” After each iteration, the pipeline compares target and actual category proportions, boosts underrepresented categories, clamps extremes, and normalizes weights for the next batch.

Weight adjustment algorithm:

def compute_adjusted_weights(current_dist, target_dist, floor=0.005):
    adjusted = []
    for category in CATEGORIES:
        target = target_dist.get(category, 0.01)
        current = current_dist.get(category, 0.0)
        if current > 0:
            ratio = target / current           # > 1 = under-represented
            ratio = max(0.2, min(5.0, ratio))  # Clamp to avoid extremes
            weight = target * ratio
        else:
            weight = target * 2.0              # Missing category: boost
        adjusted.append(max(floor, weight))
    total = sum(adjusted)
    return [w / total for w in adjusted]       # Normalize to 1.0

By iteration 82, rare categories such as “Credit Ratings” and “Product Approval” closely match their 1% targets, while “Other” remains overrepresented. The correction loop still reduces model bias enough to preserve coverage across all categories.

Results

Pipeline run summary: The 500K target is slightly overshot because generation proceeds in fixed 35K batches and halts only after the threshold is crossed; the excess 2,536 examples are trimmed from the over-represented “Other” category.

• Total unique headlines: 502,536 (target: 500,000)
• Total iterations: 82
• Batch size: 35,000 headlines per iteration
• Total generated (pre-deduplication): ~2.87M headlines
• Overall deduplication rate: ~82% (cumulative)
• Net new per iteration: ~5,000–6,000 unique headlines
• Categories: 12 broad topics + Other
• Similarity threshold: 90% cosine similarity
• Few-shot examples per iteration: 39 (3 per category)

Deduplication efficiency

Semantic deduplication was the dominant filter, and its role shifted over the run. In iteration 1, seeded by only one manual example per category, the pipeline retained 17,348 of 50K headlines; the remaining 32,652 (65%) were near-duplicates. Iterations 2–3 used farthest-from-centroid examples from previous outputs and maintained higher yields of 15K–17K new headlines per 50K batch, though cross-batch collisions began rising as the corpus grew.

From iteration 4 onward, we reduced batch size from 50K to 35K once the deduplication drop rate exceeded ~80%. Larger SLURM jobs were mostly producing headlines discarded downstream, while smaller 35K batches delivered similar net-new yield with less wasted compute.

After the corpus passed ~100K headlines, each new headline had to be distinct from a much larger pool. By iteration 40+, yield stabilized at ~5K–6K new headlines per 35K batch, or roughly 15%, with about 30K discarded per batch. Table 4 shows the transition: early duplicates are mostly within-batch repetition, while later duplicates are mostly collisions with the accumulated corpus. The steady nonzero yield near 500K shows that evolving few-shot examples continued to steer generation into novel regions.

Downstream application: Model distillation

A practical path

Four principles emerged for iterative synthetic data generation with global semantic duplication: 

• Iterate, don’t batch. A single 500K generation pass produces mostly duplicates. Eighty-two smaller iterations with evolving prompts yield a far more diverse corpus.
• Deduplicate globally, not locally. Each iteration must deduplicate against the full accumulated dataset to prevent cross-batch duplicates.
• Few-shot examples are the steering wheel. Farthest-from-centroid selection with semantic similarity filtering is the most impactful lever for generation diversity.
• Distribution correction is essential. Without dynamic weight adjustment, models over-generate popular categories and starve rare ones.

Getting started

• Use the dataset — Download FinHeadlineMix on Hugging Face and use it directly for headline classification, fine-tuning, or as training data for your own distillation pipeline.
• Run the recipe — Reproduce or extend this pipeline using NeMo Data Designer for structured synthetic generation and NeMo Curator for scalable semantic deduplication.
• Train a distilled model — Follow the AI Model Distillation for Financial Data developer example to fine-tune a compact student model on this dataset.

Going further 

• Nemotron-3-Nano-30B-A3B on NVIDIA Build — the MoE model used for generation, approved for distillation workflows
• Scaling Laws for Task-Specific LLM Distillation  — research paper leveraging the produced FinHeadlineMix dataset to derive distillation scaling laws
• Build Efficient Financial Data Workflows with AI Model Distillation — companion blog
• How to Build License-Compliant Synthetic Data Pipelines — related reading