About

Dataset Generator is a desktop app that automates the full dataset generation pipeline — topic planning, multi-turn conversation generation, quality validation via LLM Judge, deduplication, and HuggingFace Hub upload. No scripts to write, no ML infra to configure.

Under the hood it runs a three-stage engine: instead of a single "generate 100 examples" prompt, the app first decomposes the job into unique topics and outlines, only then generating the actual examples. The result: diverse, coherent data without the repetitive patterns of naive generation.

Everything stays local. API keys live in SQLite on your device, datasets land in ~/.datasetgenerator/. All model traffic goes through OpenRouter (~300 models, one key).

Note on provider terms. Users are responsible for complying with the terms of service of the LLM providers they use through OpenRouter. Some providers restrict using model outputs for training competitive models — check the ToS of your chosen model before generating datasets for fine-tuning.

Why I built this

I recently started fine-tuning open-source LLMs as a hobby, and build software with AI coding agents. I wanted a simple way to generate training datasets without writing custom scripts every time — pick categories, configure the pipeline, click Generate, get a JSONL ready for training. There's plenty of datasets on HuggingFace, but sometimes you want one tailored to your specific categories and proportions. So I built the tool I wanted to use.

Benchmark

Datasets generated by this app were used to fine-tune Qwen2.5-Coder-7B-Instruct and evaluated against the base model on HumanEval / HumanEval+ (pass@1, average across 5 runs). Every model in the pipeline — topic planner, example generator, and LLM Judge — was open-source (Llama, Qwen, DeepSeek, Mistral via OpenRouter). No proprietary APIs.

+4.5 pts on HumanEval, +5.0 pts on HumanEval+ vs base. Error bars don't overlap — the difference is statistically significant.

🤗 Artifacts: fine-tuned model · V1 dataset (750 samples) · V2 dataset (1135 samples)

_{This benchmark validates the pipeline on a coding-focused dataset (multi-turn coding assistance with explanations). The tool itself is domain-agnostic — define any categories (writing, Q&A, math, customer support, etc.) and the same workflow applies. Results depend on category configuration, judge criteria, and model selection — your mileage may vary.}

Demo

Features

Actively developed — bug reports and feature requests welcome via Issues. General questions and ideas → Discussions.

• Plan-then-Execute pipeline — three stages (topics → outlines → examples), each can use a different model
• Tests - 270-test suite (unit + integration + E2E) — internal
• Per-category configuration — any number of categories with custom proportions, descriptions, and dedicated models
• LLM Judge — a second model scores every example 0–100 against editable criteria; rejected examples are regenerated
• Real-time SSE dashboard — global and per-category progress, live example feed, running cost
• Three export formats — ShareGPT, Alpaca, ChatML
• Multi-turn conversations — 1–5 turns generated coherently in one LLM call
• Actual cost tracking — pulls real usage tokens from every response, multiplies by live pricing
• Embedding-based deduplication — cosine similarity over OpenRouter embeddings
• Quality Report — judge histogram, token stats, efficiency, export to JSON/CSV
• Dataset history + in-app preview — turn-by-turn rendering, code highlighting, dataset merging
• HuggingFace Hub upload — one-click push to your repo

Use cases

• Fine-tuning a domain-specific assistant — coding, legal, medical, customer support. The benchmark above is exactly this flow.
• Instruction datasets at any scale — SFT-ready JSONL for models from 7B edge deployments up to 70B+; merge multiple jobs to grow the corpus.
• Experimenting with fine-tuning — quickly test how different category compositions affect model behavior without weeks of data curation.
• Multi-turn conversation datasets — generate 3–5 turn dialogues for training agentic behaviors.

Download

Pre-built binaries are on the Releases page — no Python, no Node.js required.

chmod +x DatasetGenerator-x86_64.AppImage
./DatasetGenerator-x86_64.AppImage

sudo apt install libfuse2t64   # Ubuntu 24.04+
sudo apt install libfuse2      # Ubuntu 22.04 and older

tar -xzf DatasetGenerator-linux-x64.tar.gz
cd DatasetGenerator
./DatasetGenerator

sudo apt install libgtk-3-0 libwebkit2gtk-4.1-0

Quick start

git clone https://github.com/AronDaron/dataset-generator.git
cd dataset-generator

# Backend
cd backend
python3 -m venv venv
./venv/bin/pip install -r requirements.txt
./venv/bin/uvicorn app.main:app --reload --port 8000

# Frontend (new terminal)
cd frontend
npm install
npm run dev

Backend on http://localhost:8000, frontend on http://localhost:3000. Open Settings → enter your OpenRouter API key → pick a category → click Generate.

Windows: replace ./venv/bin/pip and ./venv/bin/uvicorn with venv\Scripts\pip.exe and venv\Scripts\uvicorn.exe.

Requirements: Python 3.10+, Node.js 20+, an OpenRouter API key.

Stack: Next.js 16 + React 19, FastAPI + Pydantic v2, SQLite (aiosqlite), SSE for progress, Pywebview + PyInstaller for packaging.

FAQ

Can I use local models (Ollama, LM Studio)? Not currently — the pipeline talks to OpenRouter. Adding a local / OpenAI-compatible endpoint is on the roadmap; open an Issue if you need it sooner.

Is Linux fully supported? Yes — the app ships AppImage and tar.gz builds and all features work cross-platform. That said, day-to-day development and manual testing happens on Windows; Linux builds are verified with automated smoke tests but don't get the same amount of hands-on time. If something feels off on Linux, please open an Issue — I'll take a look.

How much does it cost to generate 1000 examples? Depends on model choice, turn count, and judge strictness. With open-source models available on OpenRouter (Llama 3.x, Qwen 2.5, DeepSeek, Mistral) expect single-digit dollars per 1000 multi-turn examples. Note that the UI shows the cost of accepted examples only — real spend includes rejected and skipped examples plus retries, typically 1.5-2x the displayed cost depending on judge threshold.

Is my API key safe? Keys are stored locally in SQLite (~/.datasetgenerator/database.sqlite). No telemetry, no remote calls except to OpenRouter and (optionally) HuggingFace Hub. Nothing leaves your machine unless you push a dataset.

Why AGPL-3.0 and not MIT? To prevent closed-source SaaS forks. You're free to use, modify, and self-host — but if you deploy a derivative as a hosted service, your users have the right to receive your source code. Commercial licensing is negotiable — open an Issue or contact me directly.

License

GNU Affero General Public License v3.0 — see LICENSE.

Strong copyleft: you're free to use, modify, and redistribute, but any derivative work — including SaaS / network-deployed versions — must release its full source under the same license. For proprietary commercial use, open an issue or contact me directly.