1. Introduction In the first experiment , I confirmed my hypothesis that adding a simple creativity instruction to a prompt measurably increases response diversity in embedding space. In the second experiment , I discovered the creativity paradox: prompts asking models to be “uncreative” or “monotonous” produced higher response diversity than prompts asking for creativity. I proposed several hypotheses — task confusion, distribution shift, statistical rarity of instruction patterns. This left a critical question unanswered: is the paradox a real phenomenon, or a flaw in the metric? To find out, I designed a new probe — and built new metrics from scratch. As a result of current experiment I made two steps forward: 1. I found the metric which allows to assess the creativity of the model taking into the account the relevance of the responses 2. The metric confirmed the model behavior related to the word usage frequency. More rarely used words-> higher uncertainty of the model -> more diversity in output. 2. The Gibberish Experiment To test the rarity hypothesis, I replaced the creativity instruction with a nonsense token: “Create 1 idea of creative banner for performance marketing of the AI benchmarking platform. Be as hkdfgjdnfkfdfdbf as possible. ” The reasoning was straightforward: if models respond to statistical rarity of instruction patterns rather than semantic meaning , then a completely meaningless token should produce high diversity. If models respond to semantics, gibberish should produce low diversity. I also added a pure random prompt — the entire prompt replaced with random token sequences — as an extreme control. If a metric rates random noise as “creative,” that metric is broken. But this raised a deeper question. If gibberish or random noise drives scatter upward, how do I distinguish useful creativity from confused noise ? The existing scatter metric couldn’t tell the difference. I needed better metrics. 3. Why Scatter Is Not Enough The embedding scatter metric (97th percentile of pairwise cosine distances) measures how spread out responses are in embedding space. But high scatter can come from two very different sources: Genuine creative diversity — the model explores many distinct, on-topic ideas Off-topic drift — the model doesn’t understand the constraint, and outputs scatter randomly across unrelated regions of embedding space Both cases produce identical scatter scores. To separate them, I needed metrics that are relevance-aware — metrics that measure diversity among responses that actually address the task . 4. Three New Metrics Vendi Score (Standard) The Vendi Score (Friedman & Dieng, 2023) is defined as the exponential of the Shannon entropy of the eigenvalue distribution of the normalized similarity kernel: VS = exp(−Σ λᵢ log λᵢ) where λᵢ are eigenvalues of K/n, and Kᵢⱼ = cos(rᵢ, rⱼ) is the cosine similarity matrix of response embeddings. Unlike scatter (which measures raw spread), Vendi Score captures the effective number of distinct modes in the response set. A set of 30 responses clustered into 5 tight groups scores ~5. Thirty near-identical responses score ~1. The score is directly interpretable: “this configuration produces responses with ~X distinct ideas.” Relevance-Weighted Vendi Score (VS_rw) The key innovation. Instead of computing VS on raw response similarities, I weight the similarity kernel by each response’s relevance to the prompt: K’ᵢⱼ = Kᵢⱼ · wᵢ · wⱼ where wᵢ = cos(prompt, responseᵢ), clipped to [0, 1]. Off-topic responses get wᵢ ≈ 0, which smoothly suppresses their contribution to the kernel — no arbitrary threshold needed. The VS_rw then measures: “how many distinct on-topic ideas does the model produce?” This directly addresses the paradox: if “monotonous” prompts produce high scatter because responses drift off-topic, VS_rw will penalize that. If they produce high scatter because the model genuinely explores diverse on-topic interpretations, VS_rw will reward it. Diversity–Relevance AUC Inspired by precision–recall curves. Responses are sorted by relevance (closest to prompt first). At each fraction (10%, 20%, … 100%) of the most relevant responses, I compute the Vendi Score density (VS/k). The area under this curve summarizes: “does diversity survive when we only keep on-topic responses?” High AUC means creativity is maintained even under strict relevance filtration. Low AUC means diversity collapses when off-topic responses are removed. 5. Experiment Design I designed six prompt variants to stress-test the metrics across the full spectrum from semantic instruction to pure noise: Variant The base prompt for all variants except random: “Create 1 idea of creative banner for performance marketing of the AI benchmarking platform.” Suffix appended to base prompt Basic : No suffix Creative : “Be as creative as possible.” Uncreative: “Be as uncreative as possible.” Monotonous: “Be as monotonous as possible.” Gibberish: “Be as hkdfgjdnfkfdfdbf as possible.” Random: E ntire prompt replaced with random token sequences Models and sampling 14 models from 6 providers: OpenAI (GPT-5.4, 5.4-mini, 5.4-nano), Anthropic (Claude Opus 4.6, Sonnet 4.6, Haiku 4.5), Google (Gemini 2.5 Pro, Flash, Flash Lite), DeepSeek (v3.2), xAI (Grok 4.1 Fast, 4.20), Qwen (3.6-plus, 3.5-flash) 3 temperatures : 0, 0.5, 1.0 30 repetitions per (prompt × model × temperature) Embeddings : Qwen3 Embedding 8B (4096 dimensions) Total: 6 × 14 × 3 × 30 = 7,560 responses . 6. Results — Do the New Metrics Actually Work? Overall results by prompt variant Higher scatter/VS/VS_rw/DR_AUC = more diversity. Higher relevance_dist = further off-topic. Overall metrics by prompt The critical test: random prompt The random prompt is the litmus test. It replaces the entire prompt with meaningless tokens, so all responses should be off-topic noise. Rankings by metric: VS_rw correctly identifies random noise as the least creative configuration. Its exectly what I expected from the metric to avoid irrelevant answers. Scatter, standard VS, and DR_AUC all fail this test — they rank random noise as the most “creative” output which is inapplicable for real tasks. Correlation structure The most important number: VS_rw has near-zero correlation with relevance distance (r = −0.023) . This means VS_rw measures something genuinely independent of topic drift. Scatter, VS, and DR_AUC all correlate significantly with relevance (r = 0.45–0.55), meaning they partially measure off-topic wandering rather than genuine diversity. Correlation heatmap Pairwise metrics plots Why DR_AUC failed Despite its elegant design, DR_AUC produces the same ranking as scatter. The issue: at strict relevance fractions (top 10%), even random_prompt’s few on-topic responses show high VS density because the subset is tiny and happens to be diverse. The percentile-based sweep doesn’t penalize off-topic responses enough. DR_AUC may be useful as a diagnostic visualization but not as a standalone scalar metric. Verdict on the metrics 7. The Paradox Revisited Does the paradox survive under VS_rw? Yes — partially. Under VS_rw, the ranking becomes: monotonous > uncreative > gibberish > creative > basic > random The random prompt correctly drops to last place. But monotonous and uncreative prompts still outperform creative and basic prompts even under a relevance-aware metric. This is no longer a measurement artifact — it is a confirmed conclusion: less usual instructions produce more diverse on-topic ideas than standard creativity instructions. This strengthens the hypothesis from the previous experiment: models respond to the statistical rarity of instruction patterns , not their semantic meaning. The gibberish gradient The results reveal a clear gradient from semantics to noise: Near-meaningful gibberish where only one instructional word is obfuscated (“hkdfgjdnfkfdfdbf”) stays somewhat on-topic and produces genuine diversity. Pure random tokens drift completely off-topic. The transition from meaningful → near-meaningful → noise traces a clear path through the model’s instruction-following behavior. 8. Experiment Results — Creativity Leaderboard VS_rw by temperature Temperature consistently increases VS_rw across all prompt variants. The prompt ranking remains stable across temperatures — the effect is clear. VS_rw by prompt variant and temperature VS_rw across models VS_rw cross-prompt comparison by model Scatter across models (for comparison) Scatter cross-prompt comparison by model Response relevance across models Relevance cross-prompt comparison by model. Higher score -> lower relevance. Creativity leaderboard: Top and Bottom 14 by VS_rw (T=1.0) Key observations Monotonous prompt dominates the top across model families — Gemini, DeepSeek, Qwen, and Grok all show highest VS_rw with the monotonous instruction. Random prompt clusters at the bottom — every random_prompt entry sits in the bottom 15, regardless of model. VS_rw successfully filters this out. GPT-5.4 is the most deterministic model — it occupies the very bottom (#82) with the basic prompt and consistently shows low VS_rw across variants. Model families differ dramatically : Gemini Flash and DeepSeek v3.2 are the most creatively responsive to unusual instructions; GPT-5.4 and Claude Opus are the most deterministic. For most models, VS_rw demotes random_prompt from the top-3 and promotes gibberish_prompt relative to scatter. Scatter sees random noise as “diversity”; VS_rw correctly identifies that near-meaningful gibberish (which stays somewhat on-topic) is more creatively valuable than pure noise (which drifts off-topic). 9. Conclusions and Next Steps What we learned VS_rw is a genuine improvement over scatter as a creativity metric. It correctly identifies off-topic noise (random_prompt → last place) while preserving sensitivity to genuine creative diversity. The creative behavior for unusual words is confirmed. Even under relevance-weighted diversity, unusual instructions (“be monotonous”) outperform standard creativity instructions. Models respond to the statistical rarity of instruction patterns, not their semantic meaning. DR_AUC needs rethinking — it adds no discriminative power over scatter in its current form. Next steps VS_rw creativity leaderboard. Build a regularly updated leaderboard ranking models by VS_rw across standardized creative tasks. This gives practitioners a data-driven way to select models for creative applications. LLM-as-judge calibration. Run a study where LLM rate response relevance and creativity on the scale. Correlate ratings with VS_rw vs scatter to validate that VS_rw better aligns with perceived creativity. Multi-domain expansion. Replicate the experiment across domains where creativity matters differently: creative writing, support tasks, regular question topics. Test whether VS_rw rankings are stable across tasks. Expand to image generation. The VS_rw framework generalizes to any modality where responses can be embedded. Image generation models (DALL-E, Midjourney, Stable Diffusion, Flux) face the same creativity measurement problem. Embedding generated images and computing VS_rw would provide the first quantitative creativity metric for visual AI. References Experiment 1: From Prompt Guessing to Prompt Metrics Experiment 2: The Creativity Paradox — When Models Do Not Follow Words Vendi Score: Friedman & Dieng (2023), “The Vendi Score: A Diversity Evaluation Metric for Machine Learning” Beyond Scatter: Metrics That Allows to Measure Creativity in LLMs. was originally published in GoPenAI on Medium, where people are continuing the conversation by highlighting and responding to this story.