View PDF HTML (experimental)

Abstract:Deploying natural language search systems presents a critical cold-start challenge: no real user queries to learn linguistic patterns, and no relevance labels to train ranking models. We present CASTLE (Contrastive And Seed-guided Training for natural Language sEarch), an LLM-based framework for generating synthetic queries and relevance labels from structured catalog data, powering Airbnb's natural language search across its full lifecycle.
CASTLE makes three contributions. First, we generate realistic queries by combining structure-guided prompting with seed queries from user research, using template, few-shot, and attribute-grounded prompt variants together with explicit variety mechanisms to prevent query collapse. Second, we produce relevance labels by construction via contrastive listing pairs derived from booking sessions, achieving near-zero false positives without LLM judgment. Third, CASTLE's structured input design is flexible: incorporating richer signals such as guest reviews and photo captions alongside listing attributes enables generation of niche, long-tail queries that reflect subjective user preferences (e.g., "cozy cabin with fireplace") beyond what catalog attributes alone can express.
Compared against InPars-style, Promptagator, and contrastive-only baselines, CASTLE achieves KL 1.01 vs. real users -- a 9.2x improvement over the best baseline (9.33) -- and the lowest attribute-type KL divergence (0.08), outperforming even survey seed queries (0.09). A human evaluation on 200 sampled triplets confirms label quality: annotators agree with CASTLE labels at 91-93%. We deploy production pipelines generating synthetic examples daily for embedding-based retrieval and ranking evaluation. Synthetic data remains valuable beyond cold-start: it targets tail queries underrepresented in organic traffic and extends naturally to multi-turn conversational search.

Submission history

From: Wendy Ran Wei [view email]
[v1] Wed, 20 May 2026 23:18:49 UTC (237 KB)
[v2] Sat, 29 Aug 2026 07:25:41 UTC (331 KB)