Abstract:Natural Language Interfaces for Databases (NLIDBs) let users query data by asking questions in everyday language instead of writing SQL, and recent systems translate those questions accurately. Accuracy says little about the cognitive effort of querying: does an NLIDB reduce that effort, or only relocate it? We report a mixed-method, between-subjects user study comparing SQL-LLM, a GPT-4o-backed NLIDB, with Snowflake, a traditional SQL analytics platform. Twenty SQL-literate professionals and graduate students (70% intermediate, 30% advanced), ten per interface, each completed 12 querying tasks drawn from BIRD, a dataset of realistic natural-language questions mapped to SQL. Behavioral coding of the 11 recorded think-aloud sessions showed where the effort went: SQL-LLM users handed schema navigation to the model and spent their effort verifying that the generated SQL matched what they meant, while Snowflake users spent comparable effort exploring the schema and constructing syntax by hand. Frustration episodes and reformulation counts were similar across groups. The interface changes the kind of work, not the amount. It does move the clock: a linear mixed-effects model controlling for query difficulty and database found SQL-LLM users 212 s faster per query on average (p = 0.044). The speed gain, however, did not buy accuracy: graded against the BIRD gold answers, SQL-LLM users were correct on 46% of queries versus 64% for Snowflake, a difference favoring Snowflake that did not reach significance (p = 0.06). For our SQL-literate participants, a verification burden survives the switch to natural language, so an NLIDB that hides the generated SQL would remove the very step these users relied on to trust the answer.
From: Panos Ipeirotis [view email]
[v1]
Tue, 18 Nov 2025 18:04:24 UTC (630 KB)
[v2]
Sun, 19 Jul 2026 04:55:06 UTC (597 KB)
此内容由惯性聚合(RSS阅读器)自动聚合整理，仅供阅读参考。 原文来自 — 版权归原作者所有。