Academic researchers using commercial AI tools for literature review and idea generation are sending unpublished research questions, draft hypotheses, and proprietary domain knowledge into systems whose data handling they do not understand.

A think-aloud study of 15 researchers documents the workarounds these users have built to manage what they see as unresolved confidentiality and output verification problems in tools including Research Rabbit and Elicit AI.

The study, conducted by researchers at the University of Texas at Austin and Microsoft, observed participants in real-time as they completed literature exploration, synthesis, and ideation tasks. These gaps map closely onto concerns familiar to enterprise security functions managing employee use of generative AI.

Prompt content treated as a disclosure vector

Two of the 15 participants raised direct concerns about the confidentiality of prompt content. One participant said AI platforms “will leverage the prompt you share for training, which has the potential to leak your research question or research data.” Another cited “not knowing how much of my personal data is being stored, where it is being stored, and who has access to it.”

The number is small, but the underlying behavior was widespread across the sample. Participants routinely entered draft research questions, descriptions of work in progress, and unpublished analytical framings into the tools. The study describes this as an institutional answerability problem, where end users have no visible forum through which AI vendors can be held responsible for collected, stored, or repurposed inputs.

For organizations governing employee AI use, the parallel is direct. Staff who paste internal documents, code, or strategic plans into commercial LLMs are exposed to the same opacity around retention, training reuse, and access controls.

Output verification gaps drive heavy manual review

Nine of the 15 participants reported difficulty establishing where AI-generated content came from. Retrieval pipelines, training data coverage, and curation logic were opaque, making it impossible to confirm sources. One participant described the black-box nature of the tools as a limitation for rigorous work, since sources and underlying data can’t be reported with certainty.

Seven participants treated hallucinations as a transparency failure rather than a discrete accuracy issue. The study identifies two failure modes. Attribution displacement occurs when accurate information is tied to the wrong source. Synthetic blending integrates fabricated claims alongside legitimate citations in a single output, making verification slow and error-prone.

One researcher described challenging ChatGPT about a non-existent citation and receiving an apology followed by more fabricated references. The same researcher noted a separate failure mode: citations that exist but have no connection to the topic. The study calls this a provenance problem distinct from outright fabrication.

To compensate, all 15 participants developed mitigation strategies, including social credibility heuristics such as recognizing author names or publication venues. Eight defaulted to redundant manual verification, repeatedly checking names, dates, and citations. Ten restricted AI use to low-stakes tasks and kept core analytical work outside the tools.

Implications for enterprise AI governance

The compensatory strategies documented in the study consume time and depend on domain expertise that newer staff may lack. The authors note that early-career researchers are more vulnerable to being misled by confidently stated yet poorly grounded outputs, since they have less baseline knowledge against which to calibrate.

The same dynamic appears in corporate environments where employees use LLMs for tasks outside their expertise. Confident output combined with opaque sourcing creates conditions where errors propagate without detection.

The authors recommend slower, more measured AI adoption supported by verification pipelines, metadata exposure, and clearer data governance disclosures from vendors. The study’s limitations include its small sample, an academic-only participant pool, and the fact that both tools studied have been updated since data collection. The authors call for longer-term, naturalistic research to track how user practices and vendor policies develop.

Read more:

• Even cybersecurity researchers are exposing secrets in their arXiv LaTeX source
• Indirect prompt injection is taking hold in the wild