Abstract

The rapid proliferation of large language models (LLMs) has stimulated researchers to seek effective and efficient approaches to deal with LLM hallucinations and low-quality outputs. Uncertainty quantification (UQ) is a key element of machine learning applications in dealing with such challenges. However, research to date on UQ for LLMs has been fragmented in terms of techniques and evaluation methodologies. In this work, we address this issue by introducing a novel benchmark that implements a collection of state-of-the-art UQ baselines and offers an environment for controllable and consistent evaluation of novel UQ techniques over various text generation tasks. Our benchmark also supports the assessment of confidence normalization methods in terms of their ability to provide interpretable scores. Using our benchmark, we conduct a large-scale empirical investigation of UQ and normalization techniques across eleven tasks, identifying the most effective approaches.

Author Biography

Roman Vashurin

Research engineer at the NLP department.

Ekaterina Fadeeva

MSc student at HSE University

Artem Vazhentsev

PhD student, research engineer.

Lyudmila Rvanova

PhD student, research engineer.

Akim Tsvigun

MSc student at University of Amsterdam.

Daniil Vasilev

MSc student at HSE University.

Rui Xing

Research assistant at the NLP department of MBZUAI.

Abdelrahman “Boda” Sadallah

MSc student at Mohamed bin Zayed University of Artificial Intelligence: MBZUAI.

Sergey Petrakov

Independent researcher, PhD scholarship seeker.

Alexander Panchenko

Leading research scientist at AIRI.

Timothy Baldwin

Professor of the NLP department, Provost.

Preslav Nakov

Professor of the NLP department, the head of the department.

Maxim Panov

Assistant Professor of the Machine Learning department at MBZUAI.

Artem Shelmanov

Sr. Research Scientist at the NLP department of MBZUAI.