Building confidence in LLM outputs

A crucial aspect of machine learning systems is having confidence in the output of the models. Confidence enables us to trust that model outputs are reliable and to use those outputs in downstream applications and for decision-making. However, it is well known that large language models (LLMs) sometimes hallucinate the answers to questions, which creates uncertainty about their answers and their utility. This white paper discusses some of the approaches that can be used to elicit confidence in LLM outputs and discusses some of the ways we build confidence in model outputs at Alkymi.

Note: This white paper deals with the issue of confidence in LLM outputs. Confidence is one of the components of trust in machine learning. Other components include bias, stereotypes, ethics, privacy, confidence, and fairness (Wang et al., 2023). While this white paper only focuses on whether we can be confident that the answers provided by an LLM are factually correct, at Alkymi we acknowledge the importance of the other aspects of trust and continue to focus on improving trust of all types in our machine learning models.

Large language models (LLMs) have demonstrated their ability to complete complex natural language understanding tasks, e.g., translation and summarization, despite not explicitly being trained to perform those tasks (Radford et al., 2023; Wei et al., 2022). LLMs typically perform inference in 3 common settings:

1. Zero-shot learning, where the underlying model is not changed and the LLM is not provided with any representative examples. Despite this, LLMs have still demonstrated an impressive ability to produce high-quality answers in a zero-shot setting (Radford et al., 2023).
2. In-context learning, where the LLM is provided with a few representative examples of the task to be performed as part of the prompt. These representative examples guide the LLM in completing the current task, but the underlying model weights are not changed as they are when a model is fine-tuned.
3. Fine-tuning, where the underlying weights of a pre-trained LLM, such as Llama 2 (Touvron et al., 2023), are adapted through the use of additional training. In this way, the underlying model is changed to perform better at a specific task.

Despite the impressive performance of LLMs on many downstream tasks in each of these inference settings, there is still significant concern about the possibility of LLMs hallucinating answers to questions. This should occur less frequently in the fine-tuning and in-context learning settings as the model is more familiar with the downstream tasks; however, there is no guarantee that hallucinations will not happen. Previous research has identified 3 main types of hallucination for LLMs (Zhang et al., 2023):

1. Input-conflicting hallucinations, where LLM output conflicts with user input.
2. Context-conflicting hallucinations, where LLM output conflicts with something that it has previously output (a lack of internal consistency).
3. Fact-conflicting hallucinations, where LLM output conflicts with facts about the real world.

Hallucination is an important and active research area for LLMs (Zhang et al., 2023). At Alkymi, we apply several measures to minimize the risk of hallucinations when using our LLM-powered tools. The main issue that arises due to hallucinations is that they degrade trust in LLM outputs. In addition to hallucinations, LLMs are black boxes, where we don’t fully understand their internal working and reasoning, and thus it can be challenging to explain how they arrive at a given answer.

Building confidence in LLM output is important as it allows us to trust them, despite their black box nature and the fact that they occasionally hallucinate.

There are different ways that one could build confidence in LLM outputs. In this white paper, we discuss three: model explainability, confidence scoring, and attribution. We then discuss some practical implications and describe how we build confidence in LLM-powered tools at Alkymi.

Model explainability

Model explainability in machine learning refers to mechanisms that can be used to explain model outputs to build trust and confidence (1) (Zhao et al., 2023). There has been significant research on explainability in machine learning and natural language processing (Zhao et al., 2023; Zini & Awad, 2022; Danilevsky et al., 2020), and explainability is crucial when using machine learning in domains, such as medicine, law, and finance, where being able to document the reasons for decisions is important.

Explainability in NLP can act as a debugging aid when analyzing LLM output and can also be used to measure improvements in models (Zhao et al., 2023). Some traditional machine learning models often make use of hand-designed features, such as decision trees and logistic regression models. These models can sometimes be explained by analyzing split criteria or feature weights, which can provide insight into the importance of certain features in model decisions. However, LLMs and deep neural networks typically do not use hand-crafted features but learn them automatically from the data, thus making them harder to interpret and explain. This has led to the development of other explanation methods that attempt to pry open the black box, such as perturbation methods, which slightly modify the model input and observe what happens to the output; or by visualizing model attention (Vig, 2023). Other approaches make use of surrogate models, which are simpler and more explainable models that attempt to emulate the decisions of complex large models (Zhao et al., 2023).

The key idea behind model explainability is that the goal is to explain how a model came to an answer.

Confidence scores

An alternative to model explainability is confidence scoring. In this approach, the goal is to produce a score that quantifies the extent to which you should trust the LLM’s output. When the model confidence is high then the answer can be trusted, and when the model confidence is low, further investigation is warranted. A requirement for this, however, is that model confidence is well-calibrated with model accuracy (Wightman et al., 2023) so that model confidence can be trusted as a predictor of answer quality.

One way to measure model confidence is with output token probabilities; however, this is an unreliable predictor of confidence for LLMs, especially closed-source LLMs (Xiong et al., 2023), thus necessitating the need for other approaches. Another approach involves training LLMs to generate natural language explanations for their reasoning (Rajani et al., 2019). This approach might not only help with explainability but could potentially also reduce hallucinations because the models are required to give explanations, which requires them to be more truthful.

Given the ability of LLMs to perform well at a variety of tasks, one might wonder if it’s possible to have an LLM explain its output. Lin et al. (2022) demonstrated that GPT-3 was capable of reliably demonstrating its confidence for math problems, though its performance on more generic question answering (for instance, over financial documents), was left as an open research question.

Finally, other approaches focus on prompting models multiple times with variants of a prompt and using the similarities among responses as an estimator of confidence (Wightman et al., 2023).

Attribution

Attribution for LLMs is related to explainability; however, instead of focusing on why an LLM made a decision based on its underlying beliefs, the task instead is to provide the source for the decision (Bohnet et al., 2023). In the context of the document Question Answering (QA), the requirement for attribution is that the LLM shows where in a document (or in a collection of documents) it found an answer. Attribution for QA is challenging for several reasons. For instance, there could be multiple sources in a document that answer a specific question. This makes automatically evaluating the attribution capabilities of LLMs challenging, because it usually requires one to identify all possible sources and label them appropriately to perform a robust evaluation (2). Human evaluation of an answer and its source(s) has been proposed as the gold standard for evaluation, since it only requires a human to answer the question of whether the answer is supported by the source (Bohnet et al., 2023). Automatic methods have also been proposed, whereby another LLM is used to score whether the answer to a question is supported by a source (Bohnet et al., 2023). Another challenge for Attribution for QA is when an answer is a synthesis of information from multiple sources. In this case, all of those sources need to be identified and cited appropriately (Bohnet et al., 2023).

There are several methods proposed for attribution in LLMs. One method is known as retrieve-then-read. In this approach, a retrieval system retrieves passages in a database for a given question and then ranks them. An answer is generated from the highest ranked passage and, because the passage that generated the answer is known, that answer can be attributed to that passage (Izacard et al., 2022). This approach can be used with Retrieval Augmented Generation systems, but the utility may be dependent on the size of the chunks retrieved (3). An alternative to retrieve-then-read is post-hoc retrieval. In this setting, the answer to a question is first generated, and then a retrieval system is used to retrieve supporting facts. Another option is the LLM-as-a-retriever approach, where an LLM is trained to answer questions and perform retrieval, which can be used for attribution (Tay et al., 2023). A full discussion and evaluation of these different approaches is available in Bohnet et al. (2023).

The main goal of attribution is to build confidence by allowing an LLM user to see the support for an answer without needing to check it independently (Bohnet et al., 2023). Attribution also can be used for auditing purposes.

(1) In some contexts, explainability refers to all mechanisms that can be used to describe the reasons for model outputs. In this white paper, we use explainability to refer only to the underlying model mechanisms that can be used to explain outputs, e.g., weights, features, etc.

(2) Typically for automatic evaluation of LLM attribution, you would want to label all possible sources for answering a question. Then, when performing automatic evaluation you would assess the extent to which all of those sources were identified.

(3) Full page sources may be less useful than smaller paragraphs for attribution purposes.

Having confidence in the outputs produced by LLMs is important, especially in financial document processing, where having an answer may not be sufficient unless the reason for the LLM producing that answer is provided. As described above, there is a large amount of research into building confidence in LLM outputs, including methods for explaining underlying model mechanisms and providing attribution for answers.

Many LLM applications require that confidence information is available at run time. Additionally, confidence information should be easily accessible and interpretable as part of the application user interface. In many cases, these requirements rule out model explainability as a method for eliciting model confidence since it usually requires one to probe the underlying model mechanisms and the interpretation may not be straightforward. Therefore, model confidence scores and attribution may be more appropriate for eliciting model confidence in LLM applications.

For model confidence scores and attribution, the reasons for the model confidence must be interpretable to the user. For model confidence scores, this means providing evidence that the model scores are well-calibrated (Wightman et al., 2023) and that models are not acting over-confident when wrong (Feng et al., 2018). In many cases, this may mean providing a natural language description for the model confidence. In practice, attribution methods have the potential to be the most useful as they have the benefit of linking directly to sources, which allows users to validate the output in real-time.