Detecting LLM Hallucinations with Semantic Entropy
You cannot prevent an LLM from hallucinating. It’s not a bug in the Transformer architecture, but a mathematical consequence of probabilistic generative modeling, an innate limitation we need to live with.
However, there are techniques we can use in our applications to prevent LLM hallucinations from getting to the users. Retrieval-augmented Generation (RAG) is one of the most well-known mechanisms we can use to have responses grounded in specific domain knowledge.
In this article, I will discuss another technique that uses “semantic entropy” to detect when an LLM is not completely sure about the generated response.
The problem of hallucination
The paper “Why Calibrated Language Models Must Hallucinate”, published in 2024 by Adam Tauman Kalai and Santosh S. Vempala demonstrates that AI hallucinations are an inherent statistical byproduct of pretraining language models for predictive text accuracy. There is a mathematical lower bound on the hallucination rate for arbitrary facts (such as obscure personal details). We can use post-training alignment techniques (such as RLHF/PPO) to suppress hallucination rates, but they do so by deliberately degrading the model’s statistical calibration, hurting its performance.
Also in 2024, Sourav Banerjee, Ayushi Agarwal and Saloni Singla published “LLMs Will Always Hallucinate, and We Need to Live With This”. They assert that AI hallucinations are not temporary engineering flaws, but rather “Structural Hallucinations”, an ineliminable mathematical reality rooted in computational theory and formal logic, and formally prove that every stage of the LLM pipeline is susceptible to hallucination.
“Hallucination is Inevitable: An Innate Limitation of Large Language Models”, published in the same year by Ziwei Xu, Sanjay Jain, and Mohan Kankanhalli, formally proves that hallucinations cannot be eliminated from large language models regardless of architecture, training data, or prompting strategies.
It is clear that we need to live with hallucinations, and accept that they are an intrinsic behavior of LLMs. But large language models are never exposed directly to users, just like databases aren’t either. We build software systems around them that sanitize user inputs, handle access and implement business logic. In agentic contexts we call these systems “the harness around the model”, and we need to add safeguards in them to prevent model hallucinations from reaching the user.
As we will see in this article, we can use the concept of “semantic entropy” to detect when the certainty of the model is too low, indicating that its answer is probably a hallucination.
What is Semantic Entropy?
In 2024, Sebastian Farquhar, Jannik Kossen, Lorenz Kuhn and Yarin Gal published in Nature a paper titled “Detecting hallucinations in large language models using semantic entropy” where they introduce a very powerful method to detect hallucinations (confabulations) that does not depend on any domain knowledge or specific data set. It’s a purely mathematical verification that works in any context or application.
The original companion code can be found in Kossen’s GitHub repository, here and here. I have created a simplified version for this article that you can find here.
The paper explains how when LLMs don’t know the answer to a question, they tend to invent answers that make sense but are completely arbitrary. If you ask a model several times using different values of temperature, it will give conceptually different answers (for example: sometimes “Paris”, other times “Rome” or “Berlin”).
We cannot use traditional token entropy (the authors use the term naive entropy) because the same idea can be expressed in multiple ways. To the question “Where is the Eiffel Tower?” the model can generate:
- “Paris”
- “It’s in Paris”
- “It’s in France, in Paris”
Token entropy will say that the model is highly uncertain because the three answers are very different. However, it is absolutely sure of the meaning.
Semantic entropy is not calculated on the tokens, but rather on the meanings. We cluster equivalent answers and then calculate entropy on the clusters:
- If 100% of the answers fall in the “Paris” cluster, the semantic entropy is 0 (high certainty, no confabulation).
- If answers are scattered among “Paris”, “Rome” and “Tokyo”, then the semantic entropy will be very high (the model is guessing, confabulating).
Mathematical definition
LLMs generate text in an autoregressive manner, one token at a time. For a given context $x$, the probability of a specific sequence $s = (s_1, s_2, \dots, s_N)$ is:
$$P(s \mid x) = \prod_{i=1}^N P(s_i \mid s_{<i}, x)$$
To avoid penalizing very long sequences, the authors normalize the probability by getting the average of the log-probabilities:
$$\frac{1}{N} \sum_{i=1}^N \log P(s_i \mid s_{<i}, x)$$
If we partition the response space into a set of meanings $C$, then the theoretical probability of a given meaning $c$ given a context $x$ will be the sum of the probabilities of all the sequences that express the same meaning:
$$P(c \mid x) = \sum_{s \in c} P(s \mid x)$$
We can use the Shannon entropy formula to calculate the semantic entropy for a given context $x$:
$$SE(x) = - \sum_{c \in C} P(c \mid x) \log P(c \mid x)$$
Since we cannot evaluate infinite sequences, we take $M$ samples $[s^{(1)}, \dots, s^{(M)}]$ and group them into $\vert{}C\vert{}$ discrete semantic clusters, so we have:
$$SE(x) \approx - \sum_{k=1}^{\vert{}C\vert{}} P(C_k \mid x) \log P(C_k \mid x)$$
where the normalized probability for cluster $k$ is
$$P(C_k \mid x) = \frac{\sum_{s \in c_k} P(s \mid x)}{\sum_{c \in C} \sum_{s \in c} P(s \mid x)}$$
The challenge is we typically don’t know these probabilities unless we have access to the internal token probabilities. The authors propose using discrete semantic entropy as a good estimator, and prove in the paper that it produces almost equivalent results:
$$P(C_k \mid x) \approx \frac{\vert{}c_k\vert{}}{M}$$
where $\vert{}c_k\vert{}$ is simply the number of samples that belong in cluster $k$.
Implementation
The algorithm proposed in the paper is as follows:
- Generate the first answer $s^{(1)}$ and initialize the first cluster $C = { {s^{(1)}} }$.
- Generate $M - 1$ new answers to the same question, with different values of temperature.
- For each answer $s^{(m)}$ (where $m = 2, \dots, M$), compare the answer $s^{(m)}$ with the first element of each cluster $s^{(c)}$ using a Natural Language Inference model to calculate if they follow each other:
- $s^{(c)} \implies s^{(m)}$ (is $s^{(m)}$ a consequence of $s^{(c)}$)?
- $s^{(m)} \implies s^{(c)}$ (is $s^{(c)}$ a consequence of $s^{(m)}$?)
- If they are both true, then add $s^{(m)}$ to the cluster
- If $s^{(m)}$ cannot be added to any cluster, create a new cluster.
nli.py contains utility functions to check entailment and calculate the semantic entropy value:
from collections import Counter
import math
from typing import Sequence
import torch
from transformers import AutoModelForSequenceClassification, AutoTokenizer
MODEL_NAME = "microsoft/deberta-large-mnli"
ENTAILMENT_LABEL_ID = 2 # In MNLI, label index 2 corresponds to 'ENTAILMENT'
device = "cuda" if torch.cuda.is_available() else "cpu"
tokenizer = AutoTokenizer.from_pretrained(MODEL_NAME)
model = AutoModelForSequenceClassification.from_pretrained(MODEL_NAME).to(device)
model.eval()
def check_entailment(premise: str, hypothesis: str) -> bool:
"""Return True if premise entails hypothesis."""
if premise.strip() == hypothesis.strip():
return True
inputs = tokenizer(
premise, hypothesis, return_tensors="pt", truncation=True, max_length=512
).to(device)
with torch.no_grad():
logits = model(**inputs).logits
predicted_class = torch.argmax(logits, dim=-1).item()
return predicted_class == ENTAILMENT_LABEL_ID
def are_equivalent(text_a: str, text_b: str) -> bool:
"""Return True if both texts bidirectionally entail each other."""
return check_entailment(text_a, text_b) and check_entailment(text_b, text_a)
def get_semantic_ids(texts: Sequence[str]) -> list[int]:
"""Assign a cluster ID to each text based on bidirectional entailment."""
cluster_representatives: list[str] = []
semantic_ids: list[int] = []
for text in texts:
# Check if text matches an existing cluster's representative
assigned_id = None
for cluster_id, representative in enumerate(cluster_representatives):
if are_equivalent(text, representative):
assigned_id = cluster_id
break
# If it doesn't match any existing cluster, start a new one
if assigned_id is None:
assigned_id = len(cluster_representatives)
cluster_representatives.append(text)
semantic_ids.append(assigned_id)
return semantic_ids
def cluster_strings(texts: Sequence[str]) -> list[list[str]]:
"""Group texts into clusters of semantically equivalent meanings."""
ids = get_semantic_ids(texts)
clusters: dict[int, list[str]] = {}
for text, cluster_id in zip(texts, ids):
clusters.setdefault(cluster_id, []).append(text)
return list(clusters.values())
def calculate_semantic_entropy(semantic_ids: Sequence[int]) -> float:
if not semantic_ids:
return 0.0
total = len(semantic_ids)
counts = Counter(semantic_ids)
entropy = 0.0
for count in counts.values():
p = count / total
if p > 0.0:
entropy -= p * math.log(p)
return float(entropy)
Then, the semantic_entropy.py file defines the main function (comments and print calls removed for clarity):
def main(
question: str | None = None,
temperatures: Sequence[float] = (0.2, 0.5, 0.7, 0.9, 1.0),
) -> tuple[str, float]:
# 1. Ask initial question (at low temperature to obtain the primary response)
primary_response = ask_llm(question, temperature=0.1)
# 2. Repeat question across different temperature values
sampled_answers: list[str] = [primary_response]
for temp in temperatures:
ans = ask_llm(question, temperature=temp)
sampled_answers.append(ans)
# 3. Cluster answers using NLI (bidirectional entailment)
semantic_ids = get_semantic_ids(sampled_answers)
clusters: dict[int, list[str]] = {}
for ans, cluster_id in zip(sampled_answers, semantic_ids):
clusters.setdefault(cluster_id, []).append(ans)
# 4. Calculate semantic entropy
total_answers = len(sampled_answers)
counts = Counter(semantic_ids)
for cluster_id, count in sorted(counts.items()):
prob = count / total_answers
print(f" P(Cluster {cluster_id}) = {count}/{total_answers} = {prob:.3f}")
entropy = calculate_semantic_entropy(semantic_ids)
if entropy < 0.3:
print(" Confidence Interpretation: LOW ENTROPY.")
print(" -> High semantic consensus across temperatures (likely factual / correct).")
else:
print(" Confidence Interpretation: HIGH ENTROPY.")
print(" -> Semantic divergence across temperatures (likely hallucination / confabulation).")
return primary_response, entropy
With this approach, we can choose an entropy threshold and prevent the final response from getting to the user if there is not enough confidence.
Use the following command to execute the code:
uv run python -m semantic_entropy "<your question>"
For example, if we use the question “What is the capital of Japan?”, we will get a similar output to this:
======================================================================
DEMONSTRATION: SEMANTIC ENTROPY FOR HALLUCINATION DETECTION
======================================================================
[Step 1] Input Question:
"What is the capital of Japan?"
[Step 2] Asking primary question (temperature=0.1)...
Primary Response: "Tokyo is the capital of Japan."
[Step 3] Repeating question with 5 different temperature values:
- Temp 0.1 (primary): "Tokyo is the capital of Japan."
- Temp 0.2: "Tokyo is the capital of Japan."
- Temp 0.5: "Tokyo is the capital of Japan."
- Temp 0.7: "Tokyo is the capital of Japan."
- Temp 0.9: "Tokyo is the capital of Japan."
- Temp 1.0: "Tokyo is the capital of Japan."
[Step 4] Clustering 6 answers by semantic equivalence using NLI...
Cluster 0 (6/6 answers):
* "Tokyo is the capital of Japan."
* "Tokyo is the capital of Japan."
* "Tokyo is the capital of Japan."
* "Tokyo is the capital of Japan."
* "Tokyo is the capital of Japan."
* "Tokyo is the capital of Japan."
[Step 5] Calculating Semantic Entropy (discrete / black-box):
P(Cluster 0) = 6/6 = 1.000
Calculated Semantic Entropy: 0.0000 nats
Confidence Interpretation: LOW ENTROPY.
-> High semantic consensus across temperatures (likely factual / correct).
Whereas “What is the capital of Narnia?” will produce an output similar to this:
======================================================================
DEMONSTRATION: SEMANTIC ENTROPY FOR HALLUCINATION DETECTION
======================================================================
[Step 1] Input Question:
"What is the capital of Narnia?"
[Step 2] Asking primary question (temperature=0.1)...
Primary Response: "The capital of Narnia is the fictional city of Narnia itself."
[Step 3] Repeating question with 5 different temperature values:
- Temp 0.1 (primary): "The capital of Narnia is the fictional city of Narnia itself."
- Temp 0.2: "The capital of Narnia is Narn, located in the fictional kingdom of the Narnia region in the book *The Lion, the Witch and the Wardrobe*."
- Temp 0.5: "The capital of Narnia is not a city but a fictional island in the middle of the ocean."
- Temp 0.7: "The capital of Narnia is Bogsfield."
- Temp 0.9: "Narnia is a fictional series of books by C.S. Lewis and J.R.R. Tolkien, so it has no real-world capital."
- Temp 1.0: "The capital of Narnia is Ludgate Circus."
[Step 4] Clustering 6 answers by semantic equivalence using NLI...
Cluster 0 (1/6 answers):
* "The capital of Narnia is the fictional city of Narnia itself."
Cluster 1 (1/6 answers):
* "The capital of Narnia is Narn, located in the fictional kingdom of the Narnia region in the book *The Lion, the Witch and the Wardrobe*."
Cluster 2 (1/6 answers):
* "The capital of Narnia is not a city but a fictional island in the middle of the ocean."
Cluster 3 (1/6 answers):
* "The capital of Narnia is Bogsfield."
Cluster 4 (1/6 answers):
* "Narnia is a fictional series of books by C.S. Lewis and J.R.R. Tolkien, so it has no real-world capital."
Cluster 5 (1/6 answers):
* "The capital of Narnia is Ludgate Circus."
[Step 5] Calculating Semantic Entropy (discrete / black-box):
P(Cluster 0) = 1/6 = 0.167
P(Cluster 1) = 1/6 = 0.167
P(Cluster 2) = 1/6 = 0.167
P(Cluster 3) = 1/6 = 0.167
P(Cluster 4) = 1/6 = 0.167
P(Cluster 5) = 1/6 = 0.167
Calculated Semantic Entropy: 1.7918 nats
Confidence Interpretation: HIGH ENTROPY.
-> Semantic divergence across temperatures (likely hallucination / confabulation).
======================================================================
Limitations
The main downside of this technique is the impact on latency. We need to make several calls to the LLM for each question, then compute the semantic clustering by comparing all the answers. In the worst case, the time complexity will be $\mathcal{O}(M^2)$, where $M$ is the number of calls we make to the LLM.
The performance impact is even worse when working with long texts or biographies, if we extract factoids and generate multiple questions that produce several answers for each one.
There are some things we can do to mitigate this impact:
-
Parallelized generation: Since all the $M$ answers are generated from the same initial context with different temperatures, if the infrastructure supports it we can use batch inference to have a latency similar to a single call.
-
Use smaller values of $M$. Depending on the target audience and the type of application, you can reach a compromise between computational cost and reliability by reducing the number of LLM calls.
In any case, this method is not designed for casual chatbots that require millisecond-rate streaming, but for scenarios where factual accuracy is critical:
-
Asynchronous or offline processing: clinical summary generation, extracting structured data from legal documents, or financial reports. In these contexts, waiting a few additional seconds is a negligible cost compared to the cost of a medical or legal hallucination.
-
Selective triggering: Instead of calculating semantic entropy on every turn, implement a light signal (like a quick classification, or entropy on the first tokens only) to detect high uncertainty.
Conclusion
I am on a journey to learn about modern AI tools and the principles that underpin them. Years ago, I wrote about Deep Learning and built a simple artificial neural network in TypeScript, to understand how these models are trained and how the gradient descent algorithm works. More recently, I wrote about retrieval-augmented generation and built a multi-agent system to introduce myself to LLMs and how we can make sure their responses are grounded in factual data when working on a specific domain.
In this article, I’ve examined how pure mathematical approaches can also be used to detect model hallucinations and prevent them from reaching users, even when we don’t have an external data store with domain knowledge.
Thanks for reading!