Skip to content
All posts
·3 min read

Prompt Risk Density: measuring hallucination risk before the model answers

LLM ReliabilityHallucination Mitigation

Most work on LLM hallucination targets the model: better training data, improved architectures, constrained decoding. All of that matters, but it quietly assumes the prompt is a fixed input we can't do anything about.

That assumption is wrong. Every LLM output has two actors, the model and the user. The prompt is the one input surface the user fully controls, and it has a real influence on whether the model ends up fabricating. This was the starting point of my bachelor's thesis at TUM, Echo, and it's what "shift-left" means in this context: catch the risk before generation, not after.

Two kinds of risk hiding in a prompt

The first thing Echo needed was a vocabulary. When you look at prompts that reliably produce hallucinations, the problems fall into two distinct categories:

Prompt Risk lives at the token level: ambiguous phrasing, vague references, presuppositions that smuggle in unverified claims. The classic example is a question that asserts something false in passing ("Why did X acquire Y in 2021?") and invites the model to build on it.

Meta Risk is structural: multi-hop questions that chain several inferences, scope overload where one prompt asks for five deliverables, constraints that quietly conflict with each other. None of these are wrong at the word level. The prompt is just shaped in a way that pushes the model past what it can answer reliably.

The distinction matters because the fixes are different. Prompt Risk you resolve by clarifying tokens. Meta Risk you resolve by restructuring or splitting the request.

Making it a number

A taxonomy is useful, but "this prompt feels risky" doesn't scale. So Echo introduces Prompt Risk Density (PRD): a weighted score across the risk categories, normalized by prompt complexity, that quantifies hallucination potential before any generation happens.

Normalization is the important part. A long, detailed prompt naturally contains more flaggable spans than a short one; what you care about is the density of risk relative to how much the prompt is trying to do, not the raw count.

The pipeline

Echo implements this as a multi-agent pipeline with four roles: an Analyzer that detects and classifies risks, an Initiator and Conversation agent that drive iterative refinement with the user, and a Preparator that assembles the improved prompt. Outputs are structured XML, and the UI renders risk color-coded on the prompt text, so you can see exactly which spans are the problem rather than getting a generic "your prompt is unclear" back.

Human-in-the-loop was a deliberate choice. Fully automatic rewriting changes what the user meant often enough to be dangerous; the refinement loop keeps the user in control of intent while the system handles risk detection.

What I took away from it

Two things stuck with me beyond the thesis itself.

First, catching risk before generation is cheaper than fixing bad outputs after. Post-hoc verification of a hallucinated answer means you already paid for the generation, and you now need a second system to check it. Prompt-side analysis runs on a much smaller input, before the damage.

Second, better prompts narrow the gap between model tiers. In the thesis evaluation, refined prompts helped smaller open-source models close some of the distance to expensive closed-source ones. For high-stakes domains like law, healthcare, and finance, where deployment constraints often rule out the biggest models anyway, that's a practical result, not just an academic one.

The thesis, source code, and a demo are on the research page. If you're working on LLM reliability and want to compare notes, get in touch.

Thoughts on this? I usually discuss posts on LinkedIn.

This website can use privacy-friendly analytics to improve content quality. No advertising cookies are used.

You can change this anytime in the privacy page. Read more in Privacy Policy.