TalksHyderabad

From Traces to Verdicts: When Tracing Isn’t Enough for LLMs

Evaluation and failure detection for production AI systems, where request-level tracing tells you what happened but not whether it was right.

LLM observabilityEvaluationOpenTelemetryLangfuse
Varun speaking at a podium next to a slide asking who has set up observability for an LLM app
Opening with a show of hands: who has set up observability for an LLM app?
Varun presenting a Jupyter notebook showing the demo agent’s system prompt
Live demo, piece 1: the instructions that make the agent always sound certain.
Notebook on the projector showing the app model, judge model and tracing configuration
The setup: an app model, a separate judge model, and every call traced.

Abstract

Traditional monitoring does not catch most failures in LLM applications. A model can return a confident but wrong answer, or an agent can retry in a loop, and the request still succeeds. Tracing shows what happened during a request, but it does not tell you whether the result was correct.

The talk walks through turning raw LLM traces into verdicts: attaching evaluations to the traces you already collect, grading behaviour with a second model, and deciding what counts as a failure before it reaches users. The live demo used a customer-support agent whose instructions push it to always sound certain, an independent judge model, and Langfuse tracing every call.

What the talk covers

  1. 01Why tracing alone misses the failures that matter in LLM applications
  2. 02Pairing traces with evaluations so every request can carry a verdict
  3. 03Using a separate judge model, and where it can and can’t be trusted
  4. 04Keeping the setup vendor-neutral: OpenAI-compatible models and open-source tracing
← All talks