A prototype runs on your laptop. You see every input, every output, every error. Production runs at scale, often at night, handling inputs you never imagined. Without observability, you are flying blind.
Observability for LLM systems is not the same as logging for a web application. The inputs are unstructured, the outputs are probabilistic, and the failure modes are subtle — a model that returns plausible but wrong answers looks healthy to every traditional monitoring tool.
What to trace
Every LLM call should capture the full prompt (with template variables resolved), the raw response, latency, token count, and cost. For retrieval-augmented systems, capture the query, the retrieved documents, and the relevance scores. For multi-step chains, capture each step with its inputs and outputs.
This is not optional instrumentation to add later. It is the foundation that makes every other improvement possible.
Dashboards that matter
Three dashboards cover most production needs: a real-time view of throughput, latency, and error rate; a cost dashboard showing spend by model, by feature, and over time; and a quality dashboard driven by your evaluation suite, showing score distributions and trends.
The goal is not to collect data. The goal is to make the system's behavior legible to the humans responsible for it.
The payoff
With observability in place, debugging goes from hours to minutes. Cost anomalies surface before the invoice arrives. And when someone asks "is the system getting better?" you have a number, not a feeling.