When a model starts behaving differently, the first question is always what changed. Without a release record, the answer may be scattered across a data scientist’s laptop, a prompt edited in a console and a library upgrade nobody connected to the problem. A release record that pins data, code, configuration and scores turns that investigation from days into minutes.
For LLM-based features the same discipline applies to prompts, retrieval settings and the provider’s model version, which can change underneath you if it is not pinned.
Uptime is not quality
A model endpoint can return fast, error-free responses that are steadily getting worse. Traditional service monitoring will show green throughout. We add quality signals: shifts in input data, rising user corrections, falling scores from automated graders and, where they exist, real outcome labels. Each signal has an owner and a documented first response.
Investigating drift without guessing
Not every distribution change is a problem. Seasonal patterns, a new marketing campaign or a broken upstream feed can all look like drift. We separate missing or malformed inputs, genuine population shifts and real outcome degradation, because each needs a different fix and only some justify retraining.
Counting the full cost
The price of a model in production is more than its compute bill. Retries, human review of uncertain outputs and rejected results all cost money. Tracking cost per useful result shows whether a cheaper model with slightly lower accuracy is actually more expensive once review effort is included, or the reverse.
Rollback as a rehearsed procedure
Reverting to the previous model sounds simple until the feature schema, retrieval index or database has moved on since that version shipped. We check compatibility as part of every release and practise rollback on staging, so on the day it matters your team follows a procedure they have already run.
Rehearsal also exposes slow steps. If restoring an older embedding index takes hours, or a previous prompt depends on a tool definition that no longer exists, it is far better to learn that during a planned exercise than during a customer-facing incident. We record how long each recovery step took so your on-call engineers know what to expect and can decide early whether to roll back or hold.