Model drift, configuration drift and cost drift all show up quietly, until someone asks why. Here is how we pin, roll back and keep spend visible before that happens.
Three kinds of drift take down agents in production, and none of them announce themselves. The model changes underneath you, a configuration value in one environment quietly diverges from another, or the monthly bill arrives and nobody can say why. All three get the same answer: pin what should be pinned, and make "it just changed" impossible to miss.
An unpinned model version is a silent regression channel — a provider ships an update, behavior shifts, and nothing in your logs explains why last week's answer differs from this week's. On the shortage agent and the supplier-messaging agent, we write an explicit "no automatic model fallback" statement into the change-advisory-board pack and treat every model change as its own release, gated by a regression run — never a float.
A staging environment quietly kept serving data from a stale copy instead of the live feed, with nothing visibly broken — it just kept answering from the wrong source. The defense is layered, not singular: a deployment that can't see the data it expects should fail before taking live traffic; the day-to-day test suite re-asserts invariants against fresh data on every run; and a schema change that would silently break a downstream query should fail at deploy time, not in production.
A rollback is a deploy, not an exception to one — it goes through the same validation suite as any other release. It also has to name every component that moved together, not just one. Rolling back a model version alone can be unclean if a paired configuration changed at the same time; the rollback record specifies the matched set, written at the moment of promotion, not reconstructed from memory afterward.
Two cost lines move for different reasons and should never be collapsed into one number: model inference spend, tracking usage, and the compute an agent's tool calls trigger per turn, tracking which model is deployed. A model swap can quietly change that second number even when the model's own price hasn't moved — which is why we treat a version promotion as a cost event and watch spend on the first full business day after any change, rather than assuming a passing eval suite means cost is fine too.
Tell us what system the answer lives in and who needs it. We'll reply with a view on whether it's a two-week assessment, a five-week pilot, or something else.