Your team approved an AI tool using ordinary customer emails. Three months later, the same tool handles complaints about contract terms, a new employee reviews the output, and the vendor has changed the model.
The original evaluation may still contain useful evidence. It does not automatically cover the new work. Retest the changed assumptions before treating the old approval as permission for an expanded operating mode.
METR's February 2026 developer-productivity update explains why its newer experiment gave an unreliable estimate of the current effect. Tools, participation, task selection, and measurement changed. That software-specific research does not tell you how your workflow performs; it illustrates why leaders should preserve context when using earlier evidence.
Keep a record of the assumption that changed
Use this register for a hypothetical customer-email drafting workflow. Its earlier approval covered routine delivery questions with employee review before sending.
| Change | Earlier assumption affected | Targeted retest |
|---|---|---|
| New model version | Same system produces drafts with the approved checks | Replay representative routine and exception cases; compare correction effort |
| Contract complaints added | Work only concerns delivery information | Keep complaints out until a qualified reviewer defines acceptable handling |
| New reviewer | Reviewer recognizes unsupported commitments | Have the reviewer demonstrate corrections and escalation on known cases |
| Different order-data source | Drafts use current, correctly matched records | Test missing, stale, and mismatched order records before restoring that connection |
The register identifies evidence you need for continued use. It avoids repeating the entire original trial for a cosmetic interface change while giving a consequential data change the attention it deserves.
Choose a scope proportional to the changed work
A model update may justify replaying the cases most likely to expose a changed answer: missing facts, competing dates, and customer requests beyond the approved authority. Preserve the original expected behavior so the comparison has meaning.
A new document type requires different cases. Passing ordinary delivery emails tells you little about contract complaints. Keep the new type in the existing manual process until you define and test its expected treatment.
A reviewer change may require an operational exercise more than a model test. Can the new person identify an invented delivery promise, find the supporting order record, and route an unresolved issue? A correct draft in a demo does not answer those questions.
Use the result to change the operating permission
In the example, the new model handles routine emails within the earlier quality criteria, but contract complaints require too much judgment for the available reviewer. The leader continues routine drafting and excludes contract complaints from the AI queue.
Record that boundary beside the approval. If the tool cannot enforce the separation, add a workable intake control or pause the connected workflow until the team can prevent the unsupported use. A written restriction is weak protection if staff cannot recognize the case at intake.
Include correction time and final record quality in the retest. You could receive polished drafts that require more review than the earlier version. That result matters to capacity even when the task still passes a basic accuracy check.
Agree on triggers rather than ritual retesting
Review the register when a model, source, task, reviewer, or authority changes. Add periodic checks where operating drift can matter, with a cadence appropriate to the risk and volume. Avoid a large testing exercise simply because a month has passed if nothing relevant changed.
Our vendor proof-of-concept guide owns the initial buying test. The Ops Guide's fractional CAIO services can help maintain a clear connection between current evidence and permitted operating scope.

