How to Rotate API Keys Without Downtime (Many Agents)

To rotate an API key shared by many agents without downtime, never swap the value in one step. Make every receiver accept both the old and the new value, move every sender to the new one, prove nothing still uses the old one, and only then retire it. The overlap window is the whole trick.

We learned this the hard way in September and October 2026, while rotating shared secrets across our own fleet. Below: what a bad rotation looks like from the outside, what actually broke, the order that held, and a check you can run this week.

What does a failed key rotation look like?

A failed rotation looks like a wave of 401 errors from agents that worked yesterday. Usually not all of them. A few senders keep working, a few fail, and the failures cluster around whoever was missed. Health checks say “secret expired” while the secret file on your own laptop looks fine.

That last detail matters. The one copy you can see is rarely the copy that is wrong. A shared key lives in more places than anyone remembers: hosting dashboards, build settings for preview deploys, local files on a developer machine, a scheduled job on a second computer, a script property in a spreadsheet tool. Each is a separate holder, and each flips at a different moment.

What actually broke for us?

Four different things, none of them exotic.

A plan with a built-in outage. In late September 2026 we reviewed our first rotation script and the verdict was no-go. The receiving side could only accept one value at a time. The script updated some holders instantly (the database, local files, the code host) while the hosting platform took far longer to write every entry and redeploy.

That gap is a guaranteed window where some holders have the new key and the receivers still want the old one. A failure partway also left half the fleet on each value, with a rollback that only undid part of the change.

A test suite that wrote a real secret. The rotation tool’s own tests ran against the real filesystem. Each run on a build machine overwrote the live secret file with a test fixture. Agents on that machine then sent a junk fixture value and were rejected, a scheduled routine on another machine failed repeatedly, and nobody connected it to a rotation tool they were not even running.

The fix was a guard that refuses real file writes while a test is running. The lesson is broader: any test that can touch a production path will eventually do it.

A secret set by hand. Someone with dashboard access copied a value out of the hosting UI and pasted it into another site. Hosting dashboards mask secrets, so what got pasted was the mask, not the key. Every sender holding the real key was then rejected until we staged the previous value as an accepted fallback. Nobody did anything unusual. The tool simply let a well-meaning person do it.

A partial write nobody checked. After a rotation, one agent’s production and preview settings still held the old value while its development setting held the new one. It kept working until we retired the old value, then its daily report started returning 401. Our post-rotation check had read only the development setting, the one place that was readable and correct.

A related trap from a different incident: changing a setting on a hosting platform does not change what an already-built scheduled function sees. Environment values are bundled at deploy time, so the first run after the change used the old value and looked like a silent no-op. Rotation needs a redeploy step, and the verify step has to wait for it.

What is the safest order for rotating a shared secret?

The safest order is: accept both, switch senders, observe, then retire. Five steps, in this sequence, no shortcuts.

Strategy	How it works	Downtime risk	Best for
Hard cutover	Replace the value everywhere, hope the timing works	High: any slow holder gets rejected	Genuine emergencies (a leaked key)
Dual-accept overlap	Receiver accepts the new value and a staged previous value for a set window	Low: stragglers keep working	Shared secrets with many holders
Alternating or versioned credentials	Two identities or labelled versions; rotation updates the idle one	Low, but needs provider support	Databases, vendors with rolling keys

Vendors converged on this long ago. Stripe’s key documentation (2026) says that when you rotate a key in the dashboard, “both the old and new keys work for up to 7 days,” so you can migrate gradually, and it tells you to expire the old key only after its request volume has been at zero for a few hours or days.

AWS Secrets Manager’s documentation describes the same idea for databases: its alternating-users strategy keeps both credentials valid after a rotation, and it stages the new version under a pending label, tests it, and only then promotes it, keeping the previous version labelled as the last known good copy.

The OWASP Secrets Management Cheat Sheet (current edition, read October 2026) adds two rules that matter for fleets: automate rotation “or at least ensure that the process is sufficiently supported,” and log every attempt to use a revoked secret. That second rule is your retirement signal.

Here is how it ran for us after the no-go.

1. Teach the receivers to accept two values. Every gate that checks the shared secret now accepts the current name or a “previous” name, and logs a line whenever the previous one is used. This ships and deploys first, alone, before any rotation.
2. Stage the previous value. Copy the live value into the “previous” slot. Probe: old works, new does not exist yet. Nothing has changed for any agent.
3. Rotate, then redeploy. Write the new value to every holder, trigger the builds that bundle it, and wait for them to finish.
4. Audit every context, not one. Production, preview and development settings are separate values. Compare each holder’s fingerprint against the new value. For masked secrets, compare the hash of the mask, because the mask is all the API will show you.
5. Soak, check logs, then drop. Search the receiver logs for the “previous accepted” line. When it stops appearing, and every manual holder (the files on laptops and second machines) is confirmed, remove the previous value.

Run this way, the shared secrets we issue ourselves rotated in one working session with no agent outage. The every-context audit afterwards still found stragglers, and the previous-value fallback is the reason they had not caused an incident yet.

Is the whole job done?

No, and we would rather say so. The secrets we mint ourselves are rotated. Keys held by outside providers (model access, database service roles, messaging bots, mail relays) are parked. Each needs its own sequence, because some providers allow only one live key and some tie a key to a webhook that must change on both sides at once. The transcript clean-up for a leaked key is also still waiting on those rotations. As of October 2026 that is the honest status.

The pressure to finish is real. GitGuardian’s State of Secrets Sprawl 2026 report counted 29 million new hardcoded secrets exposed on public GitHub in 2025, up 34% on the year, and noted that remediation is not keeping pace. A rotation you can do safely is a rotation you will actually do.

What can you check this week?

You can run a rotation readiness check in about an hour, without rotating anything.

• List the holders. For one shared key, write down every place it lives: each hosting context, each local file, each scheduled job, each spreadsheet or script property. If you cannot list them, that is finding number one.
• Test whether the receiver accepts two values. If it can accept only one, a no-downtime rotation is not possible yet. Fix that first.
• Look for hand-set copies. Ask who has dashboard access to set secrets manually. Hand copies of masked values are the most common source of silent breakage.
• Check that your tests cannot write to real paths. Run the suite and check the modified time on your real secret files afterwards.
• Write the retirement rule. “Drop the old value when the previous-accepted log line has been absent for N days and every manual holder is ticked.” Pick N before you start.

If any of these fail, you have a handover problem more than a rotation problem. Our agent handover checklist covers where credentials should be recorded, and the pre-deployment security review covers who is allowed to set them.

Where this fits

Every tomte, our word for one production agent with one defined job, carries at least one credential. A fleet multiplies that: one shared key, many holders, no single owner of the list. An AI operations audit maps those holders, checks whether your receivers can accept overlap, and hands you a written rotation order. You can see the fleet this comes from on the agent fleet proof page.

Want the holder list built for you? Start async. We reply within one business day, and there are no meetings.