Password rotation without a restart
Rotating the support password needs a restart of every serverno restart.
A colleague raised the ticket. The design and the work were mine.
Every customer server has a support login for our support engineers. Each server is a single replica with its database in the same pod.
Why it was needed
The password lived in a Terraform variables file. Rotating it meant an apply, a wait for the sync and a restart of every server, and a restart is a customer outage. So it was rarely rotated.
How it was done
Redesigned the Helm chart so each secret names its own store, 1Password for secrets people rotate and Secrets Manager for machine ones.
Added a sidecar that applies a new password inside the pod and records success only after a real login.
Proved it on one staging cluster first, with the risks written down before and answered after.
Problems on the way
The new External Secrets Operator dropped the old resource API.
Every manifest still used it. Fix: Upgraded to a version that serves both, moved every manifest, then upgraded again.
Rotating the token half would have broken a third of the fleet.
Device management depended on it in a way nobody had documented. Fix: Checked every production server read-only first, shipped the password half, and specified a product fix for the rest.
A known bug in the operator's 2.x line stalls syncs without logging.
Nothing in the logs shows the stall. Fix: Watched that refresh times kept moving, not the Ready flag.
Store a secret where its reader is.
Results
- 0restarts to rotate the support password
- ~61 sfrom changed secret to a verified login
- 7 → 1per-region passwords became one entry