A schedule requests rotation work
A Secret Manager rotation schedule sends a SECRET_ROTATE notification to configured Pub/Sub topics. It does not, by itself, change a password at an external database, publish a correct replacement credential, or reload every application using the secret. Those actions belong to a rotation workflow that the system owner must implement.
Treat the notification as the start of a controlled operation. The worker needs to recognize repeated requests, coordinate with the dependency, create a new secret version, and verify adoption before retiring the previous credential. A successful notification delivery is not evidence that credential rotation completed. Monitoring must follow the whole operation to its accepted result.
Prepare a nonproduction secret, a test dependency supporting safe credential changes, Pub/Sub topic and subscription configuration, and a consumer inventory. Define the overlap policy, emergency rollback route, and owner of the external credential. Obtain metadata access without printing secret payloads.
Implement the operation lifecycle
- Inspect
gcloud secrets describe SECRET_NAME --project=PROJECT_ID. Record rotation settings, associated notification topics, and ownership labels. Confirm the Secret Manager notification path has the documented publish permission and that the topic has a usable consumer subscription. - Define a durable rotation operation record. Store the secret identity, requested rotation context, current phase, and outcome. Use this record to make repeated notifications or worker retries resume the same intended operation instead of creating uncontrolled credential changes.
- Implement dependency update and version publication in a safe order for that dependency. Preserve an overlap where supported. If changing the dependency succeeds but publishing the secret fails, the worker needs an explicit recovery phase; it must not blindly repeat every side effect from the beginning.
- Coordinate consumer adoption. Pinned environment values need an application rollout, while mounted values still require correct read or refresh behavior. Verify real authentication with the new credential before disabling the old dependency credential or secret version. Record a completed rotation only after those acceptance checks pass.
Illustrative scenario
A hypothetical integration account is rotated monthly. The schedule emits notifications reliably, so a dashboard stays green even though the rotation worker has been failing to update an external vendor credential. The team separates notification health from rotation completion, adds a phase record and overdue-completion alert, and rehearses recovery after a dependency update succeeds but secret publication fails.
| Observation | Missing boundary | Corrective direction |
|---|---|---|
| Notifications arrive, credential stays old | Rotation worker implementation | Perform and verify the dependency change |
| Multiple credentials created for one request | Retry and deduplication state | Resume a durable operation |
| Secret updates, some callers fail | Consumer adoption and overlap | Coordinate refresh or rollout |
| Worker crashes midway | Phase-specific recovery | Reconcile before repeating side effects |
Verify the entire workflow
Run a controlled test rotation and inspect every phase without exposing the value. Confirm the dependency accepts the new credential, intended applications adopt it, and the previous credential is retired only at the approved point. Test a worker retry and a failure between phases. Verify the application rollback route during the overlap period. A test that only receives SECRET_ROTATE is incomplete.
Operating procedure
Monitor notification delivery, worker failures, operation age, last accepted rotation, and consumer authentication errors. Alert on overdue completed rotation rather than merely a missed scheduled event. Review ownership when applications or vendors change. Rehearse partial-failure recovery after automation updates, and retain phase evidence so an incident responder can determine which credential state is authoritative.
Explore Clouvenix cloud security for implementation and operational handover.