A red canary can describe two different outages
A synthetic canary runs a defined check against a service. It can fail because the monitored workflow is broken, or because the runner cannot perform the test or store its evidence. Treating both outcomes as a confirmed customer outage makes alerts noisy and can hide a monitoring blind spot.
This guide follows a CloudWatch Synthetics check from execution through its user journey and artifacts. It does not assume that a loaded homepage proves the whole application is healthy. Define the small representative operation that the canary is intended to validate.
Setup: define the test and its dependencies
Record the canary runtime, script revision, execution role, schedule, target and artifact location. Identify whether it runs through a VPC and which endpoints it must reach. Include service dependencies used by the runner itself, not only the monitored application.
A useful acceptance worksheet is:
Expected user step and success assertion:
Runner runtime and execution identity:
Target network path and permitted endpoints:
Artifact bucket and encryption policy:
Failure classification and escalation owner:
Use a controlled test identity and avoid real customer side effects. Screenshots, URLs and reports can contain sensitive information, so define redaction and access policy before recording a privileged user journey.
Illustrative incident: a working page produces a failed run
In this fictional scenario, a canary reaches the application's expected page, but fails while writing artifacts to its S3 location after a policy change. The application team begins a rollback because the synthetic alert is red.
The responder inspects the failed step and artifact-delivery error before concluding that the application regressed. The monitoring owner repairs the evidenced role or bucket-policy mismatch and then reruns the complete test. Successful page access alone did not establish that the monitoring path was healthy.
Troubleshooting: classify the first failing stage
| Observation | Evidence | Boundary |
|---|---|---|
| Runner cannot initialize in a VPC | Runtime errors and required networking permissions | Canary execution infrastructure |
| Target cannot be reached | DNS, route and endpoint evidence | Monitored network path |
| Target loads but assertion fails | Step result and sanitized page evidence | User journey or test expectation |
| Artifacts cannot be written | Role, bucket and encryption-policy errors | Evidence storage |
Check supported runtime and script requirements rather than assuming a dependency compatible with another Node.js application is available in the canary. For a VPC canary, a public-looking subnet name does not demonstrate the necessary outbound path. Follow the actual network design.
Repair the runner or the service at the correct owner
Apply the narrow correction supported by the failing step. Use appropriate execution-role and storage permissions without granting broad unrelated access. If the application genuinely fails its asserted workflow, retain the evidence and hand the incident to the service owner with the exact step and timing.
If the test expectation changed because of an intentional UI release, update the canary through review and verify that it still catches a real failure. Do not remove assertions simply to restore a green dashboard. Separate monitor-health incidents from customer-impact classification in the response process.
Verification, alerting and SOP
Verify one successful complete run with stored evidence and one safe controlled failure in a test environment. Confirm that alerts explain whether execution, user workflow or evidence storage failed. Test the original representative operation rather than replacing it with a shallow endpoint check.
An internal SLO can describe valid synthetic execution and the accepted user journey separately; an external SLA needs agreed measurement locations and exclusions. The SOP should record script ownership, runtime maintenance, test identity, network and artifact dependencies, safe test data, alert routing and rollback. Synthetic results are sampled observations, not proof that every user experience is successful.
See Clouvenix observability monitoring for a reviewed implementation and operational handover.
Technical reference
AWS failed-canary troubleshooting, AWS canary creation, AWS canary script features. Reviewed 19 October 2026; use documentation for the installed product and release.