Separate permission from token validity
A private Cloud Run service checks more than whether a request contains a bearer token. The caller needs permission to invoke the receiving service, and the request must carry an appropriate Google-signed ID token. An access token intended for Google APIs is not interchangeable with an ID token intended for a service audience.
The audience identifies the service the token was issued for. It is not automatically every URL that reaches that service. In the ordinary service-to-service flow, use the receiving service URL as the audience. A request sent to a traffic-tag URL normally keeps that service audience. Custom audiences are a deliberate receiver configuration; a custom domain should not simply be substituted because it appears in the browser address bar.
Before investigating, identify the runtime service account of the caller and the project and region of the receiver. Obtain permission to read both configurations and the receiver's IAM policy. Keep tokens out of logs, tickets, browser history, and shell transcripts. Prefer client-library identity support over manually constructing credentials.
Inspect the complete call path
- Read the caller and receiver with
gcloud run services describe SERVICE --project=PROJECT_ID --region=REGION. Confirm the caller's deployed service account rather than the account used by the human who deployed it. Record the receiver's canonical service URL and any intentionally configured custom audience. - Read the receiver policy with
gcloud run services get-iam-policy receiver --project=PROJECT_ID --region=REGION. Verify that the intended caller identity has the necessary invoker access at the appropriate scope. Avoid making the service public as a diagnostic shortcut. - Inspect token acquisition in the application. Request an ID token for the expected audience, refresh it through the supported library, and send it in the supported authentication header. If debugging claims, decode only in a controlled local context and inspect the audience and expiry without copying the token into shared output.
- Test from the actual caller runtime using a harmless authenticated endpoint. Compare the requested destination, token audience, and resulting receiver logs. A test from a developer workstation can use a different identity and therefore answer a different question.
Illustrative scenario
A hypothetical invoice API calls a private rendering service through a candidate revision's tagged URL. The application requests its token with that tagged URL as the audience. Invoker IAM is correct, but authentication fails. The correction is to use the receiver's canonical service audience, unless the receiver has deliberately configured another accepted audience. The tagged destination and the identity contract are related settings, not identical strings.
| Result | Evidence to examine | Narrow correction |
|---|---|---|
| Token rejected before application handling | Audience, signature source, expiry | Use the supported ID-token flow |
| Identity recognized but invocation denied | Receiver IAM and actual caller | Grant the intended service-level access |
| Workstation succeeds, runtime fails | Runtime service account and token source | Correct the deployed caller |
| Tag works without application credentials | Receiver public access policy | Review intended service exposure |
Verify the correction
Run an allowed call and a deliberately unauthorized test from separate test identities where your environment permits it. Confirm that the allowed call reaches the expected revision and returns the correct business response. Ensure the rejected call remains rejected. Check that credentials never appear in access logs or application diagnostics. Exercise token refresh over a realistic runtime duration rather than testing only a freshly acquired token.
Operating procedure
Document each service dependency's caller identity, receiver URL, accepted audience, and IAM owner. Review those records when renaming services, adding domains, introducing tags, or changing runtime identities. Monitor authentication failures by receiver and caller context. Keep least-privilege invocation policy in the deployment source so a one-off debugging grant does not become permanent access.
Explore Clouvenix cloud security for implementation and operational handover.