Private nodes and private administration differ
A GKE cluster's node exposure and its control-plane access are separate configuration decisions. Private nodes do not, by themselves, explain which endpoint an administrator uses to reach the Kubernetes API. The control plane can expose DNS-based access and IP-based access with different network and policy requirements.
This distinction matters during a lockout investigation. A client reaching an IP endpoint needs an appropriate network route and any applicable authorized-network allowance. A client using the DNS endpoint follows the documented Google API access and identity controls for that endpoint. Once a request reaches the API, Kubernetes authorization remains a separate check. Opening a firewall cannot repair an RBAC denial.
Before making changes, identify the cluster location, current endpoint configuration, administrator network, and an existing recovery access path. Obtain read access to cluster settings. Do not disable the only working endpoint until an alternative has been tested from the place administrators actually work.
Trace the selected access path
- Inspect
gcloud container clusters describe CLUSTER --project=PROJECT_ID --location=LOCATION. Record enabled endpoints, private endpoint settings, DNS access, and authorized networks. Compare this with the endpoint stored in the current kubeconfig. A stale context can keep targeting an endpoint that the team has retired. - Identify the client environment. For private IP access, verify the VPN, Interconnect, or VPC path and return routing. Check source-address changes caused by NAT or a bastion. For DNS-based access, confirm the client can reach the relevant Google API path and satisfies applicable IAM and perimeter policies.
- Check restrictions at the endpoint actually being used. Review authorized-network ranges and whether restrictions apply to the intended IP endpoint. Do not assume an allowlist on one access method describes another. Use current cluster configuration rather than a diagram written before the latest endpoint changes.
- Test authentication and authorization separately after reachability is established.
kubectl auth can-i get deployments -n NAMESPACEis a useful read-only permission check for the active identity. A forbidden response demonstrates a different boundary from a connection timeout.
Illustrative scenario
A hypothetical platform team moves administrators onto a new corporate VPN. Cluster nodes remain healthy, but kubectl times out because kubeconfig still points to the private IP endpoint and the new VPN does not advertise the required route. Another administrator using the enabled DNS endpoint succeeds. The immediate correction follows the chosen endpoint strategy: repair private routing or deliberately migrate the client to the approved DNS path, then validate identity access.
| Evidence | Likely boundary | Next check |
|---|---|---|
| TCP connection times out to private IP | Routing or endpoint network controls | Client route, source range, return path |
| Google identity error on DNS access | IAM or API access policy | Active identity and endpoint permissions |
| Kubernetes reports forbidden | Kubernetes authorization | RBAC binding for the intended scope |
| Old context fails, new context works | Stale kubeconfig endpoint | Regenerate and review the context |
Verify before tightening access
Use a test administrator identity from the intended operational network. Confirm API reads, namespace-scoped authorization, and the documented recovery route. Test an unauthorized identity or disallowed network where practical. Record the kubeconfig context and endpoint used for each result. Only after this evidence exists should the team retire an obsolete access method through a reviewed change.
Operating procedure
Maintain an endpoint access map with network owners and identity owners. Review it after VPN changes, perimeter-policy updates, and cluster endpoint migrations. Monitor administrative access failures separately from workload health. A healthy application does not establish that the team can administer the cluster during an incident.
Explore Clouvenix containers kubernetes for implementation and operational handover.