Define the unit that can time out
A Cloud Run job execution contains one or more tasks. Each task can have multiple attempts when retries are enabled. The configured task timeout applies to an attempt, not to an overall business batch deadline. A long timeout combined with repeated attempts can therefore keep an execution active much longer than an operator initially expects.
A successful container exit is also not enough to prove that the intended business records were processed. Your application must define its input snapshot, assignment of records to tasks, durable progress, and completion check. Retrying a whole task without these boundaries can repeat already completed effects or silently skip work after an incomplete checkpoint.
Prepare a nonproduction job with representative data, read access to job and execution details, and application logs that identify task attempts. Establish the acceptable batch deadline and the dependency's timeout behavior. Consult current limits for the workload type: GPU tasks can have different timeout limits from ordinary tasks.
Design a recoverable task
- Inspect
gcloud run jobs describe nightly-import --project=PROJECT_ID --region=REGION --format=export. Record task count, parallelism, timeout, and retry settings. Read execution details to distinguish one execution's status from an individual attempt's termination reason. - Measure time per record or bounded chunk, including dependency waiting. Select a timeout that allows a useful chunk to finish but still detects stalled work. Set client-side dependency timeouts below the task boundary so the application can record a meaningful failure and clean up.
- Persist checkpoints only after the associated business operation is durably complete. Use stable record or chunk identifiers so an attempt can recognize completed work. Make repeated external effects safe through dependency-supported idempotency or a durable operation ledger.
- Establish a batch-level reconciliation step. Compare expected input identifiers with accepted results, failed chunks, and intentionally excluded records. Alert when the deadline is missed even if retries are still running. Increasing the task timeout alone does not satisfy a missed business deadline.
Illustrative scenario
A hypothetical catalog import assigns multiple source files to each task. The last file stalls on an external service, so the attempt times out. On retry, a naive implementation imports all earlier files again. A corrected implementation records each completed file using its immutable source identity, resumes from durable progress, and reports unreconciled files separately. The operator can then decide whether the batch is useful, incomplete, or safe to retry.
| Observation | What to inspect | Correct response |
|---|---|---|
| Execution runs beyond the expected deadline | Attempts, retries, and task count | Add a business deadline and reconciliation |
| Same records appear repeatedly | Checkpoint and effect ordering | Make completed units recognizable |
| Attempt hangs until platform timeout | Dependency timeouts and cancellation | Bound the waiting operation |
| Execution succeeds with missing records | Expected input versus results | Repair application completion checks |
Verify recovery deliberately
In staging, interrupt work after some chunks have completed and let the configured retry path run. Confirm that completed chunks are preserved, partial work is detected, and external effects are not duplicated. Test a dependency timeout and a permanently invalid input. Review both job status and the business reconciliation result. These tests provide more useful evidence than merely running a small happy-path batch.
Operating procedure
Monitor execution age, failed task attempts, repeated retries, progress rate, and unreconciled input. Record the job configuration with the release that produced it. Require an operator to inspect the failure class before launching another execution over the same input. Retain enough input identity and checkpoint history to explain a partial batch and resume it safely.
Explore Clouvenix managed infrastructure for implementation and operational handover.