Context
Two workers can both read the same file and then overwrite each other's changes. A retry can create a similar problem: the first upload succeeds, its response is lost, and a later attempt overwrites a newer object. Reliable object storage does not automatically decide which application's update should win. That decision belongs in the write contract.
Consider an illustrative export service that publishes a tenant's latest report under a stable object name. A slow worker finishes after a newer worker and replaces the correct report with stale data. Both uploads are individually valid. The missing control is a condition tying each write to the version the worker intended to replace.
Use generation as an object-version condition
Cloud Storage supports request preconditions based on object generation and metageneration. A generation-match condition can require that the current object match a specific version. A generation match of zero has a special create-only meaning for the live object, and a failed precondition returns HTTP 412. Use the condition supported by the particular operation and client library. 1
Think of this as “replace the version I inspected” rather than “write whenever the request arrives.” The storage service checks the condition atomically with the operation. A separate application-level existence check followed by an unconditional write leaves a race between those two steps.
Define whether the operation creates or replaces
For a new immutable artifact, use a unique name and an appropriate create-only condition. For a replacement, obtain and retain the generation that the application is authorized to replace. Keep the generation associated with the same bucket and object name; a version number without its resource identity is not a complete reference.
Avoid using a timestamp as a substitute for a storage generation. Clocks, time precision, and concurrent operations make that comparison weaker than a condition evaluated by the storage service. If the workflow updates metadata rather than object content, inspect the metageneration requirements for that operation instead of assuming all version fields mean the same thing.
Treat a conflict as a business decision
A 412 response can be an expected concurrency outcome. It means the original condition no longer holds, not necessarily that the service is unavailable. Decide whether the user should reload, the worker should discard stale output, or the application should merge and submit a new update based on current state.
Do not automatically remove the precondition after a conflict. That converts a protected write into an unconditional overwrite at the exact moment when another writer has changed the object. Log the expected generation and the operation identity without exposing sensitive content, then route the conflict through the intended product workflow.
Reconcile a request whose response disappeared
A network timeout can leave the client unsure whether a write succeeded. A safe retry policy needs the original condition and an operation identity. If a retried create-only upload receives a precondition failure, inspect the existing object and verify whether it is the artifact from the earlier attempt or a different operation.
Use a checksum and trusted application metadata where appropriate to support that decision. Matching an object name alone is insufficient when multiple producers can use it. Keep the evidence needed to distinguish “my earlier upload succeeded” from “another worker won.” This is especially important before marking a report as published in the application database.
Publish immutable artifacts through a controlled pointer
A useful design writes each completed report to an immutable object name, validates it, and then updates a small current-report pointer through a concurrency-controlled operation. Readers can continue using the previous valid artifact until the new one is ready. The exact pointer mechanism may live in the application database or in storage, depending on the consistency contract.
This pattern does not eliminate coordination; it makes the publication boundary explicit. Define what happens if the artifact upload succeeds but the pointer update fails. An orphaned artifact can be cleaned later, while an incorrectly advanced pointer can expose incomplete or stale work to customers.
Test a deliberate two-writer race
Have two test workers read the same starting generation, then release their writes together. The intended result is that the protected condition permits the appropriate update and rejects the stale one. Verify the final content and application status, not only the returned HTTP codes.
Repeat with a simulated lost response and with an object that did not previously exist. Include deletion and recreation if the product allows those operations. These tests reveal whether retry code preserves the original condition or quietly changes its behaviour after the first failure.
Keep conflict handling visible in operations
Track precondition failures separately from transient storage errors. A sudden rise may indicate more legitimate concurrency, a stale client, or a scheduling bug that runs overlapping jobs. Measure how many conflicts are resolved automatically and how many leave customer work incomplete.
A practical example is a nightly report and a manual regeneration running together. The product should define whether the newest requested report or the newest completed report becomes current. Store that rule in the job metadata and publication logic so support can explain the result. Storage generations enforce a condition; they do not choose the customer's business priority.
A focused operator exercise: A stale-report publication exercise
Start two test report jobs from the same initial object generation. Let the newer business request finish first, then release the older worker's publication attempt. Verify the configured publication rule prevents the old result from becoming current, or records the intended alternative if the product explicitly chooses a different priority.
Repeat with the first upload response hidden from the client. The retry should reconcile the existing artifact using trusted operation metadata and content evidence rather than dropping its precondition. Record both the final storage object and the application's report status, because protecting one without the other can still leave customers seeing the wrong completed report.
Quick diagnostic map
| Evidence | Review | Acceptance boundary |
|---|---|---|
| Observed generation | Record the object version read by the worker | Attach the intended precondition |
| Concurrent writer | Simulate a competing update | Treat a conflict as a decision point |
| Publication retry | Re-read relevant state before retrying | Avoid silently overwriting newer work |
Frequently asked questions
Should every 412 be retried? Only through a workflow that reevaluates the conflict safely. Blind retries can defeat the original protection.
Clouvenix can review concurrent upload and publication paths, helping teams preserve useful retry behaviour without allowing an old worker to overwrite a newer result.
For a scoped review of this workflow, discuss the relevant Clouvenix engineering service.