Context
A warehouse ingestion job times out after sending a batch. The client cannot tell whether BigQuery accepted the rows, so it resubmits them. Depending on the stream and retry design, the result may include duplicates. The Storage Write API provides stream-level tools for controlling append identity, but the ingestion service must preserve enough state to use them correctly after a restart.
Name the guarantee precisely
Google documents different Storage Write API stream types. For committed streams, clients can supply offsets to support exactly-once write semantics within a stream; omitting offsets permits a retry to append records again. The default stream has different delivery semantics. Review the current API guidance and chosen client library before designing the checkpoint protocol.
This stream guarantee is narrower than “every business event appears once in the warehouse.” Two upstream events representing the same business operation can still be appended at distinct valid offsets. A re-created stream may also receive data that was written through an earlier stream. Keep transport append identity and business-event identity as separate concepts.
An illustrative uncertain append
Imagine a worker sending a batch of purchase events and losing its connection before receiving the result. It restarts with no record of the stream or intended offset. The replacement creates another stream and resends the data. The application has abandoned the information needed to resolve the original append.
Persist the stream identity, intended offset, batch identity, and source checkpoint using a documented sequence. The exact design depends on the source and writer architecture. It should let a replacement worker determine what it may retry and how to recognise an already accepted append without guessing from row counts.
Establish a single writer contract
Decide which process owns each stream and how ownership moves after failure. Concurrent writers assigning offsets independently can conflict or produce a checkpoint that no longer describes the accepted sequence. Use a durable ownership mechanism appropriate to the deployment.
Keep the mapping between source partitions or batches and destination streams explicit. If a source partition moves to another worker, transfer the relevant append state with it. A container restart should not silently become a new ingestion identity unless the recovery design accounts for possible overlap.
Handle append outcomes deliberately
Distinguish confirmed success, confirmed rejection, and an uncertain response. A schema-related rejection requires correction; a transient connection failure may require retry with the same intended identity. Do not advance the upstream acknowledgement solely because the client finished sending bytes.
Check how the client reports row errors, offset conflicts, and asynchronous completion. Follow the library's supported recovery behaviour. Log identifiers, counts, and error categories rather than complete sensitive records. Preserve rejected-record references so the team can repair data without replaying an entire day's ingestion unnecessarily.
Manage schema changes as releases
A source schema change can turn a stable stream into a recurring failure. Review compatibility, field defaults, and the destination table's actual schema before promoting producer changes. Test representative records from both sides of the transition.
Avoid treating schema coercion as a generic reliability fix. Converting a failed numeric field to text or dropping unknown fields may make ingestion succeed while damaging analytics. Route invalid records to an explicit exception process with ownership and a method to reintroduce corrected data.
Build business deduplication separately
Retain a stable event identifier and relevant business version where the analytical model needs them. Decide whether duplicate source events are valid, should be collapsed in a curated table, or indicate an upstream defect. The answer differs between an immutable event log and a current-state reporting table.
Document how reprocessing behaves. A warehouse rebuild may deliberately write another physical copy into a new destination before switching queries. An in-place repair may require a keyed merge or another controlled approach. Do not apply a broad distinct operation without understanding which fields define a duplicate business fact.
Test crash boundaries
Inject failure before append, after append submission, after acknowledgement, and before the source checkpoint is committed. For each boundary, restart the worker and compare the destination with the intended event set. This exercise exposes gaps that a simple throughput benchmark misses.
Test an offset conflict and a lost writer state record. The system should stop with actionable evidence rather than inventing a new offset or stream to keep the pipeline green. Recovery must preserve data meaning, not merely restore ingestion rate.
Quick diagnostic map
| Evidence | Interpretation | Next check |
|---|---|---|
| Rows repeat after timeout | Append outcome unresolved | Restore stream state |
| Offsets conflict | Writer ownership inconsistent | Inspect append checkpoint |
| Duplicate business facts remain | Source identity differs | Apply business deduplication |
A focused operator exercise
Prepare a small immutable input manifest with event identifiers, expected row values, stream identity, and append positions. Run the writer through the intended ownership mechanism and retain each confirmed append result. Include one schema-invalid item in a separate controlled test so rejection handling is observable without corrupting the normal acceptance population.
Break the client connection immediately after submitting a known append and before its response is processed. Restart from durable writer state and follow the library's documented outcome-resolution path. Confirm the destination contains the expected physical records and the source checkpoint advances only after the required handoff. Repeat after a confirmed acknowledgement but before source progress is saved.
Then supply two different source records representing the same business event. Verify that the analytical deduplication policy treats them as designed independently of stream offsets. The exercise passes when append uncertainty, worker replacement, schema rejection, and business repetition have separate evidence and decisions. Keep the manifest and final reconciliation so the pipeline's guarantee can be reviewed precisely.
Monitor progress and correctness
Track accepted rows, rejected rows, retry categories, checkpoint age, and the distance between source and warehouse progress. Add reconciliation by business event identifier for a representative or complete population as appropriate. A high write rate can coexist with duplicated or missing business records.
Clouvenix can help define stream ownership, durable checkpoints, and warehouse acceptance tests. The useful result is an ingestion pipeline whose retry behaviour remains explainable after a timeout or worker replacement.
Frequently asked questions
Can offsets remove duplicate events already present upstream?
Offsets identify append positions within a stream. They do not decide whether two different source records represent the same purchase or update. Keep business identifiers and apply an explicit analytical rule for duplicate source events where required.
Should a restart always create a new stream?
Choose that behaviour only with a recovery design that accounts for previously accepted writes. Retaining the original stream and append state can be essential to resolving uncertain outcomes. Test the exact client and stream lifecycle instead of assuming restarts are harmless.
Explore Clouvenix cloud architecture for implementation and operational handover.