Retention changes affect data already present
Partition expiration is a data-retention control, not just a rule for partitions created after a setting changes. For time-partitioned tables, a new expiration duration applies to existing partitions as well. Partitions already older than the new rule can expire immediately. Late-arriving data written into an old partition is subject to that same eligibility.
Distinguish partition expiration from the table's own expiration and from dataset defaults. The resulting data availability depends on the actual table configuration. Integer-range partitioning has different support, so confirm the partitioning type before borrowing a time-based retention procedure. Physical cleanup and visible data expiry are also different stages; delayed cleanup does not mean expired rows remain available to normal queries.
Prepare read access to table metadata and partition information, the application retention requirement, and an approved recovery method. Confirm the dataset's actual time-travel window. If longer preservation is required, evaluate a snapshot or separate retained copy before applying a shorter policy.
Preview the effect
- Read
bq show --format=prettyjson PROJECT_ID:DATASET.TABLE. Record partitioning field or ingestion-time mode, expiration settings, dataset defaults where applicable, and table expiration. Confirm the table identifier and location before continuing. - Query the dataset's
INFORMATION_SCHEMA.PARTITIONSfor the target table with a narrowly scoped metadata query. List partition identifiers, row counts, and relevant modification information. Translate those identifiers into the cutoff implied by the proposed policy. Do not infer partition age solely from the last write timestamp. - Compare the proposed cutoff with business queries and backfill requirements. Identify reports that need older dates and pipelines that deliver late data. Create and verify the approved recovery artifact before the change, with its own retention and access policy.
- Apply the reviewed expiration change through the team's normal source of configuration. Re-read metadata and test known boundary dates. Watch the next ingestion cycle so an apparently successful backfill is not immediately expired because it targets an old partition.
Illustrative scenario
A hypothetical analytics team wants to retain a shorter period of event data. It assumes the change will only affect future partitions. The preview shows that many currently queried partitions are already beyond the proposed cutoff. The team first preserves required reporting data and changes dependent reports, then shortens retention. It also updates a late-event pipeline so users understand which older events fall outside the supported window.
| Unexpected result | Evidence to inspect | Correction |
|---|---|---|
| Old dates disappear immediately | New expiration and partition boundaries | Restore approved data and revise the policy |
| Table disappears before partitions should | Table-level expiration | Correct the table retention contract |
| Backfill completes but rows vanish | Target partition age | Align late-data handling with retention |
| Storage metrics lag the change | Cleanup and storage billing behavior | Separate logical expiry from physical cleanup |
Verify recoverability and consumers
Run safe queries for a retained date, the cutoff boundary, and a date intentionally outside retention. Confirm the result with the reporting owner. Test the snapshot or historical recovery method in an isolated destination before relying on it. Time travel is bounded by dataset configuration and elapsed time; it should not be treated as an indefinite backup.
Operating procedure
Keep retention ownership with the data contract. Review partition policies when business reporting periods, backfill windows, or dataset defaults change. Monitor ingestion failures and unexpected empty-date ranges in critical reports. Record each retention change with the calculated cutoff and preserved recovery artifact so later investigators can distinguish intentional expiration from a pipeline defect.
Explore Clouvenix cloud architecture for implementation and operational handover.