We had a customer with corruption in some partitions of a RANGE partitioned table. Ideally they wanted to just skip those partitions and fix them at their leisure. Instead we had a laborious process of tricking Offload into thinking certain partitions had been offload by inserting up to a corrupt partition and then inserting a dummy row in BigQuery to make the next incremental offload skip the partition.
We should be able to simplify this and Late Arriving PBO should allow skipped partitions to be filled in afterwards.
We would not store skipped partition values in metadata because we will eventually want to come back and offload them. So it would be per-operation option.
We had a customer with corruption in some partitions of a RANGE partitioned table. Ideally they wanted to just skip those partitions and fix them at their leisure. Instead we had a laborious process of tricking Offload into thinking certain partitions had been offload by inserting up to a corrupt partition and then inserting a dummy row in BigQuery to make the next incremental offload skip the partition.
We should be able to simplify this and Late Arriving PBO should allow skipped partitions to be filled in afterwards.
We would not store skipped partition values in metadata because we will eventually want to come back and offload them. So it would be per-operation option.