Skip to content

fix: replace instead of drop on managed Iceberg full refresh - #1669

Open
moomindani wants to merge 3 commits into
databricks:mainfrom
moomindani:fix/managed-iceberg-full-refresh-replace
Open

fix: replace instead of drop on managed Iceberg full refresh#1669
moomindani wants to merge 3 commits into
databricks:mainfrom
moomindani:fix/managed-iceberg-full-refresh-replace

Conversation

@moomindani

Copy link
Copy Markdown
Contributor

Resolves #1662

Description

A full refresh of an incremental model on a Unity Catalog managed Iceberg table ran drop table if exists and then the CTAS, so the table was absent for the whole rebuild — minutes on a large table — and concurrent queries failed with TABLE_OR_VIEW_NOT_FOUND.

The replaceability check keyed its Iceberg arm off file_format:

{% set is_iceberg = (catalog_relation.file_format == 'iceberg' and existing_relation.is_iceberg) %}

An Iceberg model never has file_format == 'iceberg'. iceberg_table_properties raises for any file_format other than delta, and iceberg is not in the accepted file_format list at all, so this arm was unreachable. The Delta arm then failed too, because the existing table reports Provider = iceberg. Both false, so the macro dropped a table that CREATE OR REPLACE handles fine — which is exactly the statement it ran next.

This keys the arm off table_format plus the use_managed_iceberg flag, the same condition file_format_clause uses to decide on using iceberg. The flag is read through get_behavior_flag_no_warn so projects that never opt in do not get a warning (#1266).

The Delta arm is unchanged, so a project that has just switched the flag on and still has a Delta table keeps replacing rather than dropping. The only case whose behaviour changes is managed Iceberg over managed Iceberg.

The predicate moved into format_allows_create_or_replace so the V1 and V2 paths share one definition and it can be tested without a warehouse.

Testing

The functional test asserts on the table's history rather than its id or creation time — both of those change on create or replace too, so only the first history entry surviving distinguishes a replace from a drop and recreate. It fails on the unfixed macro with history restarted, so the full refresh dropped and recreated the table and passes with the fix.

Against a serverless SQL warehouse: tests/functional/adapter/iceberg/ 11 passed, tests/functional/adapter/incremental/ 108 passed with 3 failures that reproduce identically on main (TestAppendParquet, and the two column-tag tests). Unit suite 1349 passed; code-quality clean.

Checklist

  • I have run this code in development and it appears to resolve the stated issue
  • This PR includes tests, or tests are not required/relevant for this PR
  • I have updated the CHANGELOG.md and added information about my change to the "dbt-databricks next" section.
  • [Optional] I have run the dbt-databricks-pr-ready project skill for this PR and addressed its merge-readiness feedback

moomindani and others added 2 commits September 8, 2026 16:58
A full refresh of an incremental model on a Unity Catalog managed Iceberg
table dropped the table and then ran the CTAS, leaving it absent for the
duration of the rebuild and losing its history.

The replaceability check keyed its Iceberg arm off `file_format`, which an
Iceberg model never sets to iceberg -- `iceberg_table_properties` raises
for anything but delta, and `iceberg` is not an accepted `file_format` at
all, so that arm was unreachable. The Delta arm then failed too, because
the existing table reports `Provider = iceberg`. Key the arm off
`table_format` plus the `use_managed_iceberg` flag instead, matching the
condition `file_format_clause` uses to emit `using iceberg`.

Extracted into `format_allows_create_or_replace` so both the V1 and V2
paths share it and it can be tested without a warehouse.

Resolves databricks#1662

Co-authored-by: Isaac <no-reply@databricks.com>
Asserts on the table's history rather than its id or creation time: both of
those change on `create or replace` as well, so only the first history entry
surviving distinguishes a replace from a drop and recreate.

Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Incremental full refresh on UC managed Iceberg drops the table instead of create-or-replace (v1 materialization)

1 participant