fix: replace instead of drop on managed Iceberg full refresh - #1669
Open
moomindani wants to merge 3 commits into
Open
fix: replace instead of drop on managed Iceberg full refresh#1669moomindani wants to merge 3 commits into
moomindani wants to merge 3 commits into
Conversation
A full refresh of an incremental model on a Unity Catalog managed Iceberg table dropped the table and then ran the CTAS, leaving it absent for the duration of the rebuild and losing its history. The replaceability check keyed its Iceberg arm off `file_format`, which an Iceberg model never sets to iceberg -- `iceberg_table_properties` raises for anything but delta, and `iceberg` is not an accepted `file_format` at all, so that arm was unreachable. The Delta arm then failed too, because the existing table reports `Provider = iceberg`. Key the arm off `table_format` plus the `use_managed_iceberg` flag instead, matching the condition `file_format_clause` uses to emit `using iceberg`. Extracted into `format_allows_create_or_replace` so both the V1 and V2 paths share it and it can be tested without a warehouse. Resolves databricks#1662 Co-authored-by: Isaac <no-reply@databricks.com>
Asserts on the table's history rather than its id or creation time: both of those change on `create or replace` as well, so only the first history entry surviving distinguishes a replace from a drop and recreate. Co-authored-by: Isaac <no-reply@databricks.com>
moomindani
requested review from
jprakash-db,
saishreeeee and
sd-db
as code owners
September 8, 2026 20:00
Co-authored-by: Isaac <no-reply@databricks.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Resolves #1662
Description
A full refresh of an incremental model on a Unity Catalog managed Iceberg table ran
drop table if existsand then the CTAS, so the table was absent for the whole rebuild — minutes on a large table — and concurrent queries failed withTABLE_OR_VIEW_NOT_FOUND.The replaceability check keyed its Iceberg arm off
file_format:An Iceberg model never has
file_format == 'iceberg'.iceberg_table_propertiesraises for anyfile_formatother than delta, andicebergis not in the acceptedfile_formatlist at all, so this arm was unreachable. The Delta arm then failed too, because the existing table reportsProvider = iceberg. Both false, so the macro dropped a table thatCREATE OR REPLACEhandles fine — which is exactly the statement it ran next.This keys the arm off
table_formatplus theuse_managed_icebergflag, the same conditionfile_format_clauseuses to decide onusing iceberg. The flag is read throughget_behavior_flag_no_warnso projects that never opt in do not get a warning (#1266).The Delta arm is unchanged, so a project that has just switched the flag on and still has a Delta table keeps replacing rather than dropping. The only case whose behaviour changes is managed Iceberg over managed Iceberg.
The predicate moved into
format_allows_create_or_replaceso the V1 and V2 paths share one definition and it can be tested without a warehouse.Testing
The functional test asserts on the table's history rather than its id or creation time — both of those change on
create or replacetoo, so only the first history entry surviving distinguishes a replace from a drop and recreate. It fails on the unfixed macro withhistory restarted, so the full refresh dropped and recreated the tableand passes with the fix.Against a serverless SQL warehouse:
tests/functional/adapter/iceberg/11 passed,tests/functional/adapter/incremental/108 passed with 3 failures that reproduce identically onmain(TestAppendParquet, and the two column-tag tests). Unit suite 1349 passed;code-qualityclean.Checklist
CHANGELOG.mdand added information about my change to the "dbt-databricks next" section.dbt-databricks-pr-readyproject skill for this PR and addressed its merge-readiness feedback