feat: support table stages and staged file paths - #424
Open
arnaldojvg wants to merge 4 commits into
Open
arnaldojvg wants to merge 4 commits into
arnaldojvg wants to merge 4 commits into
Conversation
This was referenced Sep 10, 2026
arnaldojvg
force-pushed
the
narr-6-table-stages
branch
from
September 12, 2026 11:35
67ca462 to
47da4a0
Compare
tekumara
added a commit
that referenced
this pull request
Sep 19, 2026
Alternative to #420, trimmed as discussed there: same unquoted-PUT fix, but only `AUTO_COMPRESS` is honoured — `parallel`, `sourceCompression` and `overwrite` keep their hardcoded defaults (with the existing TODO). We only need `AUTO_COMPRESS=FALSE`: our job PUTs an already gzipped csv, and with `autoCompress: True` it was gzipped a second time so the staged file no longer matched what `COPY INTO` expected. `put_options` stays generic so other options can be wired later. If this one is preferred, #420 can be closed and #422 / #424 rebased onto this branch instead. Part of #409, umbrella #403. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: arnaldo-cai <arnaldo.varela@clarity.ai> Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com> Co-authored-by: Oliver Mannion <125105+tekumara@users.noreply.github.com>
Owner
|
@arnaldojvg FYI this now has merge conflicts |
In server mode the PUT response told the connector to write the file to a local path, which only works when the client shares the server's filesystem. Return a GCS-style stage info with a presigned url instead, so the connector uploads the file over http to a new /fs_bucket route that stores it in the stage's backing directory. GCS is the only location type the connector uploads to via a plain http url, and it re-requests the url with the destination file name, so PUT now accepts a bare file://name.gz source. A PUT with a bound ? target keeps LOCAL_FS because that re-request runs without bindings. Part of tekumara#409, umbrella tekumara#403. Supersedes part of tekumara#413. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Let COPY INTO apply a named file format via FILE_FORMAT = (FORMAT_NAME = ...), looking its stored options up and treating them like inline options. Inline and named CSV options gain SKIP_HEADER counts, NULL_IF, EMPTY_FIELD_AS_NULL, ESCAPE_UNENCLOSED_FIELD = NONE and COMPRESSION, mapped onto duckdb's read_csv arguments. ON_ERROR also accepts a quoted string as the connector sends it. Part of tekumara#408, umbrella tekumara#403. Supersedes part of tekumara#412. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
A table stage (@%table or @db.schema.%table) exists implicitly for every table, so PUT, LIST and COPY INTO resolve it against the table catalog rather than the stages table. A stage reference can also carry a path suffix (@stage/dir/file.csv.gz) which COPY INTO treats as a prefix match within the stage, as Snowflake does. Internal stage globs now use the plain path instead of a file:// uri, because duckdb does not decode percent-encoded uris. Adds end-to-end server-mode tests for the stage/PUT/COPY/MERGE bulk-load pipeline and for loading a gzipped csv from a table stage. Part of tekumara#409, umbrella tekumara#403. Supersedes part of tekumara#413. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
arnaldojvg
force-pushed
the
narr-6-table-stages
branch
from
September 21, 2026 12:32
47da4a0 to
277b1c0
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Depends on #422 and #423
A table stage (
@%tableor@db.schema.%table) exists implicitly for every table, soPUT,LISTandCOPY INTOnow resolve it against the table catalog instead of the stages table, and a PUT to a table stage for a missing table raises Snowflake'sStage ... does not exist or not authorized. A stage reference can also carry a path suffix (@stage/dir/file.csv.gz), whichCOPY INTOtreats as a prefix match within the stage, as Snowflake does. Internal stage globs use the plain path rather than afile://uri because duckdb does not decode percent-encoded uris.This is the last slice, so it also adds the end-to-end server-mode tests that need everything above: the stage/PUT/COPY/MERGE bulk-load pipeline run twice (
test_server_bulk_load_pipeline) and loading a gzipped csv with VARIANT columns from a fully qualified table stage (test_server_table_stage_bulk_load).Tests:
test_copy_internal_stage_path,test_copy_internal_table_stage,test_put_table_stage_non_existent_tableintests/test_copy_into.py;test_server_bulk_load_pipeline,test_server_table_stage_bulk_loadintests/test_server.py.Part of #409, umbrella #403. Supersedes part of #413.
🤖 Generated with Claude Code