feat: PUT over http in server mode, unquoted PUT urls, PUT options, and table stages - #413
Closed
arnaldojvg wants to merge 9 commits into
Closed
arnaldojvg wants to merge 9 commits into
arnaldojvg wants to merge 9 commits into
Conversation
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
A replaced stage starts empty, as in Snowflake. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Stores named file formats in the info schema so they can be referenced by name, eg: from COPY INTO FILE_FORMAT = (FORMAT_NAME = ...). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- FILE_FORMAT = (FORMAT_NAME = ...) resolves named file formats - SKIP_HEADER > 1, NULL_IF, EMPTY_FIELD_AS_NULL, ESCAPE_UNENCLOSED_FIELD and COMPRESSION are mapped onto duckdb's read_csv - ON_ERROR accepts a string literal - a stage reference can include a path suffix, eg: @stage1/file.csv.gz Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
eg: PUT file://data.csv @db1.schema1.%mytable and COPY INTO t FROM @db1.schema1.%mytable/data.csv.gz Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
sqlglot only parses PUT as exp.Put when the source is a quoted literal, so an unquoted url, as sent by eg: snowflake-sqlalchemy, fell through to duckdb as a raw command. Quote it before parsing. Honour AUTO_COMPRESS, OVERWRITE, PARALLEL and SOURCE_COMPRESSION, and don't recompress an already gzipped source. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Runs the full pipeline twice on one server to prove CREATE OR REPLACE STAGE/FILE FORMAT/TABLE work on rerun. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The server returned a LOCAL_FS stage location, so the connector wrote the file to the client's own filesystem. When the client doesn't share the server's filesystem (eg: the server runs in a container) a subsequent COPY INTO silently found no files. Return a GCS-style stage info with a presigned url pointing back at the fakesnow server instead, so the connector uploads the file over http into the stage's backing storage. A PUT with a ? placeholder target keeps LOCAL_FS because the connector re-requests the presigned url by executing the command without bindings. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
arnaldojvg
force-pushed
the
feat-put-server-mode-and-table-stages
branch
from
September 4, 2026 07:57
b2a525c to
404c557
Compare
This was referenced Sep 4, 2026
Contributor
Author
arnaldojvg
pushed a commit
to arnaldojvg/fakesnow
that referenced
this pull request
Sep 12, 2026
Accept PUT commands as the connector and most snowflake clients issue them: an unquoted file:// source, which sqlglot otherwise parses as an opaque command, and options such as AUTO_COMPRESS, OVERWRITE, SOURCE_COMPRESSION and PARALLEL. AUTO_COMPRESS = FALSE and already gzipped sources are stored as-is rather than being gzipped again, and the upload result reports the real compression of each side. Part of tekumara#409, umbrella tekumara#403. Supersedes part of tekumara#413. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
arnaldojvg
pushed a commit
to arnaldojvg/fakesnow
that referenced
this pull request
Sep 12, 2026
In server mode the PUT response told the connector to write the file to a local path, which only works when the client shares the server's filesystem. Return a GCS-style stage info with a presigned url instead, so the connector uploads the file over http to a new /fs_bucket route that stores it in the stage's backing directory. GCS is the only location type the connector uploads to via a plain http url, and it re-requests the url with the destination file name, so PUT now accepts a bare file://name.gz source. A PUT with a ? placeholder target keeps LOCAL_FS because that re-request runs without bindings. The existing server-mode PUT test (test_copy_internal_stage_server) now exercises the http upload. Part of tekumara#409, umbrella tekumara#403. Supersedes part of tekumara#413. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
arnaldojvg
pushed a commit
to arnaldojvg/fakesnow
that referenced
this pull request
Sep 12, 2026
A table stage (@%table or @db.schema.%table) exists implicitly for every table, so PUT, LIST and COPY INTO resolve it against the table catalog rather than the stages table. A stage reference can also carry a path suffix (@stage/dir/file.csv.gz) which COPY INTO treats as a prefix match within the stage, as Snowflake does. Internal stage globs now use the plain path instead of a file:// uri, because duckdb does not decode percent-encoded uris. Adds end-to-end server-mode tests for the stage/PUT/COPY/MERGE bulk-load pipeline and for loading a gzipped csv from a table stage. Part of tekumara#409, umbrella tekumara#403. Supersedes part of tekumara#413. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
arnaldojvg
pushed a commit
to arnaldojvg/fakesnow
that referenced
this pull request
Sep 21, 2026
In server mode the PUT response told the connector to write the file to a local path, which only works when the client shares the server's filesystem. Return a GCS-style stage info with a presigned url instead, so the connector uploads the file over http to a new /fs_bucket route that stores it in the stage's backing directory. GCS is the only location type the connector uploads to via a plain http url, and it re-requests the url with the destination file name, so PUT now accepts a bare file://name.gz source. A PUT with a bound ? target keeps LOCAL_FS because that re-request runs without bindings. Part of tekumara#409, umbrella tekumara#403. Supersedes part of tekumara#413. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
arnaldojvg
pushed a commit
to arnaldojvg/fakesnow
that referenced
this pull request
Sep 21, 2026
A table stage (@%table or @db.schema.%table) exists implicitly for every table, so PUT, LIST and COPY INTO resolve it against the table catalog rather than the stages table. A stage reference can also carry a path suffix (@stage/dir/file.csv.gz) which COPY INTO treats as a prefix match within the stage, as Snowflake does. Internal stage globs now use the plain path instead of a file:// uri, because duckdb does not decode percent-encoded uris. Adds end-to-end server-mode tests for the stage/PUT/COPY/MERGE bulk-load pipeline and for loading a gzipped csv from a table stage. Part of tekumara#409, umbrella tekumara#403. Supersedes part of tekumara#413. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Depends on #412 (based on its branch) — contains its commits until it merges. Please merge #412 first; I'll rebase this PR right after so only its own 4 commits remain. (If this one is merged first instead, #412 can simply be closed as already included.)
exp.Putwhen the source is a quoted literal, so an unquoted url (as sent by eg: snowflake-sqlalchemy) fell through to duckdb as a raw command; the source is quoted before parsing.AUTO_COMPRESS,OVERWRITE,PARALLELandSOURCE_COMPRESSIONare honoured, and an already gzipped source is no longer recompressed.?placeholder target keeps LOCAL_FS because the connector re-requests the presigned url by executing the command without bindings.%is a table stage, which exists implicitly for every table: existence is checked against the table catalog rather than the stage registry, and its files live in the internal-stage backing storage. Out of scope:LIST @%table, which sqlglot cannot parse.Fixes #409. Split out of #400 per maintainer request.
🤖 Generated with Claude Code