Skip to content

feat: support a stage/PUT/COPY/MERGE bulk-load pipeline in server mode - #400

Closed
arnaldojvg wants to merge 9 commits into
tekumara:mainfrom
arnaldojvg:feat-bulk-load-pipeline
Closed

feat: support a stage/PUT/COPY/MERGE bulk-load pipeline in server mode#400
arnaldojvg wants to merge 9 commits into
tekumara:mainfrom
arnaldojvg:feat-bulk-load-pipeline

Conversation

@arnaldojvg

@arnaldojvg arnaldojvg commented Aug 28, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Implements the Snowflake features needed to run acceptance tests of a stage/PUT/COPY/MERGE bulk-load pipeline (Postgres → Snowflake data-sync job using snowflake-sqlalchemy) against fakesnow in server mode.

Fixes #403

Features

  • CREATE OR REPLACE STAGE replaces an existing stage (and empties its internal storage), and CREATE STAGE IF NOT EXISTS succeeds without error when the stage exists.
  • CREATE [OR REPLACE] FILE FORMAT [IF NOT EXISTS]: named file formats are stored in the info schema (_fs_global._fs_information_schema._fs_file_formats). Previously this crashed the server with a 500 (AssertionError: Unexpected parent kind: FILE FORMAT in checks.py).
  • COPY INTO ... FILE_FORMAT = (FORMAT_NAME = '...') resolves a named file format to its stored options. Previously: "FILE_FORMAT without TYPE is not currently implemented".
  • More CSV options in COPY INTO / file formats, mapped onto duckdb's read_csv: SKIP_HEADER=<n>, NULL_IF, EMPTY_FIELD_AS_NULL, ESCAPE_UNENCLOSED_FIELD='NONE', COMPRESSION (AUTO/GZIP/NONE). ON_ERROR also accepts a string literal (ON_ERROR = 'ABORT_STATEMENT').
  • PUT with an unquoted file url (as sent by eg: snowflake-sqlalchemy) now works: sqlglot only parses PUT as exp.Put when the source is a quoted literal, so the statement fell through to duckdb raw and failed with Parser Error: syntax error at or near "PUT". The source is now quoted before parsing (both embedded and server mode).
  • PUT options: AUTO_COMPRESS, OVERWRITE, PARALLEL, SOURCE_COMPRESSION are honoured (and returned to the client in server mode, so the connector's file transfer agent respects them). An already gzipped source is no longer recompressed.
  • COPY from a stage path suffix, eg: COPY INTO t FROM @stage/file.csv.gz (prefix match, like Snowflake).
  • Table stages for PUT and COPY, eg: PUT file://data.csv @db.schema.%mytable and COPY INTO t FROM @db.schema.%mytable/data.csv.gz.
  • Load history is scoped to the target table: the same file loaded into a different table is not skipped, and recreating a table (eg: CREATE OR REPLACE TEMPORARY TABLE staging) resets its load history, matching Snowflake's per-table load metadata.

An end-to-end server-mode test runs the full pipeline (CREATE OR REPLACE STAGE / FILE FORMAT / TEMPORARY TABLE, PUT AUTO_COMPRESS=FALSE, COPY with FORMAT_NAME, MERGE) twice on one server to prove reruns work.

Out of scope

  • LIST @%table_stage — sqlglot cannot parse LIST with a table stage reference; table stages are supported for PUT and COPY only.
  • ESCAPE_UNENCLOSED_FIELD values other than 'NONE', and EMPTY_FIELD_AS_NULL=FALSE combined with NULL_IF=(''), raise NotImplementedError as they have no faithful duckdb mapping.
  • SHOW FILE FORMATS / DROP FILE FORMAT.

🤖 Generated with Claude Code

@arnaldojvg arnaldojvg mentioned this pull request Aug 31, 2026
Open
@tekumara

tekumara commented Aug 31, 2026 •
edited
Loading

Copy link
Copy Markdown
Owner

thanks for the contribution @arnaldojvg - fyi this is a large PR and will take me some time to review each commit, if you wanted to progress some of these more quickly then I recommend raising individual PRs

@arnaldojvg

arnaldojvg commented Sep 1, 2026

Copy link
Copy Markdown
Contributor Author

Thanks @tekumara, let me know of anything 🙏

arnaldo-cai and others added 9 commits September 2, 2026 09:41
@arnaldo-cai @claude
feat: support CREATE OR REPLACE STAGE and CREATE STAGE IF NOT EXISTS
bb2793b 
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@arnaldo-cai @claude
feat: support CREATE FILE FORMAT for named file formats
c0086eb 
Stores named file formats in the info schema so they can be referenced
by name, eg: from COPY INTO FILE_FORMAT = (FORMAT_NAME = ...).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@arnaldo-cai @claude
feat: resolve FORMAT_NAME and support more CSV options in COPY INTO
6d1d153 
- FILE_FORMAT = (FORMAT_NAME = ...) resolves named file formats
- SKIP_HEADER > 1, NULL_IF, EMPTY_FIELD_AS_NULL, ESCAPE_UNENCLOSED_FIELD
  and COMPRESSION are mapped onto duckdb's read_csv
- ON_ERROR accepts a string literal
- a stage reference can include a path suffix, eg: @stage1/file.csv.gz

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@arnaldo-cai @claude
fix: PUT with an unquoted file url and support PUT options
23234c7 
sqlglot only parses PUT as exp.Put when the source is a quoted literal,
so an unquoted url, as sent by eg: snowflake-sqlalchemy, fell through to
duckdb as a raw command. Quote it before parsing.

Honour AUTO_COMPRESS, OVERWRITE, PARALLEL and SOURCE_COMPRESSION, and
don't recompress an already gzipped source.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@arnaldo-cai @claude
fix: scope load history to the target table
c623c8f 
Snowflake tracks load metadata per table, so the same file loaded into
a different table is not skipped, and recreating a table resets its
load history.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@arnaldo-cai @claude
feat: support table stages for PUT and COPY INTO
1693087 
eg: PUT file://data.csv @db1.schema1.%mytable and
COPY INTO t FROM @db1.schema1.%mytable/data.csv.gz

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@arnaldo-cai @claude
test: stage/PUT/COPY/MERGE bulk-load pipeline in server mode
83a6546 
Runs the full pipeline twice on one server to prove CREATE OR REPLACE
STAGE/FILE FORMAT/TABLE work on rerun.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@arnaldo-cai @claude
fix: MERGE returns integer counts when no rows are affected
b5402e2 
duckdb's COUNT_IF returns NULL over an empty merge_candidates table,
which crashed clients that read the counts, eg: the snowflake python
connector calls int() on them. Snowflake always returns integers.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@arnaldo-cai @claude
fix: transfer PUT files over http in server mode
8cd4caf 
The server returned a LOCAL_FS stage location, so the connector wrote
the file to the client's own filesystem. When the client doesn't share
the server's filesystem (eg: the server runs in a container) a
subsequent COPY INTO silently found no files.

Return a GCS-style stage info with a presigned url pointing back at the
fakesnow server instead, so the connector uploads the file over http
into the stage's backing storage. A PUT with a ? placeholder target
keeps LOCAL_FS because the connector re-requests the presigned url by
executing the command without bindings.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@arnaldojvg
arnaldojvg force-pushed the feat-bulk-load-pipeline branch from ca7d9b4 to 8cd4caf Compare September 2, 2026 07:46
This was referenced Sep 2, 2026
Merged
Merged
Closed
Closed
@arnaldojvg

arnaldojvg commented Sep 2, 2026

Copy link
Copy Markdown
Contributor Author

Thanks for the review and the suggestion — agreed this is a lot to review at once. I've split it into four issues and PRs, all cherry-picked verbatim from this branch, each independently green (full suite + hooks):

Issue PR Scope Depends on
#406 #410 fix: MERGE returns integer counts when no rows are affected —
#407 #411 fix: scope load history to the target table —
#408 #412 feat: CREATE OR REPLACE STAGE, named file formats, and more CSV options in COPY INTO —
#409 #413 feat: PUT over http in server mode, unquoted PUT urls, PUT options, and table stages (+ end-to-end pipeline test) #412, #411

#410, #411 and #412 are independent and can be reviewed in any order; #413 is based on #412's branch and carries #411's commit until they merge, so it is green standalone and its diff shrinks to its own changes as the others land.

This PR remains as the integrated reference — happy for it to be closed in favour of the splits whenever you prefer.

🤖 Generated with Claude Code

tekumara added a commit that referenced this pull request Sep 4, 2026
@arnaldojvg @arnaldo-cai
@claude @tekumara
fix: MERGE returns integer counts when no rows are affected (#410)
755a29f 
duckdb's `COUNT_IF` returns NULL over an empty `merge_candidates` table,
which crashed clients that read the counts, eg:
snowflake-connector-python calls `int()` on them. Snowflake always
returns integers, so wrap each count in `COALESCE(..., 0)`.

Regression tests cover both local mode and the server/connector path
(where the `int()` crash happened).

Fixes #406. Split out of #400 per maintainer request.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: arnaldo-cai <arnaldo.varela@clarity.ai>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: Oliver Mannion <125105+tekumara@users.noreply.github.com>
tekumara pushed a commit that referenced this pull request Sep 4, 2026
@arnaldojvg @arnaldo-cai @claude
fix: scope load history to the target table (#411)
efd0ea2 
Snowflake tracks load metadata per table, so the same file loaded into a
different table is not skipped, and recreating a table resets its load
history.

The dedup check now matches on file name, table name and schema, and
`CREATE TABLE` clears the table's load-history rows.

Fixes #407. Split out of #400 per maintainer request.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: arnaldo-cai <arnaldo.varela@clarity.ai>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
@arnaldojvg

arnaldojvg commented Sep 4, 2026

Copy link
Copy Markdown
Contributor Author

I'll close this since we are managing this on the smaller PRs

@arnaldojvg arnaldojvg closed this Sep 4, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Server mode cannot run a stage/PUT/COPY/MERGE bulk-load pipeline

3 participants

@arnaldojvg @tekumara @arnaldo-cai