fix: transfer PUT files over http in server mode - #422
Open
arnaldojvg wants to merge 1 commit into
Open
arnaldojvg wants to merge 1 commit into
arnaldojvg wants to merge 1 commit into
Conversation
arnaldojvg
force-pushed
the
narr-5-gcs-presigned-url
branch
from
September 12, 2026 11:32
195f6c4 to
407c3cd
Compare
tekumara
added a commit
that referenced
this pull request
Sep 19, 2026
Alternative to #420, trimmed as discussed there: same unquoted-PUT fix, but only `AUTO_COMPRESS` is honoured — `parallel`, `sourceCompression` and `overwrite` keep their hardcoded defaults (with the existing TODO). We only need `AUTO_COMPRESS=FALSE`: our job PUTs an already gzipped csv, and with `autoCompress: True` it was gzipped a second time so the staged file no longer matched what `COPY INTO` expected. `put_options` stays generic so other options can be wired later. If this one is preferred, #420 can be closed and #422 / #424 rebased onto this branch instead. Part of #409, umbrella #403. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: arnaldo-cai <arnaldo.varela@clarity.ai> Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com> Co-authored-by: Oliver Mannion <125105+tekumara@users.noreply.github.com>
Owner
|
@arnaldojvg FYI this now has merge conflicts |
In server mode the PUT response told the connector to write the file to a local path, which only works when the client shares the server's filesystem. Return a GCS-style stage info with a presigned url instead, so the connector uploads the file over http to a new /fs_bucket route that stores it in the stage's backing directory. GCS is the only location type the connector uploads to via a plain http url, and it re-requests the url with the destination file name, so PUT now accepts a bare file://name.gz source. A PUT with a bound ? target keeps LOCAL_FS because that re-request runs without bindings. Part of tekumara#409, umbrella tekumara#403. Supersedes part of tekumara#413. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
arnaldojvg
force-pushed
the
narr-5-gcs-presigned-url
branch
from
September 21, 2026 12:24
407c3cd to
cc57b24
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
In server mode the PUT response told the connector to write the file to a
LOCAL_FSpath, which only works when the client shares the server's filesystem and breaks when fakesnow runs in a container. The server now returns a GCS-stylestageInfowith a presigned url pointing at a newPUT /fs_bucket/{path}route, so the connector uploads the file over http and the server stores it in the stage's backing directory. GCS is the only location type the connector uploads to via a plain http url, and it re-requests the url with the destination file name (file://data.csv.gz), soput_stagenow includes the url netloc in the source path. A PUT with a bound?target keepsLOCAL_FS, because the connector re-runs that command without bindings.Tests:
test_server_put_presigned_url,test_server_put_qmark_target_stays_local,test_server_bucket_upload_rejects_path_outside_bucketintests/test_server.py. The existing server-mode testtests/test_copy_into.py::test_copy_internal_stage_serveralso goes through the http upload.Part of #409, umbrella #403. Supersedes part of #413.
🤖 Generated with Claude Code