Skip to content

Mark HTTP request functions VOLATILE to prevent double execution on constant arguments - #37

Merged
rustyconover merged 1 commit into
Query-farm:v1.5from
asubbarao:fix/v1.5-volatile-http-functions
Sep 8, 2026
Merged

Mark HTTP request functions VOLATILE to prevent double execution on constant arguments#37
rustyconover merged 1 commit into
Query-farm:v1.5from
asubbarao:fix/v1.5-volatile-http-functions

Conversation

@asubbarao

Copy link
Copy Markdown
Contributor

Problem

When an HTTP function call with constant (literal) arguments is nested inside a larger expression, the request is sent twice per query:

LOAD json; LOAD http_client;
SELECT http_post('http://my-endpoint/token', headers => MAP {}, params => MAP {})->>'access_token';
-- my-endpoint receives TWO POST requests for this one call

The query result is still a single, correct row — the duplicate is only visible on the server side. For idempotent GETs that is silently doubled load; for side-effecting requests it is a correctness bug with no client-side symptom. The sharpest example is an OAuth authorization-code exchange: http_post(token_url, ...)->>'access_token' is exactly the idiom this fires on, and the second send replays a one-time code.

Two things narrow when it manifests:

  • Only constant arguments. When any argument comes from a column or a prepared-statement bind, the request is sent once.
  • Only when nested in a larger constant expression. A bare SELECT http_post(...) sends once; http_post(...)->>'status' sends twice.

Root cause

On current v1.5 (f0fedb2), http_get / http_post / http_post_form / http_head are registered with the default FunctionStability::CONSISTENT. That tells DuckDB's optimizer the function is pure, so an all-constant sub-expression is evaluated during constant folding at plan time — and evaluated again at execution time. Two evaluations → two network requests.

HTTP requests are side-effecting, so these functions should be VOLATILE (the same classification DuckDB uses for random(), nextval(), uuid()), which excludes them from constant folding and pins one evaluation per logical call.

main / v1.5.5 already do this via HttpFn (function.SetVolatile()), landed with PR #36. This PR is the same flag on the four functions for the v1.5 line only — not a reopen of #34 (that branch mixed telemetry, submodules, and a merge from main).

Fix

SetVolatile() on each registered ScalarFunction (http_head, both http_get overloads, http_post, http_post_form). No telemetry, submodule, or Vector-API changes.

No behavior change for column/bind arguments (already one send); nested constant-argument calls now send once instead of twice.

Verification

A local HTTP target that logs every inbound request, fired with the nested constant-arg call above (same query, same target, only the extension binary differs):

  • stock build: target logs 2 requests
  • patched build (this PR): target logs 1
# terminal 1 — count inbound requests
node -e 'const fs=require("fs");require("http").createServer((q,s)=>{
  fs.appendFileSync("reqs.log",q.method+" "+q.url+"\n");
  q.on("end",()=>s.end("{}"))}).listen(18476,"127.0.0.1")'

# terminal 2 — note the ->> nesting; a bare SELECT http_post(...) does NOT reproduce
duckdb -unsigned -c "LOAD json; LOAD http_client;
  SELECT http_post('http://127.0.0.1:18476/token', headers => MAP {}, params => MAP {})->>'status';"
grep -c 'POST /token' reqs.log   # stock: 2, patched: 1

…onstant arguments

http_get/http_post/http_post_form/http_head register with the default
FunctionStability::CONSISTENT, which lets the optimizer evaluate an
all-constant call at plan time (constant folding) in addition to execution
time. Concretely, nesting a constant-arg call in a larger expression --
SELECT http_post(url, ...)->>'access_token' -- sends the request twice per
query, while still returning a single correct row, so the duplicate is only
visible server-side. For non-idempotent endpoints (OAuth code exchange,
webhooks, row-creating POSTs) that is a correctness bug.

Network requests are side effects, so these functions must be VOLATILE (same
class as random()/nextval()). Verified against a request-counting server: the
nested constant-arg call sends 2 requests on a stock build and exactly 1 after
this change; bare calls and column/bind arguments already sent once and are
unaffected.
@rustyconover
rustyconover merged commit 9e57133 into Query-farm:v1.5 Sep 8, 2026
9 checks passed
@rustyconover

Copy link
Copy Markdown
Contributor

I'll get the release out shortly.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants