Skip to content

fix(gfql): temporal comparison and arithmetic conformance (#1915) - #1919

Open
lmeyerov wants to merge 6 commits into
fix/gfql-1913-stale-index-rebindfrom
fix/gfql-1915-temporal-and-union
Open

fix(gfql): temporal comparison and arithmetic conformance (#1915)#1919
lmeyerov wants to merge 6 commits into
fix/gfql-1913-stale-index-rebindfrom
fix/gfql-1915-temporal-and-union

Conversation

@lmeyerov

Copy link
Copy Markdown
Contributor

Stacked on #1914. Round-010's temporal/UNION findings, B-1 through A-3.

B-1 (severe) — datetime() filters silently matched ZERO rows on tz-naive datetime columns, across ~10 spellings. Two compounding defects:
(a) _gfql_safe_mixed_comparison_op turned a per-element TypeError into False; now pd.NA, per openCypher and per this same file's own stated intent 200 lines above. This is general, not temporal — it is also why RETURN '1.0' < 1.0 answered false. Independent confirmation: two openCypher TCK scenarios flipped from success_wrong_rows to success_matches_expected, and the wrong-row debt bucket is now empty.
(b) A native-datetime detector feeding the temporal comparison path. Deliberately kept separate from order_detect_temporal_mode rather than extending it: that function also drives ORDER BY, and routing real datetime columns through the temporal-key sort would have changed null placement for a working feature.

B-2 — temporal + was string concatenation. Chose real arithmetic over a decline. Temporal literals lower to ISO text before the AST exists, so provenance is gone by evaluator time — but it is intact at fold time, so the arithmetic is constant-folded before engine dispatch and both engines get identical values with no per-engine implementation. Duration month/day/second groups stay separate, so date + PT25H is a no-op, date + P1D advances a day, and date('2020-01-31') + P1M clamps to 02-29. The non-foldable column form now declines typed exactly like its - sibling.

B-3/B-4 — Z-suffix mismatch. polars drops a trailing UTC Z from both operands when a Z-suffixed literal meets a String column (restoring equality and preserving lexicographic order); pandas routes IN [temporal] through the same temporal = with Cypher 3-valued OR. Pinned per engine, not by parity — B-4 was wrong on both engines, so parity testing could never have caught it.

A-1/A-2/A-3 — UNION. Empty branches no longer contribute a type (which alone fixes ['7','8','9'][7,8,9]); NaN dedups against None; BOOLEAN keys apart from INTEGER, with concat no longer upcasting True to 1 before dedup runs. Genuinely unrepresentable mixed-type branches decline typed on polars (a polars column cannot hold both '7' and 7) — pinned in both directions.

83 new pins. Zero pygraphistry test changes — nothing asserted the old behavior. Gates: full-tree failure set byte-identical (96 local GPU-less failures both sides), TCK 0 failed with 2 promotions cascaded, typecheck 329, ruff + guards clean.

Honest residuals: B-5 untouched (fixing it would contradict the text-path convention that naive text denotes UTC, which B-1b and B-3 both rely on). B-7's tz-aware-vs-naive column case still raises a raw pandas TypeError — it hits a pushdown path before the row evaluator. And a new cuDF divergence to verify on GPU: the mixed-comparison early-return at row/pipeline.py:608-611 yields all-False for cuDF to avoid a host round-trip, so pandas now returns null where cuDF still returns false — deliberately left rather than blind-editing GPU code that could not be executed locally.

🤖 Generated with Claude Code

https://claude.ai/code/session_01MF7uRZLKZaD6Q9FGWSmyXi

Round-010 probe (~282 queries x both engines, oracles written before the sweep).
Five silent-wrong defects, each pinned on both engines in
graphistry/tests/compute/gfql/test_temporal_and_union_semantics_1915.py (83 cases).

B-1 (severe) `MATCH (n) WHERE n.ts > datetime('...')` matched ZERO rows on a
tz-naive datetime64 column, across ~10 spellings, and `NOT (...)` returned every
non-null row. Two compounding defects:
(a) row/pipeline.py `_gfql_safe_mixed_comparison_op` swallowed a per-element
    TypeError into False. openCypher: an incomparable comparison is NULL -- the
    rule this same file already states 200 lines up for cross-type ordering. Now
    pd.NA. This is a GENERAL trap, not only temporal: it is exactly why
    `RETURN '1.0' < 1.0` answered false. Two openCypher TCK scenarios
    (expr-comparison2-6-3/6-4) flip to matching their oracle as a result.
(b) row/ordering.py recognised only TEXT temporals, so a real datetime column
    skipped the temporal comparison path. Added order_detect_native_temporal_mode
    + a datetime_native branch in build_temporal_sort_columns that keys off UTC
    epoch nanoseconds rebased onto the text path's Julian-day scale (tz-aware
    normalises to UTC). Kept separate from the text detector on purpose so ORDER
    BY keeps its native-dtype sort path. The sibling spellings that already
    worked -- localdatetime(), date(), a plain ISO string, a tz-AWARE column --
    are pinned as the regression fence.

B-2 `date('2020-01-02') + duration('P1D')` was Python string concatenation
('2020-01-02P1D'), and inside WHERE the concatenated text changed the ROW SET,
while its sibling `-` declined typed. CHOSE real arithmetic over a symmetric
decline: temporal literals are constant-folded in temporal/folding.py
(_fold_temporal_arithmetic), so date/datetime/localdatetime/time +/- duration,
duration +/- duration and duration * | / number all evaluate before engine
dispatch -- both engines get the same value with no per-engine implementation.
Month arithmetic clamps to end-of-month and a Duration's month/day/second groups
stay separate (date + PT25H is a no-op, date + P1D advances one day), via the new
parse/format_duration_calendar_components in temporal/durations.py. The residual
COLUMN form is not foldable, so `+` now declines typed exactly like `-`
(row/pipeline.py); ORDER BY keeps its own column+duration path, and ordinary
string concatenation is untouched because only ISO-duration-shaped operands
engage.

B-3 polars compared a String ISO column against the Z-suffixed literal
lexicographically, so `=` never matched and `>=` silently degraded to `>`.
B-4 `IN [datetime('...')]` returned nothing on BOTH engines -- structurally
invisible to differential testing, so it is pinned per engine.
Normalized both: polars drops a trailing UTC 'Z' from both operands when a
Z-suffixed ISO literal meets a String column (comparison + is_in), pandas routes
IN through the same temporal comparison its `=` already used.

A-2 polars UNION concat (vertical_relaxed) stringified a non-string branch and
then deleted rows via the DISTINCT -- an EMPTY branch alone was enough
(['7','8','9'] instead of [7,8,9]). Empty branches now contribute neither rows
nor type, and a genuinely unrepresentable mixed-type UNION declines typed
(a polars column cannot hold both branches' values); numeric-vs-numeric widening
stays served since openCypher 1 = 1.0 is true.

A-1 pandas UNION failed to dedup NaN against None (2 rows where openCypher gives
1); A-3 BOOLEAN and INTEGER were conflated across branches on both engines
(openCypher true = 1 is FALSE). UNION DISTINCT now dedups on an openCypher
identity key on pandas, and bool-vs-numeric branches widen to object instead of
upcasting True to 1.

NOT attempted (noted as untouched): A-4 UNION column-name ordering, B-7 raw
backend exceptions (n.dt_utc > n.dt_native still raises a raw pandas TypeError),
B-8 keyword property names.

Gates: graphistry/tests/compute/gfql 96 failed / 8676 passed / 707 skipped /
35 xfailed -- failure set BYTE-IDENTICAL to the base (96 failed / 8593 passed,
all cudf on this GPU-less box); test_compute_chain + test_compute_hops +
fast-path/index/semantics suites 79 failed / 1060 passed, identical at base;
cypher/test_lowering.py -k "not cudf" 1441 passed; ruff clean; bin/typecheck.sh
329 files clean; ci_type_hygiene_guard + ci_cypher_surface_guard pass.
TCK (scratchpad tck-gfql, committed separately): 0 failed, 4143 passed /
6 skipped / 690 xfailed after promoting expr-comparison2-6-3/6-4.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MF7uRZLKZaD6Q9FGWSmyXi
@lmeyerov

Copy link
Copy Markdown
Contributor Author

GPU sweep @211c2885b: 9430 passed, 6 failed — with the parent c6fc1fcda measured in the same session at 9347 passed, 6 failed. Delta +83 passed, 0 new failures, matching the 83 new pins exactly. The 6 failures are byte-identical on both commits and are not from this PR: they are a test-helper bug in the parent's newly-GPU-promoted #1913 pins (AttributeError: 'Series' object has no attribute 'tolist'). Fixed on that branch as be80784b4 — the helper assumed pandas-vs-polars, but a cuDF/polars-gpu request can be served back as a polars Series, and cuDF's Series raises from both to_list and tolist by design, so attribute probing can't discriminate; it now sniffs the module like dtypes.py does and uses to_arrow().to_pylist().

cuDF lane genuinely executed: -k "cudf or gpu" -rs → 1344 passed, single skip is a missing igraph, zero GPU-availability skips.

The flagged cuDF divergence is CONFIRMED and newly introduced by this PR. row/pipeline.py:607-620 early-returns False for any cuDF operand while the pandas path now correctly appends pd.NA. At the parent both engines agreed (both wrong); this fix moved pandas to the openCypher answer and left cuDF behind:

shape oracle pandas cuDF
RETURN '1.0' < 1.0 null None ✓ False
RETURN NOT ('1.0' < 1.0) null None ✓ True
WHERE NOT (n.str_col < 1.0) [] [] ['a','b']
WHERE n.str_col < 1.0 (bare) [] [] [] ✓ masked

Only genuinely-incomparable ordering comparisons, and only when the boolean is observed — via NOT or by projecting it. A bare positive WHERE masks it because dropping-on-null and dropping-on-false coincide. The new pin file is pandas/polars only, so nothing catches this; a follow-up will either extend the cuDF path or pin the divergence explicitly.

Good news on the severe finding: B-1 is fixed on both engines — the const-fold and ordering work is engine-agnostic — and at the parent those same cuDF cases raised NotImplementedError: Binary operations with timezone aware operands is not supported, so this PR also repaired cuDF there.

lmeyerov and others added 2 commits August 15, 2026 07:57
The #1915 fix grew temporal/folding.py and temporal/durations.py with the
constant-fold for openCypher temporal arithmetic. The end-to-end query paths
were pinned in test_temporal_and_union_semantics_1915.py, but the branch
structure was not, and CI's coverage audit failed both files:

  folding.py    53.89% vs floor 90.00% (-36.11)
  durations.py  65.68% vs floor 75.47% (-9.79)

Adds 232 direct unit tests. Every helper folded here is pure (literal in ->
literal or None out), so inputs are built straight as Literal/BinaryOp/
FunctionCall nodes and results are asserted structurally, no engine execution
-- the style of cypher/test_aggregate_identity_branches.py and
cypher/test_flatten_pure_carry_optional.py. DECLINE (helper returned None,
leave the node for the engine) is distinguished from a folded NULL
(Literal(None)) by an explicit sentinel, because conflating the two is exactly
how B-2 shipped.

Covered: the month/day/second component-group split (date + PT25H is a no-op,
date + P1D advances one day, P0.5D spills into the seconds group); month-end
clamping incl. leap years (2020-01-31 + P1M -> 2020-02-29, 2019 -> 02-28,
2020-02-29 + P1Y -> 2021-02-28); the full operand-type matrix for
+ - * / including every pair that must NOT fold (number / duration, duration *
duration, duration - temporal, bool operands, ordinary string concatenation);
malformed and unparseable duration text; per-token and prefix sign handling;
fractional and negative multipliers, division by zero, and the fractional-month
decline; the wide-year duration.between/inSeconds fallback with its month and
year borrows; the epoch constructors; resolve_duration_text_property; and the
pre-parse text rewriter.

Measured with
  python3 -m coverage run --include="*/temporal/folding.py,*/temporal/durations.py" \
    -m pytest -q <audit args>
(one comma-separated --include: repeating the flag silently keeps only the last
pattern) over the ci-pandas-py3.12 audit slice:

  folding.py    84.97% (164/193) -> 98.96% (191/193)
  durations.py  79.54% (241/303) -> 99.01% (300/303)

Local before-numbers read higher than CI's for both files; the after-numbers do
not depend on that, because the new file alone reaches the same 191/193 and
300/303 and the remaining lines are unreachable. Floors are set just under the
measured values (98.90 / 98.95) so a rounding difference cannot fail the gate
while a single lost statement (0.52 / 0.33 points) still would.

The 5 uncovered lines left are unreachable by construction and each is a
type-narrowing guard mypy requires:
  folding.py:91  _shift_temporal_value's date_value-is-None return -- only
                 time/localtime have no date and they return above it
  folding.py:234 _replace_current's normalize-failed return -- the regex only
                 matches the five names _current_temporal_literal always serves
  durations.py:54-56 the Y arm of parse_temporal_sort_duration_components --
                 that function lexes with _DAY_TIME_DURATION_TOKEN_RE, which has
                 no Y unit, so the arm is dead; the decline is pinned instead.
                 Recommend deleting the arm in a follow-up.

Gates: graphistry/tests/compute/gfql 96 failed / 8908 passed / 707 skipped /
35 xfailed -- failure set BYTE-IDENTICAL to the 96 at 211c288, passed +232;
test_temporal_and_union_semantics_1915.py 83 passed; ruff clean;
bin/typecheck.sh 329 files clean; ci_type_hygiene_guard and
ci_cypher_surface_guard pass.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MF7uRZLKZaD6Q9FGWSmyXi
…verflowError

Both carry sites used timedelta, which raises OverflowError rather than the
ValueError the constructors raise, so a raw OverflowError escaped the fold for
date('9999-12-31') + P1D and localdatetime('9999-12-31T23:00:00') + PT2H.
Found by the coverage sweep of this module and reported rather than pinned.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MF7uRZLKZaD6Q9FGWSmyXi
lmeyerov and others added 3 commits August 15, 2026 09:16
The B-1 native-datetime comparison key assumed `series.astype("int64")` on a
datetime column returns epoch NANOSECONDS. It returns the column's own storage
ticks, and pandas 3 dropped the datetime64[ns] coercion: `pd.to_datetime([...])`
now yields datetime64[us] where pandas 2 yielded datetime64[ns]. Treating
microsecond ticks as nanoseconds divided by 86_400e9, so every 2020-2021 instant
collapsed onto Julian day 2440606 (1970-01-19) and the whole family of
`WHERE n.ts <op> datetime('...')` spellings compared equal instead of ordering
-- including the localdatetime()/tz-aware fences that predate the B-1 fix, since
they share the same native-dtype key.

This is a resolution bug, not a version bug: on pandas 2 the same query already
returned [] for a datetime64[us]/[ms]/[s] column: pandas 3 only made the coarser
unit the default, which is why CI's test-polars (3.11) lane saw it and the 3.10
lane did not. Fixed behaviorally by asking the dtype for its unit
(_native_temporal_unit_nanoseconds) and keying in ticks, with the nanosecond
duration shift split into whole ticks plus a sub-tick nanosecond remainder.
Keying in ticks also keeps the full datetime64[s]/[ms] range, which an int64
nanosecond count would overflow. No pandas version gate.

Pinned over all four resolutions (s/ms/us/ns) rather than over pandas versions,
so the pin fails on pandas 2 as well if the assumption returns.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MF7uRZLKZaD6Q9FGWSmyXi
The pandas-3 fix added unit-detection branches that the resolution-parametrized
end-to-end pins cannot reach: the string-fallback path (a dtype exposing no
.unit), a non-string .unit attribute, and the unparseable default. Also pins the
invariant the fix rests on -- ticks * unit_nanoseconds is constant across
resolutions -- and tz normalisation to UTC.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MF7uRZLKZaD6Q9FGWSmyXi
Cascading base update after #1914 took its base. ZERO conflicted files --
the base moved but every hunk merged cleanly, so there is nothing to resolve
and no side was taken over another. Committed anyway so the PR stops being
CONFLICTING against its moved base and CI can build refs/pull/1919/merge.

Gates: no conflict markers, ruff clean, type-hygiene guard clean, cypher
surface guard pass, mypy shows only the 4 known polars-skew errors.
test_native_temporal_resolution + test_temporal_arithmetic_folding_branches +
test_temporal_and_union_semantics_1915 + test_polars_lane_completeness =
347 passed, 0 failed.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MF7uRZLKZaD6Q9FGWSmyXi
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant