fix(gfql): temporal comparison and arithmetic conformance (#1915) - #1919
fix(gfql): temporal comparison and arithmetic conformance (#1915)#1919lmeyerov wants to merge 6 commits into
Conversation
Round-010 probe (~282 queries x both engines, oracles written before the sweep).
Five silent-wrong defects, each pinned on both engines in
graphistry/tests/compute/gfql/test_temporal_and_union_semantics_1915.py (83 cases).
B-1 (severe) `MATCH (n) WHERE n.ts > datetime('...')` matched ZERO rows on a
tz-naive datetime64 column, across ~10 spellings, and `NOT (...)` returned every
non-null row. Two compounding defects:
(a) row/pipeline.py `_gfql_safe_mixed_comparison_op` swallowed a per-element
TypeError into False. openCypher: an incomparable comparison is NULL -- the
rule this same file already states 200 lines up for cross-type ordering. Now
pd.NA. This is a GENERAL trap, not only temporal: it is exactly why
`RETURN '1.0' < 1.0` answered false. Two openCypher TCK scenarios
(expr-comparison2-6-3/6-4) flip to matching their oracle as a result.
(b) row/ordering.py recognised only TEXT temporals, so a real datetime column
skipped the temporal comparison path. Added order_detect_native_temporal_mode
+ a datetime_native branch in build_temporal_sort_columns that keys off UTC
epoch nanoseconds rebased onto the text path's Julian-day scale (tz-aware
normalises to UTC). Kept separate from the text detector on purpose so ORDER
BY keeps its native-dtype sort path. The sibling spellings that already
worked -- localdatetime(), date(), a plain ISO string, a tz-AWARE column --
are pinned as the regression fence.
B-2 `date('2020-01-02') + duration('P1D')` was Python string concatenation
('2020-01-02P1D'), and inside WHERE the concatenated text changed the ROW SET,
while its sibling `-` declined typed. CHOSE real arithmetic over a symmetric
decline: temporal literals are constant-folded in temporal/folding.py
(_fold_temporal_arithmetic), so date/datetime/localdatetime/time +/- duration,
duration +/- duration and duration * | / number all evaluate before engine
dispatch -- both engines get the same value with no per-engine implementation.
Month arithmetic clamps to end-of-month and a Duration's month/day/second groups
stay separate (date + PT25H is a no-op, date + P1D advances one day), via the new
parse/format_duration_calendar_components in temporal/durations.py. The residual
COLUMN form is not foldable, so `+` now declines typed exactly like `-`
(row/pipeline.py); ORDER BY keeps its own column+duration path, and ordinary
string concatenation is untouched because only ISO-duration-shaped operands
engage.
B-3 polars compared a String ISO column against the Z-suffixed literal
lexicographically, so `=` never matched and `>=` silently degraded to `>`.
B-4 `IN [datetime('...')]` returned nothing on BOTH engines -- structurally
invisible to differential testing, so it is pinned per engine.
Normalized both: polars drops a trailing UTC 'Z' from both operands when a
Z-suffixed ISO literal meets a String column (comparison + is_in), pandas routes
IN through the same temporal comparison its `=` already used.
A-2 polars UNION concat (vertical_relaxed) stringified a non-string branch and
then deleted rows via the DISTINCT -- an EMPTY branch alone was enough
(['7','8','9'] instead of [7,8,9]). Empty branches now contribute neither rows
nor type, and a genuinely unrepresentable mixed-type UNION declines typed
(a polars column cannot hold both branches' values); numeric-vs-numeric widening
stays served since openCypher 1 = 1.0 is true.
A-1 pandas UNION failed to dedup NaN against None (2 rows where openCypher gives
1); A-3 BOOLEAN and INTEGER were conflated across branches on both engines
(openCypher true = 1 is FALSE). UNION DISTINCT now dedups on an openCypher
identity key on pandas, and bool-vs-numeric branches widen to object instead of
upcasting True to 1.
NOT attempted (noted as untouched): A-4 UNION column-name ordering, B-7 raw
backend exceptions (n.dt_utc > n.dt_native still raises a raw pandas TypeError),
B-8 keyword property names.
Gates: graphistry/tests/compute/gfql 96 failed / 8676 passed / 707 skipped /
35 xfailed -- failure set BYTE-IDENTICAL to the base (96 failed / 8593 passed,
all cudf on this GPU-less box); test_compute_chain + test_compute_hops +
fast-path/index/semantics suites 79 failed / 1060 passed, identical at base;
cypher/test_lowering.py -k "not cudf" 1441 passed; ruff clean; bin/typecheck.sh
329 files clean; ci_type_hygiene_guard + ci_cypher_surface_guard pass.
TCK (scratchpad tck-gfql, committed separately): 0 failed, 4143 passed /
6 skipped / 690 xfailed after promoting expr-comparison2-6-3/6-4.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MF7uRZLKZaD6Q9FGWSmyXi
|
GPU sweep @211c2885b: 9430 passed, 6 failed — with the parent cuDF lane genuinely executed: The flagged cuDF divergence is CONFIRMED and newly introduced by this PR.
Only genuinely-incomparable ordering comparisons, and only when the boolean is observed — via Good news on the severe finding: B-1 is fixed on both engines — the const-fold and ordering work is engine-agnostic — and at the parent those same cuDF cases raised |
The #1915 fix grew temporal/folding.py and temporal/durations.py with the constant-fold for openCypher temporal arithmetic. The end-to-end query paths were pinned in test_temporal_and_union_semantics_1915.py, but the branch structure was not, and CI's coverage audit failed both files: folding.py 53.89% vs floor 90.00% (-36.11) durations.py 65.68% vs floor 75.47% (-9.79) Adds 232 direct unit tests. Every helper folded here is pure (literal in -> literal or None out), so inputs are built straight as Literal/BinaryOp/ FunctionCall nodes and results are asserted structurally, no engine execution -- the style of cypher/test_aggregate_identity_branches.py and cypher/test_flatten_pure_carry_optional.py. DECLINE (helper returned None, leave the node for the engine) is distinguished from a folded NULL (Literal(None)) by an explicit sentinel, because conflating the two is exactly how B-2 shipped. Covered: the month/day/second component-group split (date + PT25H is a no-op, date + P1D advances one day, P0.5D spills into the seconds group); month-end clamping incl. leap years (2020-01-31 + P1M -> 2020-02-29, 2019 -> 02-28, 2020-02-29 + P1Y -> 2021-02-28); the full operand-type matrix for + - * / including every pair that must NOT fold (number / duration, duration * duration, duration - temporal, bool operands, ordinary string concatenation); malformed and unparseable duration text; per-token and prefix sign handling; fractional and negative multipliers, division by zero, and the fractional-month decline; the wide-year duration.between/inSeconds fallback with its month and year borrows; the epoch constructors; resolve_duration_text_property; and the pre-parse text rewriter. Measured with python3 -m coverage run --include="*/temporal/folding.py,*/temporal/durations.py" \ -m pytest -q <audit args> (one comma-separated --include: repeating the flag silently keeps only the last pattern) over the ci-pandas-py3.12 audit slice: folding.py 84.97% (164/193) -> 98.96% (191/193) durations.py 79.54% (241/303) -> 99.01% (300/303) Local before-numbers read higher than CI's for both files; the after-numbers do not depend on that, because the new file alone reaches the same 191/193 and 300/303 and the remaining lines are unreachable. Floors are set just under the measured values (98.90 / 98.95) so a rounding difference cannot fail the gate while a single lost statement (0.52 / 0.33 points) still would. The 5 uncovered lines left are unreachable by construction and each is a type-narrowing guard mypy requires: folding.py:91 _shift_temporal_value's date_value-is-None return -- only time/localtime have no date and they return above it folding.py:234 _replace_current's normalize-failed return -- the regex only matches the five names _current_temporal_literal always serves durations.py:54-56 the Y arm of parse_temporal_sort_duration_components -- that function lexes with _DAY_TIME_DURATION_TOKEN_RE, which has no Y unit, so the arm is dead; the decline is pinned instead. Recommend deleting the arm in a follow-up. Gates: graphistry/tests/compute/gfql 96 failed / 8908 passed / 707 skipped / 35 xfailed -- failure set BYTE-IDENTICAL to the 96 at 211c288, passed +232; test_temporal_and_union_semantics_1915.py 83 passed; ruff clean; bin/typecheck.sh 329 files clean; ci_type_hygiene_guard and ci_cypher_surface_guard pass. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MF7uRZLKZaD6Q9FGWSmyXi
…verflowError
Both carry sites used timedelta, which raises OverflowError rather than the
ValueError the constructors raise, so a raw OverflowError escaped the fold for
date('9999-12-31') + P1D and localdatetime('9999-12-31T23:00:00') + PT2H.
Found by the coverage sweep of this module and reported rather than pinned.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MF7uRZLKZaD6Q9FGWSmyXi
The B-1 native-datetime comparison key assumed `series.astype("int64")` on a
datetime column returns epoch NANOSECONDS. It returns the column's own storage
ticks, and pandas 3 dropped the datetime64[ns] coercion: `pd.to_datetime([...])`
now yields datetime64[us] where pandas 2 yielded datetime64[ns]. Treating
microsecond ticks as nanoseconds divided by 86_400e9, so every 2020-2021 instant
collapsed onto Julian day 2440606 (1970-01-19) and the whole family of
`WHERE n.ts <op> datetime('...')` spellings compared equal instead of ordering
-- including the localdatetime()/tz-aware fences that predate the B-1 fix, since
they share the same native-dtype key.
This is a resolution bug, not a version bug: on pandas 2 the same query already
returned [] for a datetime64[us]/[ms]/[s] column: pandas 3 only made the coarser
unit the default, which is why CI's test-polars (3.11) lane saw it and the 3.10
lane did not. Fixed behaviorally by asking the dtype for its unit
(_native_temporal_unit_nanoseconds) and keying in ticks, with the nanosecond
duration shift split into whole ticks plus a sub-tick nanosecond remainder.
Keying in ticks also keeps the full datetime64[s]/[ms] range, which an int64
nanosecond count would overflow. No pandas version gate.
Pinned over all four resolutions (s/ms/us/ns) rather than over pandas versions,
so the pin fails on pandas 2 as well if the assumption returns.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MF7uRZLKZaD6Q9FGWSmyXi
The pandas-3 fix added unit-detection branches that the resolution-parametrized end-to-end pins cannot reach: the string-fallback path (a dtype exposing no .unit), a non-string .unit attribute, and the unparseable default. Also pins the invariant the fix rests on -- ticks * unit_nanoseconds is constant across resolutions -- and tz normalisation to UTC. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MF7uRZLKZaD6Q9FGWSmyXi
Cascading base update after #1914 took its base. ZERO conflicted files -- the base moved but every hunk merged cleanly, so there is nothing to resolve and no side was taken over another. Committed anyway so the PR stops being CONFLICTING against its moved base and CI can build refs/pull/1919/merge. Gates: no conflict markers, ruff clean, type-hygiene guard clean, cypher surface guard pass, mypy shows only the 4 known polars-skew errors. test_native_temporal_resolution + test_temporal_arithmetic_folding_branches + test_temporal_and_union_semantics_1915 + test_polars_lane_completeness = 347 passed, 0 failed. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MF7uRZLKZaD6Q9FGWSmyXi
Stacked on #1914. Round-010's temporal/UNION findings, B-1 through A-3.
B-1 (severe) —
datetime()filters silently matched ZERO rows on tz-naive datetime columns, across ~10 spellings. Two compounding defects:(a)
_gfql_safe_mixed_comparison_opturned a per-elementTypeErrorintoFalse; nowpd.NA, per openCypher and per this same file's own stated intent 200 lines above. This is general, not temporal — it is also whyRETURN '1.0' < 1.0answeredfalse. Independent confirmation: two openCypher TCK scenarios flipped fromsuccess_wrong_rowstosuccess_matches_expected, and the wrong-row debt bucket is now empty.(b) A native-datetime detector feeding the temporal comparison path. Deliberately kept separate from
order_detect_temporal_moderather than extending it: that function also drives ORDER BY, and routing real datetime columns through the temporal-key sort would have changed null placement for a working feature.B-2 — temporal
+was string concatenation. Chose real arithmetic over a decline. Temporal literals lower to ISO text before the AST exists, so provenance is gone by evaluator time — but it is intact at fold time, so the arithmetic is constant-folded before engine dispatch and both engines get identical values with no per-engine implementation. Duration month/day/second groups stay separate, sodate + PT25His a no-op,date + P1Dadvances a day, anddate('2020-01-31') + P1Mclamps to 02-29. The non-foldable column form now declines typed exactly like its-sibling.B-3/B-4 — Z-suffix mismatch. polars drops a trailing UTC
Zfrom both operands when a Z-suffixed literal meets a String column (restoring equality and preserving lexicographic order); pandas routesIN [temporal]through the same temporal=with Cypher 3-valued OR. Pinned per engine, not by parity — B-4 was wrong on both engines, so parity testing could never have caught it.A-1/A-2/A-3 — UNION. Empty branches no longer contribute a type (which alone fixes
['7','8','9']→[7,8,9]); NaN dedups against None; BOOLEAN keys apart from INTEGER, with concat no longer upcastingTrueto1before dedup runs. Genuinely unrepresentable mixed-type branches decline typed on polars (a polars column cannot hold both'7'and7) — pinned in both directions.83 new pins. Zero pygraphistry test changes — nothing asserted the old behavior. Gates: full-tree failure set byte-identical (96 local GPU-less failures both sides), TCK 0 failed with 2 promotions cascaded, typecheck 329, ruff + guards clean.
Honest residuals: B-5 untouched (fixing it would contradict the text-path convention that naive text denotes UTC, which B-1b and B-3 both rely on). B-7's tz-aware-vs-naive column case still raises a raw pandas TypeError — it hits a pushdown path before the row evaluator. And a new cuDF divergence to verify on GPU: the mixed-comparison early-return at
row/pipeline.py:608-611yields all-Falsefor cuDF to avoid a host round-trip, so pandas now returns null where cuDF still returns false — deliberately left rather than blind-editing GPU code that could not be executed locally.🤖 Generated with Claude Code
https://claude.ai/code/session_01MF7uRZLKZaD6Q9FGWSmyXi