Skip to content

fix(descriptors): honor TextMatch whole-word boundaries - #1972

Open
March-7 wants to merge 1 commit into
evidentlyai:mainfrom
March-7:codex/round116-text-match-word-boundaries
Open

March-7 wants to merge 1 commit into
evidentlyai:mainfrom
March-7:codex/round116-text-match-word-boundaries

Conversation

@March-7

@March-7 March-7 commented Oct 22, 2026

Copy link
Copy Markdown

TextMatch(word_boundaries=True) still used substring membership after tokenizing the text, so cat matched bobcat. The same false positives reached the word descriptors and their report counts.

Match literal items with word-character boundaries in the contains/not-contains path. Preserve the existing text normalization, case and lemma pipeline, substring mode, exact/regex modes, and empty-item/list behavior. Multiword, punctuation, Unicode, whitespace and literal regex-character cases are covered.

Fixes #1971.

Validation on Python 3.12.13, pandas 2.2.3 and NumPy 2.5.4:

  • Focused descriptor file: 17 failed / 63 passed before the product change, then 80 passed; 6 skipped and 5 deselected in both runs. The 47 new cases include 17 that failed on the base.
  • Separate public-API oracle: 15 failed / 45 passed before, then all 60 row assertions passed. The same 10-row CategoryCount report changed from 6 matches to the expected 3.
  • Existing compatibility subset: 24 passed, 26 deselected.
  • Ruff 0.3.7 lint/format checks and git diff --check passed.

Configured mypy 1.1.1 is blocked by a syntax error in the installed NumPy type stubs; base and patched runs produce the identical error before effective checking. WordNet is absent, so actual lemmatization cases were excluded; no fake lemmatizer was used. The full repository suite and upstream CI were not run.

Prepared with OpenAI Codex assistance. The reproduction and checks above were run locally.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

TextMatch word_boundaries=True still matches substrings inside words

1 participant