Skip to content

Match LIFT list values ignoring surrounding whitespace - #1200

Draft
imnasnainaec wants to merge 4 commits into
mainfrom
lift-import-trim-possibility-names
Draft

imnasnainaec wants to merge 4 commits into
mainfrom
lift-import-trim-possibility-names

Conversation

@imnasnainaec

@imnasnainaec imnasnainaec commented Oct 23, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

Semantic domain lists from FieldWorks templates before 2017 have trailing spaces in 16 English names ("Necessary ", "Imperative ", "Animism ", most of 9.6.x, ...). Projects created then still have them. A LIFT file from another program (here, The Combine; see sillsdev/TheCombine#4410) writes the trimmed 9.4.2.3 Necessary. That doesn't match the key 9.4.2.3 Necessary that FLEx builds from the project, so import adds a duplicate domain 9.4.2.3 Necessary - 9.4.2.3 Necessary.

 TryGetPossibilityMatchingTrait(value)
-  try value, lower(value)
+  TryGetPossibilityMatchingExactly(value)       // value, lower(value)
+  else TryGetPossibilityMatchingTrimmed(value)  // scan for key.Trim() == value.Trim() (or lowercase)

-FindComplexFormType(old entry-type) / FindVariantType(old entry-type)
+FindOldEntryType(old entry-type)
+  exact complex form type
+  else exact variant type
+  else trimmed complex form type
+  else trimmed variant type

The possibility maps are unchanged, so an exact match always wins over a trimmed one. The scan also finds items created earlier in the same import, so whitespace variants within one file don't create duplicates. The scan runs only on a miss, which happens about once per distinct new value, since the created item is then in the map.

Evidence

New tests in LiftMergerTests:

existing domain "9.4.2.3" / "Necessary ";  import "9.4.2.3 Necessary"
  -> no new domain; sense links to the existing one
existing domains "Necessary " then "Necessary";  import "9.4.2.3 Necessary"
  -> sense links to "Necessary" (exact beats trimmed)
no existing domain;  import "9.4.2.3 Necessary " and "9.4.2.3 Necessary", both orders
  -> one domain created, both senses link to it
existing variant type "Foo ";  import entry-type "Foo" with a component relation
  -> variant reference to "Foo "; no new complex form type
existing complex form type "Foo " and variant type "Foo";  import entry-type "Foo"
  -> variant reference to "Foo" (exact variant beats trimmed complex form)
  • Before:

    • ImportSemanticDomainTrait_MatchesExistingDomainWithUntrimmedName on main: Should not create a new semantic domain. Expected: property Count equal to 1 But was: 2
    • ImportOldEntryTypeTrait_MatchesExistingVariantTypeWithUntrimmedName without trimmed entry-type lookup: Expected: 0 But was: 1 (complex form reference instead of variant)
    • ImportOldEntryTypeTrait_PrefersExactVariantTypeOverTrimmedComplexFormType with trimmed lookup but complex form types checked first: Expected: 0 But was: 1

    After: all of LiftMergerTests: 69 passed, 0 failed, 2 skipped.

Merge Danger

Door: two-way

Import-time matching only. No model or data migration.

Blast Radius: LIFT import

The trimmed fallback applies to every list whose LIFT traits go through TryGetPossibilityMatchingTrait: semantic domains, parts of speech, complex form and variant types (including the old entry-type trait), anthropology codes, dialects, status, domain, sense, usage and translation types, locations, languages and publication types. A value that differs from an existing item only by surrounding whitespace, and has no exact match, now links to that item instead of creating a new one.

Out of scope: reversal parts of speech and lexical relation types use their own lookups and still match without trimming.

Considered and rejected

  • Storing trimmed aliases in the possibility maps. An alias could claim a key before a later item whose name matched exactly, and items created during import got no alias.
  • Matching lift-ranges semantic domains by label. ProcessSemanticDomain matches range elements only by guid. A range element with an unknown guid creates a new domain whatever its label says, trimmed or not. That's a separate, pre-existing behavior and isn't changed here.

🤖 Generated with Claude Code


This change is Reviewable

LIFT import matched list items, such as semantic domains, only on exact
text. Semantic domain lists from FieldWorks templates before 2017 have
trailing spaces in some English names (e.g., "Necessary "), so a LIFT
value of "9.4.2.3 Necessary" from another program created a duplicate
domain instead of linking to the existing one.

Also key possibility lookups on names and abbreviations trimmed of
surrounding whitespace, and fall back to the trimmed value when looking
up a trait.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions

github-actions Bot commented Oct 23, 2026 •

Copy link
Copy Markdown

NUnit Tests

    1 files  ±0      1 suites  ±0   14m 14s ⏱️ + 5m 40s
6 822 tests +6  6 737 ✅ +6  85 💤 ±0  0 ❌ ±0 
6 831 runs  +6  6 746 ✅ +6  85 💤 ±0  0 ❌ ±0 

Results for commit 13023b8. ± Comparison against base commit d0d85cb.

♻️ This comment has been updated with latest results.

@codecov-commenter

codecov-commenter commented Oct 23, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 80.64516% with 6 lines in your changes missing coverage. Please review.
✅ Project coverage is 39.90%. Comparing base (d0d85cb) to head (13023b8).

Files with missing lines Patch % Lines
Src/LexText/LexTextControls/LiftMergerRanges.cs 80.00% 3 Missing and 3 partials ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #1200      +/-   ##
==========================================
+ Coverage   39.88%   39.90%   +0.01%     
==========================================
  Files        1542     1542              
  Lines      355361   355378      +17     
  Branches    41320    41326       +6     
==========================================
+ Hits       141753   141807      +54     
+ Misses     184253   184222      -31     
+ Partials    29355    29349       -6     
Files with missing lines Coverage Δ
...LexTextControls/LiftMergerSupportCodeAndClasses.cs 58.87% <100.00%> (+1.01%) ⬆️
Src/LexText/LexTextControls/LiftMergerRanges.cs 73.77% <80.00%> (+1.22%) ⬆️

... and 3 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Storing trimmed aliases in the possibility maps let an alias claim a
key before an item whose name matches exactly, and items created during
import got no alias, so whitespace variants within one file still
created duplicates.

Keep the maps as built from the project, and only when the exact and
lowercase lookups miss, scan for a key that matches once trimmed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
imnasnainaec and others added 2 commits October 23, 2026 11:34
Resolving the old entry-type trait looked up complex form and variant
types by exact or lowercase text only, so an existing variant type
"Foo " was missed and a new complex form type "Foo" was created. Use
the same lookup as the other LIFT trait values.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Resolving an old entry-type value checked complex form types first,
including a trimmed match, so a complex form type "Foo " beat a variant
type that matched "Foo" exactly. Try exact matches on both lists before
trimmed matches on either; complex form types still win at equal
strength.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants