Skip to content

Speed up LevenshteinDistance and spellcheck corrections - #798

Open
TheNetherWatcher wants to merge 2 commits into
NaturalNode:masterfrom
TheNetherWatcher:perf/levenshtein
Open

TheNetherWatcher wants to merge 2 commits into
NaturalNode:masterfrom
TheNetherWatcher:perf/levenshtein

Conversation

@TheNetherWatcher

@TheNetherWatcher TheNetherWatcher commented Nov 4, 2026 •

Copy link
Copy Markdown

LevenshteinDistance and Spellcheck.getCorrections spend most of their time on bookkeeping that doesn't affect the result. This PR removes it; outputs (including the order of corrections) are unchanged.

Levenshtein distance: the plain path (no search, no damerau) built a full matrix where every cell allocated a few objects and picked the cheapest parent with _.min. Only the final cost is returned, so two rows of numbers are enough. Search mode, Damerau variants, and non-string inputs or non-numeric costs still go through the original code.

Spellcheck: edits() deduplicated with filter + indexOf (quadratic in the number of candidate edits), and editsWithMaxDistanceHelper rebuilt its array with concat for each word. It now uses a Set, which keeps first-occurrence order, and push.

Benchmarks

Node 24, Apple M4 Pro. Old and new code compared side by side, with outputs checked equal on 5,000 random distance pairs (mixed costs, including negative and fractional ones), plus search/Damerau pairs and spellcheck expansions. Dictionary: 4,000 words from /usr/share/dict/words.

Case Before After Speedup
Levenshtein, 6 chars (10,000 pairs) 32.8 ms 5.4 ms 6.0x
Levenshtein, 20 chars (3,000 pairs) 75.0 ms 5.5 ms 13.5x
Levenshtein, 100 chars (300 pairs) 180.3 ms 10.6 ms 17.0x
Levenshtein, 500 chars (12 pairs) 290.4 ms 9.7 ms 30.1x
getCorrections, distance 1 (5 words) 0.7 ms 0.3 ms 2.4x
getCorrections, distance 2 (3 words) 208.2 ms 42.7 ms 4.9x

Tests

  • Randomized check of the fast path against a full-matrix reference across costs and strings that include surrogate pairs.
  • Edge cases: empty strings, identical strings, emoji.
  • Spellcheck: edit order, first-occurrence dedup, and depth/order of editsWithMaxDistance.

npm test: 904 specs, 0 failures.

Proof

There's a Lean 4 proof that the change preserves output: lean-perf-proofs/natural-levenshtein. It covers:

  • The two-row loop returns the same final cell as the full matrix, for any inputs and costs.
  • Set dedup returns the same list as filter + indexOf.
  • Pushing edits builds the same list as repeated concat.

It uses Lean core only, has no sorry, and relies on no extra axioms.

The non-search, non-Damerau path built a full matrix of cell objects and
picked the cheapest parent with _.min for every cell, even though plain
LevenshteinDistance only returns the final cost. Use two rows of numbers
instead. Search, Damerau and non-string or non-numeric-cost inputs keep the
original path, so results are unchanged.
edits() removed duplicates with filter + indexOf, which is quadratic in the
number of candidate edits, and editsWithMaxDistanceHelper rebuilt its array
with concat for every word. Use a Set (keeps first-occurrence order) and
push. Returned corrections and their order are unchanged.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant