Repository navigation
perf: avoid quadratic placeholder handling in SentenceTokenizer - #799
Open
TheNetherWatcher wants to merge 1 commit into
Open
TheNetherWatcher wants to merge 1 commit into
TheNetherWatcher wants to merge 1 commit into
Conversation
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
SentenceTokenizerreplaces abbreviations, URIs, numbers and delimiters with placeholders like{{DELIM_3}}, then reverts them. The revert step builds a newRegExpfor every placeholder in the map and runs it over every sentence, andsplitOnPlaceholdersbuilds one alternation of all delimiter keys. Placeholder count grows with input length, and so does sentence count, so tokenizing is roughly quadratic. A 120 KB text takes about 0.7 s.The revert also passes the original text as a replacement string to
String.prototype.replace, so$&,$1and similar sequences get interpreted. For exampleSee https://example.com/?a=$&b=$1 now.comes back asSee https://example.com/?a={{URI_0}}b=$1 now.Fix
{{NUMBER|URI|ABBREV_n}}and{{DELIM_n}}).replaceper sentence with a callback that looks the match up in the map (unknown matches are left as-is). The callback also keeps$sequences literal.splitOnPlaceholderswalks the delimiter matches withmatchAlland only splits on keys that are in the map, same as before.The order of the two revert passes is unchanged.
Benchmark
Synthetic English-like text (12 words per sentence, some numbers and URLs), abbreviations
['Mr.', 'Dr.', 'e.g.']. Median CPU time oftokenize()over 9 runs (3 for the largest), old and new alternated, Node on macOS (machine was under other load, so treat absolute numbers loosely).Tests
$&and$1(fails on master, passes here).npm test: 899 specs, 0 failures.standardandeslintclean,jscpdfinds no clones.{{/}}/{{DELIM_1}}fragments, each run with and without abbreviations and withtrimSentenceson and off. Output was identical to master for all 80,000 runs.