Skip to content

perf: avoid quadratic placeholder handling in SentenceTokenizer - #799

Open
TheNetherWatcher wants to merge 1 commit into
NaturalNode:masterfrom
TheNetherWatcher:perf/sentence-tokenizer
Open

TheNetherWatcher wants to merge 1 commit into
NaturalNode:masterfrom
TheNetherWatcher:perf/sentence-tokenizer

Conversation

@TheNetherWatcher

Copy link
Copy Markdown

Problem

SentenceTokenizer replaces abbreviations, URIs, numbers and delimiters with placeholders like {{DELIM_3}}, then reverts them. The revert step builds a new RegExp for every placeholder in the map and runs it over every sentence, and splitOnPlaceholders builds one alternation of all delimiter keys. Placeholder count grows with input length, and so does sentence count, so tokenizing is roughly quadratic. A 120 KB text takes about 0.7 s.

The revert also passes the original text as a replacement string to String.prototype.replace, so $&, $1 and similar sequences get interpreted. For example See https://example.com/?a=$&b=$1 now. comes back as See https://example.com/?a={{URI_0}}b=$1 now.

Fix

  • Two module-level regexes match any placeholder ({{NUMBER|URI|ABBREV_n}} and {{DELIM_n}}).
  • Reverting is one replace per sentence with a callback that looks the match up in the map (unknown matches are left as-is). The callback also keeps $ sequences literal.
  • splitOnPlaceholders walks the delimiter matches with matchAll and only splits on keys that are in the map, same as before.

The order of the two revert passes is unchanged.

Benchmark

Synthetic English-like text (12 words per sentence, some numbers and URLs), abbreviations ['Mr.', 'Dr.', 'e.g.']. Median CPU time of tokenize() over 9 runs (3 for the largest), old and new alternated, Node on macOS (machine was under other load, so treat absolute numbers loosely).

Sentences Chars Before After Speedup
10 1.2 K 0.18 ms 0.07 ms ~2.6x
200 25 K 28.0 ms 1.16 ms ~24x
1000 123 K 680 ms 6.1 ms ~112x

Tests

  • Added a spec for a URL containing $& and $1 (fails on master, passes here).
  • npm test: 899 specs, 0 failures. standard and eslint clean, jscpd finds no clones.
  • Differential check: 20,000 random inputs built from words, delimiters, quotes, brackets, abbreviations, URLs, emails, numbers and literal {{/}}/{{DELIM_1}} fragments, each run with and without abbreviations and with trimSentences on and off. Output was identical to master for all 80,000 runs.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant