Skip to content

fix: RegexpTokenizer keeps empty tokens when discardEmpty is false - #804

Open
OsamaAnsar wants to merge 1 commit into
NaturalNode:masterfrom
OsamaAnsar:fix/regexp-tokenizer-discard-empty
Open

OsamaAnsar wants to merge 1 commit into
NaturalNode:masterfrom
OsamaAnsar:fix/regexp-tokenizer-discard-empty

Conversation

@OsamaAnsar

Copy link
Copy Markdown
Contributor

The discardEmpty option of RegexpTokenizer cannot be turned off:

const { RegexpTokenizer } = require('natural')

const tokenizer = new RegexpTokenizer({ pattern: /,/, discardEmpty: false })
tokenizer.tokenize('a,,b,')
// ['a', 'b'], expected ['a', '', 'b', '']

The constructor reads it as options.discardEmpty || true, which is always true.

The change defaults to true only when the option is not given, the same way gaps is handled a few lines below. Nothing in the library passes discardEmpty: false, so the other tokenizers behave as before. A new spec/regexp_tokenizer_spec.ts covers the default, true and false; the false spec fails without the change. npm test (901 specs) and npm run lint pass.

The option was read as options.discardEmpty || true, which is always true,
so discardEmpty: false had no effect.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant