Skip to content

fix: WordNet.get gives null instead of hanging for an offset that is not a record - #800

Open
OsamaAnsar wants to merge 1 commit into
NaturalNode:masterfrom
OsamaAnsar:fix/wordnet-invalid-synset-offset
Open

OsamaAnsar wants to merge 1 commit into
NaturalNode:masterfrom
OsamaAnsar:fix/wordnet-invalid-synset-offset

Conversation

@OsamaAnsar

Copy link
Copy Markdown
Contributor

Fixes #455.

WordNetFile.appendLineChar reads one byte at a time and stops at a newline. A read past the end of the file returns 0 bytes and leaves the buffer as it was, so it never saw a newline and called itself for ever. wordnet.get(offset, 'n', cb) with an offset past the end of data.noun never called back and kept reading the file for ever. (#455 reports the heap running out on an older Node; on Node 24 I measured memory staying flat at about 140 MB for 25 s while the callback never fired.)

Reproduced on master (8.2.6) with wordnet-db:

const natural = require('natural')
const wordnet = new natural.WordNet()
wordnet.get(99999999, 'n', (r) => console.log(r)) // never calls back

An offset in the middle of a record was also accepted. get(1741, 'n') returned the entity record that starts at 1740: the line it reads is 001740 03 n ..., and parseInt of the offset ignores the missing leading zero.

Changes

  • appendLineChar ends the line when the read returns no bytes (also fixes the last line of a file without a trailing newline).
  • DataFile.get checks that the line is a record starting at the requested offset (an offset of the record, plus the | gloss part) and otherwise calls back with null. That is also what the other WordNet connector linked in wordnet invalid query freezes node #455 does.
  • getSynonyms treats null as no synonyms instead of throwing on result.ptrs.
  • DataRecordCallback includes null; the TypeScript example in io_spec/wordnet_test.ts checks for it.

Tests (io_spec/wordnet_spec.js): offsets past the end of the file and inside a record give null, a valid offset still gives its record, getSynonyms with a bad offset gives [], and appendLineChar returns the last line without a newline and an empty line past the end. Without the change 4 of the new specs fail (three never complete, one gets the wrong record). With it wordnet_spec and wordnet_index_spec pass (12 specs), tsc --noEmit is clean and standard reports nothing new.

Note on the type change: TypeScript users who call get now have to handle null under strictNullChecks. If you would rather not change the type in a patch release I can drop that part.

…not a record

appendLineChar read one byte at a time and never stopped at the end of the file, because a read past the end returns no bytes and leaves the buffer untouched. WordNet.get(offset, pos) with an offset past the end of the data file therefore never called back and kept reading the file for ever (NaturalNode#455). An offset inside a record was also accepted: 1741 returned the record at 1740, because the leading zero of its offset is dropped and the text still parses as 1740.

appendLineChar now ends the line at the end of the file. DataFile.get checks that the line is a record that starts at the requested offset and otherwise calls back with null, which getSynonyms now handles as no synonyms. The DataRecordCallback type includes null.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

wordnet invalid query freezes node

1 participant