Repository navigation
fix: WordNet.get gives null instead of hanging for an offset that is not a record - #800
Open
OsamaAnsar wants to merge 1 commit into
Open
OsamaAnsar wants to merge 1 commit into
OsamaAnsar wants to merge 1 commit into
Conversation
…not a record appendLineChar read one byte at a time and never stopped at the end of the file, because a read past the end returns no bytes and leaves the buffer untouched. WordNet.get(offset, pos) with an offset past the end of the data file therefore never called back and kept reading the file for ever (NaturalNode#455). An offset inside a record was also accepted: 1741 returned the record at 1740, because the leading zero of its offset is dropped and the text still parses as 1740. appendLineChar now ends the line at the end of the file. DataFile.get checks that the line is a record that starts at the requested offset and otherwise calls back with null, which getSynonyms now handles as no synonyms. The DataRecordCallback type includes null.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #455.
WordNetFile.appendLineCharreads one byte at a time and stops at a newline. A read past the end of the file returns 0 bytes and leaves the buffer as it was, so it never saw a newline and called itself for ever.wordnet.get(offset, 'n', cb)with an offset past the end ofdata.nounnever called back and kept reading the file for ever. (#455 reports the heap running out on an older Node; on Node 24 I measured memory staying flat at about 140 MB for 25 s while the callback never fired.)Reproduced on master (8.2.6) with
wordnet-db:An offset in the middle of a record was also accepted.
get(1741, 'n')returned the entity record that starts at 1740: the line it reads is001740 03 n ..., andparseIntof the offset ignores the missing leading zero.Changes
appendLineCharends the line when the read returns no bytes (also fixes the last line of a file without a trailing newline).DataFile.getchecks that the line is a record starting at the requested offset (an offset of the record, plus the| glosspart) and otherwise calls back withnull. That is also what the other WordNet connector linked in wordnet invalid query freezes node #455 does.getSynonymstreatsnullas no synonyms instead of throwing onresult.ptrs.DataRecordCallbackincludesnull; the TypeScript example inio_spec/wordnet_test.tschecks for it.Tests (
io_spec/wordnet_spec.js): offsets past the end of the file and inside a record givenull, a valid offset still gives its record,getSynonymswith a bad offset gives[], andappendLineCharreturns the last line without a newline and an empty line past the end. Without the change 4 of the new specs fail (three never complete, one gets the wrong record). With itwordnet_specandwordnet_index_specpass (12 specs),tsc --noEmitis clean andstandardreports nothing new.Note on the type change: TypeScript users who call
getnow have to handlenullunderstrictNullChecks. If you would rather not change the type in a patch release I can drop that part.