Skip to content

fix: count __proto__ terms in TF-IDF documents - #809

Open
xiaoyao-666-coder wants to merge 1 commit into
NaturalNode:masterfrom
xiaoyao-666-coder:fix/tfidf-proto-term-counts
Open

xiaoyao-666-coder wants to merge 1 commit into
NaturalNode:masterfrom
xiaoyao-666-coder:fix/tfidf-proto-term-counts

Conversation

@xiaoyao-666-coder

@xiaoyao-666-coder xiaoyao-666-coder commented Nov 14, 2026 •

Copy link
Copy Markdown

addDocument('__proto__ __proto__ ordinary') silently drops both occurrences of __proto__: its term frequency and TF-IDF score are zero, and listTerms() omits it. The default tokenizer preserves the token, but the plain-object frequency map cannot store it through normal assignment. Token-array input has the same problem.

Proposed Changes

  • Build frequency maps with a null prototype, retaining the existing __key metadata. This makes __proto__ an ordinary own property and removes the function-name workaround from Tfidf scoring doesnt work for the word 'constructor' #119. Prebuilt document objects are still passed through unchanged.
  • Add four Jasmine specs covering string and array input, repeated Object property names, term listings, JSON round trips, document keys, and cache updates when documents are added or removed.

For a corpus containing __proto__ __proto__ ordinary and ordinary, tfidfs('__proto__') changes from [0, 0] to [2, 0], as required by the documented TF-IDF formula.

This complements #785: that fix guarded term/cache reads, while this fixes counts lost during document construction. Generated documents entries now have no inherited Object methods, consistent with the IDF cache. JSON serialization continues to work.

Validation

On Node 24.22.1 / Windows:

  • New regression tests against unmodified source: 31 specs, 4 failures
  • Fixed TF-IDF suite: 31 specs, 0 failures
  • npm test: 902 specs, 0 failures
  • npm run build:tests and npm run lint
  • Standard JS and focused jscpd check for lib/natural/tfidf/tfidf.js
  • npm run build:esm and npm run smoke:esm
  • git diff --check

The database-backed I/O suite was not run. Companion documentation targets gh-pages in #810, as requested by CONTRIBUTING.md.

Checklist before merging

  • Added Jasmine tests
  • TypeScript declarations: no signature changes required

Prepared with Codex assistance.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant