Skip to content

lib,src: reset TextDecoder after a decoding error - #67610

Open
vedchaudhari wants to merge 1 commit into
nodejs:mainfrom
vedchaudhari:textdecoder-reset-on-error
Open

vedchaudhari wants to merge 1 commit into
nodejs:mainfrom
vedchaudhari:textdecoder-reset-on-error

Conversation

@vedchaudhari

@vedchaudhari vedchaudhari commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

The spec now says (whatwg/encoding#369) that when TextDecoder#decode() throws in fatal mode, the decoder resets and the next call starts fresh. Node currently keeps the old state, so if you keep streaming after an error:

  • a BOM at the start of the next chunk shows up as U+FEFF
  • UTF-16 keeps throwing on valid input, because half a surrogate pair is still held
  • iso-2022-jp decodes AB as U+758E, and gb18030 leaks a leftover byte

This resets the decoder when an error happens, for UTF-8 (in JS) and the ICU encodings (in C++), and nothing changes when fatal is off. I added tests for it and also checked it against the new WPT tests from the WebKit PR, which all pass now.

cc @ChALkeR since you wrote the UTF-8 fast path and opened whatwg/encoding#359.

Fixes: #67604
Refs: whatwg/encoding#369

Assisted-by: an AI coding agent. Reviewed and tested locally by me.

A fatal decode() error now resets the decoder, as the Encoding
Standard requires, so the next call does not continue with the bytes
or BOM state from before the error.

Signed-off-by: vedchaudhari <vedc2853@gmail.com>
@nodejs-github-bot nodejs-github-bot added c++ Issues and PRs that require attention from people who are familiar with C++. encoding Issues and PRs related to the TextEncoder and TextDecoder APIs. i18n-api Issues and PRs related to Node.js internationalization support. needs-ci PRs that need a full CI run. labels Nov 19, 2026
@vedchaudhari
vedchaudhari marked this pull request as ready for review November 20, 2026 20:13
@codecov

codecov Bot commented Nov 20, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 90.60%. Comparing base (d9e18d5) to head (c565285).
⚠️ Report is 50 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main   #67610      +/-   ##
==========================================
- Coverage   90.72%   90.60%   -0.13%     
==========================================
  Files         801      801              
  Lines      280809   280969     +160     
  Branches    54212    54226      +14     
==========================================
- Hits       254776   254563     -213     
- Misses      16209    16582     +373     
  Partials     9824     9824              
Files with missing lines Coverage Δ
lib/internal/encoding.js 96.75% <100.00%> (+<0.01%) ⬆️
src/node_i18n.cc 78.39% <100.00%> (+0.12%) ⬆️

... and 69 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

c++ Issues and PRs that require attention from people who are familiar with C++. encoding Issues and PRs related to the TextEncoder and TextDecoder APIs. i18n-api Issues and PRs related to Node.js internationalization support. needs-ci PRs that need a full CI run.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Small TextDecoder decode() change

2 participants