Skip to content

fix(ws): decode TEXT payloads per code point, not per byte - #12

Merged
ehsanmok merged 1 commit into
mainfrom
fix/ws-text-payload-utf8
Aug 27, 2026
Merged

fix(ws): decode TEXT payloads per code point, not per byte#12
ehsanmok merged 1 commit into
mainfrom
fix/ws-text-payload-utf8

Conversation

@ehsanmok

Copy link
Copy Markdown
Owner

Summary

WsFrame.text_payload() validated its payload as UTF-8 per RFC 6455 8.1 and then decoded it with s += chr(Int(b)) per byte, which maps each byte to the code point of the same value. Every non-ASCII TEXT frame came back with one character per byte:

WsFrame.text("café 日本語").text_payload()  # -> "café æ¥æ¬èª"

This is user-visible on both sides of the connection today. WsClient.recv() builds its WsMessage from text_payload() (flare/ws/client.mojo:766), and both WsServer echo paths go through it (flare/ws/server.mojo:513, and the documented handler example at line 331).

The validation on the preceding line already guarantees well-formed UTF-8, so the loop collapses into the bulk constructor with no new failure mode. Worth stating explicitly, since it is the reason this substitution is safe here and not everywhere: String(unsafe_from_utf8=...) runs the stdlib validator internally despite its name (documented in this tree at docs/benchmark.md:705-708 and flare/http/proto/ascii.mojo:51-61), and on invalid input that validator aborts the process under -D ASSERT=all. It cannot fire here because line 496 has already rejected invalid payloads.

Same shape as the Response.text() fix in #9.

Not in scope

Request.text() (flare/http/request.mojo:419) and MultipartPart.text() (flare/http/multipart.mojo:232, reached publicly via MultipartForm.value()) carry the identical loop and are identically broken, confirmed locally. Their bytes are caller-supplied rather than pre-validated, so the plain substitution would trade mojibake for a remotely-triggerable abort on an assert-enabled build. They need validate-then-copy and are tracked separately.

Test plan

  • test_encode_decode_roundtrip_text_utf8 added, covering 2-, 3- and 4-byte sequences; verified it FAILS on the unfixed decoder and passes with the fix
  • pixi run test-ws 34/34
  • pixi run test-ws-permessage-deflate 12, test-ws-h2 6, test-ws-h2-roundtrip 1, test-ws-stateful-handler 1, test-conformance-ws OK
  • pixi run format-check clean

Made with Cursor

text_payload() validated the payload as UTF-8 (RFC 6455 8.1) and then
decoded it with ``s += chr(Int(b))`` per byte, which maps each byte to
the code point of the same value. Every non-ASCII TEXT frame came back
with one character per byte: "café 日本語" arrived as "café æ¥æ¬èª".
That reached every caller of WsClient.recv() (client.mojo builds
WsMessage from text_payload()) and both WsServer echo paths.

The RFC 6455 validation on the line above already guarantees
well-formed UTF-8, so the loop collapses into the bulk constructor
without introducing a new failure mode: String(unsafe_from_utf8=...)
runs the stdlib validator internally despite its name, and that
validator aborts the process under -D ASSERT=all on invalid input,
which cannot happen here because line 496 has already rejected it.

Same shape as the Response.text() fix in #9. Request.text() and
MultipartPart.text() carry the same per-byte loop but their bytes are
caller-supplied, so they need validate-then-copy rather than this
substitution; tracked separately.

Co-authored-by: Cursor Agent <cursoragent@cursor.com>
@ehsanmok
ehsanmok merged commit 98bd9dd into main Aug 27, 2026
5 checks passed
@ehsanmok
ehsanmok deleted the fix/ws-text-payload-utf8 branch August 27, 2026 18:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant