fix(ws): decode TEXT payloads per code point, not per byte - #12
Merged
Conversation
text_payload() validated the payload as UTF-8 (RFC 6455 8.1) and then decoded it with ``s += chr(Int(b))`` per byte, which maps each byte to the code point of the same value. Every non-ASCII TEXT frame came back with one character per byte: "café 日本語" arrived as "café æ¥æ¬èª". That reached every caller of WsClient.recv() (client.mojo builds WsMessage from text_payload()) and both WsServer echo paths. The RFC 6455 validation on the line above already guarantees well-formed UTF-8, so the loop collapses into the bulk constructor without introducing a new failure mode: String(unsafe_from_utf8=...) runs the stdlib validator internally despite its name, and that validator aborts the process under -D ASSERT=all on invalid input, which cannot happen here because line 496 has already rejected it. Same shape as the Response.text() fix in #9. Request.text() and MultipartPart.text() carry the same per-byte loop but their bytes are caller-supplied, so they need validate-then-copy rather than this substitution; tracked separately. Co-authored-by: Cursor Agent <cursoragent@cursor.com>
6 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
WsFrame.text_payload()validated its payload as UTF-8 per RFC 6455 8.1 and then decoded it withs += chr(Int(b))per byte, which maps each byte to the code point of the same value. Every non-ASCII TEXT frame came back with one character per byte:This is user-visible on both sides of the connection today.
WsClient.recv()builds itsWsMessagefromtext_payload()(flare/ws/client.mojo:766), and bothWsServerecho paths go through it (flare/ws/server.mojo:513, and the documented handler example at line 331).The validation on the preceding line already guarantees well-formed UTF-8, so the loop collapses into the bulk constructor with no new failure mode. Worth stating explicitly, since it is the reason this substitution is safe here and not everywhere:
String(unsafe_from_utf8=...)runs the stdlib validator internally despite its name (documented in this tree atdocs/benchmark.md:705-708andflare/http/proto/ascii.mojo:51-61), and on invalid input that validator aborts the process under-D ASSERT=all. It cannot fire here because line 496 has already rejected invalid payloads.Same shape as the
Response.text()fix in #9.Not in scope
Request.text()(flare/http/request.mojo:419) andMultipartPart.text()(flare/http/multipart.mojo:232, reached publicly viaMultipartForm.value()) carry the identical loop and are identically broken, confirmed locally. Their bytes are caller-supplied rather than pre-validated, so the plain substitution would trade mojibake for a remotely-triggerable abort on an assert-enabled build. They need validate-then-copy and are tracked separately.Test plan
test_encode_decode_roundtrip_text_utf8added, covering 2-, 3- and 4-byte sequences; verified it FAILS on the unfixed decoder and passes with the fixpixi run test-ws34/34pixi run test-ws-permessage-deflate12,test-ws-h26,test-ws-h2-roundtrip1,test-ws-stateful-handler1,test-conformance-wsOKpixi run format-checkcleanMade with Cursor