fix(http): decode text() bodies losslessly, never abort on bad UTF-8

[ehsanmok]

Completes the family of text() decode bugs that #9 and #12 started on. Closes the loop on the two copies #9 did not touch, and fixes the failure mode #9 introduced on the copy it did.

The two problems

Request.text() and MultipartPart.text() still mangle non-ASCII. Both carried the same per-byte out += chr(Int(b)) loop, which maps each byte to the code point of the same value. Confirmed on main before this change:

Request("POST", "http://x/", _bytes("café 日本語")).text()  # -> "café æ¥æ¬èª"
form.value("n")                                              # -> "café æ¥æ¬èª"

This is the server-side ingest path, reached through the public MultipartForm.value() and Request.json().

Response.text() can abort the process. String(unsafe_from_utf8=Span) runs the stdlib UTF-8 validator internally despite its name, as this tree already documents at docs/benchmark.md:705-708 and flare/http/proto/ascii.mojo:51-61. What it does on failure is the problem. Measured with a body of a <0xFF> b:

build	before this PR	after
default	97 255 98 - raw 0xFF survives into an ill-formed String	97 239 191 189 98
-D ASSERT=all	abort: Assert Error: String: span is not valid UTF-8	97 239 191 189 98

Since the peer chooses those bytes, that is a remotely-triggerable abort on any assert-enabled build, and pixi run tests-asserts-all is a supported posture here.

The fix

proto/utf8.utf8_lossy_string() makes the decode total: well-formed input is bulk-copied, and each maximal ill-formed subpart becomes one U+FFFD per Unicode 15 3.9. The result is always well-formed UTF-8, so it can neither abort nor produce an ill-formed String. All three call sites use it.

This also makes the promise Response.text()'s docstring has carried all along - "invalid UTF-8 sequences are replaced with the Unicode replacement character" - true for the first time. Request.text() and MultipartPart.text() now document it as well.

WsFrame.text_payload() is deliberately left on validate-and-raise: RFC 6455 8.1 requires rejecting invalid UTF-8 on TEXT frames rather than substituting, so #12 is the right shape there and this is not.

One intentional shortcut is marked with a ponytail: comment: the happy path validates twice, once in the scan and again inside the constructor. The upgrade path (unsafe_uninit_length + memcpy, as proto/ascii.mojo does) is named in the comment. text() is not a hot path today; the parser is, and it already uses ascii_unchecked_string.

Test plan

• test_response_text_invalid_utf8_replaced - a <0xFF> b becomes a U+FFFD b
• test_response_text_truncated_sequence_is_one_replacement - truncated E6 97 collapses to a single U+FFFD, pinning maximal-subpart behavior rather than one U+FFFD per byte
• test_test_post_body_utf8_roundtrip and test_text_part_utf8_value - non-ASCII round-trips through Request.text() and a multipart field value; both fail on main
• Verified no abort with and without -D ASSERT=all, where the pre-change code aborted
• test-http 44/44 (also 44/44 under -D ASSERT=all), test-multipart 20, test-request-factories 10, test-server 84, test-auth-extract 40, test-extractors-concrete 31, test-extractors 11, test-typed-extractors 9, test-request-chunks 10, test-handler 14, test-form 25, test-session 24, test-sse, test-inbound-body, test-proto-reexports all pass
• fuzz-multipart 0 crashes, check-no-http-http2-cycle clean, format-check clean

Made with Cursor