Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/release.yml
Original file line number Diff line number Diff line change
Expand Up @@ -258,7 +258,7 @@ jobs:
mv /tmp/artifacts/milo-parser-bin-macos-intel/milo-parser-bin-macos-intel.tar.gz /tmp/release/
- name: Push release commit and tag
run: |
git add -f package.json CHANGELOG.md parser/Cargo.toml parser/Cargo.lock macros/Cargo.toml macros/Cargo.lock references/rust/Cargo.toml references/rust/Cargo.lock parser/src/wasm/package.json
git add -f package.json CHANGELOG.md parser/Cargo.toml parser/Cargo.lock macros/Cargo.toml macros/Cargo.lock references/rust/Cargo.toml references/rust/Cargo.lock parser/wasm/src/package.json
git commit -m "chore: Updated version."
git tag -f "v${{ inputs.version }}"
- name: Publish macros on crates.io
Expand Down
67 changes: 67 additions & 0 deletions AGENTS.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,67 @@
# Milo development guide

## Parser internals

Milo leverages Rust's [procedural macros](https://doc.rust-lang.org/reference/procedural-macros.html), [syn](https://crates.io/crates/syn), and [quote](https://crates.io/crates/quote) crates to define actions and matchers for the parser.

See the [macros](./macros/README.md) internal crate for more information.

## Build WebAssembly and C++ locally

Required tools:

- [cargo-make](https://github.com/sagiegurari/cargo-make).
- The pinned Rust nightly toolchain, installed via [rustup](https://rustup.rs/).
- [rust-cbindgen](https://github.com/mozilla/cbindgen).
- [Binaryen](https://github.com/WebAssembly/binaryen), providing `wasm-opt`.

Install the pinned toolchain, the Rust sources required by the WebAssembly release build, and the WebAssembly target:

```sh
rustup toolchain install nightly-2026-07-29 --component rust-src
rustup target add wasm32-unknown-unknown
```

Run from the repository root:

```sh
makers
```

This produces debug and release builds for each language in the top-level `dist` folder.

Build tooling is compiled from `scripts` into standalone Rust binaries. Node.js and npm dependencies are not required to build the parser or generate its C++ and WebAssembly packages.

The WebAssembly release build uses immediate-abort panics to keep the artifact smaller. Panics trap without unwinding or rich panic messages. The debug build also enables the `on_state_change` callback and provides more detailed WebAssembly errors.

For JavaScript linting and formatting, install the development dependencies with `pnpm install`.

## Run tests

Run `makers test` from the repository root for the Rust and WebAssembly suites, or `makers test:wasm` to build and test only WebAssembly.

The WebAssembly suite uses Node.js's built-in test runner and tests the release SIMD package by default. Set `MILO_VARIANT=no-simd` to select the non-SIMD package instead. After building, run `pnpm test:wasm` to rerun it without rebuilding.

Tests live in `parser/wasm/test` and mirror the Rust integration tests in `parser/tests` with the same case names: basic, benchmark, compliance, issue, undici, and upgrade. Issue regressions use the `issue_<number>__<description>` naming convention. The llhttp suite is not yet ported.

## Build WebAssembly with Docker

The repository includes a Docker image for building the WebAssembly packages without changing the working tree. Build the image from the repository root, then mount the sources read-only and choose a host directory for the generated artifacts:

```sh
docker build -t milo-wasm .
mkdir -p /path/to/milo-wasm-output
docker run --rm \
-v "$PWD:/src:ro" \
-v "/path/to/milo-wasm-output:/output" \
milo-wasm
```

The container builds both the debug and release profiles in its temporary workspace. The output directory receives the resulting `debug` and `release` packages; the mounted source tree remains read-only.

## Contributing

- Check the latest default branch to make sure the feature hasn't been implemented or the bug hasn't been fixed yet.
- Check the issue tracker for existing requests and contributions.
- The contribution workflow uses a fork and a feature or bugfix branch, followed by commits and a push when the contribution is ready.
- Add tests for changes to prevent regressions.
6 changes: 6 additions & 0 deletions Makefile.toml
Original file line number Diff line number Diff line change
Expand Up @@ -8,8 +8,14 @@
script = ["cd parser", "makers build"]

[tasks.test]
dependencies = ["test:rust", "test:wasm"]

[tasks."test:rust"]
script = ["cd parser", "cargo test"]

[tasks."test:wasm"]
script = ["cd parser", "makers test:wasm"]

[tasks.format]
dependencies = ["format:rust", "format:js", "format:cpp"]

Expand Down
72 changes: 4 additions & 68 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -91,6 +91,8 @@ milo.dealloc(ptr, message.length)

The default JavaScript entry point uses the SIMD WebAssembly build. Use `@perseveranza-pets/milo/no-simd` when SIMD is not available, and add `/unbundled` to either entry point to load the external `.wasm` file instead of the bundled JavaScript module.

The WebAssembly release build uses immediate-abort panics: panics trap without unwinding or rich panic messages. The debug build also enables the `on_state_change` callback and provides more detailed WebAssembly errors.

CommonJS projects can use the same entry points from `@perseveranza-pets/milo-cjs`:

```javascript
Expand Down Expand Up @@ -233,42 +235,6 @@ clang++ -std=c++11 -o example main.cc libmilo.a
# Pos=38 Body: abc
```

### Build milo (WebAssembly and C++) locally

If you want to build it locally, you need the following tools:

- [cargo-make][cargo-make]
- Rust toolchain - You can install it via [rustup].
- [rust-cbindgen](https://github.com/mozilla/cbindgen)

Make sure you have the pinned nightly toolchain installed locally:

```bash
rustup toolchain install nightly-2026-07-29
```

Make sure you have the `wasm32-unknown-unknown` target:

```bash
rustup target add wasm32-unknown-unknown
```

After all the requirements are met, you can then run:

```bash
makers
```

The command above will produce debug and release builds for each language in the top-level `dist` folder.

Build tooling is compiled from `scripts` into standalone Rust binaries. Node.js and npm dependencies are not required to build the parser or generate its C++ and WebAssembly packages.

For JavaScript linting and formatting, install the development dependencies with `pnpm install`.

The WebAssembly release build uses immediate-abort panics to keep the artifact smaller. Panics trap without unwinding or rich panic messages.

The debug build also enables the `on_state_change` callback and is more verbose in case of WebAssembly errors.

## How to use it (CLI)

Install it from crates.io:
Expand Down Expand Up @@ -336,11 +302,7 @@ Milo validates HTTP/1.1 syntax, message framing, protocol switching, connection

## How it works?

Milo leverages Rust's [procedural macro], [syn] and [quote] crates to allow an easy definition of actions and matchers for the parser.

See the [macros](./macros/README.md) internal crate for more information.

The resulting parser is a simple state machine which copies data in only one optional case: automatically handling the unconsumed portion of the input data.
Milo is a simple state machine which copies data in only one optional case: automatically handling the unconsumed portion of the input data.

In all other cases, no data is copied and the memory footprint is very small as only a few dozen `bool`, `uintptr_t`, or `uint64_t` fields can represent the entire parser state.

Expand All @@ -358,33 +320,13 @@ To see the rationale behind the replacement of llhttp, check Paolo's talk at [Va

To see the initial disclosure of milo, check Paolo's talk at [NodeConf EU 2023][nodeconf-talk] in November 2023 ([slides][slides]).

## Building WebAssembly with Docker

The repository includes a Docker image for building the WebAssembly packages without changing the working tree. Build the image from the repository root, then mount the sources read-only and choose a host directory for the generated artifacts:

```sh
docker build -t milo-wasm .
mkdir -p /path/to/milo-wasm-output
docker run --rm \
-v "$PWD:/src:ro" \
-v "/path/to/milo-wasm-output:/output" \
milo-wasm
```

The container builds both the debug and release profiles in its temporary workspace. The output directory receives the resulting `debug` and `release` packages; the mounted source tree remains read-only.

## Sponsored by

[![NearForm](https://raw.githubusercontent.com/ShogunPanda/milo/main/docs/nearform.jpg)][nearform]

## Contributing to milo

- Check out the latest master to make sure the feature hasn't been implemented or the bug hasn't been fixed yet.
- Check out the issue tracker to make sure someone already hasn't requested it and/or contributed it.
- Fork the project.
- Start a feature/bugfix branch.
- Commit and push until you are happy with your contribution.
- Make sure to add tests for it. This is important so I don't break it in a future version unintentionally.
See [AGENTS.md](./AGENTS.md) for development setup, local and Docker builds, tests, and contribution guidelines.

## Copyright

Expand All @@ -402,12 +344,6 @@ Licensed under the ISC license, which can be found at https://choosealicense.com
[nodeconf-talk]: https://youtube.com/watch?v=dcHbAeO_ccY
[slides]: https://talks.paoloinsogna.dev/milo
[isc]: https://choosealicense.com/licenses/isc
[procedural macro]: https://doc.rust-lang.org/reference/procedural-macros.html
[syn]: https://crates.io/crates/syn
[quote]: https://crates.io/crates/quote
[match]: https://doc.rust-lang.org/rust-by-example/flow_control/match.html
[match-slice]: https://doc.rust-lang.org/rust-by-example/flow_control/match/destructuring/destructure_slice.html
[cargo-make]: https://github.com/sagiegurari/cargo-make
[rustup]: https://rustup.rs/
[Clang]: https://clang.llvm.org/

5 changes: 3 additions & 2 deletions package.json
Original file line number Diff line number Diff line change
Expand Up @@ -19,8 +19,9 @@
"private": true,
"type": "module",
"scripts": {
"format": "prettier -w \"parser/**/*.js\" \"benchmarks/**/*.js\" \"references/**/*.js\"",
"lint": "eslint --cache \"parser/**/*.js\" \"benchmarks/**/*.js\" \"references/**/*.js\""
"test:wasm": "node --test parser/wasm/test/*.test.js",
"format": "prettier -w \"parser/wasm/**/*.js\" \"benchmarks/wasm/**/*.js\" \"references/wasm/**/*.js\" references/reference.js",
"lint": "eslint --cache \"parser/wasm/**/*.js\" \"benchmarks/wasm/**/*.js\" \"references/wasm/**/*.js\" references/reference.js"
},
"dependencies": {
"@cowtech/eslint-config": "^11.1.3",
Expand Down
9 changes: 8 additions & 1 deletion parser/Makefile.toml
Original file line number Diff line number Diff line change
Expand Up @@ -11,10 +11,17 @@
dependencies = ["cpp", "wasm"]

[tasks.test]
dependencies = ["build"]
dependencies = ["build", "test:rust", "test:wasm"]

[tasks."test:rust"]
command = "cargo"
args = ["test"]

[tasks."test:wasm"]
dependencies = ["wasm:release"]
command = "node"
args = ["--test", "wasm/test/*.test.js"]

[tasks.cpp]
dependencies = ["cpp:headers", "cpp:libs"]

Expand Down
8 changes: 5 additions & 3 deletions parser/src/matchers.rs
Original file line number Diff line number Diff line change
Expand Up @@ -318,8 +318,9 @@ pub fn find_header_line_end(ptr: *const u8, len: usize) -> HeaderLineScanResult
let eq_7f = u8x16_eq(x, v_7f);

// Header lines stop at CR; other control bytes are invalid except HTAB.
let ctrl = v128_andnot(eq_tab, lt_20);
let invalid = v128_andnot(eq_cr, v128_or(ctrl, eq_7f));
// WASM andnot(a, b) computes a & !b, unlike the x86 intrinsic.
let ctrl = v128_andnot(lt_20, eq_tab);
let invalid = v128_andnot(v128_or(ctrl, eq_7f), eq_cr);
let found = v128_or(eq_cr, invalid);

if v128_any_true(found) {
Expand Down Expand Up @@ -440,7 +441,8 @@ pub fn validate_token_value(ptr: *const u8, len: usize) -> bool {
let eq_7f = u8x16_eq(x, v_7f);

// Field values allow HTAB but reject the remaining C0 controls and DEL.
let ctrl = v128_andnot(eq_tab, lt_20);
// WASM andnot(a, b) computes a & !b, unlike the x86 intrinsic.
let ctrl = v128_andnot(lt_20, eq_tab);
let invalid = v128_or(ctrl, eq_7f);

if v128_any_true(invalid) {
Expand Down
11 changes: 0 additions & 11 deletions parser/tests/compliance.rs
Original file line number Diff line number Diff line change
Expand Up @@ -555,17 +555,6 @@ fn compliance_chunk_extension_quoted_pair_control_rejected() {
assert_error(&parser);
}

// Bare LF is rejected in HTTP framing.
#[test]
fn compliance_bare_lf_rejected() {
let mut parser = response_parser();
let message = "HTTP/1.1 200 OK\r\nHeader: value\nContent-Length: 0\r\n\r\n";

parse(&mut parser, message);

assert_error(&parser);
}

// Bare CR is rejected in HTTP framing.
#[test]
fn compliance_bare_cr_rejected() {
Expand Down
84 changes: 84 additions & 0 deletions parser/tests/issue.rs
Original file line number Diff line number Diff line change
@@ -0,0 +1,84 @@
mod helpers;

use milo_parser::{ERROR_NONE, Parser, STATE_ERROR};

use crate::helpers::{create_parser, parse};

fn response_parser() -> Parser {
let mut parser = create_parser();
parser.autodetect = false;
parser.is_request = false;
parser
}

fn field_messages(value: &[u8]) -> [Vec<u8>; 3] {
let fields: [(&[u8], &[u8]); 3] = [
(b"HTTP/1.1 200 OK\r\nX-Long: ", b"\r\nContent-Length: 0\r\n\r\n"),
(
b"HTTP/1.1 200 OK\r\nTransfer-Encoding: chunked\r\n\r\n0\r\nX-Long: ",
b"\r\n\r\n",
),
(b"HTTP/1.1 200 ", b"\r\nContent-Length: 0\r\n\r\n"),
];
fields.map(|(prefix, suffix)| [prefix, value, suffix].concat())
}

// Cover SIMD block boundaries and scalar tails in both field scanners.
#[test]
#[allow(non_snake_case)]
fn issue_22__field_values_reject_controls() {
let mut parser = response_parser();
parser.active_callbacks = 0;

for byte in (0u8..0x20).chain([0x7f]).filter(|byte| *byte != b'\t') {
for offset in [0, 7, 8, 15, 16, 20, 31, 32] {
for trailing in [0, 20] {
let mut value = vec![b'a'; offset];
value.push(byte);
value.extend(vec![b'a'; trailing]);
for message in field_messages(&value) {
parser.reset(false);
parser.parse(message.as_ptr(), message.len());
assert_eq!(parser.state, STATE_ERROR, "Accepted invalid field: {message:?}");
}
}
}
}
}

// HTAB and every obs-text byte remain valid, including across SIMD boundaries.
#[test]
#[allow(non_snake_case)]
fn issue_22__field_values_allow_tab_and_obs_text() {
let mut parser = response_parser();
// Raw obs-text is not necessarily UTF-8, so bypass the text-decoding callbacks.
parser.active_callbacks = 0;

for byte in [b'\t'].into_iter().chain(0x80..=0xff) {
for offset in [0, 7, 8, 15, 16, 20, 31, 32] {
for trailing in [0, 20] {
let mut value = vec![b'a'; offset];
value.push(byte);
value.extend(vec![b'a'; trailing]);
for message in field_messages(&value) {
parser.reset(false);
parser.parse(message.as_ptr(), message.len());
assert_ne!(parser.state, STATE_ERROR, "Rejected valid field: {message:?}");
assert_eq!(parser.error_code, ERROR_NONE);
}
}
}
}
}

// Bare LF is rejected in HTTP framing.
#[test]
#[allow(non_snake_case)]
fn issue_22__bare_lf_rejected() {
let mut parser = response_parser();
let message = "HTTP/1.1 200 OK\r\nHeader: value\nContent-Length: 0\r\n\r\n";

parse(&mut parser, message);

assert_eq!(parser.state, STATE_ERROR);
}
File renamed without changes.
File renamed without changes.
Loading
Loading