Repository navigation
zaml elixir +python - #23
Open
niranjanaryan wants to merge 60 commits into
Open
niranjanaryan wants to merge 60 commits into
niranjanaryan wants to merge 60 commits into
Conversation
Adds a Zig NIF wrapping libyaml for fast YAML 1.2 parsing in Elixir, alongside the existing Python prototype. Zaml.load/1 returns native Erlang terms (maps, lists, ints, floats, bools, :nil) and is ~3x faster than fast_yaml on nested data and ~1.4x faster on 1M-line flat mappings (see benchmark/RESULTS.md). Build is driven by mix zaml_nif.build and a Makefile; libyaml is auto-detected on common paths. fast_yaml is included as a dev/test dep for the head-to-head bench task only.
Ship an MIT license, slim the Hex tarball, and compile the Zig NIF through elixir_make so mix compile works for Hex consumers.
Accumulate key/value pairs per mapping frame and flush them with one enif_make_map_from_arrays call instead of re-putting each pair with enif_make_map_put, which rehashes the growing map on every insert (O(n^2) for a wide mapping). Parsing a 1M-key flat mapping drops from ~1.8 s to ~0.85 s.
Extend mix zaml_nif.bench to benchmark every installed parser (fast_yaml, glazer, yaml_elixir, yamerl), skipping any whose dep is missing, with a warm-up pass and a forced GC between timed runs. Measured on an Apple M3 Pro (avg of 5 runs), zaml is ~3x faster than fast_yaml and ~35-48x faster than the pure-Erlang parsers. The hand-rolled glazer NIF wins on nested data (~1.35x) and loses on the flat 1M-key map (~1.8x). Refresh benchmark/RESULTS.md, the README, and the release notes with the new comparison, add usage snippets for each alternative, and drop the stale PyYAML/ruamel numbers from the headline docs.
Commit 8eb350e accidentally captured an intermediate state (from a concurrent edit) in which parse_yaml took an input_term argument it did not use. Zig rejects unused parameters, so the pushed HEAD no longer compiled. Restore the original parse_yaml(env, input) signature while keeping the single-pass map build; mix test and the benchmarks pass.
Build maps with one enif_make_map_from_arrays call instead of per-pair enif_make_map_put, removing the O(n^2) rehashing on wide maps. Return string scalars as sub-binaries of the input rather than copied binaries, guarded by a pointer-in-input check with a copy fallback for quoted / block / folded scalars that some libyaml versions may write to their own buffer. 100k-key nested: ~0.65s -> ~0.53s; 1M-key flat: ~0.85s -> ~0.63s. Lead over fast_yaml widens to ~3.5x (nested) and ~4.6x (flat). Also aligns the docs to Zig 0.16+ and libyaml's YAML 1.1 scalar resolution (yes/on/no/off booleans alongside the 1.2 core spellings), and records the new benchmark numbers in RESULTS.md.
Anchor names were stored as slices into libyaml's event buffer, which yaml_event_delete frees after every event, so alias lookups missed and returned :parse_error (or read freed memory). Copy anchor names into a parse-scoped arena and register collection anchors at MAPPING/SEQUENCE start, binding them to the finished term on close. Adds real tests for scalar, list, and map anchors.
- Zig 0.10+ -> 0.16+ (the NIF uses ArrayList.empty and callconv(.c)) - clarify the reproduce comments are Zaml's parse time, not the whole run - the NIF builds via elixir_make + Makefile, not a Mix task; list the benchmark task in the project layout
Re-measure the combined build (zero-copy scalars + anchors) and use one canonical dataset everywhere: flat 0.84 s vs glazer 1.73 / fast_yaml 2.54; nested 0.53 s vs glazer 0.41 / fast_yaml 1.86. Fixes the README/release-note 0.85/0.51 tables that disagreed with benchmark/RESULTS.md, and records the anchor fix in the changelog.
Bump the version and mark the changelog. 0.2.0 adds single-shot map construction, zero-copy scalar sub-binaries, and working anchors/aliases, and re-measures the benchmarks. Update README, forum, hex, and blog drafts to 0.2.0 and fix the changelog release links.
The tasks were @moduledoc false, which made make: Nothing to be done for `all'. Generating docs... View html docs at "doc/index.html" View markdown docs at "doc/llms.txt" View epub docs at "doc/zaml.epub" warn that benchmark/RESULTS.md referenced the hidden module. Expose short docstrings with usage examples so the docs build is clean.
Give mix zaml_nif.bench/build real moduledocs so ExDoc no longer warns when benchmark/RESULTS.md autolinks the task name, and align the README flat benchmark comment with RESULTS.md (~0.8 s).
…drafts The forum post still described YAML 1.2 core-schema scalar resolution, which the README and hex press had already corrected to libyaml's YAML 1.1 rules (yes/on/no/off booleans alongside the 1.2 core spellings). Bring it in line. Also draft Bluesky / X / LinkedIn posts for the 0.2.0 launch.
A shorter, punchier variant of forum_elixirforum.md for the same thread — same facts, tighter framing. Both drafts stay on YAML 1.2 framing.
…tore YAML 1.2 core wording
These are marketing drafts (Hex press, Elixir Forum, blog, social posts) that aren't part of the shipped package or generated docs. Keep them on disk for editing but out of the repo history.
…e); scalar/tag/anchor support
…n vs 608ms CPython
- build: use full 2.x build identifiers ("cp3*-manylinux_x86_64",
"cp3*-musllinux_x86_64"); the 1.x "cp*x86_64" glob selects
nothing under cibuildwheel 2.22, so no wheels were being built.
- before-all: install libyaml with apk on the Alpine/musllinux image
(yum only exists on the CentOS-based manylinux image).
- ci: allow manually dispatching the workflow, and always upload the
wheelhouse so partial (manylinux-only) builds are still recoverable.
…nner
- zamlmodule.zig: CPython 3.12/3.13 keep the reference count in an
anonymous union ({ob_refcnt, ob_refcnt_split}) and 3.14 renamed the
wide member to ob_refcnt_full, so the previous code only compiled
against 3.10/3.11 and 3.14+. Detect the layout with @Hasfield and
initialise the refcount accordingly.
- benchmark: reformat with black 22.3.0 (the pre-commit hook) so the
lint job passes.
- ci: build wheels on ubuntu-22.04 (ubuntu-20.04 runners were retired)
and pin manylinux2014 to keep the glibc 2.17 baseline.
- The benchmark fixture (benchmark/big.yml) is generated and gitignored, so the CI job has to create it before running the benchmark. - zig's C importer cannot parse the glibc 2.17 (CentOS 7) headers, so build the manylinux wheels against the manylinux_2_28 image instead.
The PyPI project 'zaml' is owned by a third party and rejects uploads from this account, so the Python distribution moves to 'zamlpy'. - pyproject.toml: name = zamlpy; cibuildwheel test-command imports zamlpy - setup.py: Extension(zamlpy, sources=[zamlmodule.zig]) - zamlmodule.zig: module name zamlpy, PyInit_zamlpy - test.py + benchmark scripts: import zamlpy; call zamlpy.load(...) - README/CHANGELOG/SPEED_EVAL: Python references now say zamlpy The Elixir/Hex package remains named 'zaml' (independent namespace).
The Erlang NIF and the CPython extension each carried their own copy of the rules that decide what a YAML scalar is: which spellings are booleans, which are null, which are numbers, and what an explicit tag does to them. Two copies of the same rules are two rules that drift apart. The rules now live in core/scalar.zig. The NIF reaches them through a Zig module (it is built from native/, which cannot import ../core on its own) and the extension through core/native.zig, which exposes only the parts a native build can use: the WebAssembly core reaches libyaml through a C bridge that has no place in a native build. The values themselves are still built by the host, because an Erlang term and a Python object are made in very different ways, and the NIF builds them in a single pass for speed. The behaviour of both hosts is unchanged: the old and the new NIF, and the old and the new extension, produce the same result for the same document. The one exception is a fix: an explicitly tagged integer that is not a number (!!int abc) used to leave a pending Python exception behind, which turned into a SystemError instead of the string "abc".
Two WebAssembly targets, both from the same sources: zig build wasm the module the JavaScript package loads zig build wasm-wasi the command that turns YAML into JSON or CBOR The browser module is compiled for wasm32-freestanding, so it has no imports at all: it grows its own memory and carries the handful of C functions libyaml needs (malloc, free, strcmp and the file stubs it never calls). The vendored libyaml sources are compiled into it, so the module is self contained and works the same in a browser, in Node and in Bun. The WASI command uses the C library of the runtime instead, and talks to it through the raw WASI calls: the higher level helpers of the standard library walk the preopened directories of the runtime, and not every runtime reports them the same way. Two bugs are fixed along the way: the freestanding strcmp and strncmp overflowed when the compared strings had different lengths, which crashed the parser on tagged scalars, and the parser failed on an empty document because it made a pointer out of a null address.
The npm package (zamljs) is ESM only TypeScript with no runtime dependencies and it ships its own types. The browser and the node entry points are picked by the export conditions, so a bundler gets the one that fetches the module and node the one that reads it from the disk. The raw CBOR and JSON of a document are available as well, which is what the command writes. The Wasmer package (niranjanaryan/zaml) is the same parser as a command: YAML on standard input, JSON or CBOR on standard output. A release workflow publishes a tag to GitHub, npm and the Wasmer registry, and the CI workflow builds and tests the WebAssembly targets on every pull request.
The function was never exported and never used; the version of the package is in package.json.
The CBOR and the JSON encoders used to walk a tree of values that the parse filled in first. The tree is expensive: every scalar is copied into it and every value is a node of its own, so a document of 14 MB needs 500 MB of memory and two thirds of the parsing time. Both encoders are now sinks of the event stream: they write the document while the events arrive, so nothing is buffered but the output. The same document needs 70 MB of memory now, and the parse of the 14 MB fixture is 27% faster as JSON and 18% faster as CBOR. The bytes do not change. The tree and the encoders that walk it stay in the tree as the reference of a differential test, which parses a corpus of documents and the benchmark fixture both ways and compares the results. Two details the streaming encoders needed: the name of an anchor is copied when the container starts, because the buffer of the event does not outlive the event, and the offset of the end of the first document is kept, because a stream may hold more than one document and only the first one is written out. The tests of the core run again as well: they were imported but never pulled into a test block, so the test binary had no tests at all.
The browser build of the package was never loaded by a test, so a break in the loader or in the export conditions would only show up in a browser. The test replaces fetch with one that answers with the module, so the browser entry point is exercised the way a bundler loads it. The types of the package are checked as well: a file uses the public API the way a user would and `tsc` fails when a type of the package is wrong. The CI workflow and the release workflow run it.
The changelog, the readme and the CI workflow follow the change: the output formats are written while the events of the parse arrive, so a 14 MB document needs 70 MB of memory instead of 500 MB. The CI workflow builds the WebAssembly modules with the same optimisation as the release does, so the package the tests run against is the package that is published.
An alias can name the key of a pair, and a merge key is a plain key that holds an alias. Both go through the same path as an alias in a value, so the corpus of the differential test holds them now.
The benchmark reads the same fixture with zamljs, js-yaml and the yaml package, and writes the table of the last run. The raw CBOR is measured next to the value that parse() returns, so the time the WebAssembly boundary costs is visible: the document crosses it as CBOR and is turned into a JavaScript value afterwards.
The message of a failure said "invalid YAML" when the parser ran out of memory, because the message is built from the place of the problem in the document and there is none.
The plan holds the decisions the JavaScript package, the Wasmer command and the shared core were built on.
The repository was renamed to zaml, and the links still pointed at the old name in the mix project, in the Python metadata, in the changelog and in the release notes. The version of the Elixir package and of the Python extension goes to 0.3.0, which is the release that carries the JavaScript package, the Wasmer command and the shared core. The changelog moves its section from Unreleased to 0.3.0. The plan of the multi-ecosystem work follows what was built: the output formats are written while the events of the parse arrive, and the value tree stays as the reference of the differential test.
The release workflow published the WebAssembly modules to GitHub, the package to npm and the command to Wasmer, but not the Elixir package to Hex. It has a job now, and the version of the mix project follows the tag. The Python distribution takes its version from the tag as well, so a release cannot publish a wheel under the version of the last one. The secrets the jobs need are written down at the top of the two workflows: HEX_API_KEY is in the repository, NPM_TOKEN, WASMER_REGISTRY_TOKEN and pypi_password are not.
The C importer of Zig cannot translate the headers of the Windows SDK, so the CPython extension does not build on Windows. The job keeps reporting the problem, but it no longer holds the source distribution back, which blocked the release of the Python package. The release workflow skips a version that is already on Hex or on npm, so a release that fails halfway can be run again. The versions of the npm package and of the Wasmer command follow the release, and the docs tell that the extension needs Zig 0.16.x and does not build on Windows yet.
A release that the release workflow publishes does not trigger the CI workflow, because it is published with the token of the workflow. The tag is the trigger that always works, so the CI workflow listens to it and publishes the distribution to PyPI from there. The Hex job fetches the dependencies before it builds the package, because `mix hex.publish` refuses to run without them.
The manifest of Wasmer 7 wants a list of modules and a list of commands, not a source and a command. The module is the WebAssembly build of the command, which the release workflow downloads from the release of the tag before it publishes the package. The lock file of the npm package follows its version.
The dependencies of the development environment (the documentation and the parsers that the benchmarks compare with) are not needed to publish the package, and one of them needs rebar3, which the runner does not have.
The pattern replaced the whole line, so the mix project lost its version and the step failed.
The wasmerio/wasmer-action action no longer exists; the action is now wasmerio/setup-wasmer.
Node 22 does not treat a bare directory as a test root the way Node 26 does, so the CI job failed with a module-not-found error. The glob works on both versions.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.