Repository navigation
Add opt-in Winnow E4B image decisions through libmtmd - #52
Conversation
- docs: the measurement report joins docs/families/winnow.md, where every family keeps its parity numbers, as an Images section with the parity table, the coverage and the reproduce commands. The review-follow-up notes stay in the pull request. The Limits line no longer says images are out of scope. - catalog: the e4b-vision tag is defined in the winnow entry with the other tags, not patched in at the end of the file. - registry: the hand-assembled manifest goes. The maintainers write it with package.py when the tag ships, together with its config blob, so the registry never names a blob ollaya.dev lacks. - library page: the image section sits before the license and names the first release with it.
cobanov
left a comment
There was a problem hiding this comment.
@Txoka, thank you. This is careful work: same-device stock references per ADR 0003, unchanged tolerances, the cross-device numbers explained rather than hidden, the prompt-bearing mtmd logs suppressed, and the fallback path covered. We want it in.
I pushed one commit to your branch (31975f2) so it matches how the rest of the repository is laid out. No code changed:
- Docs. The measurement report is folded into
docs/families/winnow.md, where every family keeps its parity numbers. It is now an Images section with the parity table, the coverage and the reproduce commands; the review-follow-up notes stay here in the PR. The Limits line no longer says images are out of scope. - Catalog.
e4b-visionis defined in the winnow entry next to the other tags instead of being patched in at the end ofcatalog.py. - Registry. The manifest is removed for now. We write manifests with
package.pywhen a tag ships, together with its config blob, so the registry never names a blob ollaya.dev does not serve yet. - Library page. The image section moved before the license and names the first release with it (0.10.0).
Before merging, on our side:
- Our GPU. Parity on our RTX 4090 against its own stock llama-server, per ADR 0003. Our GPUs are busy until Monday evening, so this is early next week.
- Packaging. We package the tag and check that the release archives and the desktop app carry
libmtmdon every platform. - Merge timing. We merge right before the release, so the docs never mention a tag the registry does not have yet.
Two small things, if you have a moment (otherwise we will add them):
- SAFETY comment. The
from_raw_partsover the logits inrun_winnow_imagesneeds one, like the call above it. Paramsinvision.rs. A comment that it mirrorsmtmd_context_paramsof the pinned b11146tools/mtmd/mtmd.h, and that the load-timellama_versioncheck is what keeps it from meeting another build.
|
Thank you for your commit. I will have a look at those comments when I have a minute. I'm also planning to open a separate PR to extend the validation and support to the 12b version soon enough. |
|
@Txoka, merged, thank you. Our RTX 5090 run with the shipped 8,192-token context matches yours (65/65, 9.5e-6; the 505 text questions too), and the release dry run carries libmtmd on every platform. It ships in 0.10.0 today. Looking forward to the 12B PR. |
winnow:12b-vision is the 12b tag (same Q8_0 GGUF, prompt with its empty thought block, decision and calibration) plus the author's matching vision projector from the same pinned revision (EldanRing/Winnow-12B@b6ac22b0), read through libmtmd as e4b-vision is since #52. The text tags and the latest alias are unchanged. Parity against stock llama-server b11146 with the projector on the same device: 21 image requests and 65 questions, every decision the same, option logits within 7.7e-6 on an x86-64 CPU and 7.6e-6 on an RTX 5090 at the shipped 8,192-token context; the 505 text questions of winnow:12b also pass on the RTX 5090 (option logits within 1.1e-5). It peaks at 16,263 MiB on the RTX 5090, so a 16 GB GPU is marginal. The packaging tests cover both projector tags.
Winnow E4B publishes a vision projector, but Ollaya currently exposes only its text path. This adds an opt-in
winnow:e4b-visiontag: PNGs supplied through the CLI or native/api/decideparticipate in typed decisions alongside the state and questions. Existing text tags retain their inference path.Implementation
/v1/systemoneremains compatible with the text-only TypeSafe contract.docs/measurements/winnow-e4b-vision.md. Shared projector infrastructure can support future families; rendering/readout integration is currently Winnow-specific.Validation
On an RTX 4070 and x86-64 CPU, against stock llama-server from the same pinned build and on the same device:
Scope
PNG still images only. Image calibration/benchmark accuracy, Metal, Windows, physical multi-GPU operation and Winnow 12B vision have not been validated. Existing text calibration is retained. The numerical reference is stock pinned llama-server under ADR 0003; no parity claim is made against the author's separately optimized image server.