Skip to content

Add opt-in Winnow E4B image decisions through libmtmd - #52

Merged
cobanov merged 8 commits into
ollaya-dev:mainfrom
Txoka:feat/winnow-image-input
Oct 5, 2026
Merged

cobanov merged 8 commits into
ollaya-dev:mainfrom
Txoka:feat/winnow-image-input

Conversation

@Txoka

@Txoka Txoka commented Oct 2, 2026 •

Copy link
Copy Markdown
Contributor

Winnow E4B publishes a vision projector, but Ollaya currently exposes only its text path. This adds an opt-in winnow:e4b-vision tag: PNGs supplied through the CLI or native /api/decide participate in typed decisions alongside the state and questions. Existing text tags retain their inference path.

Implementation

  • Reuses the image request and CLI plumbing merged in Library: decider:2b-vision, Mapika's vision decider, and image input (#14) #29. /v1/systemone remains compatible with the text-only TypeSafe contract.
  • Adds an optional projector layer to the existing GGUF packaging pipeline, pinned to the same author revision as the language model.
  • Loads libmtmd from the existing pinned llama.cpp distribution. Prepares the image/state prefix once per request and evaluates question continuations with the original Winnow readout, preserving tokenizer boundaries and image batch ordering.
  • Keeps projector ownership, locking, selected GPU device and automatic CPU fallback within the runner. Suppresses prompt-bearing mtmd informational/debug logs while retaining warnings and errors.
  • Documents packaging, API use, numerical checks and reproduction commands in docs/measurements/winnow-e4b-vision.md. Shared projector infrastructure can support future families; rendering/readout integration is currently Winnow-specific.

Validation

On an RTX 4070 and x86-64 CPU, against stock llama-server from the same pinned build and on the same device:

  • Image parity: 21 requests, 65/65 decisions on each device; maximum normalized option-logit error 9.55e-6 CUDA and 1.15e-5 CPU, below the unchanged 1e-3 tolerance.
  • Text regression: 505/505 decisions against the same-device stock reference. The report separately records differences against the maintainer's cross-device RTX 4090 fixture.
  • Native HTTP checks: multiple images including 16-image batch rollover, malformed inputs, context overflow, recovery, text compatibility and unload.
  • A real GPU request with debug logging verifies that private state/question markers and mtmd prompt/tensor dumps are absent.
  • Formatting, workspace clippy with warnings denied, workspace tests, projector packaging test and site build pass. The branch incorporates current main, including GGUF models on auto: start the runner again on the CPU when it dies on the GPU #47's process-level GPU fallback.

Scope

PNG still images only. Image calibration/benchmark accuracy, Metal, Windows, physical multi-GPU operation and Winnow 12B vision have not been validated. Existing text calibration is retained. The numerical reference is stock pinned llama-server under ADR 0003; no parity claim is made against the author's separately optimized image server.

Txoka and others added 6 commits October 2, 2026 01:30
- docs: the measurement report joins docs/families/winnow.md, where every family keeps its
  parity numbers, as an Images section with the parity table, the coverage and the reproduce
  commands. The review-follow-up notes stay in the pull request. The Limits line no longer says
  images are out of scope.
- catalog: the e4b-vision tag is defined in the winnow entry with the other tags, not patched in
  at the end of the file.
- registry: the hand-assembled manifest goes. The maintainers write it with package.py when the
  tag ships, together with its config blob, so the registry never names a blob ollaya.dev lacks.
- library page: the image section sits before the license and names the first release with it.

@cobanov cobanov left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Txoka, thank you. This is careful work: same-device stock references per ADR 0003, unchanged tolerances, the cross-device numbers explained rather than hidden, the prompt-bearing mtmd logs suppressed, and the fallback path covered. We want it in.

I pushed one commit to your branch (31975f2) so it matches how the rest of the repository is laid out. No code changed:

  • Docs. The measurement report is folded into docs/families/winnow.md, where every family keeps its parity numbers. It is now an Images section with the parity table, the coverage and the reproduce commands; the review-follow-up notes stay here in the PR. The Limits line no longer says images are out of scope.
  • Catalog. e4b-vision is defined in the winnow entry next to the other tags instead of being patched in at the end of catalog.py.
  • Registry. The manifest is removed for now. We write manifests with package.py when a tag ships, together with its config blob, so the registry never names a blob ollaya.dev does not serve yet.
  • Library page. The image section moved before the license and names the first release with it (0.10.0).

Before merging, on our side:

  • Our GPU. Parity on our RTX 4090 against its own stock llama-server, per ADR 0003. Our GPUs are busy until Monday evening, so this is early next week.
  • Packaging. We package the tag and check that the release archives and the desktop app carry libmtmd on every platform.
  • Merge timing. We merge right before the release, so the docs never mention a tag the registry does not have yet.

Two small things, if you have a moment (otherwise we will add them):

  • SAFETY comment. The from_raw_parts over the logits in run_winnow_images needs one, like the call above it.
  • Params in vision.rs. A comment that it mirrors mtmd_context_params of the pinned b11146 tools/mtmd/mtmd.h, and that the load-time llama_version check is what keeps it from meeting another build.

@Txoka

Txoka commented Oct 3, 2026

Copy link
Copy Markdown
Contributor Author

Thank you for your commit. I will have a look at those comments when I have a minute. I'm also planning to open a separate PR to extend the validation and support to the 12b version soon enough.

@cobanov
cobanov merged commit ddb136b into ollaya-dev:main Oct 5, 2026
7 checks passed
@cobanov

cobanov commented Oct 5, 2026

Copy link
Copy Markdown
Member

@Txoka, merged, thank you. Our RTX 5090 run with the shipped 8,192-token context matches yours (65/65, 9.5e-6; the 505 text questions too), and the release dry run carries libmtmd on every platform. It ships in 0.10.0 today. Looking forward to the 12B PR.

cobanov pushed a commit that referenced this pull request Oct 6, 2026
winnow:12b-vision is the 12b tag (same Q8_0 GGUF, prompt with its empty thought block, decision and calibration) plus the author's matching vision projector from the same pinned revision (EldanRing/Winnow-12B@b6ac22b0), read through libmtmd as e4b-vision is since #52. The text tags and the latest alias are unchanged.

Parity against stock llama-server b11146 with the projector on the same device: 21 image requests and 65 questions, every decision the same, option logits within 7.7e-6 on an x86-64 CPU and 7.6e-6 on an RTX 5090 at the shipped 8,192-token context; the 505 text questions of winnow:12b also pass on the RTX 5090 (option logits within 1.1e-5). It peaks at 16,263 MiB on the RTX 5090, so a 16 GB GPU is marginal. The packaging tests cover both projector tags.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants