Add the arbiter family (hiteshluke/arbiter-4b)

[codemanhitesh]

Add the arbiter family (hiteshluke/arbiter-4b)

Arbiter is Codekins Pvt Ltd / Zyot Lab's 4B decision model: Gemma 3 4B IT + LoRA + a fixed 24-slot pointer head
(arbiter-fixed-v1). One forward pass per question; noul reads slots 0..1, choice slots 2..17 (up to 16
options), score slots 18..23 (exactly 6 levels). Full write-up: docs/families/arbiter.md.

This update addresses the review point by point.

Review items

1. Rust runtime. Layout in crates/ollaya-decision/src/arbiter.rs (port of layout.py, with unit tests);
   engine in crates/ollaya-runner/src/arbiter.rs, registered in engine.rs. A choice with more than 16 options
   and a score with other than 6 levels are rejected with a 422, never truncated.
2. Parity.
   • check.py: the Rust-layout port against the reference prompt, id for id, on the shared request set plus 6
     extra Arbiter cases, with two tokenizer library versions: 420 rows identical, 0 mismatches.
   • goldens.py: the reference model in fp32 (token ids, last position, all 24 slot scores, option logits,
     probabilities).
   • crates/ollaya-runner/examples/parity_arbiter.rs: identical rows and rejections, every decision the same,
     slot scores within 1e-3.
3. Registry. registry/v2/library/arbiter/manifests/* removed.
4. Licensing. The catalog license field now reads "Apache-2.0 (LoRA adapter and head) and the Gemma Terms of
   Use (Gemma 3 base model)"; docs match. unsloth/gemma-3-4b-it@bf46152 is byte-identical to Google's
   google/gemma-3-4b-it for both safetensors shards and tokenizer.json (same sha256 and size per the Hub API);
   only metadata files differ. The original is gated (manual approval) and Ollaya's client has no Hugging Face
   token support, which is why the manifest points at the copy. Happy to repoint if you prefer.
5. Conventions. docs/pr/arbiter.md folded into docs/families/arbiter.md; scripts/arbiter_hash_hf.py
   removed.

Found while porting

• Training adds <bos>; the earlier layout did not. Fixed.
• The head has no bias; the earlier reference loader expected one. Fixed.
• The model repo's tokenizer.json truncates at 255 tokens, so the runtime takes the tokenizer from the base repo.

Accuracy

Measured on a T4 with the training prompt, one row at a time. Model-card numbers use the 4-bit base the adapter was
trained on; the same adapter and head were also run on the same rows with the unquantized base (fp32 compute, as
Ollaya runs it):

Benchmark	n	4-bit base	Unquantized base
BoolQ (validation)	1,000	0.849	0.853
ARC-Challenge (test)	500	0.738	0.762
CommonsenseQA (validation)	500	0.706	0.720
OpenBookQA (test)	500	0.722	0.748

Running the full BF16 base does not cost accuracy relative to the model card.

Status

• Not run yet: the export, goldens and both parity runs on the real weights (they need a GPU), so there are no
  parity numbers yet; and typed-decisions accuracy on the 400 test states. Both are runnable with the commands
  in the docs. The pipeline (export, goldens, ONNX parity) was exercised end to end on a small random stand-in
  model: ONNX within about 1e-5 of the reference, all decisions agree.
• Rust: the new Rust code has not been compiled or linted on our side (no Rust toolchain on our machine), so
  please treat CI as the first check; we will fix anything it reports.
• Coverage: every score question in the shared set's 40 typed-decisions rows has 4 or 5 levels, so the fixed
  6-level head cannot answer them and they are rejected with a 422.

[cobanov]

@codemanhitesh, thanks for submitting Arbiter, and sorry for the wait. The Python side is a good start. Every family in Ollaya goes through the same path before it ships, and this PR is not there yet. Here is what it needs, in order.

1. The Rust runtime. Nothing runs this model today. arbiter-fixed-v1 needs:

• a layout in crates/ollaya-decision that builds the prompt rows from a request (port layout.py);
• an engine in crates/ollaya-runner that feeds the graph and reads the 24 slots per question type, registered in engine.rs.

kev.rs and jeeves.rs in both crates are close templates (decoder rows, last-position readout). Requests the fixed head cannot answer must be rejected with a clear 422, not truncated: a choice with more than 16 options, a score with other than 6 levels.

2. Parity, which is the gate.

• Prompt check. A check.py that compares the layout port with your reference prompt, id for id, on our shared request set: the edge cases and 40 typed-decisions rows, as a JSONL here.
• Goldens. goldens.py, written from your reference model in fp32 on those requests: token ids, slot scores and probabilities.
• Runtime parity. A crates/ollaya-runner/examples/parity_arbiter.rs that runs the Rust runtime against those goldens: identical rows and rejections, every decision the same, scores within 1e-3. parity_jeeves.rs is a template.

We run the export, the goldens and the parity again on our own GPU before merging, so they only need to be runnable.

3. Leave the registry to us. Please remove registry/v2/library/arbiter/manifests/*. We write manifests with package.py when a tag ships, from the export we checked, so a manifest never carries placeholder digests.

4. Licensing.

• License field. The base is Gemma 3, under the Gemma Terms of Use, so the catalog's "license" cannot say Apache-2.0 alone. Your license_text already names both; the field and the docs should match it.
• Base repository. Ollaya pulls weights from the original author's repository. Please say whether unsloth/gemma-3-4b-it@bf46152 is byte-identical to Google's google/gemma-3-4b-it, and why the manifest should point at the copy.

5. Repository conventions.

• Docs. Docs for a family live in docs/families/arbiter.md: sequence, graph, differences from upstream, parity, quality. Please fold docs/pr/arbiter.md into it or into this PR's description.
• Hashing script. One-off tools like scripts/arbiter_hash_hf.py stay out of the repository. package.py already verifies every file against the Hub's sha256.
• Accuracy. Please add typed-decisions accuracy (all 400 test states, argmax against the majority label), the number every model in the library is compared on. BoolQ and ARC can stay next to it.

Once 1 and 2 are in, we will do the GPU checks and help with the rest. Thanks again for the contribution.

[cobanov]

@codemanhitesh, thanks for working through the review point by point. The remaining steps needed weights and a GPU, so we ran them on our side and pushed the results to this branch:

• Merged main (0.11.0). The only conflicts were engine.rs and lib.rs, where clef and decima also register; fmt, clippy and the 251 workspace tests pass.
• Parity, on the shared set (127 requests, 6 accepted whole, 121 rejected by both, 420 rows), with transformers 4.57.6 and peft 0.19.1 for the reference:
  • check.py: 420 rows identical, 0 mismatches, as you reported.
  • Export (parity.py): every decision the same, slot scores within 5.6e-5, probabilities within 9.3e-6.
  • Runtime (parity_arbiter): identical rows and rejections, 420/420 decisions, slot scores within 6.0e-5 on the CPU and 8.5e-5 on CUDA (RTX 4090), probabilities within 1.0e-5. A request of three or more questions takes 154 ms in the runner.
• Typed-decisions (eval_refs.py): the head answers 1,200 of the 2,000 questions (none of the 800 score questions has 6 levels); on those, 0.620 (choice 0.563, noul 0.677).
• Presets. Every built-in preset has a 3- or 4-level score question, so Arbiter rejects them all. The new library page says so, and its usage example passes its own questions.
• Docs and site: the numbers in docs/families/arbiter.md, the catalog's parity text, the library page, the README row and the Gemma license note.

The unsloth/gemma-3-4b-it copy is fine with us, since you checked it is byte-identical to Google's. Marking this ready; it merges once CI is green and ships in the next release.

[cobanov]

The review points are addressed and the parity gate passes on the CPU and on CUDA (see the comment above). Thanks.

[cobanov]

Merged, thanks @codemanhitesh. arbiter:4b goes into the registry with the next release, together with its library page; the release notes will credit you.